Don't Panic
You will have time to look at the Hot New Thing, I promise
So you've just heard about this crazy new technology that's going to revolutionize everything and you need to stop what you're doing right now and holy mother of all things wonderful if you don't know what this technology is you and your team are going to be out of jobs when some hotshot in a garage who may or may not be wearing a turtleneck but has tremendous charisma (rizz?) and has partnered with another wunderkind who apparently can see through time and space to find the answers to life the universe and everything goes ahead and steals your business!
Calm down. Take a deep breath.
If you decide to go off the deep end and disrupt all those roadmaps you just worked on because yet another LLM came out, you will be causing a lot of swirl.
The good news is that you won't be alone if you succumb to temptation. For as long as I've been in this game, there have been complaints about VPs going this way and that, chasing trends and generally disrupting the best laid schemes. My favorite complaint about this behavior is in the Tao of Programming (1987):
7.1
A novice asked the Master: "In the East, there is a great tree-structure that men call 'Corporate Headquarters'. It is bloated out of shape with vice presidents and accountants. It issues a multitude of memos, each saying 'Go Hence!' or 'Go Hither!' and nobody knows what is meant. Every year new names are put onto the branches, but all to no avail. How can such an unnatural entity exist?"
The Master replied: "You perceive this immense structure and are disturbed that it has no rational purpose. Can you not take amusement from its endless gyrations? Do you not enjoy the untroubled ease of programming beneath its sheltering branches? Why are you bothered by its uselessness?"
The best I can get StableDiffusionXL to depict the Corporate Tree, sending memos to and fro.
To me, the feeling that technology is about to pass me by is a combination of FOMO, worrying about aging out of a business that has traditionally been very prejudiced against people over the age of 35, and a general sense that I should be an expert in this stuff because I've been doing it for a long time and not knowing challenges my authority. Throw in a dash of concern about teams being hired then cut (ie, the year 2022), and now I've got a pretty good recipe to panic and chase after all the new trends.
Those sensations are exactly what bullshit artists feed on. They are trying to take advantage of those insecurities to sneak past any safeguards to get a better deal when they should get no deal at all. Unfortunately, the crafting of bullshit, when artfully done, looks exactly like legitimate opportunities and new technologies that you, as a data scientist or data science manager, really should know about, and learning to distinguish between the two is a skill that must be developed in the data science manager. More to the point, no one else at the company is equipped or tasked with sifting useful new technology from smoke and mirrors more than you and your team. Your company is almost certainly being contacted by people offering all kinds of services to enhance customer experiences through AI or blockchain or what have you, some of which are legitimate technologies. A more stressful version of this is doing due diligence for mergers & acquisitions (aka M&A)-- if the company to be bought is basically full of scam artists, your reputation as an expert will be ruined if you let that one go through and the business loses a lot of money.
So, how to tell the difference?
Let's go with a flow chart:
Starting from the top, who is leading the charge to uncover the secrets behind this latest press release or YCombinator News article?
You
If it's you, congratulations, you have bought yourself some time to get ahead of the game. Perhaps all you need is to read an article or three to be prepared when the topic comes up.
Someone Else In the Company
Other people are likely just as caught up in it as you are. I strongly urge you to channel that enthusiasm to get them to do the investigation under your guidance (if you have the time). If that other person is one of your direct reports, then I would argue that guiding them through that investigation (or refocusing them on other priorities, if needs be) is a fundamental part of your job. Excitement, enthusiasm, and curiosity are fantastic attributes to have in an employee, so allow and encourage them to do their own investigations. They will uncover new things you have not seen before. If you have a regularly scheduled 1:1 with them, you will have a perfect place to talk about those innovations without disrupting anyone else and devise a plan to approach this new technology, if at all. Better yet, schedule regular team meetings where everyone can share what they’ve learned.
If that person is not your report, I would talk with their manager about how to proceed with their investigations. It's not a great look to appear to be managing everyone else's reports or encouraging them to ignore the work they've been tasked with doing.
Salesperson
If the new technology is being brought to me by a salesperson, then my guard is up. I get cold-called once or twice a day on my personal cell phone by salespeople, and I'm not very interested in engaging with any salespeople at all. Having said that, I know that there are other managers I work with who set aside time to get demos from various companies to stay abreast of what those vendors offer.
When I do vendor evaluation, I start with these principles:
Are they making any claims that are too good to be true? The combination of fantastic claims and no time to evaluate them is a huge warning signal to me. That kind of timing is designed to generate urgency in the decision making process, not to allow a good decision to be made in the first place. Time pressure is a great way to get people to buy things they don't need.
When they are showing you graphs, have they labeled their axes? Labeled axes are a sign of taking data seriously; whenever I see a graph with unlabeled axes, I do not think the presenter is a scientist but rather someone who just assumes I know what they are talking about-- or worse, someone who wants me to think we're all on the same wavelength, but they are actually trying to avoid hard questions like "why are you using that as your metric of success?" and "why are you talking about response time in milliseconds when your graph is showing response time in days?"
Do they provide any studies showing the efficacy of their product? How thorough is the presentation of those studies, and how rigorous is the science behind the studies? Have they committed any of the mistakes listed in Statistics Done Wrong? To be fair, a lot of well-meaning people make those same mistakes, and making a mistake in statistical reasoning is not an immediate indicator of someone acting in bad faith-- but bad faith isn't really what I'm trying to avoid, what I'm trying to avoid is finding myself with technology that does not solve my problem.
When evaluating a technology vendor, can I give them a test/holdout set? I like to have a test/holdout set used across all of my experiments for any domain where I need to develop models; can I just send them that test/holdout set for evaluation and get a report or the data on each item? The inability to process data other than their own canned data is also a huge red flag since that lack suggests that much of their technology is hand-crafted rather than automated.
These are just my starting points; I generally find that my analysis will wander through the technology, and so remembering the touchpoints of why I would even want to investigate their product in the first place helps to ground this exploration.
Exec
If it's someone higher up in the chain (maybe even a C-Level Exec), you may feel caught out and flat-footed, perhaps with a dash of amygdala stress response thrown in. If/when that happens, I'd first determine where the exploration lies in their priority list and how much detail they want to know. Ideally, all you'll need to do is read an article or three as if you found out about the New Thing all on your own and get back to that exec in a reasonable timeframe without disrupting the work of your team or any adjacent teams. If they want to know Right Now and they want to pivot the entire roadmap and they are aware of how much this will impact all those carefully laid plans, well, it's probably time to get the rest of the Basic Building Blocks together and let them know about the new task ahead of you. If you've done everything you can to let that exec understand that this new plan will be very disruptive and they accept the subsequent turmoil of their decisions, you should definitely spend some time talking to that executive about their reasoning. Almost certainly, they know something you don't know, and getting that additional context will help you to understand the need for the sudden change. It may be that those carefully crafted roadmaps should absolutely be changed because the business climate has changed radically in the past few weeks.
(As a side note: most of the execs I’ve enjoyed working with telegraph their moves well in advance so this kind of fire drill would be avoided. If I’m working with that consideration, the kind of incident that would cause a drop-everything-now-and-work-on-this response is almost certainly a real emergency that will be known shortly throughout the company)
So what are you (or that exec) looking for? Is there a significant and immediate business need that must be filled or a problem to be addressed? Is there a way to leapfrog 5-10 years of development to get to a "dream" system, or some significant boon you can develop for a product line, or create an entirely new product line? Or do you think this technology will become table stakes in your business, like recommendation engines for e-commerce or chatbots for customer service? If the answer to those questions is no, then maybe the right approach is to wait and see. I tend to watch for new developments, but I typically don't leap on to be an early adopter. The tech may fizzle out to nothing or take decades to mature; after all, the current LLM trend is the culmination of decades of neural network research, and investing heavily in an immature space is a huge gamble. If your leadership team is comfortable making the gamble, perhaps you could be like NVidia in 2016 going all-in on AI; or perhaps you could be caught up in an AI Winter.
But if the answer is yes to any of those questions, then it behooves you to investigate further. The format of that investigation is a judgment call; if you need to do a deep dive into comparing new tech to an existing product's functionality, then maybe a three week sprint to prepare a report is necessary for other business stakeholders to understand the implications of the technology.
Communicating What You’re Doing
I strongly urge you to let people know what you're up to but that you don't expect them to change what they are doing. New AI advances seem to pop up like mushrooms after a storm, creating tension and excitement. I caution that, as a manager, I don't disrupt sprints, plans, or anything like regularly scheduled business in a large meeting involving Product, Project, Engineering, etc.
Instead, focus on having 1:1 and/or team conversations with your reports and other managers and leaders to get a feel for how the new technology or idea could impact your business if you think that there’s some there there. This approach combines courtesy and getting pointed feedback on your own reaction. No one likes to feel blindsided, and people walking into a regular sprint planning meeting getting told that this entirely new thing will derail the next month's worth of work is the very definition of blindsiding the team.
You may even find yourself in a regular exploration cadence, where you have allocated time for you or your team to investigate new technologies. If you have the enthusiasm and support of your team (and, presumably, your boss), you can schedule a time for anyone interested to come and talk about what they've learned. The more academically inclined tend to call these meetings journal clubs, with a regularly scheduled rotation of who will be presenting on given dates.
The only real exception to this more relaxed atmosphere is if you learn drastic information from that C-Level conversation I alluded to earlier; in that case, I would expect that executive to tell the team (or pointedly let you know that you're going to tell the team) about the change in priorities. I would definitely work with that executive to at least let your management peers know ahead of time about the change, again, to avoid surprising them.
Meanwhile, don't unnecessarily disrupt anyone outside of your org; let them program under the sheltering branches of the corporate tree.