The data science and AI market may be out for a recalibration

Being a data scientist was supposed to be "the sexiest job of the 21st century". Whether the famous Harvard Business Review aphorism of 2012 holds water is somewhat subjective, depending on how you interpret "sexy". However, the data around data scientists, as well as related data engineering and data analyst roles, are starting to ring alarms.

The subjective part about HBR's aphorism is whether you actually enjoy finding and cleaning up data, building and debugging data pipelines and integration code, as well as building and improving machine learning models. That list of tasks, in that order, is what data scientists spend most of their time on.

Some people are genuinely attracted to data-centered careers by the job description; the growth in demand and salaries more attracts others. While the dark sides of the job description itself are not unknown, the growth and salaries part was not disputed much. That, however, may be changing: data scientist roles are still in demand but are not immune to market turmoil.

Mixed signals

At the beginning of 2022, the first sign that something may be changing became apparent. As an IEEE Spectrum analysis of data released by online recruitment firm Dice showed, in 2021, AI and machine learning salaries dropped, even though, on average, U.S. tech salaries climbed nearly 7%.

Overall, 2021 was a good year for tech professionals in the United States, with the average salary up 6.9% to $104,566. However, as the IEEE Spectrum notes, competition for machine learning, natural language processing, and AI experts softened, with average salaries dropping 2.1%, 7.8%, and 8.9%, respectively.

It's the first time this has occurred in recent years, as average U.S. salaries for software engineers with expertise in machine learning, for example, jumped 22% in 2019 over 2018, then went up another 3.1% in 2020. At the same time, demand for data scientist roles does not show any signs of subsiding -- on the contrary.

Developer recruitment platforms report seeing a sharp rise in the demand for data science-related IT skills. The latest IT Skills Report by developer screening and interview platform DevSkiller recorded a 295% increase in the number of data science-related tasks recruiters were setting for candidates in the interview process during 2021.

CodinGame and CoderPad's 2022 Tech Hiring Survey also identified data science as a profession for which demand greatly outstrips supply, along with DevOps and machine-learning specialists. As a result, ZDNet's Owen Hughes notes, employers will have to reassess both the salaries and benefits packages they offer employees if they hope to remain competitive.

Plus, 2021 saw what came to be known as the "Great Resignation" or "Great Reshuffle" -- a time when everyone is rethinking everything, including their careers. In theory, having a part of the workforce redefine their trajectory and goals and/or resign should increase demand and salaries -- analyses on why data scientists quit and what employers can do to retain them started making the rounds.

Then along came the layoffs, including layoffs of data scientist, data engineer and data analyst roles. As LinkedIn's analysis of the latest round of layoffs notes, the tech sector's tumultuous year has been denoted by daily announcements of layoffs, hiring freezes and rescinded job offers.

About 17,000 workers from more than 70 tech startups globally were laid off in May, a 350% jump from April. This is the most significant number of lost jobs in the sector since May 2020, at the height of the pandemic. In addition, tech giants such as Netflix and PayPal are also shedding jobs, while Uber, Lyft, Snap and Meta have slowed hiring.

According to data shared by the tech layoff tracking site Layoffs.fyi, layoffs range from 7% to 33% of the workforce in the companies tracked. Drilling down at company-specific data shows that those include data-oriented roles, too.

Looking at data from FinTech Klarna and insurance startup PolicyGenius layoffs, for example, shows that data scientist, data engineer and data analyst roles are affected at both junior and senior levels. In both companies, those roles amount to about 4% of the layoffs.

High-tech coolies coding themselves out of their jobs

What are we to make of those mixed signals then? Demand for data science-related tasks seems to be going on strong, but salaries are dropping, and those roles are not immune to layoffs either. Each of those signals comes with its own background and implications. Let's try to unpack them, and see what their confluence means for job seekers and employers.

As Dice chief marketing officer Michelle Marian told IEEE Spectrum, there are a variety of factors likely contributing to the decreases in machine learning and AI salaries, with one important consideration being that more technologists are learning and mastering these skill sets:

"The increases in the talent pool over time can result in employers needing to pay at least slightly less, given that the skill sets are easier to find. We have seen this occur with a range of certifications and other highly specialized technology skills", said Marian.

That seems like a reasonable conclusion. However, for data science and machine learning, there may be something else at play, too. Data scientists and machine learning experts are not only competing against each other but also increasingly against automation. As Hong Kong-based quantitative portfolio manager Peter Yuen notes, quants have seen this all before.

Prompted by news of top AI researchers landing salaries in the $1 million range, Yuen writes that this "should be more accurately interpreted as a continuation of a long trend of high-tech coolies coding themselves out of their jobs upon a backdrop of global oversupply of skilled labour".

If three generations of quants' experience in automating financial markets are anything to go by, Yuen writes, the automation of rank-and-file AI practitioners across many industries is perhaps only a decade or so away. After that, he adds, a small group of elite AI practitioners will have made it to managerial or ownership status while the remaining are stuck in average-paid jobs tasked with monitoring and maintaining their creations.

We may already be at the initial stages in this cycle, as evidenced by developments such as AutoML and libraries of off-the-shelf machine learning models. If history is anything to go by, then what Yuen describes will probably come to pass, too, inevitably leading to questions about how displaced workers can "move up the stack".

The bursting of the AI bubble

However, it's probably safe to assume that data science roles won't have to worry about that too much in the immediate future. After all, another oft-cited fact about data science projects is that ~80% of them still fail for a number of reasons. One of the most public cases of data science failure was Zillow.

Zillow's business came to rely heavily on the data science team to build accurate predictive models for its home buying service. As it turned out, the models were not so accurate. As a result, the company's stock went down over 30% in 5 days, the CEO put a lot of blame on the data science team, and 25% of the staff got laid off.

Whether or not the data science team was at fault at Zillow is up for debate. As for recent layoffs, they should probably be seen as part of a greater turn in the economy rather than a failure of data science teams per se. As Data Science Central Community Editor Kurt Cagle writes, there is talk of a looming AI winter, harkening back to the period in the 1970s when funding for AI ventures dried up altogether.

Cagle believes that while an AI Winter is unlikely, an AI Autumn with a cooling off of an over-the-top venture capital field in the space can be expected. The AI Winter of the 1970s was largely due to the fact that the technology was not up to the task, and there was not enough digitized data to go about.

Today much greater compute power is available, and the amount of data is skyrocketing too. Cagle argues that the problem could be that we are approaching the limits of the currently employed neural network architectures. Cagle adds that a period in which brilliant minds can actually rest and innovate rather than simply apply established thinking would likely do the industry some good.

Like many others, Cagle is pointing out deficiencies in the "deep learning will be able to do everything" school of thought. This critique seems valid, and incorporating approaches that are overlooked today could drive progress in the field. However, let's not forget that the technology side of things is not all that matters here.

Perhaps recent history can offer some insights: what can the history of software development and the internet teach us? In some ways, the point where we are at now is reminiscent of the dot-com bubble era: increased availability of capital, excessive speculation, unrealistic expectations, and through-the-ceiling valuations. Today, we may be headed towards the bursting of the AI bubble.

That does not mean that data science roles will lose their appeal overnight or that what they do is without value. After all, software engineers are still in demand for all the progress and automation that software engineering has seen in the last few decades. But it probably means that a recalibration is due, and expectations should be managed accordingly.