Data scientist vs data engineer: How demand for these roles is changing | ZDNet

Research suggests many companies can’t find the talent they need, as they struggle to deal with turning their vast supplies of data into usable information.

Generally this means a hunt for data scientists, which has sent demand for recruits who can fill this particular job title skyrocketing. But while hiring more people who call themselves data scientists is one way to fix the problem, companies are also coming up with alternatives that don’t mean joining the race to hire a few of these elusive individuals.

Tech analyst Forrester warned five years ago that while companies were busy directing huge resources to attracting data science talent, there was a risk that they were forgetting to invest in the engineering capability that would help scientists create value from data. Now, it looks as if some companies are starting to address that imbalance. 

SEE: Software skills will get you far, but you don’t have to be a coder to make it big in tech

Loïc Giraud, global head of digital platform and product delivery at life sciences giant Novartis, recognises that the battle for talent was a huge issue not so long ago. But today, it’s less of a concern.

“I think there’s hype,” he says. “Two years ago, it was very difficult to get data scientists.”

Novartis has about 2,000 data scientists and Giraud says his battle for talent is now focused on other areas, including snaring data engineering talent and honing business analyst capability – and he expects other companies to come to similar conclusions, too.

“I don’t think the demand for data scientists is going to increase. I think you will find more technologies, which are easier to consume and for business analysts to do the science,” he says. 

“In fact, even in our organisation, we’re not trying to look for more data scientists. We are trying to build software solutions that can be used by more people and to democratise data science with business analysts.” 

Novartis is focused on finding the full-stack engineering capability it needs to help business analysts around the organisation make the most of the data it holds.

While data scientists use their skills to create models and solve problems, data engineers build and manage the infrastructure that sits between data sources and data analytics. Both are important, yet there’s increasing evidence to suggest that too much emphasis has been placed on data science at the expense of data engineering.

Another industry commentator has suggested “a course correction” is taking place. Data scientist Maruf Hossain wrote in a blog post last year that many organisations hire data scientists and then present them with work more commonly associated to data engineers. 

He suggests that this misalignment occurs because many data scientists join companies that don’t have strong technological foundations in place to run analytics.

The task then falls to data scientists to help build those foundations. So, when they should be coding or creating algorithms, some scientists end up fulfilling technical roles that are unlikely to fit snugly with their existing capabilities.

It’s worth nothing that, regardless of the role they end up fulfilling, companies are still on the lookout for data science talent: CodinGame and CoderPad’s recent Tech Hiring Survey identified data science as a profession where demand greatly outstrips supply.

Of course, whether those companies require full-blow data scientists or something more akin to a full-stack engineer is something that many candidates might only discover once they start working in the role.

To that end, the work that Giraud and his colleagues at Novartis have already undertaken presents some important pointers for managers looking to hire data scientists and for professionals looking to take on these roles.

SEE: How to get promoted: Five ways to climb the ladder and have a successful career

The company’s approach to ensuring its data scientist skills gaps have been filled during the past few years has involved a journey of discovery that’s now leading to a new focus on engineering and business analysis.

The company took a cloud-based approach and adopted Snowflake in 2017 as part of an all-encompassing effort – known as Formula One – to digitise every aspect of its operations.

Part of this approach included the creation of a new chief data office to promote the use of technology and data to improve decision-making processes in the organisation. 

“When we created our CDO office, we recruited talent from across the industry. We created a Data Science Academy and then we started to recruit a lot of people. We had a lot of statisticians in our organisation that we also converted into becoming data scientists,” says Giraud.

One of the key things his organisation learned quickly is that data science is of no use if you don’t have good data. 

For the first year and a half, data scientists at Novartis spent as much as 60% to 70% of their time identifying and curating data – rather than writing algorithms. 

That’s when the company started to think much more carefully about the talent it needed, and the crucial role played by data engineers. 

“Ultimately, as a data engineer, we want people who are good at integrating our data sets together – and the full-stack engineer makes your entire stack work in an integrated manner,” he says.

Today, the company’s 2,000 data scientists use tools from companies such as Snowflake, Databricks, Data IQ and Sage Maker to find smart answers to business challenges.

Those scientists are part of a team that is using data to help bring life-changing medicines to market quicker than ever before.

SEE: ‘I used to run a pub, now I work in tech’. Career changers on why they made the switch to IT

From initial research to manufacturing, trials and onto distribution, it traditionally takes as many as 12 years to bring a new drug to market. By applying data and artificial intelligence to these processes, Novartis believes it can reduce the time to nine years.

Giraud says the company’s tight grip on data science is helping it to decide which of its 500 trials a year should be taken forward and developed as a drug that can be pushed to market. And as the company’s data engineering platform continues to be honed, Giraud expect professionals across the business to take even more responsibility for the insight they create.

Six or seven years ago, his team used to create all the dashboards used in Novartis. Today, there are almost 3,000 people across the business who create their own dashboards. 

Data science, therefore, is being democratised – and Giraud wants to ensure his talented data scientists and engineers are focused on high-level activities that make the most difference.

“I don’t want my team to create a dashboard, as that has no value,” he says. “I want the business analysts and the business users to have a platform from which they can self-serve.”