The Difference Between a Data Scientist and Data Engineer and Why You Should Care

April 19, 2018August 31, 2018

This post was featured in our Cognilytica Newsletter, with additional details. Didn’t get the newsletter? Sign up here

As Artificial Intelligence (AI) continues to become a focus for an increasing number of enterprises, these organizations are realizing how important it is to have the right people and skills in place. In particular, we have seen a significant increase in demand for data scientists in organization as AI, various applications of machine learning (ML), non-ML predictive analytics, and other so-called “big data” approaches continue to gain traction in the enterprise. In fact, the significant demand of data scientists has led to the talent crunch that we’re seeing across many enterprises and organizations — something we’ve discussed in our previous newsletters and podcasts.

Companies are searching for and competing for increasingly scarce data scientists. Salaries and signing bonuses for skilled data scientists continue to skyrocket, and the sheer number of code academies that are now focusing on data science is evidence of the significant demand for data science skills. However, are data scientists always needed by these organizations? Many enterprises, vendors, and startups often confuse the role of data scientist and data engineers. While these different roles share some traits and skills, at their core these are job descriptions that have two very different skill sets that are not easily interchangeable.

Data Scientists vs Data Engineers

About a half decade ago, we saw the emergence of the Data Scientist position. From the above linked article: “This increase in the demand for data scientists has been driven by the success of the major Internet companies. Google, Facebook, LinkedIn, and Amazon have all made their marks by using data creatively: not just warehousing data, but turning it into something of value.” Not surprisingly, any organization that has data of value is looking at data science and data scientists to increasingly extract more value from that information.

Originating from roots in statistical modeling and data analysis, data scientists have backgrounds in advanced math and statistics, advanced analytics, and increasingly machine learning / AI. The focus of data scientists is, unsurprisingly, data science — that is to say, how to extract useful information from a sea of data, and how to translate business and scientific informational needs into the language of information and math. Data scientists need to be masters of statistics, probability, mathematics, and algorithms that help to glean useful insights from huge piles of information. These data scientists usually have learned programming out of necessity more than anything else in order to run programs and run advanced analysis on data. As a result, the code that data scientists have usually been tasked to write, is of a minimal nature – only as necessary to accomplish a data science task (R is a common language for them to use) and work best when they are provided clean data to run advanced analytics on. A data scientist is a scientist who creates hypothesis, runs tests and analysis of the data, and then translates their results for someone else in the organization to easily view and understand.

On the other hand, data scientists can’t perform their jobs without access to large volumes of clean data. Extracting, cleaning, and moving data is not really the role of a data scientist, but rather that of a data engineer. Data Engineers have programming and technology expertise, and have previously been involved with data integration, middleware, analytics, business data portal, and extract-transform-load (ETL) operations. The data engineer’s center of gravity and skills are focused around big data and distributed systems, with experience with programming language such as Java, Python, Scala, and scripting tools and techniques. Data engineers are challenged with the task of taking data from a wide range of systems in structured and unstructured formats, and data which are usually not “clean”, with missing fields, mismatched data types, and other data-related issues. These data engineers need to use their programming, integration, architecture, and systems skills to clean all the data and put it into a format and system that data scientists can then use to analyst, build their data models, and provide value to the organization. In this way, the role of a data engineer is an engineer who designs, builds and arranges data.

Can there Be a Combined Data Scientist-Engineer Role?

While it might seem that the roles of data scientist and data engineer are distinct, data scientists and data engineers share many traits and skill sets. These overlapping skills include the necessity to work with and manipulate big data sets, programming skills to apply operations to the data, data analytics skills, and general fluency with systems operations. The Big Data Institute produced a good chart that helps visualize the roles of data scientists and engineers, and where their skill sets overlap.

While the overlap is substantial, it’s clear that the emphasis of the roles of the data scientist and data engineer are still distinct, and as a result, they’re not particularly interchangeable. Even more importantly, when interviewing and hiring data scientists and data engineers you need to make sure you’re asking the right questions and seeking the right skills from your candidate. Are you asking your data scientists to spend most of their time data engineering? Are you demanding more data science capabilities than your data engineers have had experience, training, and even aptitude or desire to do? Are you confusing your job candidates by asking them engineering questions in an interview for a data science position, or data science questions in an interview for what’s fundamentally a data engineering job?

More importantly, the rise of data science code academies, workshops, and training begs the question: are these training and code academy activities focused on the science behind data science, or the engineering and programming behind what is fundamentally data engineering? Or worse… are these activities muddling the mix by doing a bit of engineering with data science and not adequately focusing, or screening, their attendees and participants by determining which area of important big data and ML analysis that these individuals should focus on?

While it might seem that you can do a bit of engineering in a science role, or a bit of science in an engineering role, mixing the roles could be very detrimental to your organization’s success with an ML or data science initiative. Data scientists who are pushed to do engineering roles without the background, skills, or aptitude can easily misconfigure or misuse technology or write programs that are inefficient, costly, and waste time. Likewise, asking individuals who have fundamentally an engineering background and aptitude to learn the complicated mathematics of data science can result in incorrect conclusions for an organization about its information that can lead to disastrous outcomes. Specialization is important – this is why doctors perform checkups and phlebotomists draw blood. It’s possible for the doctor to draw your blood and the phlebotomist to understand the lab results, but why would you want to risk your comfort and health this way?

Where Does the Data Scientist fit in your Organization?

So, most organizations need both data scientist and data engineer roles if they are trying to address problems that require data science solutions. Since these roles are not interchangeable, it is improper to try to seek the single, magical, data scientist-engineer unicorn. Yet, while you might need multiple data scientist and engineers in your organization, the ratio between the two is rarely 1:1. For most organizations, it makes sense to have more data engineers than data scientists. The reason for this is that data scientists have learned to operate with large volumes of clean data, but getting lots of clean data from many disparate systems can be multiple full-time jobs. It simply takes more work to move and clean data than it does to conceptualize data models and run analyses against the data sets.

Also, the organizational reporting structure for the data scientist is incorrect in most organizations. We frequently see the data scientist roles reporting to the technical team. However, this doesn’t make sense. The data scientist isn’t (usually) asking technology-specific, implementation-specific questions and data analyses. Often the challenges the data scientist is facing are line-of-business specific. As such, the data scientist should report to the strategic decision-making parts of the business that represent the specific lines of business that the data scientist is assisting.

If Data Scientists are Business-Centric Roles, Will we see Business-centric Tools for Data Scientists?

If we truly see data science and engineering as separate roles in the organization, then it makes sense to think of the tools they need as separate as well. Many vendors entering the data science / machine learning world are muddying the waters and making things even more confusing. They’re saying their tools are for data scientists, but everything about those tools are primarily for data engineers, with a sprinkling of data science on top: “algos for programmers”.But this doesn’t make sense. The natural environment for a data scientist is in an analytic, data-oriented, model-centric tool, not something that has big buttons for cleaning data, moving data, and moving stuff from your private environment to the cloud. This is like giving a driving instructor the pieces of a car and saying “build this car yourself, and then teach others how to drive it”.

Rather than engineering and programming-centric tools, data scientists need data science-centric tools. Right now there’s a growing collection of these tools, often emerging from data or predictive analytics environments, that suit the needs of data scientists. However, it’s possible that even more business-centric tools might be appropriate, especially as the data scientists become more embedded with the line of business. For example, decades ago if you wanted to operate on large volumes of data in a spreadsheet-like format, this involved programming, but tools like Excel introduced things like pivot tables and now business managers are going hog-wild with all sorts of analyses. Will we start to see tools like Excel embed data science capabilities, or business-centric data mining and analysis tools likewise start incorporating ML models? We think so! In which case, technology-centric companies, prepare for a run for your money. Your data engineering-centric “ML platform” will have real competition.

As the talent gap for data scientists continues to widen, there is no doubt that we will see new tools created out of necessity to allow non-technical (read: business) people to run, test, and analyze data. Strategic business managers will begin to learn data science, without needing or wanting programming or data integration experience. Traditional data scientists will still be needed to run very complex analysis of data. But for the most part, basic analysis will more to the business unit due to increasingly easy-to-use tools. This means we have still yet to see which tool or technology will be the dominant one for ML and data science in the enterprise.

As you can tell, Cognilytica is paying very close attention to what is happening with data scientists, data engineers, and tooling for both. We’ve analyzed the vendors, and we know where the vendors are heading. We know that many vendors are misleading and muddying the market with tools that blur the lines. We’re seeing everyone enter this market trying to plant a flag as the “ML / AI platform”. We know that there will only be a few real winners, and we’ve already started to sort who is winning and who is not. If you want to know more about the players, how the roles of data scientist and data engineer fit into your organization, and want to know how all these trends are changing, reach out to us, and be one of the many firms we’re helping navigate their way through this journey to AI / ML success.

Get weekly research, insights, analysis, vendor reports, end-user use cases, exclusive event discounts, and other information delivered once a week to your inbox. Sign up to be one of the thousands getting Cognilytica Insights.