Data Scientist: The Sexiest Job No One Has

The data scientist has been called the sexiest job of the 21st century, but it's largely going unfilled. That's a huge problem for the business world.

I've spent most of my career working with new technology, most recently helping companies make sense of mountains of incoming data. This means, as I like to tell people, that I have the sexiest job in the 21st century. That's not me talking. SmartData Collective, an online community moderated by Social Media Today, has called big data scientist "the sexiest job of the 21st century." Who am I to disagree?

Besides making folks like me feel good, that designation might offer another benefit. It'll help draw new talent into our field. This is currently a big problem, and it's getting bigger every day.

The role of data scientist has changed dramatically. Data used to reside on the fringes of the operation. It was usually important but seldom vital -- a dreary task reserved for the geekiest of the geeks. It supported every function but never seemed to lead them. Even the executives who respected it never quite absorbed it.

But not anymore. Today the role set aside for collating, sorting, and annotating data -- a secondary responsibility in most environments -- has moved to the forefront. In industries ranging from marketing to financial services to telecommunications, the data scientists of today don't just crunch numbers. They view the universe as one large data set, and they decipher relationships in that mass of information. The analytics they develop are then used to guide decisions, predict outcomes, and develop a quantitative ROI.

Data used to be the backup for instinct and intuition, but now it's instrumental in how companies know which marketing campaign is working, which new markets have the best potential, and which consumers represent the best prospects for growth. Data is the ultimate roadmap, and job candidates who can navigate it are poised for success.

But here's the best part: Since it's not a one-dimensional discipline, data scientists can emerge from just about any field. A good data scientist is someone who has the right tools (math, programming, critical thinking), is self-sufficient (doesn't need someone else to implement his or her ideas) and has an interest in understanding the context in which the skills can be applied. This is what the marketplace seeks.

A marine biologist with a PhD might have a strong grasp of both quantitative and qualitative research methodologies, and those skills perfectly meet the needs of many companies looking to mine data for intelligence they can use to take action. Baseball statisticians can leverage their passion and apply it to solve problems in corporate America. Many of these professionals probably don't even think of themselves as data scientists per se, but they should. There's a huge need, and here's why.

The all-purpose big data market is booming. Many news stories beat the drum of chronic unemployment, but Gartner estimates that there will be 4.4 million IT jobs created to support data analysis in just the next two years. Almost half these jobs will be in the US, and each will lead to three more. This field alone could help turn the job market around.

However, it's clearly not happening now. Countering the good news, Gartner also sees a staggering data scientist shortfall in the pipeline. There are only enough qualified applicants to fill a third of those positions. In other words, despite the importance of the data scientist position, they are set to become a "scarce, valuable commodity."

This is why decision makers in corporate America -- in particular the technology and business-centric professionals reading this -- need to refocus their attention on the institutions they left behind so long ago: middle and high school. That's where the problem begins, and it has to do with a simple acronym: STEM (science, technology, engineering, and math). The private sector needs to fund initiatives that bring young STEM prospects into the workplace to show them how the real world uses the information that people generate and demonstrate how to apply their interests to this discipline -- for instance, data visualization.

Deficiencies in school today will have a crippling effect on the workplace tomorrow, and that's why this is so important. The so-called sexiest job of the century isn't going away soon. Data scientists will lead the way in innovation and economic growth, but we have to help develop the next generation.

IT groups need data analytics software that's visual and accessible. Vendors are getting the message. Also in the State Of Analytics issue of InformationWeek: SAP CEO envisions a younger, greener, cloudier company (free registration required).

the hard sciences have some fields that are data-intesive like bioinformatics and astronomy. the practitioners necessarily have skills to deal with (computational) and analyze (stats, machine learning) large data.

Yes you are right if HR continues to run a checklist to find candidates then anything that is new and innovative will always have a candidate pool that is limited, even if the population in general has all the necessary skills needed to implement technology which will result in efficiency. I guess, this would be a human error rather than market disequilibrium.

It is not to say that markets are not in disequilibrium, as we hear so many times that individuals wanted to get into a computer specialized field and somehow ended up getting an IT degree over a CS degree, the later being the one which the market desires. And ironically, I feel that it is big data itself that needs to help students find degrees that will be in high market demand.

The problem is that firms seem to want to hire only experienced data scientists, but very few people actually have that experience. There are people out there (not lots of them, but some) who have the data and statistics skills as well as the proper aptitude to explore data and find elusive insights - Companies should start to consider these people next.

I think we need to leverage the resource/talent from other fields such as mathematics, statistics, etc. to get more data scientists. You need strong background in mathematics to become qualified for this kind of sexy job. The short-term professional training is far from enough to transform an IT professional to a data scientist.

I can only liken this to how scientists have started to involve gamers in solving complex scientific problems. I love the idea of looking at candidates from other backgrounds, not just IT, to work with Big Data. It's a very tricky thing, trying to look at mass amounts of data and draw correlations. If you ask me, it's the creative types that would make strides in this, as they tend to think of different ways to relate information, and can often mentally visualize how data is connected. The problem is that these roles still scream IT geek, as sexy as you may make it seem, so many folks who would do wonders in this type of role would still question how they could possibly be qualified for what they deem to be a technical role.

What I take from this is you're really hiring for a skill -- a way of thinking -- and not some specific training. Is the risk here similar to where the HR team hires technical people based on a checklist of software they've worked on, rather than a broader ability to apply tech to our company's needs?

Your views mesh with a recent guest column that explained how a data science team benfitted from hiring a biologist. What other specialties teach people to process large patterns and make connections -- and this might produce valuable data science team members? I even think of musical composition here.

To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.

IT pros at banks, investment houses, insurance companies, and other financial services organizations are focused on a range of issues, from peer-to-peer lending to cybersecurity to performance, agility, and compliance. It all matters.

Join us for a roundup of the top stories on InformationWeek.com for the week of November 6, 2016. We'll be talking with the InformationWeek.com editors and correspondents who brought you the top stories of the week to get the "story behind the story."