Survey Shows Big Data Projects Getting by Without Data Scientists, For Now

You hear it all the time: There simply aren’t enough trained data scientists to support the demand for Big Data analytics.

But here’s an interesting fact from TDWI’s best practices report on “Managing Big Data”: The data scientists aren’t really managing it now.

Actually, there’s an incredible range of job titles that manage Big Data. Out of 297 responses from 166 respondents (they could choose multiple options), only 6 percent said data scientists manage Big Data in their organizations.

Application developer, business analyst and systems analyst or systems architect also received 6 percent of the responses.

The top choice right now for managing Big Data is the data architect, at 16 percent, followed by the data analyst at 10 percent. Database administrators garnered 7 percent of the tally, with the rest going to a motley crew of IT jobs, including domain expert (seriously?) and ETL developer.

You could argue that that’s because of a shortage of data scientists. However, it’s worth noting that this particular survey attracted “unusually large percentages from the two kinds of organizations that are most prone to Big Data,” such as midsize to large Internet firms and corporations with $10 billion or more in annual revenue.

In other words, the very type of organizations you would expect would be able to attract data scientists. (Whatever that means.)

The data warehouse group also houses most Big Data projects, TDWI found, followed closely by central IT. Business units or departments are the next highest environments for housing Big Data, with about half as many responses as central IT.

TDWI also asked respondents which data management disciplines or teams were needed to manage Big Data.

Interestingly enough, BI and data warehousing skills topped the list of “strongly involved” skills, followed by data integration. DBA skills ranked third in the survey, with enterprise data architecture and data quality as the next most-popular choices.

So it’s interesting to see that most companies aren’t even using data scientists right now.