How Search Became Your Company’s Most Valuable Programmer (Part 3)

CHISEL

This is a series of posts about how software development and programmers have become increasingly reliant on developer communities like Stack Overflow and search engines to help them develop and ship code. We think this is a fundamental shift in the nature of software development that has wide ranging implications.

Part 3 – How Many Programmers are SAPs?

At a guess, 10s of millions, but estimating the total number of Search Addicted Programmers (SAP) is complicated by not really having an accurate estimate of the total number of programmers on the planet, or for that matter, how to definea programmer.

One quick way to get a lower bound on the number of SAP programmers is to look at the traffic to websites like Stack Overflow which as of this writing has approximately 22 million unique visitors globally per month, 85M visits and 386M page views. These numbers are just simply staggering, not only completely blowing estimates for the total number of “programmers” in the world out of the water but also pointing to a level of reliance of programmers on the Stack Overflow community and knowledge base that is honestly a little frightening.

And Stack Overflow is just one part of much larger ecosystem of online resources which developers interact with and have become dependent on. Many languages and frameworks have their own communities with similarly large repositories of knowledge specific to their community. In our recent MSR 2013 paper analyzing the types of links that programmers post on Stack Overflow we attempted to sketch out this online programmer ecosystem by analyzing of the top 20 most frequently linked to domains.

Domain

Unique

Total

stackoverflow.com

301,134

449,365

microsoft.com

128,875

364,315

wikipedia.org

40,989

178,578

google.com

67,274

140,190

github.com

71,761

132,037

jsfiddle.net

119,330

122,964

php.net

27,922

110,790

oracle.com

31,438

78,467

jquery.com

4,534

76,415

apple.com

31,270

66,456

android.com

11,073

64,814

sourceforge.net

19,328

56,758

apache.org

18,148

56,081

python.org

12,510

49,460

sun.com

13,640

38,555

blogspot.com

20,049

35,461

msdn.com

16,170

33,814

codeplex.com

9,414

33,643

mozilla.org

7,771

33,464

mysql.com

6,201

32,262

We have only begun to analyze this data and there are likely issues with some domains not being entirely programming related (microsoft.com, apple.com, google.com for example) but given that these links are being posted to Stack Overflow we can assume that a large number of them are referencing programming related topics. Unfortunately as few of these domains publish their website traffic, this data does not really help us improve our estimate of the number of programmers that use these resources. However, it does give us a glimpse into the large and complex ecosystem of online resources that developers rely on and how developers share useful programming knowledge stores with other developers.

This programmer dependency on the web is obviously not a new phenomenon, to plagiarize myself “(email, bulletin boards, Usenet, IRC, the web) were most often first colonized by programmers. The emergence of Stack Overflow is the latest evolution of this historical trend with programmers inventing or adopting a technology to meet their need to discuss what it is they do with other programmers.” (Treude et al, 2011) But what is new is the sheer scale of the phenomenon both in terms of the numbers of programmers visibly participating in this ecosystem and the terabytes of stored programming knowledge indexed and made easily available through powerful search engines.

This is a graphic I produced for our poster at Future of Collaborative Software Development workshop at CSCW 2011. It shows the growth in the number of Stack Overflow registered users, visits and views since its launch to early 2012. What’s really interesting about this graph is the explosion that started about September 2010 when the number of views started to grow relative to the number of unique visits. From a research perspective we don’t yet know what is causing this explosion or what it’s doing to programmers and how they write code. But we have some ideas, which we discuss in our next post.