Harvesting Codeplex Statistics

Update: The method described below does not work anymore. The page on CodePlex that was POSTed to was closed to external queries and so doesn't allow CORS any longer... Alas, it was fun while it lasted!

For a while now, all MOBZystems code is hosted on Codeplex. That has really proven to be a good move, if only for the download statistics Codeplex provides for each project. There is a Statistics page containing detailed historic information about the project, including graphs. That's fascinating information.

Harvesting Codeplex statistics for a single project

But of course, I need (want?) more: an aggregated view of all our projects on Codeplex. But there's no way to get that from Codeplex - that is: not from the web site itself. The number of page views, visits, and downloads is shown on every project page, though, for the last 7 or 30 days, or the lifetime of the project. That information is harvested using the following Javascript code:

Basically, that means we can harvest those statistics as well, by POSTing to http://runnet.codeplex.com/stats/getActivity and supplying a period argument. Some more digging around reveals that the argument can be 7, 30 or -1 for 'All'.

Aggregating Codeplex statistics

The following code snippet is the heart of a console application that retrieves those statistics for a series of projects (the ones we host on Codeplex, of course):

''' <summary>
''' Data returned from GetActivity
''' </summary>
Private Class ActivityInfo
Public pageViews As Integer
Public visits As Integer
Public downloads As Integer
End Class

The code that does the heavy lifting is basically adapted from MSDN. It creates a POST web request to /stats/getActivity on the project's domain on Codeplex, adds a period argument to the POST data and reads the response into a string. That string is then deserialized into an ActivityInfo object using JSON.NET.

Here's the code:

''' <summary>
''' Get the activity information for a Codeplex project
''' </summary>
''' <param name="projectName">The name of the project (as in [projectname].codeplex.com)</param>
''' <param name="period">7, 30 or -1 ("All")</param>
''' <returns>A populated ActivityInfo object</returns>
''' <remarks>No error handling!</remarks>
Private Function GetActivityForProject(
projectName As String,
period As Integer
) As ActivityInfo
' Set up a WebRequest
Dim request As WebRequest = WebRequest.Create(
String.Format("http://{0}.codeplex.com/stats/getActivity", projectName)
)
' Use POST to supply the parameters
request.Method = "POST"
request.ContentType = "application/x-www-form-urlencoded"
' Set up the POST data
Dim postData As String = String.Format("period={0}", period)
' Write it to the request stream in UTF8 format
Dim byteArray As Byte() = Encoding.UTF8.GetBytes(postData)
' Set Content length BEFORE writing
request.ContentLength = byteArray.Length
Using dataStream As Stream = request.GetRequestStream()
dataStream.Write(byteArray, 0, byteArray.Length)
End Using
' Get and process the response
Using response As WebResponse = request.GetResponse()
Using dataStream As Stream = response.GetResponseStream()
Using reader As New StreamReader(dataStream)
Dim responseFromServer As String = reader.ReadToEnd()
' Return an ActivityInfo from the content of the response
Return JsonConvert.DeserializeObject(Of ActivityInfo)(
responseFromServer
)
End Using
End Using
End Using
End Function

The output of the resulting console application (the complete VB.NET code can be found in Codeplex Stats) is:

Of course, the data presented here must still be imported into a database or a spreadsheet to be useful over time, but that's left as an exercise for the reader ;-)

As usual: enjoy!

PS: Since there is - to my knowledge - no public API to get these statistics, the usual caveat applies: the interface with /stats/getActivity may change without notice, so any code that depends on it (i.e. all of the above) may break anytime.