Step 1: Finding the data

Ok, the first step of making a graph is obviously to collect the information you want to created the graph from.

In the Explore-menu on the site there's a entry called Most recent that shows all instructables in groups of 15. The URL for that page looks like this: http://www.instructables.com/tag/type:instructable/?sort=RECENT . After clicking on one of the numbered pages at the bottom the string "&offset=15" is added to end of the URL.

On each of the 15 entries on the pages there are a line that displays the author and the date of that particular instructable. "posted by Parker955 on Aug 4, 2007"

So we now know the URL for getting the data that we want to use and we also have a string to search to extract the data from the page.

Sure, I can do that. But do you really want to do it this way? I mean that this technique is good for pulling data from a site that you only have web access to. If you got access to the underlying info from the database itself the correct way would be to get the data aggregated into the needed format directly and then plot the data using php or jsp or whatever language the web server is using. Or maybe plot it with Excel connected to the database using a ODBC connection. But if you want to do it "the external way" how would you like to display the data? If doing it day-by.day we would currently have about 750 point in the dataset. A 750 pixel with graph with one pixel representing each day?

Well of course I can run a query on the database and get this, but that's only me. I thought others might be interested in seeing the spike in projects right before contest deadlines, for example. Perhaps I'm just being optimistic that anyone wants to see this data besides me.

Nah, at least I'm interested in seeing this, so we're two at a minimum :-) I've added a new step4 in the instructable with the script that can be run after running the monthly script. It takes the log file already generated by the first script and extracts daily data from it. There are some graphs and the excel file with that daily data in it. I could need some help visualizing this data because the graphs I've made out of it are very "noisy" and it's a bit hard to really see any real spikes...

Yea, I agree, but the level "customibility " of the graphs in Google Docs seems to have its limitations. You can't change the colors - at least I haven't found a way of doing it. But I've added a 3D graph done in Excel in the last step for another view of the data.