How to Search in Twitter

Last week I offered anyone who might be interested a copy of the batch files and other tools (cURL, grep, my nasty little program and the Excel spreadsheet) for analyzing Twitter "Tweets" -- all you had to do was send an e-mail to gearhead@gibbs.com with the subject "TA". I've had quite a few requests and you will get the files in a few days -- the delay is because I found a bug in my nasty little program and I haven't had time to fix it.

While the Excel approach to analyzing Tweets was interesting, I admitted all along it was clumsy and ugly. So if you want to keep track of a current hot topic, say, a mildly interesting topic such as swine flu, how else might you do it?

You could search for exactly that term you're looking for, but with Twitter Tweets there's another way to look for relevant content using "hashtags."

Hashtags is a convention that uses a hash mark (#) to flag words in Tweets users want to be treated as description tags. Thus, the hashtag for swine flu could be "#swineflu" (spaces aren't allowed so hashtags usually merge the words or separate them with dashes or underscores).

While we're on the topic of tags you might want to check out "Tagging: People-Powered Metadata for the Social Web" by Gene Smith (also see Smith's Web site for more on tagging). This book contains useful information but is somewhat dry and more suited for a technical audience (it also has a fair amount of code). Curiously, it mentions nothing about Twitter or hashtags despite being copyrighted in 2008. I'll give it 3 out of 5.

Anyway, hashtags can be searched for on the aptly named hashtags.org, which is way smarter in this area than Twitter's own search service because hashtags.org will also give you a list of all of the other hashtags that contain the same string. Thus, along with #swineflu you'll see the tags #swineflu, #swineflue, #swineflu-cbc, #swineflu-related, #swineflu-ing, #swineflu-nz. This can be really useful as the great unwashed occasionally pick an incorrect spelling (#swineflu-cbc, for example, was apparently an attempt to type "#swineflu-cdc").

What's great about hashtags.org is that clicking on a specific hashtag amongst those suggested will take you to a page that shows a graph of the hashtag's popularity over the past month (which wouldn't work for the term "rovio" we've been discussing over the last few weeks as it appears no one has used it in a hashtag).

Hashtags.org also provides a news feed for the selected hashtag which is, by default, in Atom format (although it appears that if you change the URL from ending with messages.atom to messages.rss you'll get an RSS formatted feed).

There's a small problem if you want to do anything with this data in Excel: While Google Reader will happily access a hashtags.org feed in either format, it appears Excel will not. And, of course, Excel reports the problem with a cryptic error message that requires you to go and find a log file that contains the error report. Argggh.

Another interesting Twitter search tool can be found at Twitscoop. The home page shows a tag cloud of the most popular words being used on Twitter (which refreshes every 20 seconds), and offers a search box that brings up a list of Tweets mentioning that topic and a graph showing the number of mentions over time.

Click on the graph or the link at the foot of the results and the next page will be dedicated to your search results and allow you to change the graph time window from the default six hours to either one day or three days.

I'll give hashtags.org a rating of 4 out of 5 (an API would increase its score) and because they don't provide any kind of news feeds or API I'll rate Twitscoop at 3 out of 5.