Twitter Hooks

June 15, 2009

When I first gave a public talk on webhooks in 2007, the guys at Twitter were the first ones to come up to me saying they wanted to implement them. Unfortunately it never happened, and this was well before they were as popular as they are now. It wouldn’t scale, they said.

As an alternative, just a few months ago, they released a stream API using long-running HTTP. It not only included a firehose stream, but a stream that allowed you to pull updates from selected users. I knew immediately from this I could build a webhook transformer, effectively giving Twitter webhooks. Unfortunately, the stream for selected users was behind a EULA that prevented that use of their API. Something about not redistributing data.

Then a few weeks ago, they added a selective stream that was publicly available. I had already designed the system and started writing some code before when I found out about the EULA. It took me a while to find some time, but last night and today I did it. I finished a proof of concept service called Twitter Hooks.

It’s very simple and is mostly about the plumbing behind it. You sign in with your Twitter account using OAuth, then you just give it a callback URL. Behind the scenes, it has a stream consumer listening for updates from the users that have a callback URL. When it gets an update from you, it posts it to your callback URL.

Obviously, there is only so much you can do with your own tweets. The next step is to listen to updates from your friends and post them to your callback URL. After that, using some trickery not involving the stream API, it will post direct messages and new followers to your URL. Throw in keyword tracking, and you have hooks for almost every significant event in Twitter. John Herren would be proud.

The problem is… Twitter is still too limiting with their stream API. As it stands, the non-EULA stream only allows following 200 users. That means Twitter Hooks can only support 200 users. It also means there’s no way we can roll out the friend update hook because some people have over 200 friends by themselves. Until that’s resolved, Twitter Hooks will remain a proof of concept demo.

But I have to say, having this is pretty sweet. The stream API is about as real-time as you can get. While experimenting, I was getting updates from the stream quicker than the page reload from posting a new tweet. This is, of course, then posted to your URL immediately for you to do whatever you want with. Here’s a tweet of mine posted to PostBin:

If you want to play with Twitter Hooks, it’s currently limited to users I authorize since it can only have 200 users and it’s terribly early/rough. Just @ me on Twitter and I can get you set up.