Monday, 30 May 2011

3 November 2012: Interim edits (cos I'm not doing this in one go!) - I'm at twapperlyzer in the list below. Anything below that hasn't been checked for a while, everything above has been.

Post history: This post was born on 30 May 2011 at a time when there were many more free tools available with which to wrangle tweets. Since then Twitter's changed its terms of service and Google Realtime no longer exists making some of the information now wildly out of date, in October 2012. The original post will be placed HERE as a PDF for completists and archivists who want to know how things used to be. But the text below has been significantly changed.

I've found Quora to be the best resource for finding out about new tools for a couple of reasons: (1) is that Quora answers persist and don't fall off the edge of the page compared to Twitter and (2) people organise the information into questions and answers making it much easier to find! Also people tend to post this info to Quora more than on Twitter (or I'm not following those people).

What have we got here then?

Part A is the alphabetic list, with a bit more info about each individual tool.

Part B looks at a range of tools according to the function(s) they can perform - B1 for finding tweets, B2 for trapping tweets and B3 for handling conference or event hashtags.

I might have sucked all the fun out of this post by itemising everything but it was getting a bit chaotic ;)

Currently I useSearchHash, Epilogger, Chirpstory, Storify and / or Twilert to trap hashtagged event tweets. I’m also using Storify for small events because the interface is so good and it lets you contextualise things. But Chirpstory is very good as a quick trapping tool too, for larger numbers of tweets - last time I looked you can grab 20 tweets at a time with Storify and 50 at a time with Chirpstory.

Part A: alphabetic listing of tools

BackTweets– "search for links on Twitter"
I searched for brodiesnotes.blogspot.com and found quite a few tweets including a couple from 2009. The search results don't actually give you either the link to an individual blog post (only the main blog homepage address) nor do they give you the link to the original tweet, though that is discoverable.

Blackbird Pie– lets you embed an example tweet in a blogpost - my example here. Defunct as Twitter's now makes it possible to embed tweets in blogs very well. To get the embed code click on any tweet to expand it, then click on Details, then grab the code.

Chirpstory.com– suggested by @annindk. Similar to Storify.
This letss you grab a bunch of tweets (up to 50 at one go) and also lets you reorder them with one click so that readers see the earliest tweet first. Very useful tool, one of my favourites. You can also easily remove RTs and any duplicates you might have copied over twice. Here's one I made earlier http://chirpstory.com/li/1600

brilliant. This automates the saving of hashtagged tweets and can pick up tweets earlier than the time at which you set up your 'log'. Currently can't embed into blogs, but does provide a basic analysis of who's tweeted the most on a hashtag.

Favstar – lets you see favourite tweets (yours, or someone else’s) – since not every single one of someone’s tweets will be favourite this is a highly selective filter of available tweets. Here are MY tweets that OTHERS have retweeted.

Freezepage – this used to trap an exact copy of whatever web page you were looking at in case the page is changed or deleted in future, however I've noticed it doesn't work as well for Twitter and have stopped using it. Here's one I made on 13 October. My favourite example of frozen tweets marks Gillian McKeith's rudeness to someone pointing out she has no real PhD qualification.

FriendFeed – n 2008 I imported my Twitter feed (http://twitter.com/JoBrodie) into this tool and it's still capturing them. I thought it would stop what with Twitter mucking about with what it lets escape from its 'ecosystem' but so far all is well. Here'smy first tweet, which I originally found in FFeed but time has passed and it doesn't appear by searching anymore.

GrabChat.com – made by @dean_jenkins for people to 'archive your conversation'. It lets you export a whole bunch of hashtagged event tweets, really good.

Guardian Twitterfall – app for events where you want to pre-moderate the tweets that appear on the wall – it's for people who understand what JSON and AJAX mean, sadly that doesn't include me.

hashtags.org – apparently it's meant to show you tweets with a particular hashtag along with a little graph but I think you have to spend money for it to do so, so I've not bothered with it – via this slideshare presentation.

Hashtracking - via rjleaman. I follow them on Twitter (@hashtracking) and often pick up some really interesting stuff, although I've not really tried out the tool very much. I tend to go for the cheapy freebies but I hear good things about this one, probably best for corporate use if you're a cheapskate like me.

SearchHash - wonderful. Does almost exactly what WTHASHTAG used to do - lets you archive and export a pretty complete record of a hashtag.

You are also given the tweet ID and the 'from user' author in the table. To find the link for the original tweet use the formula http://twitter.com/AUTHOR/status/TWITTERid eg https://twitter.com/JoBrodie/status/75299030430777344 You can export tweets as a .csv but the data table isn’t the prettiest of things to share with people so Twilert might suit better.

SnapBird – lets you search a little further back than Twitter's search (though Storify or Chirpstory do this and I use them more)

Social Searching – no idea, log in with Facebook - can anyone tell me what it does?

Storify– I use this a lot but not quite a much as Chirpstory. Once you've trapped a tweet in a Storify story it will remain there even after the original tweet has been deleted. Use this power wisely...

tagdef – find a hashtag and see what it means, or define it yourself if you know - via this slideshare presentation. It's not possible to 'reserve' a hashtag and say 'it means this' and others can't use it, but sometimes it's helpful to know what it's supposed to mean. See also tagalus.

The Archivist – I found this quite good but you have to keep an eye on the tweets as they disappear ‘off the bottom’ – very easy to miss tweets if you don’t actively monitor it. You can download in .csv (spreadsheet) or zipped file formats. Desktop version also available.

ThinkUp – downloadable free open-source app which analyses your tweets, Fb and Google_ activity. If you have your own server you can use it on that, if not they've partnered with a cloud company PHP Fog and you can host it there instead. h/t @martinjmckenna

Topsy – search the social web: this is very good and can find old tweets (I found some from 2009) but it doesn't let you scroll back and forward like Google Realtime used to let you.

Twapperkeeper - closed on 6 January 2012 - before that it let you access two archives free, and you had to pay for more than two archives. See also YourTwapperkeeper, the open source version - its current status is unknown to me. You can also query (but not create) Twapperkeeper archives using Eduserv's Summarizr (h/t @briankelly)

TweetChat - really good for watching a live stream. You can type in your hashatag of interest and it will present you with all the tweets in real time. If you miss a few you can scroll back and it will pause for you. Used very successfully in the first Twitter Journal Club aka #twitjc. You can also sign in via Twitter authorisation and tweet directly from the programme (it will automatically add the relevant hashtag to all your tweets). It also lets you block spammers from view (hat tip @fidouglas and @silv24). See also Monitter and Twitterfall.

Tweetdoc - lets you create a PDF document bringing together not more than 500 hashtagged tweets and keep them as a permanent record h/t @martinrue

TweetScan - temporarily closed to new registrations so I can't tell you anything about it

Tweet Tunnel - lets you go back 3,200 tweets (as Twitter itself does) but you don't have to scroll through all of them but can 'page' through (random access versus linear access I suppose!)

Twitterfountain - Currently inactive though website promises a revamp. Was a wonderful way to present tweets at conferences on a display screen. Until it's back, use Twitter Fontana etc.

Twitter Fontana - pretty cool, for use in foyer display screens where you want to present tweets prettily. No use for archiving though.

Twitter Search - Twitter's own (much improved) search facility, note that there's also an Advanced Search but it's still fairly limited (note, it's very good it's just that I compare everything to the no-longer-with-us awesomeness of Google Realtime).

Twitter widgets - "Widgets let you display Twitter updates on your website or social network page" - if you want to show, on your webpage, what you're sharing on Twitter then use one of these widgets. There's an example on the linked page. See also B3.3 below.

Twilert - I like this a lot. It's good for capturing event hashtags by emailing them to you and the appearance is nice too (not sure how you can display tweets sent to you by email though). It sends in batches of 100 and you can have them sent every 15 mins if it's a very busy event. I wrote about it here. It's been around since 2009 and I only heard it about it at the end of June beginning of July 2011!

TwimeMachine – will let you access the last 3,200 tweets in a person's timeline

Twitter Advanced Search - I'm personally not a fan as it doesn't offer me any advantage over Google Realtime, but it is an exceptionally easy user interface (OK it is easier than GRealtime, but you can't scroll back and forwards in it which you can on Google).

Twitterfall - lets you watch a stream of tweets in real time. See also TweetChat and Monitter.

twittertoirc - "Twitter-to-IRC bridge: A hacky Ruby script to route Twitter status updates into IRC." by @tommorrisI said in an earlier edition of this post: Another tool I have ‘mentally’ investigated is IRC (internet relay chat) as a backchannel for conferences. I’m sure all nerdy conferences have an IRC channel but for regular conferences possibly not – these let everyone sign in, all posts are labelled as who wrote them and it’s possible to create a transcript. Seems perfect – what I’d like to do, but am not clever enough to work out how, is to export a live Twitter RSS of a hashtag into an IRC channel and capture the tweets that way (maintaining tweets’ authors identities). – then discovered Tom had written this programme.

twtr2rscbeta - ostensibly sends you tweets by email (as Twilert does) or to Evernote (as IFTTT and a Twitter RSS feed trigger does, until March 2013) but I couldn't get anywhere with it.

UgoTwitt - your Twitter replies sent by email. May not work now Twitter's tweaked things.

Visible Tweets - lovely, lets you display hashtagged event tweets in a nice way on a large screen in the foyer at your event or in the event room h/t @clangyandjammy. If you want people in the room to see tweets as they're coming in though I'd recommend something like Twitterfall or Monitter.

WordPress plugin - trap de novo, curate from this point forwards - see Twitter tools

wthashtag - What the Hashtag, no longer functioning in the way it used to alas and is now What the Trend API. If you have programming skills (JSON...?) you may get something useful from it but it's beyond me.

Part B: tools ordered by functionB1. Finding tweets either from May 2010 onwards or 3,200 tweets ago and no earlier
Note that all of your tweets still exist somewhere on Twitter's servers but they stop being amenable to its searches once they are superseded by a whole bunch of other tweets.

To prove that old tweets don't really disappear here's Jack Dorsey's tweet from 22 March 2006 https://twitter.com/jack/statuses/81 - he co-founded Twitter I think. This might be the earliest tweet that remains on Twitter.

B1.1 Scrolling back on Twitter
If you've posted fewer than 3,200 tweets in total (including RTs) then you will be able to scroll continually back (160 pages maximum at 20 tweets per page) to your first one. On New Twitter the page will continually expand with another 20 tweets as you scroll to the end of the page, but on old Twitter you'll need to click on the More button.

B1.2 Fast scroll (jump) by tweaking a web address: 'URL hacking' - old Twitter onlyYou should also be able to find your own Twitter URL that will let you jump back pages by changing numbers in the URL. It seems that this might not work if you're using new Twitter and it's pretty fiddly anyway. Scroll to the bottom of the page until you see a More button - this might not appear on new Twitter - hover over it and make a note of the URL that appears in the status window (I open up a small notepad window to the side and type it in). You'd think you could right click, save URL - but no - for some reason left clicking and right clicking both activate the More button so you will have to make a note manually (you could also take a screen shot of the page using Print Screen, copy it into Paint or Word and take down the details at leisure).

B1.3 Find your first tweet Posted more than 3,200 tweets? Then it's probably not straightforward.
If it's working (and you have fewer than 3,200) you can cheat and use http://myfirsttweet.com/, or the 'jump' function mentioned above. If you've been a bit chattier and have several thousand then you'll have some difficulties finding your first.

Google Realtime is a wonderful thing from Google that lets you find tweets going back as far as May 2010 - I use it quite regularly, more details below in the list of tools.

It was my good fortune shortly after signing up to Twitter to import all of my tweets into my FriendFeed account meaning that I could search, very easily, for any of my tweets. FriendFeed's search has become much less reliable of late - but it did (though no longer does) let me find my very first tweet https://twitter.com/JoBrodie/statuses/841825864. Oddly, searching now for this no longer works - instead of picking up the original tweet, it picked up only the later search I ran for the tweet in 2009 ;)

B1.4 Finding tweets from a week or two agoI think Google Realtime is probably the best for this, see more details and alternative in the list below. Possibly much more difficult now unless you have tweets fed into FriendFeed. You could also give Topsy a go.

B1.5 Find tweets on a topic or hashtag Twitter search is good for finding recent ones, just type in your keyword(s) or hashtag. For older ones, use Google Realtime.

B2 Trapping tweets – see also B3.2Freezepage, Chirpstory and Storify will let you capture a tweet. It will remain captured even after deletion, see Freezepage and Storify details above.

B3 Conferences and events: for events organisers and delegatesB3. 1 Follow an event hashtag in real-time, at conference or at homeTwitter itself is pretty good for this - just search for the hashtag and sit back and watch the tweets auto-refresh. Also particularly good is Tweetchat - neither requires you to have an account to watch the tweets. If you have a Twitter account you can sign in to Tweetchat and send tweets that have the hashtag of interest already written in the tweet for you. More info below, and see also Monitter and Twitterfall.

B3.2 Display a Twitterwall in the foyer or the conference roomPrettiest so far seems to be Twitterfountain, ideal for the anteroom display for people to coo over. For a backdrop that people can follow inside the event room I prefer Twitterfall (actually I prefer no Twitterwall but…). See also Guardian Twitterfall.

B3.3 Display event tweets on your website
If you want to bring in the results of a search for an event hashtag you can do that with Twitter Widgets. The 'Search' widget (the options are Profile, Search, Favorites, List) will pull in a hashtag or other keyword but if you worry that cheeky types will introduce rude words into the tag (or just want to show a selection) then you can instead use the 'Favorites' widget and as you favourite each relevant tagged tweet it will show up. This requires more work from you though... See also TweetGrid (listed above) for an option to let people search for anything on Twitter via a widget you can add to your site.

B3.4 Saving an event hashtag for later
Best in show is SearchHash or Twilert but there are a few other tools available too. Twitter's made it a great deal harder for people to download / export tweet archives. The sooner you set these up the better (ie before the event begins) because people often use the hashtag a day or even a week / month or two before an event begins. Chirpstory is great for bagging a run of tweets soon after an event and lets you capture them a page at a time whereas the similar Storify lets you capture them individually but also lets you intersperse commentary. The first two are better for big conferences, the latter are better for small things. Previously everyone used wthashtag (What the hashtag) but it no longer works, and Twapperkeeper only lets you create two free archives.

B3.5 Format tweets for conference output documents, eg blogs Storify is lovely for this – you can embed the final Storify story into most blogs, but to place an individual tweet into a blogpost directly… I need to find out what the tool is that I’ve seen used!

Found one - an example is Blackbird Pie which lets you select a tweet and 'bake' it into your blogpost. I've linked an example of me trying it out in the alphabetic list above. Although I've not tried it I understand the Twitter widget in WordPress will also do this.

Sunday, 8 May 2011

I think everyone has slightly different ways of approaching this but here are some things I do - not necessarily as coherently as I've put them here ;)

Almost every time I discuss a 'case' with fellow skeptics I seem to come away with a new tool or bit of useful information. Quite often I add them to the relevant section of the 'Skepticmedia' page I update sporadically.

I like writing lists - the following are not really in any particular order. Really we could have gone with bullet points :)

There's no guarantee that you'll get very far with this as sometimes people use third party agencies to do this for them, but always worth a shot. You can do a search for the domain name registration using one of the free DNS WHOIS lookup services, such as

3. If you find a registration name, "trading as" name or company address have a bit of a Google - you might find that they own another site. Similarly if the website has a phone number (for people to order the product) searching for that number can bring up other sites too.

4. Have a look on companies house

A company I recently reported for making misleading claims seems to have been additionally trading illegally as their company dissolved a year or so ago. You can search for companies using the free Web Check site http://wck2.companieshouse.gov.uk/

5. Have a look at the Medicines (Advertising) Regulation of 1994 (via @landtimforgot)

If the leaflet or website is offering a medicinal product (or service?? Does anyone know? ie does this only concern itself with things which are generally marketed as 'medicines'?) to treat any of the conditions listed there then they may be acting illegally.

Acting illegally rather trounces the making of unsubstantiated claims, but I'd always persist with making a claim for the simple reason that, if you blog about it later (if an adjudication etc has been made) then you're slightly increasing the amount of good information on the internet / raising awareness of scams etc.

This is a free app for anyone using the Google Chrome browser. It does two things - makes it MUCH easier to fill in the complaint forms and it keeps a record of complaints made about a site. This means you can use it to see if someone's already complained previously (this doesn't necessarily mean you shouldn't add another complaint of your own of course). I've previously written about this fab tool here http://brodiesnotes.blogspot.com/2011/04/fishbarrel-simple-google-chrome-app-to.html

For your own records you might want to copy and paste the text of the complaint before sending it off. Fishbarrel records the words on a website, but not any comments you add.

An ASA adjudication is available on a database within their website but doesn't appear to be that searchable via Google. To make it more indexable it's helpful to blog about it.

Because there's an unlimited number of people flogging nonsense and misadvertising it I doubt we'll ever stop each individual instance, far better to raise awareness and promote good information. Along with saving people money and signposting people to how they can complain about stuff I suppose my order of importance for doing this is as follows:

a/b) Raise awareness of scam sites

b/a) Slightly increase the amount of good quality information on the web

Wednesday, 4 May 2011

EDIT: 30 May 2011 - Have discovered a useful thing that is a huge help in moving tweets (or other items from left to right): double-click on an item in the left and it will snap to the top of the right hand side panel, or use SHIFT double-click to make it snap to the bottom :)

Storify (http://storify.com/) is a lovely bit of web-based kit that sort of lets you trap tweets and it appeared on my radar during my mourning period for What the Hashtag and Twapperkeeper. Those two tools would do the work for you, sniffing the tweestream and picking out the hashtagged tweets you'd asked it to save, then presenting them to you (all nicely in order) for export to a text file. Lovely. But these are more or less no longer with us.

Twapperkeeper doesn't let you export an archive of hashtags and will only let you save two archives. What the Hashtag has been taken over by What the Trend and is no longer doing that sort of thing. In fact, according to Twitter's terms of service it seems that it's not possible to do this:

Additionally, and maybe most importantly, Twitter's recent changes to its Terms of Service (TOS) now prevent us from providing transcripts of conversations around a hashtag.

Storify however, absolutely definitely lets you create an archive of hashtagged tweets, it even lets you embed it in your blog. It's awesome, but the major disadvantage is that you have to manually add each tweet that you want, and you're restricted to a number of tweets (I've not worked out how many yet). It would seem to be at odds with Twitter's TOS (as I understand it from what WTHasthag says above) but perhaps Twitter doesn't seriously envisage anyone using this as an archiving tool because of the amount of effort involved.

I've begun testing Storify to see what I can do with it and will continue to report here :)

Don't know - I have just played around with the #streettalks tweets and it let me go back (you have to keep pressing 'load more' until it stops giving you that option) a day or so - however this was a small meeting in a pub and not a large science conference where tweets tend to ticker tape out at quite a rate.

There is a limit on both the number of tweets you can access and how many days back you can go, and I think these are independent. For example if you were searching for the hashtag #madeuptag but no-one had written a tweet mentioning it for, say, a year or several months, it may well not show up in your tweets, so there is a time thing to be aware of.

While investigating Storify for use by @marilyneb just now we found that her hashtag of interest, #pas2011 could find four days' worth of tweets because it hadn't been used very much. So for capturing low volume hashtaggeed tweets Storify seems to work beautifully.

3. If you can go back to, say, 500 tweets, does stripping out the RTs (eg let's assume 200 of the tweets are RTs) allow you to access an earlier batch of tweets?

Don't know.

4. Transferring tweets from the left hand side panel to the right hand side panel is not that easy on a small laptop with a pointing device rather than a mouse. It's just a bit fiddly. For serious use I think a desktop is a lot easier. I was only half joking when I tweeted the other day that one day we might see people paid to sit in conferences and do precisely this (sweeping tweets as they come in, into a Storify to save them), unless Twitter sorts out its TOS so that third party apps can archive stuff again.

Because it's a bit harder (for me) to move them, what I've done is bung them into a draft Storify - here's what I said:

These are a series of tweets that are not in the order in which they were posted. I just grabbed them from the feed and bunged them here. It will be easier to reorder them if I let them sit for a few hours or at least more than a day.

This is because a newly posted tweet will say it was posted 'x minutes ago' then 'x hours ago' then 'about x hours ago' and finally 23.06 May 4th - and that's when I'll swoop in and reorder them.

Or, what I can do is create a fresh Storify and select from THIS Storify the elements I want to use.

5. If you're archiving tweets, it seems that the avatar of the one on the top of your 'stack' of tweets is used for the Storify's picture. This could be a bit annoying but I haven't investigated it yet - an obvious way to get around this is to add in a tweet of your own and use your avatar... but not ideal.

6. When you search on Storify for a set of tweets, eg a hashtag, the most recent tweet is at the top in the panel on the left. If you want your story to start from the beginning then the trick is (if starting from the top) to drop each tweet on the top of the next one, to reverse the order.

On my computer it's easier to wait 24 hours so that all the tweets are properly timestamped (see point 4) because Storify can be a bit fiddly to use on a very small laptop.

7. I think I might have just spotted a way to get a bit more granularity in Storify. I'm currently creating a record of #streettalks (Harry Rutter's talk on making cities more active-friendly) and am working from a rough draft which has ALL the tweets bunged in from a Twitter search.

While in edit mode, clicking on the S button in the panel on the left (which lets you search within Storify stories) for #streettalks brings up my story and I can drag the entire thing in to the panel on the right. But... I've just discovered that if I click on the Storify result while it's still in the left panel it will unfurl into the entire post, enabling me to pick out the individual elements (ie the tweets). If you hover over the link it does actually say "click to view the elements of this story".

In the pic below I've clicked on the S, highlighted by an orange circle, then typed #streettalks into the search bar and pressed 'get stories'; what shows up is a single result which is my Storify based on the #streettalks hashtag. If I hover over the link it invites me to click to view the elements, which I then did... and in the second panel you can see the entire Storify story unfurled and with all the individual tweets available for adding to the panel on the right to create a new story.