Earlier this year I spoke at the BBC’s Data Fusion Day (you can find a liveblog of the event on Help Me Investigate) about data journalism workflows. The presentation slides are embedded below (the title is firmly tongue-in-cheek), but I thought I’d explain a bit more in a series of posts – beginning here.

Data journalism workflow 1: Set up data newswires

Most newsrooms take a newswire of some sort – national and international news from organisations like the Press Association, Reuters, and Associated Press.

Data journalism is no exception. If you want to find stories in data, it helps to know what data is coming out, when it comes out.

Managing information overload

A magazine editor now has little problem finding information on a range of topics. It is likely that you will have subscribed to email newsletters, RSS feeds, Facebook groups and pages, YouTube channels and various other sources of news and information both in your field and on journalistic or management topics.

There tend to be two fears driving journalists’ information consumption: the fear that you will miss out on something because you’re not following the right sources; and the fear that you’ll miss out on something because you’re following too many sources. This leads to two broad approaches: people who follow everything of any interest (‘follow, then filter’); and people who are very strict about the number of sources of information they follow (‘filter, then follow’).

A good analogy to use here is of streams versus ponds. A pond is manageable, but predictable. A stream is different every time you step in it, but you can miss things.

As an editor you are in the business of variety: you need to be exposed to a range of different pieces of information, and cannot afford to be caught out. A good strategy for managing your information feeds then, is to follow a wide variety of sources, but to add filters to ensure you don’t miss all the best stuff.

If you are using an RSS reader one way to do this is to have specific folders for your ‘must-read’ feeds. Andrew Dubber, a music industries academic and author of the New Music Strategies blog, recommends choosing 10 subjects in your area, and choosing five ‘must-read’ feeds for each, for example.

For email newsletters and other email updates you can adopt a similar strategy: must-reads go into your Inbox; others are filtered into subfolders to be read if you have time.

To create a folder in Google Reader, add a new feed (or select an existing one) and under the heading click on Feed Settings… – then scroll to the bottom and click on New Folder… – this will also add the feed to that folder.

If you are following hundreds or thousands of people on Twitter, use Twitter lists to split them into manageable channels: ‘People I know’; ‘journalism’; ‘industry’; and so on. To add someone to a list on Twitter, visit their profile page and click on the list button, which will be around the same area as the ‘Follow’ button.

You can also use websites such as Paper.li to send you a daily email ‘newspaper’ of the most popular links shared by a particular list of friends every day, so you don’t miss out on the most interesting stories.

Social bookmarking: creating an archive and publishing at the same time

Social bookmarking tools like Delicious, Digg and Diigo can also be useful in managing web-based resources that you don’t have time to read or think might come in useful later. Bookmarking them essentially ‘files’ each webpage so you can access them quickly when you need them (you do this by giving each page a series of relevant tags, e.g. ‘dieting’, ‘research’, ‘UK’, ‘Jane Jones’).

They also include a raft of other useful features, such as RSS feeds (allowing you to automatically publish selected items to a website, blog, or Twitter or Facebook account), and the ability to see who else has bookmarked the same pages (and what else they have bookmarked, which is likely to be relevant to your interests).

Check the site’s Help or FAQ pages to find out how to use them effectively. Typically this will involve adding a button to your browser’s Links bar (under the web address box) by dragging a link (called ‘Bookmark on Delicious’ or similar) from the relevant page of the site (look for ‘bookmarklets’).

Then, whenever you come across a page you want to bookmark, click on that button. A new window will appear with the name and address of the webpage, and space for you to add comments (a typical tactic is to paste a key quote from the page here), and tags.

Useful things to add as tags include anything that will help you find this later, such as any organisations, locations or people that are mentioned, the author or publisher, and what sort of information is included, such as ‘report’, ‘statistics’, ‘research’, ‘casestudy’ and so on.

If installing a button on your browser is too complicated or impractical many of these services also allow you to bookmark a page by sending the URL to a specific email address. Alternatively, you can just copy the URL and log on to the bookmarking site to bookmark it.

Some bookmarking services double up as blogging sites: Tumblr and Stumbleupon are just two. The process is the same as described above, but these services are more intuitively connected with other services such as Twitter and Facebook, so that bookmarked pages are also automatically published on those services too. With one click your research not only forms a useful archive but also becomes an act of publishing and distribution.

Every so often you might want to have a clear out: try diverting mailings and feeds to a folder for a week without looking at them. After seven days, ask which ones, if any, you have missed. You might benefit from unsubscribing and cutting down some information clutter. In general, it may be useful to have background information, but it all occupies your time. Treat such things as you would anything sent to you on paper. If you need it, and it is likely to be difficult to find again, file it or bookmark it. If not, bin it. After a while, you’ll find it gets easier.

Do you have any other techniques for dealing with information overload?

An article by Heather Brooke giving her personal experience of not being able to record her own hearing.

A report on the launch of a new website by the Judiciary of Scotland, which I’d completely forgotten about. This also helped me avoid making the common mistake of tarring Scottish courts with the same brush as English ones.

Various useful resources for courts data.

Some context on the drop in court reporters at a regional level – but also some figures on the drop at a national level, which I hadn’t thought about.

A specialist academic who has been researching court reporting.

And all this in the space of 10 minutes or so.

If you look at the resulting post you can see how the first pars are informed by what was coming into my RSS reader and social networks, but after that it’s largely bookmark-informed (as well as some additional research, including speaking to people). The copious links provide an additional level of utility (I hope) which online journalism can do particularly well.

Excerpt from the article - most of these links came from my Delicious bookmarks

All about preparation

You can see how building this resource over time can allow you to provide context to a story quicker, and more deeply, than if you had resorted to a quick search on Google.

In addition, it highlights a problem with search: you will largely only find what you’re looking for. Bookmarking on Delicious means you can spot related stories, issues and sources that you might not have thought about – and more importantly, that others might have overlooked too.

When I describe the network infrastructure outlined below, I outline two levels: the tools themselves, and how they connect to each other. In an attempt to clarify that, I’ve created a diagram.

The icons in the diagram attempt to show clearly the purpose of each tool:

The exclamation mark representing RSS readers indicate that the tool is focused on monitoring what’s new;

The question mark representing social bookmarking indicate that that tool largely serves to answer questions, providing context and background

The facial expressions representing social networks indicate that this tool help provide access to sources who may have stories to tell (positive; negative) or who are asking important questions (confused).

Here is a further breakdown of each element, and how they connect to each other.

RSS Reader

As outlined above, this part of the structure is all about ‘What’s new?’ and is quite often the first thing a journalist checks at the start of the working day (indeed, it’s ideal for checking on a phone on the way to work). It is the modern equivalent of picking up the day’s newspapers and tuning into the first radio and TV broadcasts of the day.

The RSS Reader gathers news feeds from a range of sources. Here are just a few:

Formal news organisations

Journalistic blogs

Organisational blogs

Personal blogs of individuals in your field

In addition, an RSS reader allows you to follow customised feeds reporting any mention of key terms, organisations and individuals across a variety of platforms:

Google News

The blogosphere as a whole

Social bookmarking services such as Delicious

Forums

Microblogging services such as Twitter

Video sharing services such as YouTube

Photo sharing services such as Flickr

Audio sharing services such as Audioboo

Social networks such as Facebook Pages

This is how the RSS reader connects to the two other elements of the infrastructure: most social networks have RSS feeds of some kind, as do social bookmarking services (one of the reasons I prefer Delicious over other platforms is the fact that it has an RSS feed for every user, for every item bookmarked with a particular ‘tag’ (explained below), for tags by particular users and for any combination of tags.

But if you can follow these feeds in an RSS reader, why use a social network at all?

Social networks

Why use a social network? To follow people, not just content, and because your own contributions to those networks are a key factor in gaining access to sources.

With many social networking platforms (Twitter, for example) you can of course find individual users’ RSS feeds in an RSS reader, or a feed of people you are ‘following’ – either of which you can subscribe to in an RSS reader. But there’s little point, and your RSS reader will soon become flooded with updates. Instead, you should use the RSS reader to follow subjects and add the individuals talking about those subjects to your social networks.

The social network provides an added level of serendipity to your newsgathering: increased opportunities to encounter leads, tips and stories that you would not otherwise encounter.

It is also a three-way medium: a platform for you to ask questions or invite experiences relevant to the story you are pursuing, or to follow the public conversations of others asking questions or sharing experiences.

The final use for social networks is often the first use that journalists think of: distribution. And it is here that social networking also connects to the other 2 parts of the network infrastructure.

If you read something interesting in your RSS reader and wish to share it across social networks, you can often do so with a single click – with a bit of preparation. Twitterfeed is a tool which will automatically tweet updates on your Twitter account – all you need to know is the RSS feed for the updates you want to share. If you’re using Google Reader, for example, that feed is on your Shared Items page.

To tweet something interesting you’ve seen in your RSS Reader all you have to do then is (in the case of Google Reader) click on the ‘Share’ button below that item.

Social bookmarking

The first two parts of the network infrastructure – an RSS reader and social networks – are about the initial stages of newsgathering; the first things you check at the start of a working day.

Social bookmarking, however, is about what you do with information from your RSS reader and social networks – and information you deal with throughout your day.

Today’s news is tomorrow’s context. And social bookmarking allows you to keep a record of that context to make it quickly accessible when needed.

That’s the bookmarking part. The social part also allows you to publish information at the same time as you store it; to discover what information other people with similar interests are bookmarking; and to discover which people are bookmarking similar things to you).

Because social bookmarking is the least immediate element of this network infrastructure, it is also the aspect which the fewest students get their heads around and actually use.

Yet it is, for me, perhaps the most useful element. It takes an upfront investment of time and the development of a habit which initially doesn’t have any obvious reward.

But when you’re up against a deadline and are able to retrieve a dozen useful reports, documents and people within minutes – then you’ll get it.

Here’s the process:

You come across something of interest. It may be a useful article, blog post or official report in your RSS reader – or a document linked to by someone in your social network. You might encounter the thing of interest while working on a story. You may read it – you may not have time.

You bookmark the specific webpage containing it using a service like Delicious. You add ‘tags’ to help you find it later: these might include:

the subjects of the webpage (e.g. ‘environment’, ‘health’),

its author or publisher (e.g. ‘paulbradshaw’, ‘OJB’),

specific organisations or individuals (‘nhs’, ‘davidcameron’),

the type of document (‘report’, ‘research’, ‘video’)

or information (‘statistics’, ‘contacts’),

and even tags you have made up which refer to a specific story or event (‘croatia11′)

You can if you wish add ‘Notes’. Many people copy a key passage from the webpage here, such as a quote (if a passage is selected on the page it will be automatically entered, depending how you are bookmarking it) to help them remember more about the page and why it was important.

You can also mark your bookmark as ‘private’. This means that no one else can see it – it becomes ‘non-social’.

Once you save it, it becomes available for you to retrieve at a future date: a personal search engine of items you once encountered.

The key thing here is to think about how you might look for this in future, and make sure you use those tags. For example, the publisher might not seem important now, but if in future you need to re-read a certain report and can recall that it appeared in the FT, that will help you access it quickly.

Remember also that tags can be combined, so if I want to narrow down my search to items that I bookmarked with both ‘UGC’ and ‘BBC’, I can find those at delicious.com/paulb/UGC+BBC.

This is one of the reasons why a social bookmarking service is more effective than an RSS reader. You can, for example, search your shared or starred items in Google Reader – and you can tag them also – but as you tend to get more results it is harder to find what you are looking for. The use and combination of tags in Delicious narrows things down very effectively – but equally importantly, it allows you to bookmark pages that do not appear in your RSS reader.

That said, if you cannot find what you are looking for in Delicious, Google Reader is another option. It is also worth using a backup service which provides another way to search your bookmarks.Trunk.ly is one that does just that.

Of course, the bookmark only points to the live webpage – and it may be that in future the page is moved, changed, or deleted. If you are dealing with that type of information it is worth copying it to another webspace (I use the quote option on Tumblr) or using a (generally paid-for) social bookmarking service that saves copies of the pages you bookmark (Diigo and Pinboard are just two)

Social bookmarking: networks and cross-publishing

One of the features of social bookmarking services is that you can follow the bookmarks of other users. In Delicious this is called your network – and it’s where social bookmarking not only connects to RSS readers but also becomes a form of social network. Here’s how you build your network:

Look at your bookmarks. Next to each one will be a number indicating how many users have bookmarked this. If you click on this you will see a list of who bookmarked it, and when. (Alternatively, you could also look at all users using a particular tag – if you’re a health correspondent, for example, you might want to look at people who are tagging items with ‘NHS’). Click on any name to see all their public bookmarks.

If you would like to follow that person’s future bookmarks (because they are bookmarking items which relate to your interests), click on ‘Add to my network’

You will now be able to see their bookmarks – and those of anyone else you have added – on your ‘Network’ page. It is, essentially, a mini RSS reader.

Which is why I use Google Reader to follow my network’s bookmarks instead. Because at the bottom of your Delicious Network page is, of course, a link to an RSS feed. Right-click on this and copy the link, then paste it into your RSS reader and you don’t need to keep checking your Delicious Network separately to all your other RSS feeds.

Of course, if you find someone interesting on Delicious, you might find them interesting on Twitter or a blog. If they’ve edited their Delicious public profile (the one you found in step 1 above) it might include a link. Alternatively, there’s a good chance they’ve used the same username on other social networks – so search for them using that.

This is another example of how social bookmarking can connect to social networking.

Here’s another: you can use a service like Twitterfeed (explained above) to auto-publish every item you bookmark – or just those with a particular tag, or a combination of tags. Because Delicious provides RSS feeds for your bookmarks as a whole, those with a particular tag, and any combination of tags.

One process across your network infrastructure then starts to look like this:

Read interesting blog post on Google Reader

Bookmark using Delicious – use a tag which is automatically tweeted

Link auto-tweeted on Twitter

Conversely, if you want to automatically bookmark links that you share on Twitter, you can do so by signing up to Packrati.us. Tweeted links will be given the tag ‘packrati.us’ as well as any hashtags that you include in the same tweet (So a link tweeted with the hashtag ‘#crime’ will be tagged ‘crime’).

Another process across your network infrastructure then starts to look like this:

Read interesting link tweeted on Twitter

Retweet it, adding relevant hashtags

Link is auto-bookmarked on Delicious

Listen, connect, publish

This has turned out to be a long post – which is why I think the diagram is needed. The initial set up is simple: sign up to social networks and a social bookmarking service, and set up an RSS reader. Subscribe to feeds, and add people to your networks.

But once you’ve done the technical part, you need to develop the habit of listening and continuing to add to those networks: check your RSS feeds and networks every day (but know when to switch off), and look for new sources. Bookmark useful resources – articles, documents, reports, research and profile pages – and tag them effectively.

Finally, contribute to those networks and connect the different parts together so it is as easy as possible to gather, store, publish and distribute useful information.

It is a small – and regular – investment of time. But it will keep you in touch with your field, lead you to new sources and new stories, and help you work faster and deeper in reporting what’s happening.