Searching for a fresher taste

First it bought photo-sharing site Flickr, and now it has snapped up bookmarking phenomenon Delicious. Why is Yahoo investing so heavily in the social networking stars of Web 2.0? Bobbie Johnson reports on the new face of the classic search engine

Two years ago, New Yorker Joshua Schachter was fed up. He had a huge amount of bookmarks gathered from his travels around the web, but found keeping track of them all impossible. In the end, he decided to create a program to categorise them.

After experimenting with some ideas, he decided that other people should use it too; in fact, that they should be able to share all their bookmarks with each other in order to find more interesting websites. Soon he had opened up Delicious (http://del.icio.us), a social network for webheads around the world.

Its premise is relatively simple. The Delicious site is like a big database holding the addresses of websites each user has visited and found interesting. Each user also tags their visited sites with useful (to them) labels such as "World Cup", "Canada" or "explosion". Anyone, inside or outside Delicious, can then search on a subject by tag, and the Delicious database pulls out the URLs that other users have spotted in the same area. It is collective intelligence, pooled from thousands of people.

The site has won much acclaim since being launched in 2003. Last March, Schachter, formerly with the brokers Morgan Stanley, was able to start working on the website full time, thanks to a cash injection from Amazon, technologist Esther Dyson and publisher Tim O'Reilly. But, ultimately, Delicious was the preserve of a dedicated minority of early adopters.

Last Friday, however, Schachter's hard work paid off when web giant Yahoo announced it would be buying the nine-person company outright for a rumoured $30m - equivalent to about $100 for each registered user.

It must have been a busy week at Yahoo's expansive headquarters in Sunnyvale, California. Just days before the Delicious deal was announced, the company introduced a new service where readers solve each others' queries, Yahoo Answers. A double whammy turned triplicate with the announcement that a cross-promotional deal with weblog vendors Six Apart is in the offing.

Such a flurry of activity has suddenly drawn attention to Yahoo, which often has a lower media profile than its more vivacious competitors despite being the world's most-visited website, with 101.3m visitors in August this year, according to the research company EMarketing.

Most observers were quick to link last week's announcements with another of Yahoo's recent purchases: Flickr, the photo-sharing website and another pioneer of tagging, bought for around $30m in March this year.

Tag teams

The similarities between Delicious and Flickr - small north American startups that created popular social software services using tags - have cranked up the speculation. Even Schachter publicly drew comparisons. "We're excited to be working with the Yahoo search team - they definitely get social systems and their potential to change the web," he wrote on his weblog (http://blog.del.icio.us/blog). "We're also excited to be joining our fraternal twin, Flickr!"

Clearly there's been a lot of activity in one area - but what is Yahoo really doing?

"You can probably stitch together our plan from the moves we've made, the acquisitions we've made, the products we've put out to market," says Bradley Horowitz, Yahoo's senior director of technology development. That plan: to try and make social search the next stage in the evolution of search engines.

"Day zero was when Jerry and David [Filo and Yang, the founders of Yahoo] went into a dorm room in 1994 and decided they'd organise the web," explains Horowitz. "What came around next was the automated phase that led to companies like AltaVista and Inktomi. The third phase was really the innovation that Google came up with, looking at the topology of the web - link structures, back linking." That was in 1998.

Now, he says, the next step will be to take social bookmarking - exactly the activity promoted by Delicious - and turn it into the basis for wider web searching. His proposition involves using the social network as the channel for search results.

"In topology search, what you're really doing is conferring to webmasters the privilege of deciding what's important for everybody. They cast their votes on what's important by building links - and they do it in a way that smears it out for everybody, so we all get the same results." The concept of personal search and social search, he says, "democratises that process, and says 'why should webmasters be the only authority we trust and confer that privilege to?'. Why can't I pick other authorities of trust, like for instance my friends? What is their opinion?"

Social life

So instead of getting the same results as everybody who searches a term, you get results that are filtered through your social group. You choose your own peers - friends, family, colleagues, interesting strangers - and they provide your answers. And by including different levels of friendship, you can increase the size of your net dramatically. Even if you have just 10 contacts, and those contacts have another 10 each, that's still more than 100 potential sources within two hops. The concept is useful, perhaps, but maybe not for everyone.

"If you're trying to find the population of London, you don't need social search," admits Horowitz. "But if you're trying to find a restaurant to eat at, a blog to read, or a plumber who's reputable - the kind of things you depend on the expertise of others to know - that's where the social search phenomenon comes in."

The acquisitions of Flickr and Delicious clearly point in this direction. But social search isn't new to Yahoo. The company launched an early attempt at this kind of service in the form of My Web 2.0, a Delicious-style tool that will most likely form the basis for whatever they decide to do in the future. It has had limited success - in mitigation Horowitz calls it "a living experiment" with no promotion - but will most likely be moving forward now that Delicious is on board.

So why has a veteran web firm such as Yahoo decided that social search is worth spending millions on?

For a start, it takes a lot of the hard work out of searching the web. The very clever thing about social software is that it puts the burden on to the user, not the provider. If you aren't getting enough results, it's not because the engine isn't indexing enough pages: it's because you haven't got enough friends in your social network. Conversely, if you end up getting a lot of spam results - an increasingly important issue with sites like Google -then you have obviously let the wrong person into your network (and should excommunicate them).

Users enjoy the extra power: communities become self-policing and popular taggers become trusted sources, gaining minor celebrity along the way. Providers enjoy it, too, for taking some of the heavy lifting away from them.

In a way, what's happening is a throwback to the earliest days of Yahoo, to the dormitory-created directory model, when the web was mediated through a series of editors and power users who would pick a Site of the Day (for the web really was that small). The difference now is that we get to choose who our power users are.

More important to a huge business such as Yahoo is how social search could bring new ways to cash in. Search engine firms make money through advertising, and in the short run, a tighter focus increases the likelihood of being able to charge higher prices for ads. In the long run, social groups might emerge inside the search engine - for example, a group of doctors in Hong Kong who share their bookmarks - who could be specifically targeted by advertising campaigns.

But beyond the vagaries of search evolution and the bottom line, something else is going on. Acquiring interesting little companies such as Flickr and Delicious adds value to Yahoo (else shareholders would reject them), but it isn't just about business.

Agents of change

Through a series of hires and acquisitions, Yahoo is clearly assembling a squad of innovators and forward thinkers. Schachter joins a list of employees including Flickr co-founder Caterina Fake, former CTO of technology news service Infoworld, Chad Dickerson, and blogging technology academics such as Danah Boyd and Cameron Marlow.

Under Horowitz that team will be charged with putting some of the creativity back into Yahoo's operation, an area where rivals have pulled ahead - at least in the eyes of the public.

"We are agents of change within the organisation," says Horowitz. "We're here to cause trouble. It's largely about innovation, it's about disruption, it's about change. It's about kind of the mortar between the products. Yahoo Calendar, Yahoo Local, Yahoo Maps, Flickr - how can we knit all of those together to create some uber-applications that are extremely compelling and aren't really in the purview of any of one group to deliver?"

This is the kind of innovation culture that keeps Google, in particular, in the headlines. Not that Google has shied away from acquisitions: since February 2001 it has acquired 14 firms, which have been used to create services such as Google Maps, Picasa, Google Analytics and the Google Ride Finder (in the US). Similarly, Microsoft's MSN has bought desktop search companies, and added blogging (at MSN Spaces) and mapping capabilities. One gets a sense of the three biggest "search" companies amassing armies of services ready to roll into battle against each other, fighting for your attention - and loyalty.

Horowitz thinks having a crack team of brains focused on new ideas will help redress the balance compared to Google. To this end, spending some of Yahoo's $3.18bn cash pile (Google has $7bn) is aimed at "buying the mojo" of small, innovative startups, and the spree hasn't necessarily stopped yet. "I think you'll continue to see us do acquisitions in that model," says Horowitz. "The dozen-person company where we're buying talent rather than revenue or user base."

Despite that fanfare, it is not necessarily time for anyone with a good idea to suddenly eye a deal with Yahoo. The services brought into the fold recently have reached a certain level of usage - about 300,000 registered users for Flickr and Delicious - married with passionate online followings before being bought out.

Acquiring Delicious may have added an immediate momentum to Yahoo's plans for social search, but the real point seems to be the building of an innovative culture that can widen Yahoo's lens.

It's certainly generating the biggest buzz the company has seen for some time, but the long run is what counts. In a few years, Yahoo could have become the byword for innovation - or it could look like the company was foolishly sucked into the Web 2.0 hype. But Yahoo does not feel that it can pass up this gamble.

· If you'd like to comment on any aspect of Technology Guardian, send your emails to tech@guardian.co.uk