It was either a masterpiece of timing, or serendipitous coincidence. Either way, 500 of the world's leading developers, hackers and alpha geeks gathered in a Santa Clara hotel for the O'Reilly Emerging Technologies Conference last week. At the same time, Apple launched a new machine, Star Wars: Episode II premiered, the X-Files ended, and Napster shut down and then reopened. It was all just asking for trouble.

The conference itself was remarkable. As a publisher, O'Reilly arguably dominates the technical book field and it has a strong reputation for support of cutting-edge research. As the founder, Tim O'Reilly, said in his keynote, paraphrasing the author William Gibson, "the future is here, it's just not widely distributed".

This conference aimed to distribute some of the latest futures - peer to peer computing, web services, wireless internet, weblogging and emergent intelligence - to an extremely eager group.

But where the conference has started, as a place for new stuff in general, a main theme was quick to emerge. Many people mentioned this, the sudden realisation that each of us, no matter which area we were coming from, were all heading in the same philosophical direction: that developers now see the internet as a platform to run applications on, rather than something to connect to.

Everything, from peer to peer and web services, to the social constructs that appear around weblogs - the Distributed Republic of Blogistan as one speaker, Cory Doctorow, put it - and the increasingly popular habit of building free-for-all wireless networks, all these point to a popular realisation that the internet is growing into something greater than the sum of its parts.

In that sense, "emerging technologies" is wordplay, referring not to new stuff that is invented, but to new stuff that develops from looking at old stuff in a new way. The web's "small pieces, loosely joined" (itself the title of David Weinberger's book, much quoted last week) are being added together from the bottom up, by small groups and individuals creating clever services from the slack bits the larger companies are leaving lying around.

The search for the correct metaphor for this is increasingly intense. The internet is being described as a physical place - cyberspace, if you will - where you go to sites, pick up your email, and see each other in chatrooms, but also as an abstract: people talk of applications running in "the cloud", or out on the "edges".

With these new metaphors come new ways of thinking about computing. Doctorow argued for less optimalisation: because the system is less than perfect, more innovation is possible.

Another speaker, Geoff Cohen, argued in his talk, Toward a Biological Framework for Computation, that in the future we would be better off abandoning the computational design altogether, and design computers to act as if they were biological.

Of all the subjects discussed, the one that had the most immediate effect was wireless connectivity. O'Reilly's networking guys had rigged a wireless network around the hotel - and most people's laptops were quickly connected. Having a broadband internet connection anywhere in the conference was fantastic: while some were updating their weblogs live from the seminars, I relayed live pictures from my webcam and others were providing live commentary on the unofficial internet relay chat channel to people who couldn't be there.

Questions from people around the world were being relayed via the chat system to the speakers in Santa Clara - and the answers typed back in by audience members.

The emergent nature of new technology was no more apparent than when the delegates discovered EtherPeg.

This much-downloaded EtherPeek software, made for Apple Macs, listens to the wireless network, with any image file that travels across the network appearing as part of an evergrowing collage on your screen. Although you couldn't see which individual was looking at what, the overall effect was a great measure of how people's minds were wandering. That is, until "peg-bombing" software was put together, allowing people to download the same picture over and over, effectively spamming EtherPeg, some times with hastily drawn adverts for their own sites.

Meanwhile, the audience was having fun of its own: British geek-in-exile and co-editor of ntk.net, Danny O'Brien, had rigged The Panopticon. An interactive map of the conference, it displayed little avatars of the great, good and generally riotous, and allowed them to be moved around by anyone. Useful, if you wanted to find a friend; indispensable if you were stalking Amazon.com's Jeff Bezos.

It wasn't hard to find most people, even Bezos. The three unique delights of one speaker, JC Herz, were percolating through the geek hive mind and activating the herding instinct. The rather troublesome facts that JC is a woman, able to talk eloquently on "the principles of interaction design in many-to-many systems" and the realisation of "ohmigod she's an expert on computer games", were turning our generation's finest minds into a fizzy mess. The hysteria was only to be repeated on Wednesday night, when Attack of the Clones opened, and renowned peer to peer developer Joey deVilla was able to make his ticket price back by busking to the queue with his ever-present accordion.

And it was with Star Wars II that a metaphor for the internet's development could finally be found. Where the late 1990s could be seen as the first half of Episode II - the banks' sappy love story with dotcoms nearly as embarrassing as the celluloid Anakin and Amidala tryst - we can all be glad that it is dropping away to reveal the second act: where the geeks return, and Yoda kicks ass.

How dumb pests speak to geeks on internet efficiency

For Eric Bonabeau and his fellow technologists at Icosystem, bugs are good. Not those pesky programming problems that hinder and crash, but the little creepy crawly, buzzy-stingy beasties such as ants, bees, termites and wasps. Because through studying their behaviour, Bonabeau is developing computing techniques that may well revolutionise the way we look at certain problems. He calls it "swarm intelligence".

"I thought this was a nice exotic thing that would never work, but would let me present things in nice conferences in Hawaii, but... it worked!" said Bonabeau, as he presented the ideas to the O'Reilly Emerging Technology conference in Santa Clara last week.

It certainly does work. By studying the way individually stupid creatures - social insects like ants and bees - can collectively solve massively complex problems, Bonabeau and others are building techniques that are proving to be massively useful in the human-sized world. When a swarm of bugs following simple rules act together, a form of intelligence emerges.

For example, if you place ants into a maze, "you have to starve the ants first," says Bonabeau, "or they don't bother looking for food". Give them different routes to a meal, the ants will always find the shortest.

It works like this: the ants wander out randomly. Some will find the food by chance and, as they wander back, again randomly as they don't remember the way home either, they lay a trail of pheromones. The first ants back to the nest are obviously the ones who took the shortest route and, since ants are naturally drawn to follow pheromone-scented trails, the remaining ants follow this trail back to the food.

Pheromone evaporates, so longer routes are harder to maintain and, over time, the shorter routes naturally become stronger.

This means that should a better route appear in the future, ants who take that route will arrive first, the trail will be fresher, and the new route will soon take over as the preferred way.

However, ants are very stupid. No single ant knows the shortest route. Indeed, not all of them follow the pheromone route very well at all - which works in their favour as it allows new routes to be found.

What works well for ants, works brilliantly for real-world problems. One problem, long a bugbear of computer scientists and mathematicians, is called the travelling salesman problem. How, when given a certain number of points - cities, say - do you plot the shortest path between them all, when you can visit each only once, and always start and finish at the same place?

By simulating the ants and their pheromones inside a computer, the travelling salesman problem is remarkably easy. You set hundreds of "ants" off, running a random tour between each of the points. When the ants get to their destination, they backtrack and lay pheromone proportional to how good their route was. The next ants are more likely to take those routes, further reinforcing that route, and allowing the most optimal route to appear. This technique is used in industries, from scheduling production in factories, to the massively complex logistics of haulage firms and airlines.

This technique also has huge implications for the internet. When data travels over the internet, it passes through pieces of hardware called routers. Routers point each packet of data in the vague direction of the stated destination, getting less and less vague as the packet gets towards where it wants to go. The routes the routers point to are held in "routing tables" inside each machine. These routing tables change as the network changes (as machines are turned off, as connections go up or down) and so the routes change, too.

Bonabeau uses ant-based technology to improve these routes. He sends little agent programs that travel from router to router, pointing backwards to where they have come from and saying how they did it. As each little agent hits each router, the router is able to build a picture of the most efficient routes between the points it is in contact with. As more and more routers learn the optimal routes, the system becomes more and more efficient, and by allowing the routes to evaporate like the pheromones in the maze, the routers still take into account any changes to the network.

By using ant behaviour to improve these routing tables, Bonabeau says he can give a 99% improvement in the data throughput - meaning a faster, more resilient internet.

Much research is being done into other social insects' emergent intelligence, too. The way wasps build their nests, for instance, proves to be very useful for the US Air Force Institute of Technology. They are using the system to make a swarm of "nanosatellites" that can self-assemble into larger satellites. Others are working on robots made of hundreds of other robots, that can rebuild themselves into the shape most suited to the task ahead.

For the hive mind that steals your picnic, it's not a bad accomplishment.

· Ben Hammersley is writing a book for O'Reilly on content syndication