Using software to monitor programs running on the page of CareerBuilder.com, the researcher for the Electronic Frontier Foundation, an advocacy group, saw data identifying his computer being whisked off to at least 10 outfits that track where people go on the internet. More troubling was his inability to tell what the companies did with the data.

His experience goes to the heart of a battle that could shape the future of life on the web – while also having very real knock-on effects in the physical world. The digital dossiers that companies are building from the browsing, searching and other habits of ordinary web users are becoming increasingly refined. At the same time, a deluge of personal information has been unleashed publicly on the web, with Facebook’s 500m users at the forefront.

With rapid inroads on both fronts being made into many traditional expectations of personal privacy, the results could prove explosive.

ONLINE BEHAVIOUR

Be careful what you tweet

Linda Lea Viken, a South Dakota divorce lawyer, says she has all but stopped using private investigators to collect evidence against clients’ spouses. “We used to use PIs, and occasionally got incriminating pictures. Now we just look on Facebook. It’s cheaper, and it’s usually better. Instead of pictures taken from outside houses, I get pictures taken inside.”

Even as new technologies paint ever finer portraits of individuals based on their web activity, the greatest risk to privacy online for many users is their own ineptitude. Sites such as Facebook and Twitter encourage the sharing of status updates, travel plans and photographs, making it easier inadvertently to wreck a marriage, friendship or job.

A psychological disconnect is growing between the real and online worlds. “No matter how much you know that what you post can be subpoenaed, if not copied and pasted and spread all over the world, you behave as if you were alone,” says Sherry Turkle of the Massachusetts Institute of Technology. Evidence from social networking sites is now common in divorce cases, according to the American Academy of Matrimonial Lawyers. “What you tweet and what you post can and will be held against you in a court of law,” says Ms Vike of the AAML.

The problem of self-exposure is particularly acute among the young. “Most young people have no idea how much they are giving away,” says Ms Turkle. “They are continually surprised, and continually taken aback by what this implies.” David Gelles

“Tiny pieces of disparate data are being mashed together to create a digital profile of you in detail you never thought imaginable,” says Michael Fertik of ReputationDefender, a company that helps people manage their online footprint. “Whether you stay up late at night or have ever complained about a company could affect your employability. Whether you have expensive spending habits may affect if someone will invest in your company or date you.”

As such concerns have spread, the heat has been turned up this year under a long-simmering debate about the limits of online privacy. As an official at one leading US internet company concedes, it would take only one big scandal to do with the misuse of personal information – involving, say, the child of a prominent politician – for the issue to boil over.

“Ultimately, there will be some chicanery that will emerge, some dreadful harm that will result from a lack of rules and a lack of strong privacy protection,” says Bobby Rush, a Democratic member of the US House of Representatives who last week introduced a bill that would give Americans the right to stop the spread of private information about themselves, as many Europeans can. “We are at a critical juncture.”

The tensions created by the twin advance of internet tracking technologies and user’s own less inhibited approach to exposing personal information online have exposed some deep philosophical differences, while also highlighting the powerful commercial interests that are at stake.

Mark Zuckerberg, the 26-year-old founder of Facebook, has become the champion of what many in the internet industry claim is a generational shift in attitudes to privacy. “People have gotten really comfortable not only sharing more information and different kinds, but more openly and with more people,” he said in a speech in January. “That social norm is just something that’s evolving.”

Yet he recently discovered that attitudes are not changing as fast he would like. Facebook ran into a barrage of complaints when it changed settings on its site so that some information about users, such as their job and home city, was published automatically, overriding any choice they had made to keep it private. The move drew the fire of European regulators and prompted an investigation by the US Federal Trade Commission, forcing Facebook into a partial retreat.

Legislators and regulators are concerned that internet users have been left with too little control of how their personal information is used and lack the information and understanding needed to make informed decisions. “Internet users must have effective control of what they put online and be able to correct, withdraw or delete it at will,” according to Viviane Reding, the European Union’s justice commissioner, who has made clear that privacy and data protection will be a main focus of her five-year term.

“If the goal is providing consumers with information about how their information is being used and giving them some control, that is not being achieved”, says Jessica Rich, deputy director of the FTC’s Bureau of Consumer Protection.

The outcome of this clash of visions will have a big impact on the profits of companies that operate online. Access to detailed information about users “is extraordinarily important” to Google, Facebook and others offering free services on the web, says Rebecca Arbogast, an analyst at Stifel Nicolaus in Washington. “Their business model depends on them being able to target advertisements more and more precisely to us. It goes to the core of what they are all about.”

“Without data, the web would be far less interesting to advertisers,” adds Michael Rappa, a computer science professor at North Carolina State University. “Web empires built on advertising dollars like we have today would not be possible.”

If fears about the limits of online privacy have been thrust into the headlines this year, internet companies have only themselves to blame. Business ambition has exacerbated the problem. “There is a scary race to the privacy bottom that we’re seeing,” says Mr Fertik.

While Facebook overplayed its hand as it changed its privacy settings in ways that advanced its own business interests, Google slipped up in its race to catch up with Facebook. A social function called Buzz, added this year to Google’s e-mail service, displayed users’ frequent e-mail contacts for others to see. The company changed its policy after an outcry.

In May, Google suffered further criticism for an apparent disregard for its users’ privacy after investigators in Germany forced it to admit that it had illicitly collected snippets of internet activity from unsecured home wireless networks with its fleet of camera-equipped cars.

While gaffes like these have made the headlines, though, many of the more insidious potential inroads into online privacy are taking place beyond the view – or understanding – of most internet users. These include the widespread use of the sort of “cookies” encountered by the EFF’s Mr Eckersley on CareerBuilder.com. These are pieces of data planted on users’ computers so that they can be identified whenever they visit a website or view an ad. “In the last few years we’ve seen the industry develop and deploy these incredibly sophisticated and hard-to-defend-against techniques for following users around the web, building records of what they’re reading and thinking and doing,” says Mr Eckersley. “Advertisers have an understandable desire to target people more effectively, but that means they end up engaging in surveillance that is often covert, creepy, and non-consensual.”

The networks are sharing data with each other to build more complete records and peddling access to millions of internet protocol addresses of individuals with specific attributes, such as those who have expressed interest in a new car. “Everybody has been moving to do more with identifying individuals or using individuals’ behaviour,” says Tom Alison, chief executive of ClearSight, an online direct marketing company.

Digital trails left by internet users are proliferating fast. Information about a user’s location will grow easier to come by as smartphones come into wider use. Those can broadcast someone’s whereabouts to the people that distribute programs that run on them.

The amount of data will grow exponentially with the arrival of what Ms Reding calls the “internet of things” – the spread of devices carrying smart chips of the sort being widely adopted by retailers. “No European should carry a chip in one of their possessions without being informed precisely of what they are used for, with the choice to remove or switch it off at any time,” Ms Reding has said.

Transparency and choice figure prominently in the responses that regulators are preparing. The effectiveness of such rules will lie in the detail – and in whether ordinary internet users understand or care enough to exercise whatever new controls they are given. Europe has moved most aggressively. A change to the EU privacy directive last year now explicitly requires consumers’ “consent” before data about them are collected, instead of giving them the “right to refuse”.

One direct impact is likely to be on the “cookies” that lie at the heart of online advertising. In an interpretation of the new requirement, The Article 29 Working Party, an influential group of European data regulators, said last month that the current practice of sending cookies as soon as someone lands on a page must stop.

A further push from Brussels is expected in the autumn. Among proposals being discussed is for users to have a “right to be forgotten”. Aimed at those who no longer want their teenage antics to be featured on Facebook – embarrassing pictures, out-of-context comments and the like – it will promise web users that they can wipe the digital slate clean at any time. “The implications over who owns the data that are posted on a social networking website will be profound,” says one EU diplomat.

The US, meanwhile, has so far left it to the industry to police itself – though the patience of regulators is clearly wearing thin. Jon Leibowitz, chairman of the FTC, told a congressional hearing this week that companies needed to “do a better job of ensuring that consumers can make clear choices and have clear notice” or face the risk of legislation.

That might sound like mere sabre-rattling. But a continuation of a setup in which internet companies have wide discretion over how they use personal data is far from assured. The next move is the industry’s – and like internet users everywhere, it will need to think carefully before it acts.

Additional reporting by Stephanie Kirchgaessner, David Gelles and Stanley Pignal

Copyright The Financial Times Limited 2010. You may share using our article tools. Please don't cut articles from FT.com and redistribute by email or post to the web.