It was 11.30pm on a cold Sunday evening when the phone rang. It had been a long shift, and the lone warehouse security guard was glad to hear a friendly voice. "Hello, mate. What time do you open in the morning for deliveries?" Just an HGV driver, obviously, probably working for himself. The security guard gave him directions to the warehouse.

"Is it better to get there early to beat the queue," asked the caller, "or are there enough loading bays to cope with demand?" The two fell into friendly conversation. "I suppose things get busy in the mornings. How many loading bays have you got? And just out of interest, who else delivers to the warehouse?"

After a few more minutes of idle chat the caller hung up, and the security guard returned to his cup of tea and his newspaper. And in a small office in north London, the delivery driver replaced the receiver with satisfaction. Bingo.

It is unlikely that the security guard ever realised the significance of the chat, or the value of the information he had so casually given away. Because the "delivery driver" was no such thing: he was, instead, an operative from Aware, an investigative consultancy hired by a competitor of the warehouse firm to find out its loading-bay capacity and the names of its major clients.

The firm knew that demand for warehouse space in the area was likely to increase, but wanted to find out its competitors' capabilities before building an extension to its own premises. And all it took was one brief, friendly chat with a junior employee.

Forget market research, strategic planning, supply chain management or whatever it was you thought made your company successful. Chances are - particularly if you work in telecommunications, or pharmaceuticals, or manufacturing, or just about any competitive environment - that your company is increasingly pinning its success on trying to work out exactly what its rivals are up to. And a growing number of consultants skilled in getting hold of that information are ready and willing to offer their services to give your company the competitive edge, by fair means or - occasionally - foul.

This is the world of competitive intelligence, once seen as a rather shady side to corporate affairs, but increasingly viewed as a legitimate tool to help firms steal a march on their rivals. Most big American companies now have an internal CI unit, dedicated to analysing the competitive marketplace; they are likely also to use external investigators to complement this function. Fuld and Co, one of the oldest CI consultancies in the US, says it has advised more than half the Fortune 500 companies.

The once whispered-about world of CI is increasingly overground and - like all good American innovations - over here. The Society of Competitive Intelligence Professionals (SCIP) now has 200 members in the UK - a fraction of its global membership of 7,000, but a figure which is steadily on the increase. Most major telecoms and pharmaceuticals companies in Britain are now believed to have their own intelligence departments, while smaller firms in a competitive marketplace are likely to have a lone CI worker who keeps an eye on what other players are up to.

In the knowledge economy, information is the commodity that really matters. "The fundamental is that there are huge amounts at stake when you make business decisions these days," says Bill Waite, managing director and senior counsel at the Risk Advisory Group, which claims to be the biggest CI consultancy in Europe.

"The more you know, not only about your own strengths but about your competitors', the more likely it is that you will reach a positive business transaction rather than a negative one. So the pressure to obtain good information is increasing. It is becoming a very big business. More and more companies are starting to do this kind of work, including the major consultancy firms, who wouldn't have touched this area five years ago."

There are good reasons why respected firms thought twice about developing intelligence-gathering wings, since activities falling into the broad category of competitive intelligence have not always earned the practice the most honourable of reputations.

In May last year an executive from car giant General Motors was indicted on charges that in 1996 he stole secret GM documents, including plans for new car models, supplier prices and details of a new assembly operation called "Plant X", and turned them over to Volkswagen. The German manufacturer paid GM $100m in compensation three years ago.

An employee working for a Canadian distributor of Gillette also paid heavily for sharing confidential information: he sold plans for Gillette's Mach-3 razor to arch-rivals Bic - and was jailed for 27 months for his efforts.

CI practitioners scramble to distance themselves from this kind of behaviour, more accurately described as corporate espionage and unequivocally illegal in most countries. Nonetheless, they are still called upon frequently to defend themselves against charges that the practice inhabits the slightly grubby margins of ethical business dealings.

This is doubtless because the juiciest examples of corporate information warfare - the ones that hit the headlines - invariably involve a level of intrigue that most chief executives would be embarrassed to admit to their shareholders. In the late 1980s, for instance, British Airways commissioned Kroll Associates, one of the biggest US-based corporate investigators, to look into rivals Air Europe as part of their well-documented "dirty tricks" against Richard Branson's Virgin Atlantic. The larger airline was later forced to pay record libel damages to the entrepreneur.

Larry Ellison, the chairman of software giant Oracle, felt no compunction in admitting in June that his company had used private detectives to snoop on contractors working for Microsoft. Ellison argued that it had been his "civic duty" to set investigators rummaging through the rubbish of the lobby group the Association for Competitive Technology, looking for evidence that Microsoft was paying the lobby group to influence its antitrust case. The practice, known as "dumpster diving", is frowned upon by those who try to gather information from more ethical sources.

"We always seek to distance ourselves from the less ethical kind of practice," says Ted Howard-Jones, business development manager at Current Analysis, a consultancy firm specialising in the telecoms industry. "In the UK, people cannot put into perspective the tiny minority of cases where illegal or unethical conduct takes place." He points to SCIP's code of ethics, to which all members sign up, and insists that the vast majority of CI work involves collating information already in the public domain.

Arthur Weiss, a managing partner at Aware, agrees. "It is not in my interests to lie in order to get information, and I'm going to be found out if I do. And if I call back I would have to remember who I said I was. There will be times that I won't get the information and someone who follows a different ethical rule will. But I prefer to be able to sleep at night."

But surely masquerading as someone else to get hold of sensitive information, or at least allowing them to believe that you are an innocent caller, is a bit ethically suspect? "We wouldn't identify the client - we never identify the client - but we will always identify ourselves. I would say I was calling from a research company. Certainly if I was asked." Talking to someone who doesn't appreciate the value of the information is often the best way to get hold of sensitive facts and figures, he admits.

SCIP insists that the worst excesses of rogue CI practitioners are a thing of the past, particularly since the US introduced the Economic Espionage Act in 1996, under which the FBI is believed to be investigating 800 cases of corporate misbehaviour. Howard-Jones argues that the Data Protection Act is an even more rigid constraint on the activities of information gatherers in this country.

But with the demand for corporate investigations booming at an exponential rate, it is inevitable that companies and consultants will continue to dance round the more precarious limits of the law in order to get hold of that crucial figure.

"It's the speed of change. Look at the product life cycle. You now want a new mobile phone every six months, not every three years, and it's going to get faster. Many of the old-fashioned methods of planning that companies once used are now completely redundant. Five-year plans do not exist any more.

"Businesses don't make serious plans more than a year, or at most 18 months ahead. That's why you need to react so quickly, and that's why it is vital that you have information about what your competitor is up to. If you get it wrong, you don't have the time to catch up."