In its relentless search for new ways to relieve taxpayers of their hard-earned cash, the government has decided to set up a new bureaucracy to educate investors. It will cost an impressive HK$50 million per year.

Oblivious to the failure of previous, cheaper schemes with the same objective, this new wheeze will cover all investment products. But what this scheme will most definitely not do is eradicate investor irrationality, a universal phenomenon.

It is doomed to failure in at least this respect because the entire experience of investment history demonstrates that irrationality or illogical investor behaviour is the norm. Only true believers in rational markets fail to acknowledge this.

The timing for this new government venture into the absurd could hardly be more appropriate as uncertainty looms in investment markets. Investors ponder whether the long-expected double dip in the global economy will really happen, as they agonise over whether burgeoning sovereign debt will propel the markets into a yet deeper mess.

All the while, what the economist John Maynard Keynes described as the ultimate barbarous relic soars to new highs. Keynes was talking about gold, a commodity that reliably gains value when investors scramble to find the least-worst resting place for their funds.

So the time has come to revisit the much discussed but not sufficiently appreciated subject of behavioural finance. Studying this subject gives clues and even some guidance as to why investors tend to be irrational, and shows how it is possible to profit from this irrationality.

The key to investor irrationality is that in many ways rationality in markets is counter-intuitive. Take one very good example - diversification. This tends to be the mantra of investment advisers and seems to make a lot of sense because it implies the spreading of risk and a rational allocation of resources.

Yet countless studies have shown that diversification is a lousy investment strategy. Peter Lynch, who ran Fidelity's Magellan Fund and is generally regarded as one of the most successful investment strategists of the past century, is credited with coining the term 'de-worse-ification', which has now entered the financial lexicon as the more succinct 'diworsification'.

What Lynch argued in essence was that diversification rarely works particularly well, and once asset prices start to fall it becomes a lousy way of preserving wealth. His case was elaborated by Charlie Munger, Warren Buffett's right-hand man at Berkshire Hathaway, who asked: 'Why would you want to put money into your 35th-best idea?' In other words, if you know that, say, five stocks are great performers and provide consistently good returns, why would you want to go out and buy another 30 stocks which are nothing more than so-so?

Behavioural finance teaches us something even more basic that everyone kind of knows but most people ignore: that investors love to buy shares when they are at their most expensive and they rush to offload stocks when prices fall. The pioneering work by Nobel laureates Daniel Kahneman and Amos Tversky, which they called the Prospect Theory, explains why this is so. They found that investors were more distressed by the prospect of loss than motivated by the possibility of gain. Thus they take greater risks to avoid loss than they would to achieve a gain.

In circumstances where gain appears to be assured, as in bull markets, investors think they should pile in because the risk of loss is minimal, whereas when a bear market takes hold they become risk averse and panic to sell. In so doing they reverse the pattern of their normal lives, when they queue up to buy goods that are cheap and are only moderately enthusiastic about buying when prices are high.

Since the Prospect Theory unlocked one of the bigger mysteries surrounding investor behaviour, many researchers have refined this theory and added to our understanding of why investors are so perverse.

There is, for example, much discussion of the so-called disposition effect, which looks at the other side of this equation showing that many investors hold on to bad investments for too long and sell winners too soon.

The failure to sell poor investments is, in part, explained by a reluctance to admit mistakes and it is this that also explains why investors are rational neither about buying nor selling.

Investors, understandably, want to feel good about their decisions and thus they filter out bad news about an asset and focus on favourable news. They also have an alarming tendency to disregard their own experience of investment failures.

Thus when, as usually happens, mutual funds underperform the market as a whole, investors fail to heed the lesson that this is a poor vehicle for investment; some passively hope that things will improve, while others seek out new mutual funds, searching for better returns.

But why do investors continue to invest in these funds? Sometimes they have no choice, as in the case of Hong Kong's hapless MPF investors, who are compelled by law to buy them. But sometimes it is because private investors place their assets in the hands of financial intermediaries who have a vested interest in recommending these funds because, unlike straightforward equity investments, the fund houses pay them commissions for their recommendations.

To pre-empt the usual response from the snake oil salesmen in this industry who conjure up exotic figures to justify their performance, practically every study of returns from mutual funds shows that on average they underperform and that they nearly always present returns shorn of the considerable charges levied on investors, making their level of underperformance seem less profound.

Yet investors persist in voluntarily placing such large sums of cash in these investment vehicles. One explanation for this is that these funds play the role of scapegoat; they absolve investors from the consequences of their decisions because others are taking the decisions.

However, this theory is less than persuasive because investors still participate in the choice of funds. It seems more likely that they are swayed by trust and proximity to the people who sell the funds.

New research throws up some even more worrying insights showing that investors can easily be swayed by factors entirely irrelevant to market conditions, such as the weather at the time of the decision, the colour of investment brochures and even the level of an individual's hormonal cycle.

Men with high levels of testosterone, for example, have been shown to have a greater risk appetite than those with less.

Then there is the way investment options are presented. Many so- called investment advisers have learned how to 'frame' a product by showing an inferior option before the one they really want to sell.

Presentation matters a lot here. This is why, for example, sellers of overseas property developments generally spend a lot of money hosting their presentations in luxury hotels and investing in fancy model layouts and literature.

This helps create an atmosphere of prosperity and a touchy-feely impression that lingers stronger than the reality, which is that overseas projects are generally offered abroad only if there is not sufficient demand in the home market where investors have more experience of the location and are likely to have visited the site.

This is not to say that most small investors are suckers, but that they have a curious way of processing information. Robert Shiller, one of the shrewdest observers of market behaviour, looked at how Japanese investors behaved before and after the big Tokyo market crash. At the height of the Japanese bull market only 14 per cent thought there would be a crash, but after it happened, 32 per cent were still expecting it to happen. In other words, they were giving too much weight to recent experience and extrapolating the future from this. Those who somehow believe that mainland markets can defy gravity for ever may care to take note.

Related to this is a phenomenon known as 'anchoring', which means investors over-focus on a single piece of information such as the price of a share at its peak. For example, someone may purchase a share at HK$50 and see the price rise to HK$75, but may be reluctant to sell it as he remembers having seen the share once at HK$100 and is fixated on the idea that this is the stock's 'real' value.

Or they may be fixated by another specific valuation measure such as the price-earnings ratio or price-to-book value and regard this single ratio as the sole determinant of a share's value.

There are many other reasons for investors' irrationality in making decisions and plenty of scope for discussion over what is rational and what is not. It should never be overlooked that investors need to feel comfortable with their investments. Therefore, if they feel comfortable with, say, taking profits after a 10 per cent rise in the share price as opposed to waiting for a 15 per cent increase, they are operating within their own comfort zone.

So there is no absolute standard for rationality, but by understanding the more blatantly irrational aspects of investment behaviour, a rational investor stands to make money and realise even bigger profits at times of market turmoil when investors predictably overreact.

Stephen Vines is the author of Market Panic: Wild Gyrations, Risks and Opportunities in Stock Markets