Warren Buffett on Saturday devoted more than four pages of his 29-page annual shareholder letter to criticism of active managers on Wall Street, excoriating what he perceived as exorbitant fees they charge for returns that fail to live up to lofty assumptions.

Meanwhile the legendary stock picker extolled the virtues of passive investing and its advantages for regular investors. The 'Oracle of Omaha' even compared active managers to monkeys, and estimated that financial advisors, in their futile search for ways to beat the market, had cost clients $100 billion in wasted fees in the last 10 years.

"When trillions of dollars are managed by Wall Streeters charging high fees, it will usually be the managers who reap outsized profits, not the clients," stated the widely-read letter released on Saturday morning. "Both large and small investors should stick with low-cost index funds."

Chris Kleponis | Bloomberg | Getty Images

'The results were dismal'

Buffett started this critical section of the letter with an update on a 10-year wager against Wall Street's active management he made nine years ago, with the proceeds going to a charity. This is how the billionaire described his original challenge:

"I publicly offered to wager $500,000 that no investment pro could select a set of at least five hedge funds – wildly-popular and high-fee investing vehicles – that would over an extended period match the performance of an unmanaged S&P-500 index fund charging only token fees. I suggested a ten-year bet and named a low-cost Vanguard S&P fund as my contender. I then sat back and waited expectantly for a parade of fund managers – who could include their own fund as one of the five – to come forth and defend their occupation. After all, these managers urged others to bet billions on their abilities. Why should they fear putting a little of their own money on the line?"

To his surprise, only one person stepped up to take the other side of the bet: Protégé Partners' Ted Seides, a 'fund of funds' manager. According to the bet, Seides selected five funds of hedge funds, whose results after fees would be averaged and compared to Buffett's selection, a Vanguard S&P index fund.

Here's what happened, according to the letter:

"The compounded annual increase to date for the index fund is 7.1%, which is a return that could easily prove typical for the stock market over time...The five funds-of-funds delivered, through 2016, an average of only 2.2%, compounded annually. That means $1 million invested in those funds would have gained $220,000. The index fund would meanwhile have gained $854,000."

In fact, none of the basket of funds came even close, according to Buffett:

"The results for their investors were dismal – really dismal. And, alas, the huge fixed fees charged by all of the funds and funds-of-funds involved – fees that were totally unwarranted by performance – were such that their managers were showered with compensation over the nine years that have passed," Buffett wrote. "As Gordon Gekko might have put it: 'Fees never sleep.'"

Investors seem to be heeding Buffett's anti-active advice, as more than $20 billion flowed out of U.S. active equity funds in January despite a rising stock market, according to Morningstar. In the last 12 months, more than half a trillion dollars have flowed into passive funds, while active funds have experienced outflows, Morningstar's data showed.

In his letter, Buffett criticized how the whole Wall Street complex is still set up to send pension funds, endowments and other investor types into under-performing active vehicles. He claimed that the wealthy investor classes are getting ripped off the most:

"In many aspects of life, indeed, wealth does command top-grade products or services. For that reason, the financial 'elites' – wealthy individuals, pension funds, college endowments and the like – have great trouble meekly signing up for a financial product or service that is available as well to people investing only a few thousand dollars. This reluctance of the rich normally prevails even though the product at issue is –on an expectancy basis – clearly the best choice. My calculation, admittedly very rough, is that the search by the elite for superior investment advice has caused it, in aggregate, to waste more than $100 billion over the past decade. Figure it out: Even a 1% fee on a few trillion dollars adds up. Of course, not every investor who put money in hedge funds ten years ago lagged S&P returns. But I believe my calculation of the aggregate shortfall is conservative."

Buffett stated that he knows of only 10 managers that he spotted early on who could outperform the S&P 500 over the long term, and they did so. He acknowledged there are more out there who may be able to beat the market, but they are the clear exception.

"Further complicating the search for the rare high-fee manager who is worth his or her pay is the fact that some investment professionals, just as some amateurs, will be lucky over short periods," Buffett wrote.

"If 1,000 managers make a market prediction at the beginning of a year, it's very likely that the calls of at least one will be correct for nine consecutive years. Of course, 1,000 monkeys would be just as likely to produce a seemingly all-wise prophet. But there would remain a difference: The lucky monkey would not find people standing in line to invest with him."

The billionaire heaped praise on Jack Bogle, the founder of the Vanguard Group who started the first index fund 40 years ago.

"If a statue is ever erected to honor the person who has done the most for American investors, the hands down choice should be Jack Bogle," the letter stated. "In his early years, Jack was frequently mocked by the investment-management industry. Today, however, he has the satisfaction of knowing that he helped millions of investors realize far better returns on their savings than they otherwise would have earned."