The “Oracle of Omaha” went on an epic rant against Wall Street this weekend.

Just before lunch at the Berkshire Hathaway Inc. annual meeting on Saturday, Warren Buffett unloaded what he called a “sermon” about hedge funds and investment consultants, arguing that they are usually a “huge minus” for anyone who follows their advice.

The Berkshire chairman has long argued that most investors are better off sticking their money in a low-fee S&P 500 index fund instead of trying to beat the market by employing professional stockpickers. He used the annual meeting to update the tens of thousands in attendance—and others watching via a webcast–about his multi-year bet with hedge fund Protege Partners. The bet, initiated by the New York fund back in 2006, was that over a decade, the cumulative returns of five fund-of-funds picked by Protege would outperform a Vanguard S&P 500 index fund, even when including fees.

Mr. Buffett showed a chart comparing the cumulative returns of the two sides of the bet since 2008. As of the end of 2015, the S&P 500 index fund had a cumulative return of 65.7%, outdoing the hedge fund teams’s 21.9% return. The S&P has outperformed in six of the eight individual years of the bet too.

The chart was preamble to the real point Mr. Buffett wanted to make: that passive investors can do better than “hyperactive” investments handled by consultants and managers who charge high fees.

“It seems so elementary, but I will guarantee you that no endowment fund, no public pension fund, no extremely rich person” wants to believe it, he said. “They just can’t believe that because they have billions of dollars to invest that they can’t go out and hire somebody who will do better than average. I hear from them all the time.”

But he was just getting started.

“Supposedly sophisticated people, generally richer people, hire consultants, and no consultant in the world is going to tell you ‘just buy an S&P index fund and sit for the next 50 years.’ You don’t get to be a consultant that way. And you certainly don’t get an annual fee that way. So the consultant has every motivation in the world to tell you, ‘this year I think we should concentrate more on international stocks,’ or ‘this manager is particularly good on the short side,’ and so they come in and they talk for hours, and you pay them a large fee, and they always suggest something other than just sitting on your rear end and participating in the American business without cost. And then those consultant, after they get their fees, they in turn recommend to you other people who charge fees, which… cumulatively eat up capital like crazy.”

Mr. Buffett said he’s had a hard time convincing people of this case.

“I’ve talked to huge pension funds, and I’ve taken them through the math, and when I leave, they go out and hire a bunch of consultants and pay them a lot of money,” he said, earning a laugh from the crowd. “It’s just unbelievable.”

“And the consultants always change their recommendations a little bit from year to year. They can’t change them 100% because then it would look like they didn’t know what they were doing the year before. So they tweak them from year to year and they come in and they have lots of charts and PowerPoint presentations and they recommend people who are in turn going to charge a lot of money and they say, ‘well you can only get the best talent by paying 2-and-20,’ or something of the sort, and the flow of money from the ‘hyperactive’ to what I call the ‘helpers’ is dramatic.”

A passive investor whose money is in an S&P 500 index fund “absolutely gets the record of American industry,” he said. “For the population as a whole, American business has done wonderfully. And the net result of hiring professional management is a huge minus.”

Mr. Buffett has long had a testy relationship with Wall Street, and he’s positioned himself for decades as an outsider to the world of New York finance. In addition to repeatedly attacking the fees charged by hedge funds and investment professionals, he’s criticized the tactics of activist shareholders, the danger of derivatives and the heavy use of debt by private-equity firms.

The antipathy can run in the opposite direction as well. As our Anupreeta Das noted in an article last year, many on Wall Street believe the Berkshire chairman to be a hypocrite, hiding behind the image of a folksy, benevolent investor while pursuing some of the tactics that are the targets of his attacks.

On Saturday, Mr. Buffett worked in a fresh plug for a book he’s been recommending for decades, “Where Are the Customers’ Yachts?,” by Fred Schwed. The title comes from the story of a visitor to New York who was admiring all the nice boats in the harbor, and was told that they belonged to Wall Street bankers. He naively asked where the bankers’ clients kept their boats. The answer: They couldn’t afford them.

“All the commercial push is behind telling you that you ought to think about doing something today that’s different than you did yesterday,” Mr. Buffett told his shareholders. “You don’t have to do that. You just have to sit back and let American industry do its job for you.”

Berkshire Vice Chairman Charlie Munger jumped in to offer a counterpoint, of a sort:

“You’re talking to a bunch of people who have solved their problem by buying Berkshire Hathaway,” he said. “That worked even better.”

From 1965 through the end of last year, Berkshire shares have risen 1,598,284%, compared to the 11,355% return on the S&P 500.

“There have been a few of these managers who have actually succeeded,” Mr. Munger said. “But it’s a tiny group of people. It’s like looking for a needle in a haystack.”

Mr. Buffett conceded that point, but concluded the first half of the day’s proceedings by saying that Wall Street was better at salesmanship than investing.

“There’s been far, far, far more money made by people in Wall Street through salesmanship abilities than through investment abilities,” he said. “There are a few people out there that are going to have an outstanding investment record. But very few of them. And the people you pay to help identify them don’t know how to identify them. They do know how to sell you.”