There was no big news at Berkshire Hathaway Inc.’s annual meeting this past weekend, but there was one great lesson for investors: Perhaps the most important thing you can do when everything seems to be going right in your portfolio is to listen to somebody who insists you are wrong.

To spice up the annual ritual, Berkshire’s chairman, Warren Buffett, invited someone who has placed a bet against the stock—short-seller Doug Kass of hedge fund Seabreeze Partners Management—to join the panel of analysts posing questions to Mr. Buffett and vice chairman Charles Munger.

Carp, if you will, that it didn’t take much bravery for Mr. Buffett to give air time to one skeptic among the more than 35,000 worshippers who would trample their grandmothers to kiss Mr. Buffett’s feet if he took his socks off. Complain, as many already have, that Mr. Kass’s questions weren’t all that tough.

Then ask yourself: When is the last time the management of a major U.S. company sought out unrestricted criticism from someone betting against the stock?

To get a sense of how unusual it was for Mr. Buffett to invite a bear to ask questions freely, consider a survey of more than 500 companies by the National Investor Relations Institute in 2011. The research found that 80% placed limits on who can ask questions during the quarterly ritual of the earnings conference call. Nearly 25% of the companies took questions only from “pre-approved lists” of callers. Only 11% permitted individual investors to ask questions; just 12% said the floor is open to everyone.

What is more, 76% of companies prepared scripted answers to the questions they expected to get. According to the NIRI survey, the prepared comments by top executives that open the typical earnings conference call are prerecorded by about one out of 12 companies, but more than 80% of them don’t disclose that the remarks have been prerecorded.

But before you start comparing U.S. corporate management to a closed system like, say, North Korea, ask yourself another question: When is the last time I tried as hard as possible to find someone to refute my own investing ideas?

Mr. Buffett is “self-confident, but he’s not afraid of a challenge,” Mr. Kass told me last week. “I believe he enjoys challenges.”

A deliberate, lifelong effort to find people to tell him why he might be wrong is one of the keys to Mr. Buffett’s success. It doesn’t come naturally to most investors.

Mr. Buffett once noted about the scientist Charles Darwin that “whenever he ran into something that contradicted a conclusion he cherished, he was obliged to write the new finding down within 30 minutes. Otherwise his mind would work to reject the discordant information, much as the body rejects transplants. Man’s natural inclination is to cling to his beliefs, particularly if they are reinforced by recent experience.”

Another vehement believer in the importance of challenging your own investing ideas is Ray Dalio, founder of Bridgewater Associates, the world’s largest hedge-fund manager, which oversees more than $150 billion.

“When two intelligent parties disagree, that’s when the potential for learning and moving ahead begins,” Mr. Dalio told me last week. “The most powerful thing that [an investor] can do to be effective is to find people you respect who have opposite, different points of view [from yours]—and have an open-minded exchange with them about what’s true and what to do about it.”

This principle, which Mr. Dalio calls “thoughtful disagreement,” is more important than ever. Few sayings on Wall Street are more true, or more universally ignored, than “don’t confuse brains with a bull market.” When markets are around record highs, as stocks and bonds alike are today, those rising prices inevitably lead to overconfidence and complacency.

“When you think you’re right, you don’t go looking,” says Mr. Dalio. “When you think you’re right, your mind isn’t open to learning. The more you think you know, the more closed-minded you’ll be.”

Adds Mr. Dalio: “The most common mistake of investing is the failure to distinguish something that’s become more expensive from something that’s still a good investment. Too many people [make] the mistake of looking to the price to gain confidence, without recognizing that that doesn’t make any sense.”

Only by deliberately seeking out thoughtful disagreement, says Mr. Dalio, can you counteract the false confidence conferred by a rising market. Investors “could really improve their probabilities of being right by 30% or 40% simply by saying, openmindedly, ‘Who disagrees with me in an intelligent way? And let me understand that disagreement,'” he says. “I think everybody can do that. They just have to make a deal with the people around them to be that way.”

How can investors build this kind of thoughtful disagreement into their decision-making?

“People on Wall Street take themselves very seriously, and in order to do this you need almost a sense of play and make-believe,” says Mariko Gordon, chief executive of Daruma Capital Management in New York, which manages roughly $2 billion.

Explains Ms. Gordon: “You have to pretend to be somebody completely different. Ask yourself: If I’m the most brilliant short-seller in the world, what is the investment case I’d be looking for? If I’m a momentum guy [who buys rapidly rising stocks], is there anything here for me?” This way, she says, you force yourself to take radically different perspectives on the same idea.

Mike Ervolini, chief executive of Cabot Research, a Boston-based firm that advises money managers on how to improve their skills and processes, measures how long it typically takes for good investments to stop working. Often, his firm has found, an investor will hang on to a winning stock even after its superior returns start to fade.

He suggests studying your account history to learn how long your average winner tends to outperform. As your latest winner comes up on that average holding period, use that as a signal to seek out a contrary opinion.

Finally, look back in your account statements, not in your fallible memory, to see what you did in mid-2007 and in early 2000, the last times the market hit record highs. If you bought more, became a fast trader or piled on other risks, then you need to invite the toughest, most critical person you respect over for a cup of coffee.

Talk about your temptations. Throw yourself wide open to criticism. When the bull market ends, as it surely will, you’ll be glad you did.