The Hard Truth: Successful Investing Involves a Lot of Luck

Of all the traits needed for success -- hard work, intelligence, creativity -- the most overlooked is luck.

Bill Gates went to one of the only high schools in the United States that had a computer. Warren Buffett was born in a capitalist country. Donald Trump had a wealthy father. All of these guys earned their success, but in slightly different circumstances outside of their control, things could have turned out far differently. To put it another way: People with just as much ambition and intelligence are broke nobodies only because they were born in a different time and place.

This same thing happens with all investors. And there's an easy way to prove it.

The long-term history of stocks tells us that:

Stocks earn a reasonable return after inflation.

The longer you stay invested, the better the odds of achieving that return.

Here's what this looks like. This chart shows the best and worst annual returns stocks generated over the last 141 years based on different holding periods:

Source: Robert Shiller, author's calculations.

This chart, which I've used before, is valuable in helping investors learn what to expect from stocks over time. But it has two flaws.

One, it assumes you start with a lump sum investment and hold it for 10, 20, 30 years. Nobody does that. They invest in little bits over time, ideally dollar-cost averaging every month or every quarter.

Two, the difference between earning 3.2% per year and 10.2% per year over 20 years might look small, but it adds up to an enormous disparity over time.

I used data from Yale economist Robert Shiller to show what would happen if someone invested $500 a month, every month, in the S&P 500 throughout every possible 20-year period from 1871 to 2013. Here are the top three and bottom three results (all of these numbers are adjusted for inflation and dividends):

Source: Robert Shiller, author's calculations.

These people invested the exact same way, with the exact same amount of money, for two entire decades. Yet one group ended up with seven times as much money. Not because they were smarter, but because they were born at a different time and saved during a different era. They got lucky.

Stretching this out to 30 years shows something similar. The lucky investors finish with five times as much money as the unlucky ones:

Source: Robert Shiller, author's calculations.

What amazes me is that these hypothetical investors would be considered some of the smartest around, investing steadily every month no matter what the market was doing, for decades on end. Doing this is emotionally taxing, and few investors can keep it up over time. In the real world, investors are more likely to buy after stocks have boomed, and to sell after a crash -- which devastates returns. Yet even with hypothetically perfect behavior, the difference in results between investors born in different generations can be the difference between no retirement and a lavish retirement. And it's mostly a factor of luck.

I found two takeaways from this.

One is the importance of diversification. Having a mix of stocks, bonds, cash, and real estate can reduce the risk of having all your assets in one investment that suffers a bad decade. A lot of the results in these charts can be explained by a simple factor: stock valuations at the end of the period were either in a bubble or a bust. Diversification can offer protection from extremes.

Two, not needing your money at a specific period of time, even if it's decades in the future, can be vital. When people say, "I need this money at 65," they're assuming the market is going to cooperate with that goal on their 65th birthday. But it might not. Having a flexible investment outlook helps put the odds of success in your favor. That could mean the ability to work a few years longer than you anticipated, or having enough liquid funds to tap for years before needing to withdraw from your stock portfolio.

In all the red scenarios above, holding stocks for just a few years longer would have dramatically improved returns. Without the ability to hold out a few years for a recovery, your experience as an investor relies on luck. And good luck with that.

Check back every Tuesday and Friday for Morgan Housel's columns on finance and economics.

Comments from our Foolish Readers

Help us keep this a respectfully Foolish area! This is a place for our readers to discuss, debate, and learn more about the Foolish investing topic you read about above. Help us keep it clean and safe. If you believe a comment is abusive or otherwise violates our Fool's Rules, please report it via the Report this Comment icon found on every comment.

I have to admit that luck has played a big part in my investing success thus far. I graduated from college and started a job in August 2007. The I began investing right away, but was eligible for my first 401k match from my company in 2009 (they paid the match in lump sum once a year, my contributions weren't eligible for the match until January of 2008). It just so happened that the match was paid and invested on the day of the market bottom. That one lump sum payment has been able to ride the bull market all the way up.

Wow...seems like an unfortunate hit to steady dollar-cost average investing. Surprising that one 30-yr period resulted in a loss of capital ($180k contributed vs value of $135k). Is it possible to show results of hypothetical diversified portfolios? Why is an inflation adjustment needed if contributions are fixed at $500/month?

So Morgan: One of your most cogent and seminal articles. It took me 20 years to learn this and another 10 to recover from the notion that I was smart enough to "out-think" randomness...ie LUCK.

The main event in this epiphany was my 80's ground floor investment in a "Chipoltle Like" restaurant chain much loved by the local populace. They made a classic but not fatal mistake when they over expanded and ran out of cash...they were recovering when the brilliant but quirky CEO made a FATAL mistake: REALLY FATAL... he crashed his amphibian aircraft and literally died. The company never recovered.

Pretty random and very discouraging...unless you learn the possibilities of randomness and its only antidotes- diversification and patience.

Five stars for a well thought out, logical and reasonably researched article that is nonetheless succinct. While tempted to quibble with your 10 year segments (the 1962 investor would have done exceedingly well if they just exercised patience for another 10 or 20 years) that would confuse the issue. Your parsing of time does illustrate the basic point quite well and clearly.

And I know this BECAUSE I was a young investor in the mid 60's, was tempted to boycott the big casino in the sky and rejected the idea doing quite well in the 80's and 90s Lucky... but also diversified and patient.

We've built our plan around the 20 year destination as there are is a faction of baby boomers who are late in their retirement asset accumulation. An "accelerated" methodology over 20 years may help them provide for at least some of their needs.

^ OK, that would be VERY interesting to me...lets see, when did you "Declare" this portfolio and how did you document it? I would love to see that list and your proof that you called it 10 or 20 years ago before the outcome was known....Alternatively, you could declare now and report back in 20 years how you did....I wont be very interested however, as I will either be deceased or close enough I wont benefit from the result....

Luck plays a huge role in everything. A great deal of a-holery results from the refusal by certain leaders and others to acknowledge this basic fact of life. They assume it away, because it doesn't fit their cartoon version of the American bootstraps narrative.

I think that people who are investing new money all of the time can ameliorate some of the issues that arise from the above data, but not all. Assume you aren't going to sell, and have the fortitude not to. You are a buy-and-holder. But when the market is doing really poorly, add more. When the market is doing very well, go back to one's original set-aside allotment, or reduce it.

By this means, one gets to take advantage of market drops, without risking losing out as a selling market timer. In the very long-run, it pays to own the market.

So far, this has worked out OK for me. I bought a ton of stuff in 2009. I bought a ton more in 2011 during the Euro crisis including BRK-B at under $70/share. I bought a ton of XOM in 2010, including some under $60/share I think. This is all recorded by me in my CAPS pitch comments. I had already owned these stocks. I knew and liked the companies. I just bought more shares. It's market-timing but it's not quite the same, because you aren't trying to time sales, or to trade in-and-out. I just try to buy a little more when I am bat**it scared and want to sell.

Recently, I have not felt this fear/excitement. I have been paying my remaining student loan debt down. I have been learning to play the electric guitar. I'm still putting my minimums of money in every month into drips of BDX, XOM, LMT, and UNP, and into my kids' 529 plans, and still contributing to my 401k (though not at the maximum, until the student loan debt is gone).

This all is of particular interest to me because I'm 34. I graduated from college in 2000. That was the worst possible time to start investing, within the last 70 years or so. It took me until about 2005, to realize that fact, even though I was always drawn to value stocks, even in 2000. For years I had no interest in stocks, and also, not a ton of money. That's why I had nearly my entire next egg of about $50K in an FDIC-insured 5% CD from 2007-2008, and why I was able to get excited instead of scared in 2008-2009 (though I started buying way too early, during 2008).

Now I'm neither excited nor scared. If you are a buy-and-hold guy or gal of around my age, don't be shy about starting monthly DRIPS, or 529 monthly contribution plans, or 401ks (though at this point I recommend debt-repayment over new DRIPS). It's very possible that the secular bear market of 2000-2009 is over. It's also possible that it is not. It's possible that it could go on for fifteen more years, which would mean the majority of my and your investing lifetimes. Nobody knows. Anyone who says otherwise is either a moron or selling something.

Prepare for both possibilities. Include some bonds in your portfolio even if though they really look like they are a terrible investment. Home-ownership is a 19-year-low, and interest rates are low, a good person is in charge of the Fed, and people HATE home-ownwership. Articles appear every day extolling the rental economy. Pay attention to that. Consider buying a modest house if you think you are going to be in one place for the next 10-15 years. It's not as good a time as 2010 when I bought mine, but it's still not horrible. Pay down student loans. Invest some.

And save. Saving matters a heck of lot more than any particular thing you choose to invest in. If the market drops 10% add a little more. If the market drops 20%, add a little more. If the Fed's actions turn out not to have been a bridge over troubled waters, or if some deluded ideologue like Fisher is put in as Fed Chair when Yellen's term expires (assuming her work is not yet done), then be ready to invest more from your savings when the deflationary fit hits the economic shan.

Separately, I also follow these guidelines: 1) When everyone sees only peace on the horizon and when people are talking about how defense contractors are toast, that's a good time if you have your eye on one. War is the only thing that is as inevitable as taxes and death, and defense spending will not be cut significantly in the long term. 2) When an oil company has a huge spill and its stock tanks, or when everyone is talking about the end of oil, that's usually a decent time to buy the shares. 3) Always own some tobacco stock, and buy on litigation-related dips, but don't make this a massive component of the portfolio (RAI is interesting now, since it has a new $23.6 billion verdict against it that I would say is 99% likely unconstitutional, yet has shaved 10% off of its shares). 4) If everyone is saying that a company like MSFT or WMT is toast, but the numbers don't reflect that it is going out of business anytime soon, pay attention to that. Know the importance of English-major-style themes, but understand that sometimes they become fads, they contradict the actual revenue and cash flow numbers, and nobody seems to care. This is amazing to me when this happens.

Due your own diligence, and don't take anything I'm saying as a guaranty or as advice you should act on. These are just the opinions of one dude with no professional investment training of any kind.

I have only the most rudimentary/tenuous understanding of how arbitrary reality is at its most basic (i.e., quantum) level; still, it's enough to make me constantly bemused by those who think that circumstances are ultimately any less contingent at "higher" levels of organization (e.g., human levels of experience---including the realm of investing).

Although Bill Gates had a computer in HS, he still put in a lot of hours at his craft. If you believe Gladwell's book Outliers it was at least 10,000 hours. And he worked hard doing it (usually late at night and weekends, something that today's youth probably wouldn't have done). There are probably dozens (hundreds?) of other people at the same school who are not billionaires. The lucky part for him was when he was born. If he had been born 1 or 2 years on either side of his actual birthdate he might have missed the early 1970's explosion in technology (and especially the invention of the microprocessor) and someone else might have written his software code instead.

Speaking of Microsoft, five years ago there were two different groups of people with widely disparate views regarding Microsoft.

On one side, which was actually the majority, were people who eagerly agreed with each other because they were all telling each other what they all already "knew"... that Microsoft was dead and investing in Microsoft was "no way to beat the market".

On the other side were people who examined Microsoft's financial statements, saw fantastic margins, a ton of cash, earnings out the kazoo, and scratched their heads in amazement at the ignorance of the first group.

Microsoft's dividend is now over double what it was then, and so is its price.

Its financial statements today bear the same message they did then. Microsoft is still a money machine with a rapidly growing dividend.

That dividend is backed by Microsoft's undiminished financial strength and ability to pay. It will continue to grow.

This will keep Microsoft's shares in demand by dividend investors enough to force the price to grow along with the dividend, just as it did in the last five years.

Microsoft's financial strength and stability is sending the same message it was five years ago to anyone who knew how to read it.

Dividend investors will supply enough demand for Microsoft's share that no way is Microsoft going to grow its dividend and the price stay where it is. No way Microsoft is going to be priced to yield 4% or 5%. Not gonna happen, outside of an occasional bad dip, which is a blip in the long run.

What a great article. I especially like the part about being flexible on your end retirement date vs. needing the money at 65.

Luck plays a part in most everything. True the harder you work, better your strategy, etc. the luckier you may get, but the world definitely has some randomness to it that makes some people luckier than others.

It still makes no sense to me. Morgan's article nor the comments seemed to have any guidelines.

How to recognize "English-major-style-themes" to recognize it as a fad. Are you talking about companies that speak English or publish their 10-Qs in English? I can't even find that term on the internet machine.

"Holding an opinion derived solely from popular crowd sentiment, showing a total lack of ability to think for oneself and do a competent financial analysis revealing the actual reality conveyed by the numbers in the financial statements".

To fully analyze a company for long term investment purposes you need two skills: 1) finance/math skills including the ability to understand the three major financial statements and conduct a discounted cash flow analysis; and 2) written analysis skills, what I call "English major" skills, since I was one, but which are critical thinking and research skills.

The reason you need the second type of skill is to think critically about how the company will perform in the subsequent ten years or so (since I assume we are buy-and-hold people who at least initially intend to hold that long). Without this skill, one ends up simply assuming that, say, the level of growth from the past five years will continue for the next ten. In other words, one makes assumptions, that may not be based on reality. Perhaps, rather than "English major" skills, I should have said, "critical thinking" skills.

Without critical thinking skills, you end up buying Kodak stock or New York Times stock in 2000, thinking it is cheap. You end up projecting Twitter to have infinite amounts of growth. You get suckered into buying Radio Shack when it looked cheap a few years ago. Doing the math does not tell you how the company will be doing in ten years. To do that requires imagination, and critical thinking, and an awful lot of knowledge about both the industry in which it operates (its competitors and customers), and things outside of that industry that might disrupt it (otherwise you end up investing in buggy whip manufacturers just as Ford is getting going). That is what I mean by "English major" analysis.

The problem is that this analysis can also get corrupted, just as a DCF analysis can get corrupted. In the case of this non-scientific type of analysis, it is susceptible to fads, to themes that become very powerful in a negative sense. (And, it is also susceptible to fadish themes that become very powerful in a positive sense, which is why people bought pets.com stock.) That happened in 2009-2011 with Microsoft, in a negative sense. It by and large never matched the actual numbers that Microsoft was reporting quarter after quarter.

notyouagain: as to my score, all of my picks are real money picks, and I have noted all additional purchases and sales in comments. I have outperformed the market over the last four years and have a beta around .85 while doing so. See how many people you can find on CAPS like that. My other player, TheDumbMoney2, actually followed the silly rules of CAPS, and still has a 98.55 rating even though I haven't touched it in over a year. Be well.

I was only trying to paraphrase what I thought you meant, but my own prejudices crept into it from when I used to find myself arguing with higher-rated players who seemed incapable of understanding the numbers years ago.

Please do not assume I was being critical of you. You have always been one of my favorite players!!

To add to my prior comment, addressed to everyone, the stock market is quite susceptible to positive faddish themes when there is no real cash flow or profits. There are no numbers to go on, there is only the "English-major" style analysis.

What was so shocking about the Microsoft case, and what I was referring to, is that in that case there WERE numbers, every quarter, for years, and the numbers almost every quarter kept negating the "it's toast in five years" thesis. Yet, people kept saying it again, and again, and again, on Twitter, in blogs, here, on tv, you name it.

This was enabled/exacerbated with Microsoft because a certain type of wired, sophisticated investor (I'm looking at the Paul Kedroskys and Josh Browns of the world) grew up hating Microsoft when it was the big bag bully on the Tech Block. These types of otherwise smart people were highly susceptible to a declinist theme. Kedrosky shorted Microsoft for years, and I don't know if he ever said when he stopped shorting it, or if he did stop shorting it before it started to rocket up in 2013. Josh Brown (who I love, by the way) was a major declinist on Microsoft. There were a ton of these people, everwhere. It was "cool" to hate Microsoft. They wanted to and thought they could talk it into the oblivion they so deeply hoped it would arrive at. It was deeply uncool to actually look at its financial statements every quarter.

Now this isn't to say Microsoft will be around forever. It still has challenges, primarily continuing to focus on devices in an in vain attempt to return to its monopolistic position of old. It bears watching. It's not something you can buy and stick in a drawer for twenty years. But, so far, the quarterly numbers have told the best story. It just took two or three years for the appraised market price to reflect that story.

By "focus on devices" I mean that Microsoft is trying to tie people into Windows and Office use on its own devices. (I.e., if you don't buy a Surface, you don't get full mobile Office that works as perfectly.) What I think it should be doing is making sure that people who use iOs and Android are using Office on those devices. The new CEO is moving in that direction, too, however....

Some people do. They misread people like Buffett and Fisher (the dead investor, not the current Fed member) and think it means you never sell. I just wanted to put in the caveat that "buy and hold" does not mean "buy and forget." That is even more true of tech companies than of companies like Coke.

I must say I like doing the numbers. I have to always be vigilant of my love for the numbers, and make sure I read enough articles and comments about companies that I don't rely only on financial analysis.

Hbofbyu said: "Also, home ownership has never outperformed the stock market over the long run. Why would you recommend it now?"

First of all, take anything I'm saying with a major grain of salt and do your own diligence, I'm just some dude. That said, here's way.

1) The investment arguments against homeownership, I think, assume you pay all cash. Nobody does, outside of investors who are planning to flip short-term. If you put 20% down, that is a leveraged investment in the property on which the house sits, coupled with ownership of a depreciating asset (the house). Because it is leveraged, the housing market as a whole does not have to significantly beat the market in order for one to come out ahead. Let's say you buy a $100K house and you put $20K down. Let's say the value of it only goes up 2% per year, matching inflation. In thirty years, your house is worth $181,136.16. But the return on your original $20K (ROIC) is not 2% per year. The return on that is closer to 5% annualized, nearly double.

Moreover, because 30-year fixed loans (which I'm assuming) are so heavily amortized, for years you are paying mostly interest. Congratulations, you get a massive portion of that back in a tax deduction, especially if you have a high income!

Yes, you pay down your loan over the thirty years, and you get a lower ROIC on these dollars, but the vast majority of it you pay down in dollars that have been drastically devalued by between 20 and 30 years of 2-3% inflation, assuming inflation does not jump (in which case they are devalued more).

You also have to deduct maintenance and repair costs, but a lot of that can be done oneself. In the past three years I have repainted two rooms, put on gutter covers, fixed rails on my porch that were dry-rotted, etc.

It's also dependent upon where you live. These average figures look at the whole country. That means they are averaging in Detroit over the last 40 years, with Los Angeles and Dallas over the last 40 years. That seems nuts to me. Real estate is local. The "investment" portion of a home purchase is really the land. It is the land that appreciates in value, and it does so when macroeconomic and regulatory pressures in a particular region or city cause it to do so. Buying a house in rural Kentucky is probably not as good an idea, generally, as buying one in Raleigh, in other words. There is little demand in rural Kentucky, and little economic growth, and there is tons of space, and there are no NIMBY neighbors putting up all kinds of zoning ordinances that restrict building and prop up their property values (whether intentionally or not) because everybody hates the gub'ment.

2) As stated above, interest rates are low, fixed rate 30-year loans are a crap deal for the lenders unless we get outright deflation. (I refinanced mine in 2012 at 3.75% -- that is INSANE my friend -- a 30-year fixed loan, you have to be kidding me.) Homeownership rates are at a 19-year low. We have a new Fed Chairman who is just as strong as Bernanke was in resisting the calls of creditors to allow deflation (whether they know they are advocating that or not, and some do not, they are actually deluded).

Buying a house is not right for everyone, and I may be wrong. And it depends upon where one is buying. And one has to make some sanity-check effort to compare it to the cost of renting (when monthly loan/tax costs in an area are historically high compared to rents in that area, that is a problem). But it is not always the investment sinkhole it is now portrayed to be. Literally tens of millions of people in this country have major assets only because they bought a home in an area where there was economic growth and immigration decades ago, and held onto it.

3) I live in California. So in addition to the above, I get Prop 13, which means my property taxes can never go up more than 2% per year no matter how much my property appreciates. That removes a lot of uncertainty. The equation is different in states like New York where do-gooder policy-makers are constantly allowed to raise taxes willynilly on property owners whenever they want to fund a new pet project. Also, like many, many states, California is a non-recourse state, which means if I ever default, my lender can't go after my personal property to recover any deficiency. So in addition to getting a great rate on my loan, I have a built in default option that can never, ever cost me more than what I paid in. (Which, incidentally, is why the people who "bought" homes in California and similar states with interest-only loans or low-payment ARM loans in 2006 and then got foreclosed on a couple of years later -- after basically paying just a low rental rate -- and then screamed bloody murder about it, are either ignorant, or are scumbag moochers.)

4) While an investment in a home may not beat the market, INVESTORS DON'T BEAT THE MARKET and virtually nobody just owns the market. Statistically, investors are likely to trade in at horrible times, and sell at horrible times. It's so easy to trade. It makes you feel so smart. With a house, you can't just freak out and sell because of a "flash crash" or because the Eurozone looks like it is falling to pieces. The very idea of that is insane. So pay attention to the psychological aspect. Ask yourself how many people who invest in the market actually beat the market over 30 years anyway. Ask yourself if you have the psychological fortitude and skill to do so. Do not ignore the psychology of all of this. As Munger says, life is all about incentives. Having a brokerage account with sub-$10 commissions creates an incentive to trade too much. Have a home that would be a massive PITA to sell and where you would have to move your whole family creates a massive disincentive to "trade" too much.

Again, everyone, DYOD. I'm just some guy with no particular formalized financial training other than a semester each of macro and microecon in college. I'm not an expert and I'm not giving advice, just my own opinions for you to consider or not.

(So those are a clarification of my original comment, obviously. My point is, you can't rely on blunt instruments like the relative appreciating value of all property in all of America versus the stock market, anymore than you can merely rely on PE or CPE or Tobin Q or what you like to buy in stores, or a broker, when investing investing in the stock market. All of life must be addressed wholistically. Thus, any investment decision must begin with: 1) a diligent and objective effort to seek out all of the factors that might impact it, whether financial, regulatory, macroeconomic, psychological, or whatever; and 2) a diligent and objective effort to evaluate as best as one can the likely impact of each of those factors as applied to one's own particular situation, in one's own spot in life, in one's own city, in one's own state, in one's own country, according to one's own finances, prospects and proclivities. There are no shortcuts. There are no easy answers. There is no golden ticket. There is no unifying theory. There is no Platonic ideal. The is only the relentless seeking out of all possible relevant facts, including trying to determine, as a starting point, who one is as a person, as to which one must apply the coldest, most objective eye of all.)

I do believe a lot more strongly than I guess you do that the forces of randomness can be minimized with a good analysis. I've done well every time I've held shares by evaluating the safety of the dividend and doing just enough "English major" type research to make sure the company was still relevant.

Average real-life return has been 35%...of course I had a significant tailwind from starting during the crash.

But I still believe. I believe in keeping an eye towards what I hope my dividend-adjusted price will be years from now because that involves dividend reinvestment and takes away some of the randomness.

Also, I can look at the "Historical Prices" listed @ Yahoo!Finance and see this working in just about every case for companies I would even think of investing in when I compare actual closing prices from years ago to the "Adjusted Close" prices for the same dates.

I think we can gain a little more certainty by carefully choosing, evaluating, researching, and holding companies we believe have durable competitive advantages, low to moderate payout ratios, above average interest coverage, and trying to pick a company that's demonstrated above-average dividend growth.

We can do our best to select companies with all the above characteristics that we believe have the financial strength to keep growing that dividend at a good pace for now, and keep an eye on them for deteriorating fundamentals.

Rule of 72 says if a dividend grows at 12% annually the dividend will double in about 6 years. If the financials haven't changed, the share price seems to just about always grow right along with the dividend, as with Microsoft in the last 5 years.

The truth is, well over half the stock buys recommended by the MF Stock Advisor service do not beat the market. So even when following MF advice, there is a luck factor--fingers crossed that I've picked the DIS, AMZN, NFLX, PCLN's, and at the right times, rather than the CEB, TWX, and KKD's. I "outsource" market research to the Fool, but the truth is there is still a greater than 50% chance that any given recommendation will not beat the market.

I should admit the company has had two stock splits during that time, but it isn't easy to find great market-leading companies with well-known brands that haven't split once or twice in time frames that long....