It’s official: Pre-retirees took it on the chin in the Great Recession, watching their net worth shrink at an inopportune moment. Surprisingly, their investment portfolios held up. The real damage was a result of the housing downturn, underscoring the axiom that for most people your house is your most important financial asset.

The retirement wealth of those who were aged 53 to 58 just before the recession fell by 2.8% the three years through 2009, according to a report from the National Bureau of Economic Research. That doesn’t sound like a big hit. But it came at a point in life when even modest growth can make a huge difference. Previous generations saw wealth grow by 5.4% at this age.

In other words, a $500,000 net worth would have shrunk to $486,000 during the downturn instead of growing to $527,000. That difference—$41,000—left to grow at 6% annually for the next 25 years would produce $181,714 of additional retirement wealth. So it’s more than a psychological blow. The authors take note of this group’s particular vulnerability:

“They have very few effective options for adjusting their behavior in the face of the recession. They can postpone retirement and save at a higher rate. But postponing retirement is of less help to those who have lost their job. Moreover, there is little time to increase saving. So any large losses from the recession are likely to be permanent, affecting welfare throughout retirement.”

Perhaps the only age group more vulnerable than pre-retirees was those who had recently retired. The study did not look at this group. But having already left work they had even fewer options and might have been stuck drawing down their nest egg as values were depressed, cutting off years of planned income.

The study looked at a comprehensive measure of wealth, taking into account the value of defined benefit and defined contribution pensions, Social Security, individual retirement accounts, the net value of housing and other accumulated financial and nonfinancial wealth. This broad look is one reason investment portfolios held up. They include a lot of stable assets, and at this age even market-based savings may be tilted toward bonds. Some 55% of total wealth for this group is tied up in Social Security and traditional pensions, the authors found.

Net assets not counting housing grew during the period—but not enough to offset the housing downturn, which clipped wealth by 4.5 percentage points. Previous generations enjoyed a net gain in housing at this point in their life. These findings underscore the importance of a housing recovery, which would restore some lost wealth if it occurs before new retirees sell.

Surprisingly, the study found that pre-retirees in the Great Recession experienced only modestly higher rates of job loss than previous generations at this age. That suggests pre-retirees, in their peak earning years, are vulnerable to layoffs no matter what the economic climate. Meanwhile, those on the cusp of retirement at the onset of the Great Recession are retiring at roughly the same pace as previous generations at comparable ages, the study found. That suggests that limited options at this age prompt many folks to retire no matter what their finances look like, and simply make do.

Dan Kadlec is a journalist who has written about personal finance for TIME and other outlets for 25 years. He is the author of three books, a leading voice in the global financial literacy movement, and strategic adviser to the National Financial Educators Council.

Kadlec's latest is A New Purpose: Redefining Money, Family, Work,Retirement, and Success