Melinda Gates was lauded for her bravery last week in stating her unequivocal support for contraception, especially for women in developing countries. She’s right, of course, that family planning saves lives, but it’s odd that a public embrace of contraception still makes headlines when more than 98% of American women have used contraception at some point in their lives. The problem, unfortunately, is that many Americans aren’t using contraception at the right point in their lives, and this oversight is especially common in the people who need it most: women who want a piece of the American Dream.

This is dangerous ground. No one wants to suggest that certain women shouldn’t have babies. But the hard truth is that certain women shouldn’t have babies if they aspire to a middle-class life.

To suggest that some women should delay (or forswear) having children harks back to the old “population bomb” anxiety of the 1960s, when the Town & Country set seemed to embrace procontraception policies in part to keep “riffraff” from solidifying their tenuous perch on the socioeconomic ladder. The history of contraception advocacy in the early 20th century carried a nasty stink of eugenics that continues to make a lot of people understandably uneasy. But that unease shouldn’t keep us from facing reality. Births to financially insecure single women who aspire to middle-class life are a hindrance to upward mobility.

Gary King, the eminent political scientist at Harvard, likes to talk about “evil hypotheses,” the vexing hot-button social problems that people find too politically or morally controversial to address systemically. This is a terrible shame because the more we put our heads in the sand about the comprehensive economic impact of unwed motherhood on working-class and middle-class families — including a thorough reckoning of its downstream financial costs to children’s and mothers’ physical, academic, psychological and social health — the more the void is filled by people who don’t have much intention of doing anything about it.

This is not to say that single motherhood takes a uniformly negative toll; its impact varies quite a bit. Some economists have suggested that for the very poorest and least educated young women, having a baby doesn’t significantly alter the already low prospects. In those cases, a baby born to a poor (usually unwed) teenage mother is often the symptom, not the cause, of a larger problem of dashed opportunity. That’s a depressing story in its own right, with its own dire policy implications.

But equally concerning are the women on the bottom rung of the middle class for whom single motherhood does, indeed, derail them. These are the women with a year or two of college headed to decent jobs: without the burden of motherhood, they would have moved up the socioeconomic ladder but now find themselves languishing, along with their children, in near poverty. According to a recent report in the New York Times, the proportion of these unwed mothers who have some college education but didn’t finish a college degree rose from 10% in 1990 to 30% today. This is a stunning finding.

In previous generations, the marriage and childbearing patterns of the middle class bore a closer resemblance to their wealthier, better-established cohorts than to the working class. But this pattern has flipped: middle-class problems today now seem more like the problems of people at the precarious bottom economic rung: fatherless children, economic insecurity, tough parenting trade-offs, high stress and limited options.

Unfortunately, people on all sides of the political spectrum fall into the morality trap when they talk about unwed motherhood, tying themselves in knots trying to parse the “real” impact of growing up in an atypical family. On the right, these conversations often rooted in religious convictions about marriage, sex and even abortion. On the left, policymakers are so quick to embrace all family structures that they refuse to acknowledge that some models really do work better than others in the long run. On both sides, there’s an emotional quality to the discourse that detracts from the financial realities.

We need to stop fighting about the ethics of out-of-wedlock births and instead focus on the economics behind them. When we do, we’ll discover that the real story isn’t that women aren’t getting married. It’s that they, and the men who impregnate them, are having children they can’t support. It may be good news that a majority of Americans (54%) think having a baby outside of marriage is morally acceptable, because for women under 30 today, the majority of births do take place outside of marriage. Whether it’s economically feasible is a whole different story.

A robust middle class is an essential feature of a thriving society, whether as part of the postwar boom in the U.S. or in 21st century China. We should worry about these single mothers; they are the canaries in the coal mine of a society that is economically unhinged.