Cruz has a point. The drop in the percent of working Americans can't all be explained by retiring Baby Boomers. Even the labor force participation rate for people 25 to 54 years old -- the working age population -- has declined for years in the U.S.

Why are working-age people leaving the job market? Some point to the absence of paid parental leave in the United States. American women sometimes have to choose between taking care of a newborn or going back to work.

In 1990, America ranked sixth among 22 developed countries for female workforce participation. In 2010, it ranked 17th. Part of the reason why other countries have surpassed the U.S. is that they've adopted "family-friendly" policies, according to a study by Cornell professors Francine Blau and Lawrence Kahn.

Canada has more women working than America: For a quick comparison, let's look north. Canada offers women 17 weeks of paid maternity leave. On top of that, there's an additional 35 weeks for "parental leave" which a father or mother can take. That's a year of paid leave for having a child.

Working-age American women and Canadian women had the same participation rate in 1997. Now, Canada's participation rate for women is 82% while it's only 74% for American women.

Put another way: if working-age American women had the same participation rate as Canadian women, there would be 5.4 million more American women at work, according to CNNMoney's analysis of Labor Department and Federal Reserve data.

The gap between U.S. and Canadian women has grown considerably since 2001 -- the year Canada extended the amount of paid parental leave it gives.

Future for paid leave laws: Perez argues that the U.S. participation rate would go up if women had paid maternity leave. Earlier this year, the Obama administration announced an initiative to promote paid leave reforms.

Republicans have staunchly opposed any type of paid leave in the past. Republican senators and presidential candidates Ted Cruz, Rand Paul and Marco Rubio did not respond to a request for comment. Paid leave could become a divisive issue between candidates as they try to win the female vote in the 2016 election.

It's important to note that paid parental leave has some downsides. The Cornell study found that parental leave can hinder a women's ability to climb the ladder to more executive positions and can lead to more part-time positions.

Still, the lack of government action on paid parental leave irks, among others, John Oliver, who lambasted Minnesota legislators in his Mother's Day tirade.

"You can't go on and on about how much you love mothers, and then fail to support legislation that makes life easier for them," Oliver concluded.