He's not the only one questioning whether wealthy universities with big endowments are charging their students way too much money. A congressional committee is currently looking into the issue, asking the richest colleges to explain how much of their endowments are actually spent on students, in a way that lowers their tuition cost.

One lawmaker has proposed requiring colleges with billion-dollar endowments to spend a specific portion directly on tuition relief for low- and middle-income students. If the school didn't spend enough, they would face losing their tax-exempt status, which allows a college to skip paying taxes on endowment investment gains.

Harvard's endowment, the biggest in the country, stands at nearly $36 billion. Besides Yale, Stanford, and Princeton, no other college comes close.

About 90 other colleges have endowments valued at more than $1 billion, according to the National Association of College and University Business Officers (NACUBO). The Ivy Leagues are near the top, but so are schools like the University of Notre Dame, the University of Michigan, and Amherst College, a small liberal arts school in Massachusetts.

Meanwhile private colleges, which are the most likely to have big endowments, charge an average sticker price of $45,370 a year for tuition, fees, room and board. That's 80% of median family income.

It seems reasonable to ask why, with all that money, these colleges need to charge so much for tuition. But universities say it would be difficult to spend more of their endowment each year, and experts argue that requiring the richest colleges to do so would do little to address the overall college affordability crisis.

"Very few schools, who educate a tiny fraction of students, could do more. But most schools don't have the resources," said Sandy Baum, a senior fellow at the Urban Institute and expert on higher education finance.

Endowments are not bottomless piles of cash

The average school spends 4.6% of its endowment each year, according to NACUBO. The rates stay pretty consistent from year to year because colleges typically set the withdrawal rate and stick with it, no matter how good or poorly investments performed in the past year.

Colleges typically set that rate between 4% and 5%, said Laura MacDonald, the founder of the fundraising consultancy firm Benefactor Group.

And schools say that's the right amount if they want to keep their endowments growing, which they're obligated to do.

That's because donations made to endowments are meant to be a gift that keeps on giving, not just a gift for any single year. So the colleges need to make sure it keeps growing.

"It's not about hoarding or conquest. It's about drawing down the right amount that meets the needs of today without sacrificing the ability to meet the needs of the future," MacDonald said.

Endowments are made up of thousands of different investment accounts, many of which can be restricted. That means the donor specified what the money could be used for. Sometimes it's scholarships. Sometimes it's the library, or maybe it's the football team.

Here's an example: A donor gives money to an endowment to fund scholarships for 100 low-income students. The fund's manager is obligated to invest that money in a way that it can fund 100 scholarships this year, and each year going forward.

If the donor wanted it to be a one-time gift to build a new student union building, they would do just that -- make a one-time gift rather than donating to the endowment.

Still, some want wealthy colleges to actively raise money that allows them to offer more scholarships and grants.

"Institutions are not passive in raising funds. If they really wanted to, they could ask for donations for financial aid specifically," said Andrew Nichols at The Education Trust.

Berea College, a small school in Kentucky, does just that and is able to offer free tuition to all of its 1,600 students.

Wealthy colleges have already cut tuition

Nobody wants to see families spend more than half of their annual income to send their child to college.

But it turns out, the wealthiest colleges are already doing a pretty good job making sure that doesn't happen, especially for students from the poorest families. It's unlikely they have to pay anything for their child to go to many of the Ivy League colleges.

At Harvard, for example, any family earning less than $65,000 pays nothing. And those making between $65,000 and $150,000 won't have to pay more than 10% of their income.

Even if a college doesn't make such a grand promise, many offer scholarships and grants so that the actual net price families pay is substantially lower than the sticker price.

At private colleges, the average family paid $26,080 for tuition, fees, and room and board -- compared to the average sticker price of $45,370, according to The College Board. Still, that's about 46% of median family income.