How much they owe depends a lot on the college. Average debt loads at four-year public and nonprofit colleges ranged from $3,000 to more than $55,000.

The advocacy group's president Lauren Asher says a tough job market has made it harder for recent grads to pay down their loans.

"Young college graduates are having a harder time finding jobs than they might have expected, and those that do find jobs may be under-employed-working part-time when they'd rather be working full-time, or working in jobs that don't require a degree."

Still, Asher says college grads are better off than those without a degree. The unemployment rate was more than twice as high last year for those who only finished high school.

The report only includes data for colleges that volunteered to share it. Very few for-profit colleges -- where debt loads tend to be higher -- responded to the survey.