"They have a hand in your pocket," said Linda Sherry, director of national priorities for advocacy group Consumer Action.

(Once Social Security benefits are in your bank account, you have additional protections against third-party debt collectors looking to garnish funds, Sherry said.)

During fiscal 2016, the Treasury Offset Program made more than 4.5 million offsets from Social Security payments to consumers, including both checks and electronic funds transfers. Net collection: Nearly $713 million.

Debts cutting into your Social Security benefits

How much you stand to lose depends on the details of your debt and Social Security benefits. For example, under the Debt Collection Improvement Act of 1996, the maximum allowable collection on a federal student loan debt is the smallest of three options: 15 percent of the monthly Social Security benefit payment, the amount of the monthly benefit payment less $750, or the outstanding amount of the debt.

But even a small collection could have a big impact on your budget. Social Security benefits account for at least half of income for 48 percent of retired married couples and 71 percent of unmarried individuals, according to government statistics. For 1 in 5 couples and 43 percent of singles, Social Security is 90 percent of their income.

If you owe a federal debt and are worried about garnishment, reach out proactively to that agency, said Bruce McClary, a spokesman for the National Foundation for Credit Counseling.

"If you see that coming down the road, do everything you can to try and prevent it," he said.

You might be eligible for repayment options that take less of a bite, he said. Advance notice also gives you time to challenge the debt (say, if you think that federal tax bill is incorrect) or apply for a hardship exemption.

A 2016 Government Accountability Office report pointed out that some Social Security recipients may be eligible for a "total and permanent disability discharge" for federal student loans.