That was made clear again today as the much reported discussion between J.P. Morgan and the Department of Justice drew flak from a third set of constituents: bondholders.

As WSJ reported,the Association of Mortgage Investors, a group of mortgage-backed security bondholders, sent Attorney General Eric Holdera letter urginghim to consider the impacts of any settlement could have on bondholders.

“The retirement security of the innocent parties whose money we manage could be harmed if these settlements follow recent precedents, in which major bank-servicers were allowed to fulfill their settlement obligations with other parties’ investments in mortgage-backed securities,” wrote Chris Katopis, executive director of AMI. “The members of AMI, as fiduciaries for our clients who were never involved in any of the alleged misconduct, are deeply concerned and oppose any such legal settlements.”

This gets to heart of the problem many have with Washington’s efforts to provide underwater homeowners relief: the impact helping struggling borrowers can have on the investment world.

Many in Washington wanted to force the banks to forgive some of the mortgage debt for homeowners who owe more than their home is worth, an attempt to keep them in their homes and reduce a plight of foreclosures and alleged foreclosure abuses.

One scenario under discussion last month between J.P. Morgan and DOJ would have required the bank to pay $11 billion, which would be split between a $7 billion cash penalty and $4 billion in consumer relief.

But so-called principal forgiveness can create a dangerous precedent. Banks have argued it is “a moral hazard,” which could lead some borrowers to default on purpose in an effort to get out of paying.

For mortgage-bond holders, who own pieces of thousands of mortgages, the forgiveness could hit their investments.

AMI adds to the debate that its clients, which include both big Wall Street names and pension funds for teachers and unions, will suffer, bringing the pain right back to “Main Street” from the banks.

“These are not unforeseen consequences, but rather an obvious scheme by Too-Big-To-Fail (TBTF) banks to evade the consequences for their misconduct by further abusing their duties to investors,” AMI wrote. “This affects our clients, and in turn, the general public, or ‘Main Street’ America.”