To get a sense of how difficult the process of overhaulingFannie Mae and Freddie Mac will be, witness the latest kerfuffle over whether the companies’ regulator should drop the maximum loan limits.

In a letter released Thursday, some 66 members of the House of Representatives—59 Democrats and 7 Republicans—called on the Federal Housing Finance Agency to drop previously announced plans to reduce the limits.

The agency said in August it was considering a reduction in the loan limits, which are set at $417,000 for most of the U.S. but rise to as high as $625,500 in high-cost housing markets such as Los Angeles and New York. Officials had initially said any decline would take effect by Jan. 1, though discussions in recent weeks have suggested that timetable could be delayed, according to people familiar with the matter.

The letter from lawmakers suggested that Edward DeMarco, the acting director, didn’t have the authority to unilaterally drop loan limits, and it used Mr. DeMarco’s prior testimony before Congress to argue their case. “I don’t intend to act unilaterally in lowering the loan limit because the Congress of the United States has been so actively and repeatedly involved in adjusting the conforming loan limit,” he said during a 2011 hearing.

The move would be “inconsistent with my responsibilities as conservator,” he added. “I really and truly believe that the Congress of the United States is the body that should make the determinations about the future path of the loan limit if it is going to be something other than what current law provides.”

Lawmakers responded thusly in their letter: “We could not agree more.”

The FHFA first announced the policy change was under consideration on August 6, after President Barack Obama’s speech on the housing market. A companion memo from the White House called on the FHFA to “closely examine” using existing authority “to reduce loan limits further consistent with the pace of the recovery [and] market developments.”

While most lawmakers have said they believe Fannie and Freddie need to be replaced, Congress has been slow to take action to press for an overhaul, though discussions have heated up in recent months. Absent clear direction from Congress and the White House, the FHFA has kept the companies in a holding pattern.

Along with raising fees that Fannie and Freddie charge to lenders, reducing loan limits are a major lever to reduce the firms’ large presence in the mortgage market. Large banks and private firms that have stepped up issuance of mortgage-backed securities without government backing have said that modest declines in the loan limits would help spur more private lending, though some have conceded borrowers without 20% down payments or pristine credit scores could face fewer options for getting a mortgage.

Earlier this week, major real-estate industry trade groups asked the FHFA to delay any scheduled declines in loan limits.

A spokeswoman for the FHFA didn’t respond to a request for comment on Thursday.