Only Mortgage Lender Growing Sees End Game as Loan 'Middleman'

Stan Middleman’s Freedom Mortgage Corp. did something last quarter that the rest of the nation’s biggest housing lenders couldn’t: increase its loan volume.

As all of the other top 15 companies posted lower originations than in the prior period amid a 23 percent industrywide drop, Freedom’s lending grew 19 percent, according to data compiled by newsletter Inside Mortgage Finance. The Mount Laurel, New Jersey-based firm’s $3.8 billion of mortgages lifted it seven spots from its 2013 ranking, to No. 12.

Freedom is boosting its mortgage volumes partly by buying loans from other lenders, an effort Middleman says is really about expanding its business of collecting payments and handling other work over the life of the debt. That function, known as servicing, provides a steady stream of income that’s attracting interest from some of Wall Street’s biggest investors because the contracts pay relatively high yields that could increase when interest rates rise and limit refinancing.

“The core of mortgage banking is the mortgage servicing right, and all of the other activities revolve around it,” Middleman, 60, said in an interview last week at an industry conference in New York.

His privately held firm now services almost $50 billion of loans, Middleman said. It ranked No. 22 in the business on March 31, according to Inside Mortgage Finance. Cherry Hill Mortgage Investment Corp., where Middleman serves as chairman and which raised $130 million in an initial public offering in October, co-invests in some of the servicing agreements.

The contracts are created as mortgages get packaged into bonds, allotting typically at least 0.25 percentage point annually of loan amounts from homeowners’ payments for servicers.

Still Value

While the values of servicing rights on new loans more than doubled from a few years ago amid a flurry of new competition, they still offer fair value, yielding 8 percent to 9 percent, and borrowed money can boost the returns, Middleman said. By helping originate the mortgages, he gets the asset more cheaply than by buying it, he said.

“We went from an MSR that was untradeable five years ago, and now we have people competing hard and vigorously in a very robust trading environment,” Middleman said. “This is the strongest MSR trading market we’ve seen in a long time.”

‘The Middleman’

Freedom, after originating $15 billion last year, is seeking to push the annualized pace to about $25 billion by the third quarter, said Middleman, who founded the firm in 1990 after working in investment sales. The Mortgage Bankers Association is forecasting a plunge in refinancing will reduce industry volumes by 40 percent this year.

Middleman referred to himself as “The Middleman” in introducing Fox News host Bill O’Reilly at the MBA event last week, an appearance that his company sponsored after paying for entertainment including a concert by the band America at the group’s annual conference in October.

A large source of Freedom’s volume gains this year have come from its correspondent business, in which it buys closed loans from other lenders. Those originations have grown to $1 billion a month from $600 million in October, he said.

Subsidizing Cost

The company uses other parts of its business to “subsidize” how much it’s paying, offering more than it might if it didn’t take into account revenue from servicing and other units, he said. Along with the value assigned to servicing rights, the prices of new mortgages are determined partly by what securities they back can be sold for, a function of how a loan’s rate compares with current bond yields.

Freedom, which is also the No. 2 lender through brokers and makes mortgages directly to servicing customers through call centers, is also looking to expand by adding retail branches through acquisitions. It bought BluFi Lending to gain offices in California and Nevada in May and has “a couple” of pending deals it should complete within several months, Middleman said.

‘Great Danger’

The “great danger” in the surge in demand for mortgage- servicing rights assets is if interest rates fall, causing borrowers to refinance, he said. The only good defense is to be able to originate enough loans when rates decline, he said.

“The last time we saw a huge run-up in the value of mortgage servicing rights, we saw a large crash in people who invested in servicing only,” he said.

In the early 1990s the dynamic “deeply affected the soundness” of numerous savings-and-loans, he said. More recently, National Australia Bank Ltd., which bought Florida-based mortgage servicer HomeSide Lending Inc. in 1998, later suffered $2.2 billion of writedowns on the unit as interest rates fell and then sold it to Washington Mutual Inc. in 2002.

“You can buy these and think interest rates will never go lower, I can’t tell you how many times I’ve heard that,” Middleman said. “But some people are going to make that call and some of them are going to get rich and some of them are going to go broke.”