Big U.S. banks have slowly withdrawn from the mortgage market as they face scores of new federal rules.

Nonbank lenders dominate the origination of mortgages insured by the Federal Housing Administration and by the Veterans Administration.

At $26 trillion, the American housing market is the largest asset class in the world, bigger than the U.S. stock market, according to The Economist. And the banking industry’s $13 trillion in total loans is the engine that drives the U.S. housing market, according to Attom Data Solutions.

But since Congress passed the Dodd-Frank Wall Street Reform and Consumer Protection Act in 2010, big U.S. banks have slowly withdrawn from the mortgage market as they face scores of new federal rules, billions of fines for misconduct and more regulation (the Dodd-Frank Act is 2,319 pages long).

Increasingly, nonbank mortgage lenders — sometimes called “shadow banks” — have overtaken U.S. commercial banks to grab a record slice of the government and conforming loan markets, after the heightened regulatory environment and billions of dollars in fines forced retail lenders out of the U.S. home loan market.

Wells Fargo is the biggest lender by far, doing nearly twice the volume of no. 2 nonbank Quicken Loans. Wells alone issued $43 billion in residential mortgage loans in the first quarter of 2016, according to The Wall Street Journal.

But Wells Fargo’s market share is shrinking, largely due to the rise of nonbank lenders like Quicken Loans, Caliber, loanDepot.com and others.

Wells is not the only retail bank that has been pulling back from home loans.

Bank of America, Chase, Citibank and U.S. Bank have downsized their market share in a market that is also shrinking.

Heightened scrutiny of U.S. commercial banks, tighter capital requirements and fear of heavy government-imposed penalties is paving the way for lightly regulated nonbanks to gain a bigger toehold in home loan market.

Who’s filling the void?

Nonbanks grab market share as banks retreat

In California, some of the largest nonbank lenders include PennyMac, AmeriHome Mortgage and Stearns.

All three are headquartered in Southern California, the epicenter of last decade’s subprime mortgage lending industry. And all three companies are run by executives who formerly worked at the once-giant Countrywide Financial, the now defunct subprime lender founded by Angelo Mozilo (Bank of America bought Countrywide for $4 billion in July 2008).

PennyMac, a fast-growing nonbank lender, is run by Stanford Kurland, a former Countrywide Home Loans executive and IndyMac director. Stearns, a Santa Ana, California-based nonbank lender, is run by Brian Hale, a former Countrywide division president. And Joshua Adler, who is AmeriHome’s managing director of secondary marketing, held similar roles at Countrywide and Bank of America.

Unlike retail banks, nonbank lenders do not take federally insured deposits from consumers to make loans.

Instead, they typically borrow from Wall Street hedge funds, private investors or banks to make loans, then quickly sell these mortgages to Fannie Mae and Freddie Mac and other buyers, so they can repay their loans and start the process over again.

Many nonbank-originated loans are being made to riskier borrowers with low FICO scores, and who make down payments as low as 3 percent of the purchase price of a home.

Nonbank lenders dominate riskier loans

Depository banks still dominate home lending, but nonbank market share is at an all-time high, according to National Mortgage News.

Nonbank lenders dominate the origination of mortgages insured by the Federal Housing Administration (FHA) and by the Veterans Administration (VA), the riskier corner of housing lending due to no down payment or low down payment loans and poor-credit buyers.

Nonbank lenders like Quicken Loans, which originated 18,753 home loans in the third quarter of 2016, are filling the gap, according to Attom Data Solutions.

Overall, U.S. financial institutions originated nearly 1.9 million loans on residential properties (1 to 4 units) in the second quarter of 2016, up 26 percentage points from a two-year low in the previous quarter, but down 4 percentage points from a year ago, according to Attom Data Solutions.

Purchase originations increased 1 percentage point from a year ago — the eighth consecutive quarter with an annual increase — and Home Equity Line of Credit (HELOC) originations increased 5 percentage points from a year ago — the 17th consecutive quarter with an annual increase.

Mortgage fraud suits continue

Guy Cecala, publisher and CEO of Inside Mortgage Finance, one of the leading mortgage trade publications, said one reason why nonbanks are gaining market share is because retail lenders keep getting sued by the federal government.

Big depository institutions don’t want to make FHA-insured loans because they frequently get prosecuted by the federal government under the False Claims Act, said Cecala.

“For a variety of reasons, banks want to reduce their footprint in the mortgage market,” said Cecala, adding that government compliance costs have risen too. “It’s a direct result of the settlements with the federal government.”

He said nonbank market share grew from 23.4 percent in 2008 to 48.3 percent in 2016.

Cecala said that over the last few years, the Justice Department has sued scores of lenders for violating the False Claims Act on government-insured loans. Since 2010, banks in the United States have paid over $56 billion to settle Justice Department fraud suits, according to The New York Times.