“The banks really like jumbo mortgages,” said Guy Cecala, publisher of the trade publication Inside Mortgage Finance. “It’s a business they’ve become increasingly reliant on.” Last year, jumbo mortgages hit their highest level since 2006 by dollar volume, according to the publication’s estimates.

Predicting the tax law’s exact effect on the jumbo market is difficult, and most banks aren’t talking yet about what they expect. Complicating matters: Wealthy consumers could benefit from other parts of the tax bill. Also, the jumbo market has already slowed down this year, as higher interest rates have crimped refinancing activity.

Still, some analysts say the tax bill could further slow the pace of jumbo mortgages, though for now they expect only a modest single-digit percentage drop from the tax bill.

Since the financial crisis, banks have been pushing hard toward jumbo mortgages, or loans that are too big to be sold to Fannie Mae and Freddie Mac. In most parts of the country, a jumbo is any loan above $424,100. Banks generally keep the loans on their own books, as wealthy borrowers who use the products are less likely to default.

There are two main ways the tax law could hinder some of that activity. It would slash the size of loans that qualify for the mortgage-interest deduction from $1 million to $750,000. It also would curb the amount of property taxes homeowners can deduct from their tax bill. Mr. Cecala said this cap could make wealthy buyers reconsider buying a second or third home, since the property-tax deduction is one of the major benefits of doing so.

In “states like New York, New Jersey, California—that’s a big deal,” said KBW analyst Christopher McGratty. Current mortgages will get to keep the $1 million cap on mortgage-interest deductibility, but that could also slow down sales by persuading some high-end homeowners not to move.

The banks’ increasing production of jumbo mortgages has been notable since they have pulled back on other types of mortgages, tightening lending standards and making way for nonbank lenders to take on riskier customers. Wells Fargo & Co., JPMorgan Chase & Co. and Bank of America Corp. are the three biggest jumbo mortgage producers.

At banks with $10 billion or more in assets, which includes all midsize and large banks, 19% of all mortgage originations were over the typical jumbo limit in 2016, up from 9% in 2006, according to an analysis using software from ComplianceTech’s LendingPatterns.com.

Over the last three months, as the tax bill came into shape, First Republic shares declined 13.6%, compared with about an 11% pickup in the KBW Nasdaq Bank index.

“I don’t think we view it as [a change] that totally stops the business and drives things to a halt,” said the bank’s chief financial officer, Michael Roffler, at a conference in early November. The bank declined to comment last week.

High-cost mortgages dominate the housing market in areas like New York City and California. In Manhattan, for example, 84% of purchase mortgages this year were for more than $500,000, according to housing-research firm Attom Data Solutions. In San Francisco, where First Republic is based, that proportion is 87%.

The mortgage-interest deduction has long been a staple of American homeownership, and the psychological effect of losing part of it can have just as much an effect as the financial. Already, many borrowers are staying put longer in their current homes, partly because they don’t want to lose the superlow interest rates they pay now that the Federal Reserve is raising short-term rates.

“We’re already living in a rate-lock world,” said Ben Graboske, an executive vice president at mortgage data and technology firm Black Knight Inc. “Now you’re going to add this.”

Black Knight calculates that the new interest-deductibility cap would cost a homeowner with a mortgage over $750,000 an average of $2,500 to $4,000 a year depending on their tax bracket. That would equal a 6% increase in monthly principal and interest payments for moving to a new home.