Bank stocks outperformed the broader market last year, but that trend may not last in 2013.

"I think we're set for a pause on bank stocks," said Andrew Marquardt, head of bank research at Evercore Partners.

Investors will hear from leaders in the banking industry this week, when Bank of America(BAC), Citigroup(C), JPMorgan(JPM), Goldman Sachs(GS) and Morgan Stanle(MS)y report quarterly results.

Wells Fargo(WFC) got the ball rolling Friday, reporting strong earnings on modest revenue growth. But shares fell as investors focused on the underlying weakness, including a slowdown in mortgage refinancing.

Overall, analysts expect mixed results from the rest of the Big Six, with relatively cautious guidance for the year.

Earnings could also be marred by various charges as banks try to enter the new year with a clean slate, said Marty Mosby, a bank analyst at Guggenheim Securities.

"We're expecting a lot of noise in the numbers," he said.

Still, banks are expected to be the bright spot in an otherwise lackluster quarter for corporate earnings.

Financial companies in the S&P 500 are expected to report earnings growth of 10%, according to S&P Capital IQ, whereas overall earnings are expected to rise only 3.2%.

Are banks still 'too big to fail'?

But much of the improvement may already be baked into bank stocks, which managed to end a volatile year with a solid fourth-quarter rally.

Shares of large-cap banks rallied between 25% and 50% last year, with Bank of America more than doubling in price.

And the KBW Bank Index(BKX) rose 30% in 2012,compared with a 13% gain for the S&P 500.

The rally came as lending rebounded and credit quality improved, driven by a wave of mortgage refinancing last year. Profits were also bolstered by the release of funds previously used for loan loss provisions.

Those trends appear to have run their course, leaving banks with few options to boost earnings as revenue growth remains subdued, said Jim Sinegal, bank analyst at Morningstar.

"I wouldn't be surprised if the next quarter or two are pretty quiet," said Sinegal. "As far as earnings go, there are no real areas for huge improvements."

Despite the challenges, analysts say bank stocks could get a boost in March, when the Federal Reserve is scheduled to releases results from last year's so-called stress tests. This could pave the way for banks to repurchase shares and possibly boost dividend payments.

The most likely banks to return capital to shareholders are US Bancorp(USB), Wells Fargo and JPMorgan, according to a report from Credit Suisse. Citi and Bank of America could also buy back shares, but at a more modest pace, said Credit Suisse.

The main headwind facing the banking industry is low interest rates, which have made it difficult for banks to make money off traditional lending.

Banks have been squeezed by low yields on interest-bearing assets such as U.S. Treasuries compared with the rates banks charge to lend money.

Some analysts expect rates to gradually move higher this year as the economy improves and investors move out of bonds and into stocks. But the Federal Reserve has said it will continue to buy longer-term Treasuries as part of its quantitative easing program, a policy designed to push rates lower.

So far, banks have been able to offset low interest rates by managing expenses and cutting costs. Morgan Stanley, for example, announced plans last week to cut 1,600 jobs. Analysts say more layoffs could be in the works as banks focus on maximizing efficiency.

Another way banks are managing expenses is by settling lawsuits. Just last week, Federal regulators announced an $8.5 billion settlement involving 10 banks over alleged foreclosure abuses. And Bank of America reached a $10.3 billion settlement with Fannie Mae to deal with questionable home loans it sold to the government-backed mortgage financer during the housing bubble.

"Banks are putting more and more of their problems behind them," said Sinegal. "Every piece of litigation that is resolved is a positive."