NEW YORK ( TheStreet) --Bank stocks have sold off with the broader market during the past week as the dreaded "fiscal cliff" draws closer, but one veteran investor believes the biggest risk is to the upside.

"Obviously if we solve the fiscal cliff what a lot of people, myself included, expect is now there's less uncertainty by business owners and there will be more borrowing and more investment in the economy," Tom Brown, head of hedge fund Second Curve Capital, told Bloomberg TV Monday morning.

The fiscal cliff refers to a series of automatic tax hikes and spending cuts set to kick in in 2013 if Congress can't reach a deal on the budget. Half of the cuts would come from defense spending, while the other half would come from a broad array of government programs. The impact on the banking industry would be indirect, Brown said.

"The only true impact it's going to have on the banking industry is how slow does the economy get. Does it get into a recession? How much more unemployment is there? So you're talking about more losses for the banking industry," Brown said.

Still, Brown believes confidence is already at rock-bottom.

"We see this through our bank positions. They talk to the small business owners and they just--between taxes, healthcare costs and this fiscal uncertainty they just are very unsure," he said.

The SPDR S&P Bank ETF ( KBE), which counted Bank of America ( BAC), Northern Trust ( NTRS) and Comerica ( CMA) as its top three holdings as of Dec. 28 lost 3.44% during the past five trading days headed into Monday. That compares to a 2.85% loss for the S&P 500. During all of 2012, however, the bank ETF had gained 18.76% as of an hour into trading on Monday, the final trading day of the year. The S&P, meanwhile, has added 11.67% over the same time period.

Brown has said Bank of America is his top pick for 2013, despite the fact that shares have already doubled in 2012. The bank's shares have not suffered as much as the broader index during the selloff, losing less than 1.5%.