In an article Monday, Sorkin outlined Buffett's criticism in the letter, including investment bankers' inclination to manipulate prices and deals to generate profits for themselves.

"The Street's denizens are always ready to suspend disbelief when dubious maneuvers are used to manufacture rising per-share earnings, particularly if these acrobatics produce mergers that generate huge fees for investment bankers," Buffett wrote in his letter.

Despite his negative views on these "money-shufflers," Buffett has done plenty of profitable business with banks in decades of investing. Sorkin noted that he's also spoken highly of several Wall Street CEOs and bankers, including Blankfein and JP Morgan's Jamie Dimon.

When asked about his comments, Buffett told the New York Times that his comments were meant to educate shareholders, but weren't personal.