Shares of J.P. Morgan rallied 1.5 percent in after-hours trading following news of Buffett's new $4 billion stake, which was reported in the company's 13-F filing. The filing documents the company's holdings as of end of September. Berkshire slightly decreased its stake in Wells Fargo.

Buffett and J.P. Morgan CEO Jamie Dimon admire each other. Buffett has said he reads Dimon's annual shareholder letter for insights into the banking industry. Dimon is the longest-tenured of CEOs leading a major U.S. bank. The pair, along with Jeff Bezos of Amazon, in January announced a partnership to cut health-care costs and improve services for their U.S. employees.

Todd Combs, one of Buffett's investing lieutenants at Berkshire who first got the company involved in Apple, sits on the board of directors of J.P. Morgan.