Warren Buffett, chairman and CEO of Berkshire Hathaway, Inc., arrives at the Allen & Company Sun Valley Conference in Sun Valley, Idaho, in 2012. Mr. Buffett called the company's $24 billion gain 'subpar' in his annual shareholders letter, because it should outperform the S&P 500 index over time.

It's the ninth time in 48 years this has happened. Buffett notes that the S&P has outpaced Berkshire over the past four years and if the market continues to gain this year the benchmark stock index could have its first five-year win ever.

"When the partnership I ran took control of Berkshire in 1965, I could never have dreamed that a year in which we had a gain of $24.1 billion would be subpar ... But subpar it was."

Buffett notes that while he believes Berkshire's intrinsic value will probably beat the S&P by a "small margin" over time, the company's relative performance is better when the market is down or unchanged. "In years when the market is particularly strong, except us to fall short."

Even with the bad news, Buffett warned that he and partner Charlie Munger "will not change yardsticks. "It's our job to increase intrinsic business value — for which we use book value as a significantly understated proxy — at a faster rate than the market gains of the S&P." If they are successful, Berkshire's share price will beat the S&P "over time."

Berkshire's boss did have praise for his two new investment managers, Todd Combs and Ted Weschler, writing that "we hit the jackpot with these two." Buffett said they both outperformed the S&P by "double-digit margins" in 2012. Using a very small font, Buffett added, "They left me in the dust as well."

Buffett also expressed disappointment that Berkshire didn't make a major acquisition last year. "I pursued a couple of elephants, but came up empty-handed."

Echoing what he told us after Berkshire announced two weeks ago that it's teaming up with 3G Capital for a $23 billion acquisition of H.J. Heinz, Buffett said even though that deal accounts for much of what Berkshire earned last year, "we still have plenty of cash and are generating more at a good clip."

Buffett criticized "hand-wringing" from CEOs who "cried 'uncertainty' when faced with capital-allocation decisions." Berkshire, he said, spent a record $9.8 billion on plants and equipment last year. "We will keep our foot to the floor and will almost certainly set still another record for capital expenditures in 2013. Opportunities abound in America."

Buffett still sees an opportunity in local newspapers. Berkshire has spent $344 million to buy 28 dailies, and Buffett expected more purchases of "papers of the type we like" at "appropriate prices."

Even though he still believes overall newspaper industry profits are "certain" to drop, Buffett likes smaller community-oriented newspapers that provide local news and information that can't be obtained elsewhere. "Wherever there is a pervasive sense of community, a paper that serves the special informational needs of that community will remain indispensable to a significant portion of its residents."

Buffett isn't a fan of giving that information away for free on the Internet, pointing out that an Arkansas paper has retained its circulation "far better than any other large paper in the country" after it was an early adopter of a pay model.For its fourth quarter, Berkshire's operating earnings per share came in at $1704, below the consensus estimate of $1755 from the few analysts who follow the firm. (Read More: We Want Your Questions for Warren Buffett)

The company's net earnings were $4.55 billion in the quarter, up from $3.05 billion in last year's Q4. That includes "paper" gains for derivatives contracts Berkshire has sold that provide insurance against losses for some corporate bonds and stock indexes.

The bond-related contracts will expire in the next year. Buffett expects they will generate a $1 billion pre-tax profit. After unwinding about 10 percent of its exposure in 2010 at a profit of $222 million, the remaining stock index contracts will expire between 2018 and 2026.