Glenn W. Welling, founder of hedge fund Engaged Capital, wrote a Dec. 3 letter to Abercrombie's board calling for Jeffries' removal when his contract comes up for renewal on Feb. 1. Welling wrote that the best option would be to sell the company to private equity, citing reports put out by FBR & Co. analyst Susan Anderson and Randal Konik of Jefferies Group.

Yet, Engaged has only 400,000 shares worth about $14 million at Tuesday's closing price of $35.99 per share. That's not a large stake in a retailer with a $2.75 billion market cap.

The letter also admits that Jeffries is a major hurdle to such a deal.

According to a source familiar with the company, Jeffries has no intentions of leaving the company and still opposes a sale of the company, preferring to remain public.

But other industry watchers posit a scenario where Jeffries, at the vulnerable time of an employment contract renewal would rather take his change-of-control bonus than face a possible showdown with investors. Besides whatever stock option grants he has piled up in two decade-plus run heading up Abercrombie, Jeffries owns slightly more than 1 million shares, a 1.32% stake in his company, according to regulatory filings.

The company did not respond to requests for comment.

A list of the usual suspects among private equity firms that could be interested - and big enough to write a check - include TPG Capital, Kohlberg Kravis Roberts & Co. LP, Blackstone, Apax Partners, Leonard Green & Partners LP and Advent International, although, of those, the last two are the ones likely to have the most interest, industry sources said.

Still, another industry source said Jeffries continues to have the support not only of management, but more importantly, its board.

A recent brouhaha over Jeffries' comments that his brand does not cater to overweight people, among other uncomplimentary reports about his management style and lifestyle, are largely superficial, and have done little to undermine that support or his position at the company, this person said.

For the year ended Feb. 2, New Albany, Ohio-based Abercrombie had nearly $600 million in EBITDA, according to data provided by Bloomberg. For its current fiscal year that will end on Jan. 31, it is estimated to have about $405 million in EBITDA.

Meanwhile, as of Nov. 2, it had cash of nearly $260 million and total debt of only about $200 million.