Chegg executives celebrate the company’s IPO in November 2013 at the New York Stock Exchange.

Chegg Inc.

Tech startups eager to land sky-high valuations from investors might want to heed the cautionary tale of Chegg Inc., the textbook rental service whose stock has languished since its IPO in 2013.

In a candid interview, an early investor in Chegg revealed how the company gunned for the highest possible valuation in several funding rounds ahead of its public offering. Chegg in exchange granted venture capitalists a favorable term called a “ratchet” that guaranteed the share price in the IPO would be higher than what they paid.

The move backfired. When Chegg went public, it was motivated to set an IPO price that met the terms of the covenant, or Chegg would have to pay the difference in shares to the early investors. The stock plummeted on the first day of trading and hasn’t recovered.

Chegg is one of a number of companies, along with Box Inc. and Kayak Software Corp., that have been forced to pay hefty penalties in recent years because they accepted covenants in their funding rounds that dictate an IPO price. These terms can also turn insiders against each other and lead management to go public too soon at too high a valuation.

As a private company, Chegg “went through three years of suffering and struggling for no reason whatsoever,” Oren Zeev, the early investor in Chegg, said in an onstage interview at a tech conference earlier this year.

Payments company Square Inc. disclosed in its IPO filing last week that a $150 million funding round valuating it at $6 billion last fall came with potentially onerous restrictions. Investors including J.P. Morgan Chase & Co. and Rizvi Traverse are entitled to receive additional shares if the IPO price isn’t 20% above what they paid.

Spokespeople for Square and Chegg declined to comment.

It isn’t clear exactly how many companies agree to such conditions because investors and entrepreneurs rarely discuss publicly the terms of funding rounds.

An analysis in March by law firm Fenwick & West found that 30% of private companies valued at $1 billion or more have agreed to give investors protection against a “down round IPO,” meaning the IPO price has to be above what investors paid for shares in a private funding round. There are at least 124 companies worldwide that are valued at $1 billion or more by venture-capital firms.

The IPO “ratchets” could become problematic in coming years, as those companies that are not able to price their shares high enough in the public market will trigger the provisions in their IPOs.

While Chegg’s story is fading from view, its difficulties navigating this period underscore the problems that can arise.

Mr. Zeev, an early angel investor in Chegg, said the company first decided to offer ratchets to Insight Venture Partners as part of a $57 million Series D round in 2009.

At the time, Chegg’s revenue from selling textbooks was growing at a torrid pace, rising by about sevenfold in 2009 from a year earlier. Chegg received numerous offers from venture-capital firms for financing, with no special protection. But Chegg took the ratchet deal with Insight because the New York venture investor agreed to value the startup at $600 million – about 20% higher than other offers, according to Mr. Zeev, who was on Chegg’s board at the time.

“We tried to maximize the valuation,” Mr. Zeev said. “But there was a catch.”

The catch was that Chegg guaranteed Insight would double its money in the deal, what is known as a “2x” ratchet. If Chegg failed to go public or sell at twice the price per share, or $26.30, the company would be forced to issue Insight additional shares to cover the difference, effectively diluting the value of its shares for all other investors.

This type of ratchet can create divisions among investors if things turn south, as they did at Chegg. By 2010, the company was struggling with a market shift from print textbook rentals, its core business, to digital education tools, a new line of business that required new investments and new competencies.

“While it turned out that the top line was great, the fundamentals of the business, or the assumptions we were making about the business, were a stretch,” Mr. Zeev said. “It was far less clear it was a great business.”

In 2010, Chegg hired a new CEO, former Yahoo Inc. executive Dan Rosensweig. He raised two additional rounds of equity, both with increasingly onerous new ratchet terms, as well as two rounds of debt financing. By 2013, it still wasn’t clear that the digital education business was growing fast enough to eventually offset declines in textbook sales.

The ratchets created a division between Insight and other later-stage investors, who were incentivized for the company to hold an IPO, ensuring they would get a return on their investment, Mr. Zeev said. Earlier-stage investors knew that they would most likely get diluted in an IPO because Chegg would not be able to sell its shares at a higher price than its ratchets guaranteed.

An Insight spokesman declined to comment.

The two investor groups “had very different objectives and incentives and [the ratchet] created difficulties in a situation that was already very difficult to begin with,” Mr. Zeev said. “It introduced an inherent conflict of interest.”

In November 2013, Chegg held an IPO. Despite setting a price range of $9.50 to $11.50 ahead of the offering, the bankers priced the deal at $12.50.

That price allowed Chegg to satisfy one of its three ratchets, but forced it to pay the penalty of the other two. The company distributed 11.7 million more shares, or about 14% of the shares outstanding after the IPO, to two investor groups.

That share price also greatly overestimated market demand. Chegg shares fell 23% to $9.68 in their opening trading session, the biggest first-day fall in 2013 for a U.S.-listed IPO, according to research firm Dealogic.

In nearly two years since the IPO, Chegg has never come close to regaining its IPO price. The stock dipped as low as $4.82 in May 2014, and currently trades under $8 — well under what investors paid in those final three rounds.