Rhino started as a record shop on Westwood Boulevard, Los Angeles, during 1973, run by Richard Foos. It became a record distributor five years later thanks to the effort of then-store manager Harold Bronson. Their early releases were mostly novelty recordings (with their first single being Wild Man Fischer's "Go To Rhino Records") and some punk rock singles; the difficulties involved in getting airplay and distribution for such material eventually caused Foos and Bronson to take the label in other directions. One of their earliest reissues was achieved by acquiring the rights to the White Whale Records label. The company's mascot character is an Elvis Presley rhinoceros wearing a black leather jacket. By the mid-1980s most of their releases were reissues of previously released recordings licensed from other companies. Superior sound quality (Audio mastering of the original tapes (where possible) was done under the direction of Bill Inglot) and creative packaging made Rhino one of the most respected reissue record labels, receiving rave reviews from music collectors, fans, and historians.[citation needed] Rhino was quick to get into the compact disc market, releasing dozens of oldies CDs at the dawn of the CD age in 1984. Their retrospective compact disc releases are often remastered to restore or improve upon the original analog release's audio quality.

The company also continued to produce new music, with releases on subsidiary labels such as RNA (Rhino New Artists) and Forward, as well as the main Rhino label. However, the company's entertainers tended to generate more critical acclaim than public interest. One of Rhino's early artists was The Twisters whose Los Angeles popularity far exceeded their album sales. For the most part, sales totals in the low five figures or less were routine for Rhino-produced albums,[citation needed] and the less costly, less risky reissue business remained the company’s primary revenue stream. One exception was the late-1986 success "At This Moment" by Billy Vera & the Beaters, a 1981 song that unexpectedly made it to the top of the U.S. Billboard charts after being featured in a 1986 episode of the hit NBC series Family Ties.

In 1985,[1] Rhino signed a six-year distribution agreement with Capitol Records. During 1989 Rhino and Capitol’s parent EMI made a deal to jointly acquire Roulette Records; Rhino received the US rights to Roulette's catalog, excluding jazz. When the distribution deal with Capitol ended in 1992, Rhino signed a new distribution deal with Atlantic Records, and in turn Time Warner bought a 50% stake in the record company. In 1998, Time Warner bought the other half of Rhino; thus the company became a wholly owned unit of Time Warner. The Rhino Records retail store, which was not part of the sale, closed in 2005.[2] In 1999, Rhino started the Rhino Handmade division of limited edition releases available primarily from their website. All of these Handmade reissues are limited to about 3000 copies or less and once sold out are not re-pressed. (Since Rhino redid their website in early 2013, the Handmade division appears to have been discontinued.)

In 2003, co-founders and longtime executives Richard Foos and Harold Bronson left Rhino, reportedly due to frustration at being unable to release compilation albums in an increasingly competitive market. In fact, Time Warner's final vesting of its 100% ownership of the label, and its subsequent 'reorganization' of label staff, which did not stop at the former owners, were the major factors in their exits. Soon after, Foos inaugurated a new label, Shout! Factory, which began releasing dozens of CDs and videos mirroring the original early-1990s Rhino philosophy.

In 2004, Time Warner spun off its music divisions and today Rhino is part of the newly organized Warner Music Group.

In addition to dealing with archive material, the label also manages the US distribution or worldwide production of compilations for more recent Warner acts, including still-active artists such as Enya, New Order, and Chicago.