Cliff House has had five major incarnations since its beginnings in 1858. That year, Samuel Brannan, a prosperous ex-Mormon elder from Maine, bought for $1,500 the lumber salvaged from a ship that foundered on the basalt cliffs below. With this material he built the first Cliff House.

The second Cliff House was built in 1863, and leased to Captain Junius G. Foster.[1][2] It was a long trek from the city and hosted mostly horseback riders, small game hunters or picnickers on day outings. With the opening of the Point Lobos toll road a year later, the Cliff House became successful with the Carriage trade for Sunday travel. Later the builders of the toll road constructed a two-mile speedway beside it where well-to-do San Franciscans raced their horses along the way. On weekends, there was little room at the Cliff House hitching racks for tethering the horses for the thousands of rigs. Soon, omnibus, railways and streetcar lines made it to near Lone Mountain where passengers transferred to stagecoach lines to the beach. The growth of Golden Gate Park attracted beach travellers, in search of meals and a look at the sea lions sunning themselves on Seal Rocks just off the cliffs, to visit the area.

In 1877, the toll road, now Geary Street, was purchased by the city for around $25,000.

In 1883, after a few years of downturn, the Cliff House was bought by Adolph Sutro, who had solved the problems of ventilating and draining the mines of the Comstock Lode and become a multimillionaire. After a few years of quiet management by J.M. Wilkens, the Cliff House was severely damaged by a dynamite explosion when the schooner, Parallel, ran aground on January 16, 1887.[3] The blast was heard a hundred miles away and demolished the entire north wing of the tavern. The building was repaired, but was later completely destroyed by fire on Christmas night 1894 due to a defective flue.[4] Wilkens was unable to save the guest register, which included the signatures of three Presidents and dozens of illustrious world-famous visitors. This incarnation of the Cliff House, with its various extensions, had lasted for 31 years.

In 1896, Adolph Sutro built a new Cliff House, a seven-story Victorian Chateau, called by some "the Gingerbread Palace", below his estate on the bluffs of Sutro Heights. This was the same year work began on the famous Sutro Baths in a small cove immediately north of the Restaurant. The baths included six of the largest indoor swimming pools, a museum, a skating rink and other pleasure grounds. Great throngs of San Franciscans arrived on steam trains, bicycles, carts and horse wagons on Sunday excursions. Sutro purchased some of the collection of stuffed animals, artwork, and historic items from Woodward's Gardens to display at both the Cliff House and Sutro Baths.[5]

The 1896 Cliff House survived the 1906 earthquake with little damage, but burned to the ground on the evening of September 7, 1907, after existing for only 11 years. Dr. Emma Merritt, Sutro's daughter, commissioned a rebuilding of the restaurant in a neo-classical style that was completed within two years and is the basis of the structure seen today. In 1914, the guidebook Bohemian San Francsco described it as " one of the great Bohemian restaurants of San Francisco. ... while you have thought you had good breakfasts before this, you know that now you are having the best of them all."[6]

In 1937, George and Leo Whitney purchased the Cliff House, to complement their Playland-at-the-Beach attraction nearby, and extensively remodelled it into an American roadhouse. From 1955 until 1961, a sky tram operated across the Sutro Baths basin, taking up to 25 visitors at a time from Point Lobos, enhanced by an artificial waterfall, to the outer balcony of the Cliff House.[7]

In 2003, as part of an extensive renovation, many of Whitney's additions were removed and the building was restored to its 1909 appearance. A new two-story wing was constructed overlooking what were by then the ruins of the Sutro Baths. (The Baths burned to the ground on June 26, 1966[9]). During the site restoration, the Musée Mécanique was moved to Fisherman's Wharf.[8]

More than thirty ships have run aground on the southern shore of the Golden Gate below the Cliff House.

When the Cliff House became part of the Golden Gate National Recreation Area in 1977, the National Park Service contracted with Dan and Mary Hountalas as official concessionaires of the property. The NPS renewed its contract with the Hountalas family, under the name Peanut Wagon, in 1998.[11] Peanut Wagon continues to manage Cliff House operations and worked with the Park Service for the extensive site restoration that was completed in 2004.

The Cliff House features two restaurants, the casual dining Bistro Restaurant and the more formal Sutro's. Additionally, the Terrace Room serves a Sunday Brunch buffet. There is a gift shop in the building, and the historic camera obscura is on a deck overlooking the ocean.

During the 2013 government shutdown, the US Park Service ordered the restaurant closed. The owners defied the order, but were forced to close again. They reopened with permission on October 12, 2013.[12]