For 56 years almost every Governor from Phillip to Gipps lived and administered the colony from the grand old building on the corner of Phillip and Bridge streets. Although it was predated by Phillip's prefabricated canvas house the foundation stone for 'Old' Government House was laid on 15 May 1788, and is to all intents and purposes was the first real Government House in Australia. For most of its 56 years, it was the centre of social life in Sydney, including the momentous events surrounding Governor Bligh's discovery in an upper bedroom during the Rum Rebellion.

But being situated at the very heart of Sydney brought pressure on the site as it transitioned from a town to a city. By 1836 the demands for more commercial space and access to Circular Quay saw the first proposals for the movement of the building although it was 1843 before Sir George Gipps was able to announce his move into the new building (still standing in the Domain). 1843 also marked the date of the demolition of the old building.

In a 1919 article in the Sydney Morning Herald, Mr McGuanne described the old building,

on the ground floor, there was a dining room 30 feet long, 16.5 feet broad and nine feet high: a parlour 20 feet long, 16.5 broad, and a kitchen room. There were three upper rooms. Subsequently a drawing room was added to the eastern end, adjoining the dining room. This room was 50ft long by 18.5 broad and 11.5 high. It was used as a ball room. In one of the upstairs rooms Bligh was arrested. In 1809 or 1810 the building had 100 foot frontage, and a verandah ran along Its whole length. The roof was supported by eight double pillars at each extremity, and slightly in front of the building stood a sentry box.

The main door was under the gable. The house stood well within 200 yards of the water's edge, the ground between being laid out in a fine garden with numerous trees. These were the trees under which was held, in 1849, the great anti-transportation meeting, while the 'Hashemy' convict ship was in full view at the mouth of the Cove, awaiting the determination of the Governor as to what was to be done with the exiles.

In I812, Governor Macquarie built a stone wall to separate his gardens from the public area but the wall was low, and people crossed it as a shortcut to Woolloomooloo. In an effort to put a stop to this Macquarie set spies to catch the trespassers and as a result six men and two women were apprehended. The men were flogged, and the women fined. One of the men, William Blake, who was a free man, gathered a petition for the recall of Macquarie. He then took his complaint to London and this is believed to have played a role in the recall of the Governor.

In October 1919, a stone tablet with an embedded porcelain tablet was unveiled on the corner of Phillip and Bridge streets to mark the original location of the Old Government House. The site is now occupied by the Museum of Sydney.