Government House, for Bowen, was a tent then a wooden hut, and Collins also lived in a tent until March 1804, when a pre-fabricated hut he had brought with him was erected. In 1807, he built a crude brick residence in Macquarie Street, near today's Franklin Square, which Davey occupied in 1813. In 1817 renovations occurred for Sorell's benefit, but by 1825, the office-residence was in a 'ruinous state'. Early plans for a replacement never came to fruition, although alterations and additions between 1829 and 1831 meant that the residence grew to 17 rooms.

In 1840, Franklin received approval for convict-architect James Blackburn's design at Pavilion Point, the present site. Work began, but Franklin and Blackburn's successors – Eardley-Wilmot and William Kay – were instructed to halt and even filled in foundations. In 1853, during Denison's tenure, Kay adapted Blackburn's plans and construction recommenced using on-site sandstone, Huon pine and teak from the convict hulk Anson.

Sale of the original residence contained costs, which reached £67,872. Considerable convict labour was used, and fine craftsmanship is evident. In 1857 Fox-Young moved in despite just ten of seventy rooms being ready, and after its completion Anthony Trollope dubbed Government House the 'best belonging to any British Colony'. In 1872 a railway line separated it from the water's edge, as did a road to the Hobart Bridge in 1943. An originally intended conservatory was added in the 1980s.