The Rocks became established shortly after the colony's formation in 1788. The original buildings were made mostly of local sandstone, from which the area derives its name. From the earliest history of the settlement, the area had a reputation as a slum, often frequented by visiting sailors and prostitutes. During the late nineteenth century, the area was dominated by a gang known as the Rocks Push. It maintained this rough reputation until approximately the 1870s.