Richmond station has been relocated and rebuilt four times. The first station in Punt Road was at ground level and opened on 8 February 1859. It closed the same year on 12 December 1859. On the same day the new station, also at ground level, opened in Swan Street. In 1885, an elevated station just north of Swan Street was opened with six platforms.

By the 1930s the station was struggling with the patronage, with the 1940 Ashworth Improvement Plan recommending that it be rebuilt but funding issues during World War II prevented it from happening.[2] The station had also deteriorated to the point that it was condemned by the local council, and so work begun in the 1950s on a replacement. Part of Operation Phoenix, the postwar rebuilding of Victorian Railways,[3] it was not until 26 March 1960 that the present station was completed. Located slightly west of the previous station, the bridges at each end of the station across Punt Road and Swan Street were also rebuilt to accommodate the ten tracks. For a period, platforms at both the old and new stations operated, before the original station finally closed and was demolished.