The bay itself is quite small, but well sheltered, with a small bark area and an extended coastline stretching out to the north. Nearby John F. Kennedy Park contains old WWIIbunkers that were used to survey the Hauraki Gulf. The northern headland of Castor Bay is the site of an old Maori Pa, Rahopara, that is believed to have been last inhabited and then abandoned in the 18th century, prior to European colonisation. Very little is known about the people who lived there but the tribe's ancient earth excavations are still visible.

Sam Hunt, one of New Zealand's most well known poets was born in Castor Bay.