He was the son of Agnes Louise (Crookenden) and The Rev. Gerard Kerr Olivier, who practiced high church, ritualist Anglicanism. Most of his ancestry was English. His great-great-grandfather, Daniel Stephen Olivier, was of French Huguenot descent. His uncles were painter Herbert Arnould Olivier and civil servant Sydney Olivier, 1st Baron Olivier, who was Governor of Jamaica (from 1907 to 1913) and Secretary of State for India (in 1924).

Laurence’s paternal grandfather was Henry Arnold Olivier (the son of Henry Stephen Olivier and Mary Milligen Dacres). Henry was the son of Daniel Stephen Olivier and Margaret Harriet Arnold. Mary Milligen was the daughter of Richard Dacres and Martha Phillips Milligen.

Laurence’s paternal grandmother was Anne Elizabeth Hardcastle Arnould (the daughter of Joseph A. Arnould and Elizabeth Hardcastle). Joseph was the son of Joseph Arnould and May Field. Laurence’s great-grandmother Elizabeth was the daughter of Joseph Hardcastle and Ann Corsbie.

Laurence’s maternal grandparents were Isaac Adolphus Crookenden (the son of Adolphus Pelham Crookenden and Jane Margaret Chapman) and Jane Sophia Mitton (the daughter of Joseph Mitton and Sarah or Sophia). Laurence’s great-grandfather Adolphus was the son of Isaac Crookenden and Elizabeth Fillany. Jane was the daughter of William Chapman.