Before the mid 15th century, it was known as Innermeath, and was the home of Sir John Stewart of Innermeath (great-grandson of John Stewart of Bonkyll), whose elder son was the first Lord of Lorne, and whose younger son was The Black Knight of Lorn, a powerful 15th century magnate, allied to the Black Douglases; both children were born at Invermay (still called Innermeath at the time of their birth). Sir John's grandson, William Stewart, surrendered the Lordship of Lorne to the king, in return for being made the first Lord Innermeath; the title became extinct in 1625, by which time the name of the location had become Invermay.

An early medieval freestanding cross was once located at Invermay (grid referenceNO05981664, but was destroyed in the 18th century and it was replaced with a modern stone.[3][4] The fragments of the original are thought to be those now found in the Porch of Forteviot church.[5]