Matthew Vassar dug the first spadeful of dirt for Main Building on the site of a former racetrack, the land purchased in 1860. James Renwick, Jr. was the architect and William Harloe the builder. "Mr. Vassar in the presence of Executive Committee, Mr. Harloe, and a few other spectators removed the first spadeful of soil from the NE corner of the proposed building.... Remarks were made by Mr. Vassar and the Reverend Doctor Howard Malcom of Philadelphia and the latter invoked the Divine blessing. A single furrow was then plowed about the outlines of the entire structure and the work left to the builder." trustee executive committe minutes, June 24, 1861.

A prominent Baptist minister and missionary,Howard Malcom was the former president of The University at Lewisburg in Pennsylvania—subsequently Bucknell University. He served as president of Hahnemann Medical College in Philadelphia from 1874 until his death in 1879. William Harloe’s eventual bankruptcy obliged Matthew Vassar to donate an additional $75,000 to the complete the building.