Having grown up on a farm in Cumberland County, Pennsylvania, he moved to Baltimore and began working in a produce business. He later founded Robert Garrett & Sons, a wholesale grocery business, then became involved in the Baltimore & Ohio Railroad. He became a developer of banks and hotels in Baltimore, and in 1848 built a steamship to link Baltimore and San Francisco.