Hirschfeld was born on June 21, 1903 in St. Louis to Isaac and Rebecca Hirschfeld, the youngest of three sons. He showed early symptoms of what he described as “a sickness for drawing.” He was befriended by a local artist, Charles Marx, who took him on sketching expeditions around the city. Marx pleaded with Hirschfeld's parents to take their son to New York to study. On that recommendation, the Hirschfeld family left 5124 Kensington Avenue (the address that Judy Garland sings about in Meet Me In St. Louis), and moved to New York in 1914.