Throngs of people crowd Long Wharf and Windmill Beach to watch teams compete in the annual whaleboat races at Sag HarborFest each September. But lost on many is the fact that whaling, the very reason the village holds the festival in the first place, was Sag Harbor’s version of the real estate industry during the half of the 19th century. Ships crowding the port provided the economic engine that powered many other businesses, from rope making to barrel making, and the village drew thousands of men seeking fortune as crew members on the ships that set sail on long and dangerous voyages from the village.

“It was better to own a whaling ship than sail on a whaling ship,” said Richard Doctorow, the collections manager at the Sag Harbor Historical and Whaling Museum. “It was like working in a combination of a slaughterhouse and an oil refinery with the added chance of drowning thrown in. Every part of this business was dangerous.”

Whalers’ Scrimshaw.

The Main Street museum, which just happens to be housed in the mansion built by Benjamin Huntting II, one the lucky Sag Harborites who made a fortune as a ship owner, offers visitors a glimpse at just how perilous, arduous and tedious the pursuit of these giants of the deep could be.

Take that whaleboat on the museum’s grounds. It came from the Concordia, a ship that sailed from Sag Harbor in the late stages of the whaling boom in the 1860s. “It’s probably the oldest whale boat extant, although it was extensively rebuilt in the 1970s,” Mr. Doctorow said.

Whaling ships typically carried three to five whaleboats, which carried a crew of six men each, who rowed out from the mother ship, harpoons, fastened to thousands of feet of line. When a whale was harpooned and tried to escape, often diving to elude capture, the line would uncoil at such speed it was often necessary to douse it with buckets of water to keep it cool. “It could literally tear your arm off,” Mr. Doctorow said.