Irvington's layout and development are rooted in its settlement patterns. The land that would become Irvington was originally populated by the Weckquaseck Native Americans, part of the Mohegan tribe of Algonquins. In the 1600s, Dutch settlers came to the area to farm and trade. Four main families settled in the area: Stephen Ecker, whose plantation was located on what is now the Sunnyside property; Jan Harmse, who settled land on the south and west sides of Dows Lane; Captain John Buckhout, who settled near the Hudson River; and Barent Dutcher, who established a farm on what would become Matthiessen Park.

In the 1700s, King's Highway (known as Albany Post Road after the Revolutionary War and now as Broadway in the Irvington vicinity) was completed to serve as a post road between New York City and Albany. The post road was for a long time the principal road through Irvington and facilitated settlement in Irvington. Maps from that time period show that many of the estates and farms were set along the east and west sides of the road.

By the mid-1800s, prosperous New York City families constructed country estates in the area. Well-known residents included James Hamilton, Alexander Hamilton's son, who constructed the Nevis estate that is now home to Columbia University's Nevis Laboratories; Cyrus Field, inventor of the Atlantic cable, who lived on the Ardsley estate; Charles Tiffany, who developed Tiffany Park which later became Matthiessen Park; and the author Washington Irvington, who developed the Sunnyside estate and for whom the village is named. Many of these estates were subdivided following World War I to make room for new residential communities, but the estate homes still stand and the park-like settings of the properties were preserved, contributing to Irvington's historic character and significant open space. Several infrastructure projects were also underway in the 1800s that shaped Irvington's growth and development. In 1837, construction began on the Croton Aqueduct, which ran parallel to and just west of Broadway. Some farms were subdivided and roads rerouted to create space for the Aqueduct. In 1849, the railroad line was completed along the Hudson River, running between New York City and Peekskill and then farther north to Poughkeepsie. In 1850, a developer purchased farmland in what would become the village center and laid out a subdivision plan that included 266 property lots and a street grid, with easy access to the railroad line. The plan included the east-west artery of Main Street with streets branching off in a north-south direction from the foot of Main Street east to Broadway. This development became the Village's main shopping district and municipal center, with the Village Hall, schools, churches and other public institutions situated in the Main Street vicinity. The homes that were developed on smaller lots around the core were occupied by artisans and the employees of the estates. Some industry developed as well along the riverfront, most notably the Lord and Burnham Company, which manufactured greenhouses, but the area preserved its residential character.

The Village took the name Irvington after Washington Irving in 1854 and was formally incorporated in 1872. With the electrification of the rail line in 1913, Irvington became more of a commuter suburb. The subdivision of the estate properties after World War I created residential developments such as Jaffrey Park, Matthiessen Park and Spiro Park. The park-like setting of these developments, along with the private institutional properties such as Nevis Laboratories, have helped Irvington retain its rural character and open space feel even as the population has grown.