With other wealthy women, including Anne Tracy Morgan (a daughter of J.P. Morgan), Harriman raised $500,000, and commissioned Stanford White of McKim, Mead & White to build the original clubhouse, later known as the "Old Colony Club". This building – at 120 Madison Avenue, between East 30th and East 31st Streets on the west side of Madison – was built between 1904 and 1908 and was modelled on eighteenth-century houses in Annapolis, Maryland.[3]

The interiors, which exist largely unchanged and have been accorded the landmark status, were created by Elsie de Wolfe – later to become Lady Mendl – a former actress who had recently opened an interior-design business, and whose companion, the theatrical agent Elisabeth Marbury, was one of the club's founders. Stanford White was slain by Harry K. Thaw months before construction of the Colony Club was completed. The building was designed in the Federal Revival style, and has unusual brickwork done in a diaper pattern[3] as a notable feature of its facade.

The club and the street in front of it were often the site of large suffrage rallies sponsored by the Equal Franchise Society to which many members of the Club belonged.[4]

The second clubhouse, located at 564 Park Avenue, also known as 51 East 62nd Street, on the northwest corner, was commissioned in 1913[7] and constructed from 1914 to 1916. It was designed by Delano & Aldrich in the Neo-Georgian style, with interiors designed by Elsie de Wolfe[8] The building has a marble base with red-brick and marble trim and columns for the upper floors.[2][6] According to Andrew Dolkart:

This is not one of Delano & Aldrich's more elegant works in the Colonial idiom, perhaps because it was nearly impossible to create a well-proportioned design for a building with the complex spatial requirements of this club. The beautifully appointed interior included the lounges, dining rooms, and bedrooms common to social clubs, but also had a two-story ballroom, a basement swimming pool and spa that connected via an express elevator to a gymnasium on the fifth floor, two squash courts, servants' rooms (in 1925 there were thirteen female servants), and even a kennel where members could leave their pets.[9]

In 1973, Secretary of State Henry Kissinger's birthday party was held at the Colony Club (among the guests were four couples whom Kissinger had ordered to wiretapped[citation needed]). In 2007, memorial services for Brooke Astor were held there. The club continues its policy of women-only membership – new members must be recommended by current members[10] — although it was unsuccessfully contested in court in 1987 by conservative radio talk-show host Bob Grant and Sidney Siller, who founded the National Organization for Men.[8]

The Club presently has approximately 2,500 members who have access to discussions, concerts, and wellness and athletic programs. The Clubhouse consists of seven stories, 25 guest bedrooms, three dining rooms, two ballrooms, a lounge, a squash court, an indoor pool, a fitness facility and three personal spa service rooms. Annual gross revenues are more than $10 million.[11]