Town Hall was built between 1919-21 as designed by Teunis J. van der Bent of McKim, Mead & White. The League for Political Education, who were energetic suffragists campaigning for the 19th Amendment (women's right to vote), created a meeting hall which would serve as forum for its members, and as a venue for educating the public on important issues of the times. The Colonial Revival-style auditorium democratically eliminated box seats, and there were no columns to obstruct sight lines, giving birth to the phrase, "Not a bad seat in the house." The 19th Amendment was passed during completion of the building, which opened on January 12, 1921. Almost immediately after its opening, the hall was in demand for concerts and theatrical events. Lectures and speaking engagements were given by prominent figures of the times, including Theodore Roosevelt, Booker T. Washington, Margaret Sanger, Buckminster Fuller, Winston Churchill, and Woodrow Wilson. Town Hall is also known for the American Town Meetings of the Air radio program broadcast during the 30s, 40s and 50s; the many historic lectures and debates; and innumerable memorable performances by classical, jazz and popular artists. The Town Hall Endowment Series featured artists including Sergei Rachmaninoff, Ignace Paderwaski, Lily Pons, Fedor Chaliapin, Yehudi Menuhin, and more. The hall's superb acoustics attracted many artists to make their New York debut on its stage, including Marian Anderson, contralto, who made her New York debut here on December 30, 1935, after being denied an operatic career elsewhere because of discrimination against African-Americans.

In 1987, Town Hall was designated a landmark by the New York City Landmarks Preservation Comission, and in 2012 was added to the list of National Historic Landmarks.

The organ in Town Hall was built in 1923 by the Skinner Organ Co., of Boston. An article in The New York Times (Dec. 29, 1922) announced that "James Speyer had donated an organ as a memorial to Mrs. Ellin Speyer, his wife. According to Arthur Hudson Marks, President of the firm which will build and install the $35,000 organ, it will be among the finest in the country."

The organ was to contain 2,913 pipes, have four keyboards with complete mechanical equipment for recitals. "In addition to the traditional organ tone, Mr. Speyer's gift will have many of the voices of an orchestra, such as the bassoon, oboe, tuba, harp, flute and French horn."

In 1935, Ernest Skinner's organ in Town Hall was completely overhauled by the Aeolian-Skinner Organ Company (Op. J-5364), under the direction of G. Donald Harrison. By the late 1950s the organ was rarely used; in 1960 it was acquired by the Takoma Park Seventh-day Adventist Church, Takoma Park, Md., where it was moved and installed by Lewis & Hitchcock, Inc., of Washington, D.C.