Designed by John. A. Wood, a prolific Poughkeepsie architect, the Calisthenium and Riding Academy, Vassar’s first gymnasium, was completed. Matthew Vassar endorsed physical education as propounded by the early physical educator Dr. Dio Lewis and his colleague, the advocate for female education Catherine Beecher. The college’s 1865 Prospectus declared that “our plan of education. . . must include Anatomy. . . Physiology. . . Hygiene, by which we are taught the laws of health and the art of preserving it.” Accordingly, Lewis’s “calisthenics”—"training calculated to develop the beauty of the human figure, and to promote elegant and graceful movement"—was a required activity for all students.

Instruction in riding was given by Baron Leopold von Seldeneck, who had been a cavalry officer in the Prussian army and had served in the same capacity in the Civil War. The riding academy proved too expensive and was closed in 1872.

After the opening of the Alumnae Gymnasium in 1889, the Calisthenium was used as a museum, an assembly hall and for many years the home of the departments of classics, drama and English. It was renamed Avery Hall in 1931, in honor of Dr. Alida Avery, Vassar's first professor of physiology and hygiene and its first resident physician.

Avery Hall was razed in 2003 to make way for the Vogelstein Center for Drama and Film, but the center’s architect, Cesar Pelli, retained the Calisthenium’s façade and front section, which serve as the entrance and lobby of its main proscenium theater.