The six-story mansion, built in 1908 to designs by the architect C. P. H. Gilbert, was constructed for the philanthropist Felix M. Warburg (d.1937). The style is a revival of early French Renaissance architecture from the period Francois I.

Warburg purchased from Perry Belmont a lot that measured 100 feet along Fifth Avenue, but required Gilbert to cover only half of it with his house, permitting a side lawn fifty feet across on the avenue. When it was completed, Warburg's father-in-law Jacob Schiff feared that it was ostentatious and would incite envy and anti-semitism.[2] The Warburgs specified that they were pleased with Harry Sinclair's house and would like something similar, with details likewise drawn from the Late Gothic Hôtel de Cluny, Paris.

His widow Frieda Schiff Warburg had tried to donate the house to a cultural institute but failed and finally sold the mansion in 1941 to developer Henry Kaufman and the architect Emery Roth, who intended to redevelop the site into an eighteen-story apartment building. After the developers' plans fell through, the mansion reverted to Mrs. Warburg.[3]

In January 1944, she donated the family mansion as a permanent home for the Jewish Museum, which opened to the public in May 1947. The building was expanded in 1963 and again in 1993 with a discrete mid-block addition by Kevin Roche that blends seamlessly with Gilbert's original French Renaissance design.[4]