The Rockefeller Museum was built on the site of Karm el-Sheikh, named after the owner of property, Sheikh Muhammad al-Halili, Mufti of Jerusalem in the late 17th century. In 1711 al-Halili built his summer residence there. The house still stands today, west of the Museum.

Al-Halili's two-story residence, known as Qasr el-Sheikh, was one of the first structures to be erected outside Jerusalem's Old City walls. From this building, it was possible to keep a watchful eye on the surrounding olive grove and garden and enjoy the picturesque landscape. The first floor housed an olive press; the second floor was the residential story.

In 1906 the Jewish National Fund considered purchasing the site of Karm el-Sheikh for the Bezalel School of Art and Crafts. It had been the dream of the school's founder, Boris Schatz, to establish a museum and university overlooking the Temple mount. This plan, however, never came to fruition. Thirteen years later, in 1919, the British Mandatory Government in Palestine selected the site for the construction of an archaeological museum. But it was only in 1930 that the British managed to purchase the plot, and another eight years would pass before the Museum opened its doors to the public.