The idea of establishing a cultural and educational centre across from the Berlin Palace dates back to the time of Friedrich Wilhelm IV, who dreamt of creating a "sanctuary for art and science" on the site. The basic architectural concept for the Alte Nationalgalerie – a temple-like building raised on a plinth decorated with motifs from antiquity – came from the king himself. The building was designed by Friedrich August Stüler, a student of Schinkel who also designed the Neues Museum. It was completed after Stüler’s death by another of Schinkel’s students, Johann Heinrich Strack.

The initial impetus for the construction of the Nationalgalerie was a bequest to the Prussian state in 1861 from the banker and consul Johann Heinrich Wilhelm Wagener, whose collection featured works by Caspar David Friedrich, Karl Friedrich Schinkel, painters from the Düsseldorf school, and history painters from Belgium. The bequest came with the stipulation that the paintings were to be publicly displayed in a "suitable location". Just one year later Stüler received the commission to draw up plans for the building. After ten years of construction the Nationalgalerie ceremoniously opened on 21 March 1876 for the birthday of Kaiser Wilhelm I, becoming the third museum on the island in the Spree.

The building suffered direct hits on several occasions during the aerial bombardment of the Second World War, sustaining heavy damage particularly after 1944. The collection itself had gradually been evacuated with the war’s onset. Among other places, it was stored in Berlin’s anti-aircraft towers near the zoo and in Friedrichshain, as well as in the salt and potash repositories in Merkers and Grasleben.

After the war’s end the building was quickly though provisionally restored; parts of it were re-opened in 1949. The second floor was made accessible to visitors one year later.

During the division of Germany, the 19th-century paintings that had survived the war in Western zones of occupation were housed in the Neue Nationalgalerie, starting in 1968, and in Schloss Charlottenburg’s Gallery of Romanticism from 1986. After the fall of the Berlin wall, the growing collections were united in their original building, now called the Alte Nationalgalerie, on Berlin’s Museumsinsel. Accommodating the collection meant repairing the damage the war had wrought to the building as well as adding new rooms. The architectural firm HG Merz Berlin was entrusted with this work in 1992. In March of 1998 the Alte Nationalgalerie was closed for renovations. The museum was finally re-opened in December 2001, marking its 125th anniversary.