The Return of the King, LoTR Appendix B, The Tale of Years: The Third Age

Ever most vigilant was Mithrandir, and he it was that most doubted the darkness in Mirkwood, for though many deemed that it was wrought by the Ringwraiths, he feared that it was indeed the first shadow of Sauron returning; and he went to Dol Guldur, and the Sorcerer fled from him, and there was a watchful peace for a long while.

The Silmarillion, Of the Rings of Power and the Third Age

Gandalf goes alone to Dol Guldur in secret, to discover the truth concerning the Sorcerer. But the Sorcerer is aware of him; and being not yet grown to great power, he fears the eyes of Gandalf, and the strength of the Wise, and he deserts Dol Guldur and hides in the East again for a while.