The great cape and land-locked firth of Umbar had been Númenórean land since days of old; but it was a stronghold of the King's Men, who were afterwards called the Black Númenóreans, corrupted by Sauron, and who hated above all the followers of Elendil. After the fall of Sauron their race swiftly dwindled or became merged with the Men of Middle-earth, but they inherited without lessening their hatred of Gondor.

The Lord of the Rings, Appendix A, Annals of the Kings and Rulers: Gondor and the Heirs of Anárion, Footnote

... Umbar, where was the mighty haven of the Númenóreans that no hand had wrought.

There they made a refuge for all the enemies of the king, and a lordship independent of his crown. Umbar remained at war with Gondor for many lives of men, a threat to its coastlands and to all traffic on the sea.

The Lord of the Rings, Appendix A, Annals of the Kings and Rulers: Gondor and the Heirs of Anárion

Dates from The Lord of the Rings, Appendix B, The Tale of Years: The Third Age
Some information from Unfinished Tales, Part 3, Ch 2, Cirion and Eorl and the Friendship of Gondor and Rohan: The Northmen and the Wainriders