The castle was built by Captain John George Adair (1823-1885), a native of Co Leix, and a member of the minor gentry. Adair had made his fortune by chancy land speculation in the United States, and he returned to Ireland and bought up vast tracts of land in Donegal. Adair had married in 1869, Cornelia Wadsworth Ritchie, a daughter of James S. Wadsworth, a Union General in the American Civil War. Together they set about the creation of the Gardens and Castle. Adair's ambition was to create an estate and castle that surpassed Balmoral, Queen Victoria's Scottish retreat. John Adair is remembered with scant affection in Donegal. On the heels of the Great Irish Famine and emigration on a par with the Highland Clearances, John Adair evicted 224 tenants from their blackhouses on his land. This was not for financial gain, but merely to improve the æsthetic aspect from the castle. These tenant clearances are known as the "Derryveagh Evictions". The name of John George Adair as a Donegal landlord has passed into history and folklore, ballad and documentary. All have one thing in common - Adair was notoriously cruel. He purchased Glenveagh and Gartan in 1859 making an estate of 28,000 acres (110 km2).

His troubles with the tenants began almost immediately. A row between them and Adair over shooting rights and trespassing sheep culminated in the murder of his Scottish steward James Murrog. Consequently Adair carried out his threat to evict the tenancy. On 3 April 1861, a considerable cortege of 200 police, three sub officers, the resident magistrate and the sub-sheriff set out from Letterkenny to undertake their duties. The evictions began at Lough Barra where a widow, Mrs Hanna McAward and her six daughters and one son were the first to suffer. The work of destruction continued for three days through Magerashangan, Staghall, Claggan, Ardator and Castletown among other townlands. In all, 44 families were evicted making a total of 244 persons.

Many of the evicted went to the work house in Letterkenny, others were helped by locals and the clergy also raised money. In Australia, the Donegal Relief Fund was revitalised and arrangements were made to help the young people aged between 16 and 28 years to emigrate. Many took advantage of the scheme. As they settled in Sydney the strong oral tradition ensured that the descendents remembered their families' bitter memories.