The King and Lady Conyngham's dislike of Bloomfield, as he tried to curb the King's excesses, became evident on the King's trip to Scotland on 10 August 1822, as the rising star of the King's entourage, Sir William Knighton, was situated next to the King's cabin, whilst Bloomfield was rather coldly relegated to a cabin far further away. Furthermore, Conyngham encouraged her son, Francis, to shoulder some of Bloomfield's responsibilities, much to Bloomfield's obvious displeasure. There was even a rumour that some of Lady Conyngham's jewels belonged to the Crown, a fact known by Bloomfield, and therefore the royal mistress felt compelled to have him removed. As Bloomfield began to be undermined by Sir William Knighton and Francis Conyngham, his self-confidence started to fade, his grip on the royal purse was weakened and he abruptly had his salary stopped by royal command- his demise was imminent. In an act of desperation he began to lobby Parliament, claiming 'royal betrayal', however, this was ineffective as Lady Conyngham's family were attached to Bloomfield's target audience- the Whig opposition- and therefore his pleas fell on deaf ears.[6]

Bloomfield's downfall was hastened further by a royal visit to Dublin in 1821. In one incident, the King visited a local theatre, and believing Bloomfield to be an important member of the King's party, the manager began playing the national anthem as Bloomfield entered his box, responding by bowing and smiling jokingly as the crowd rose and began singing 'God Save the King' (believing Bloomfield to be a member of the royal family). The King, noted for his sense of humour, was unusually furious at this act, declaring it an insult. Another plausible explanation for Bloomfield's demise is provided by a courtier, Sir William Freemantle in a letter to the Duke of Buckingham. The King's expenses from the spring of 1822 showed a considerable amount of money had been spent on an undisclosed item, which Bloomfield revealed to be the purchase of diamonds by the King. The King considered this to be damaging, and showed beyond all doubt that Bloomfield had lost his ability to protect the King's image at all costs. The diamonds were most probably for the royal mistress, an assertion which the media exposed.[7] In a last humiliating episode for Bloomfield, he was ordered by the King to pay J.L. Marks a sum of £45 to prevent the publication of a cartoon which implicated the King and his mistress in the diamond affair, after Marks sent a copy to the King's residence before its publication. Marks duly ripped up the plate before his eyes, despite having made copies sneakily beforehand. In fact Bloomfield had spent a fortune buying up caricatures.[8]

Finally, to the relief of the King, ministers agreed that Bloomfield should be removed from his position. The King wrote to Lord Liverpool, asking for the post of Private Secretary to be abolished to make Bloomfield's departure appear to be a matter of politics rather than the Crown. Bloomfield was offered the Governorship of Ceylon as compensation, or his current salary for life and the Order of the Bath. Bloomfield felt that his efforts deserved at the very least an English peerage, the King however flew into a rage when hearing Bloomfield's demand, threatening to have him alienated from society, just as his wife had been. Bloomfield pragmatically refused the position of Governor of Ceylon, but accepted the Order of the Bath, a sinecure worth £650 per annum and the Governorship of Fort Charles in Jamaica,[9] that he would later exchange for the post of Minister at Stockholm, where he served from 1823 to 1832.[7] The King invited him to the Royal Pavilion at Brighton one last time to receive the Order of the Bath from the King, but thought better of it, and did not journey to meet his former royal master for the last time.[10]

Following his turbulent years in service to the King, Bloomfield unexpectedly embraced the values of Methodism and became a devout Christian. His house in Portman Square, London amused many a passer-by as he would often have a placard on his front door, adorned with the words 'At Prayer'.[16] He was promoted to lieutenant general on 22 July 1830 and died in Ireland in 1846. He was buried at Borrisnafarney Parish Church[17] in the Bloomfield Mausoleum in County Offaly, Ireland[18] which is located 1.5 miles from the village of Moneygall beside the Loughton Estate [19]

Bloomfield married Harriott Douglas, daughter of John Douglas, on 7 September 1797. They had a son, John Arthur Douglas Bloomfield, 1st Baron Bloomfield of Ciamhaltha (1802–1879) who was created Baron Bloomfield, of Ciamhaltha in the County of Tipperary, in the Peerage of the United Kingdom, on his retirement as British Ambassador to Austria, and two daughters, Georgina and Harriott.[20] Harriott Mary Anne Bloomfield married on 5 June 1833 Colonel Thomas Henry Kingscote (b. 19 Jan 1799, d. 19 Dec 1861).[21]