Paul Chiasson/Pool/Reuters

The next few weeks are likely to be particularly trying ones for Prince Charles.

As the nation rejoices in the birth of a new royal baby—fourth in line to the throne regardless of gender—Prince Charles, never keen on sharing the limelight, has a less appealing summer ahead.

For although he is to be a grandfather again, it seems unlikely he will be seeing very much of Prince George’s new brother or sister. His PR team would dearly love a photo of Charles dandling the new baby on his knee whizzed out to the world on his fledgling social media account, but that isn’t going to happen.

William and Kate have never taken George to stay with Charles and Camilla, and sources say that Charles is planning to spend most of the next few weeks of royal baby madness in his Scottish home of Birkhall.

This will at least supply a face-saving cover story, a handy excuse to explain why William and Kate aren’t involving Charles in their children’s lives, when scarcely a week over the summer will pass without the Middleton parents—Carole and Mike—being invited to the Cambridge’s country pile, Anmer Hall, or having an invitation to their own house accepted by the new parents.

But it’s not just within his immediate family circle that Charles could be forgiven for feeling unloved.

Recent polls show that the British public remain stubbornly un-enamoured with Charles, or his consort, Camilla.

According to a poll by Comres, for the Daily Mail, just a third of the population (34%) say they like Camilla, with a greater number, 38% saying they dislike her.

These are shocking figures for Charles PR machine to contemplate after a decade of assiduously attempting via every trick in the book to soften Camilla’s image and encourage the populace to warm to her.

And while seven in ten Britons think Britain should remain a monarchy, many British people still believe Prince Charles should make way for Prince William as the next monarch. 40% of Britons say Prince Charles should give up his right to be the next king in favor of his son.

Charles has made it clear that the subject has never been up for discussion as far as he is concerned, and, sources say, finds solace in the example of Edward VII. Edward VII—who ruled from 1901 to 1910, following the death of his mother Queen Victoria—was a terribly unpopular heir but, when it finally came to it, a very good king. He was so good, in fact, that they named a historical era after him despite his short reign.

Prior to Charles’s epic 66 year wait, Edward VII was also the longest serving Prince of Wales and heir apparent to the throne.

So, Charles says, case closed. The constitution is the constitution, plus he genuinely believes in the divine right of kings; God has chosen him to be the next king of Albion, and who wants an argument with God?

The trouble is that we the people don’t seem to have got the memo. We insist on continuing to debate and contemplate and speculate about whether Charles should step aside. We just will keep discussing it.

Charles has made little secret of the fact he would like to rule with ‘Queen Camilla’ by his side, and refuses to abandon the idea despite the fact that a whopping 55% of the population are adamant that Camilla Parker Bowles should not become Queen.

Unfortunately for Charles, things look set to get worse before they get better, with a summer of further humiliations ahead on a very public scale.

In early August, the government will finally release Prince Charles’s secret letters to government officials, known as the black spider memos, because of the prince’s scrawled handwriting.

They include correspondence with seven government departments and are widely suspected to expose an arrogance of tone and a wackiness of belief. Let us not forget that Prince Charles talks to the plants on his Highgrove estate to encourage them to grow.

The Guardian has been battling for the release of these memos for a decade, but the establishment, which blocked the paper every step of the way, has now lost this battle.

If the letters—said to contain his most deeply-held private beliefs—really do expose Charles as a quack, there will be even more pressure on him to pass up his constitutional right to the throne in favor of his son.

With the Cambridges basking in the public affection extended to all new parents, Charles will need to grit his teeth and bear it as the chatter that he is not the right man for the job of being the next king starts up again.