Architects are all imbeciles. At least according to Gustave Flaubert in his ironic Dictionary of Platitudes. Prince Charles agrees. Flaubert went on, drily, to mock the tedious nostrums of the French middle classes who spluttered generalist drivel over provincial dinner tables, but we must be aware that the Prince of Wales's disdain is real. He thinks the majority of modern architects are inserting the wrong buildings in the wrong place. It is no use, as some do, dismissing the heir to the throne as a delusional, interfering narcissist. He wants modern architecture stopped and replaced by buildings more soothing to royal taste.

With superlative anachronism, he has set himself against the bulk of an entire profession. Excepting a coterie of fogeyish misfits, dreamers, forelock-tugging courtiers, DIY specialists, greasy pole-climbers, short-sighted antiquarians and people who would not recognise a titanium lock-nut if one were pushed up their dado, Prince Charles attaches to architects the sort of revulsion properly due to paedophiles. Since public opinion probably agrees with him, it is worth considering the argument, if that is not too dignified a term for the spectacle of head-banging provocation expected when the prince addresses the Royal Institute of British Architects this week, an event several leading architects yesterday said they would boycott.

It is more than 20 years since Charles last hit on architects in a robust speech at London's Mansion House that quickly won a place in the history of invective and abuse. With a mixture of masochistic dread and horrified fascination, the profession is wondering what he is going to say now. In the intervening period, HRH has discovered the consolations of organic farming, homeopathy and a happy marriage. Will his views on building design now be more mature, more considered, more rational?

In 1984, he indulged in a riot of puerile name-calling. A proposal for the National Gallery extension was said to be a "carbuncle" on the face of an old friend, the friend being Trafalgar Square. The distinguished firm of architects responsible, Ahrends Burton Koralek, never fully recovered from this astonishing assault. Instead, Charles's old friend was given a building by Robert Venturi, an American architect-academic then enjoying some esteem as the intellectual godfather of postmodernism.

Florence Nightingale said that visiting St Peter's was, her death aside, the greatest experience she expected to have. That's what great architecture should be. Not many think Venturi's Sainsbury Wing – like a Pennsylvania factory outlet with a college education – achieves such an effect, but Prince Charles approves because it has "classical" detail. This is rather like the late Queen Mother saying she enjoyed PG Wodehouse because he was so realistic.

Charles's speech went on to deride modern architecture for its obsession with "glass stumps", although one suspects that the words were provided by whomsoever has the secret Royal Warrant in Reactionary Propaganda. Glass, modern architects believe, is as useful and appropriate to contemporary needs as pietra serena was to Renaissance builders. But to Charles, the rhetorical use of glass suggests insensitive, brute, inhumane technophilia. Strange, then, that the entire history of modern architecture was founded on the belief that new materials, new systems and a design theory based on creativity rather than copying should be the basis of a new building.

The context of Charles's emphatically embargoed speech is an argument now going on in Chelsea, which has echoes and reflections of the Trafalgar Square debate. The vast Chelsea site was an old army barracks, built in the style which has no name, but which may be attributed to hopeless slaves in the Ministry of Works during the 60s. It was horrible. The site was acquired by Candy & Candy and great potential benefits accrued to civilisation when it was demolished. Richard Rogers was hired to design swish apartments. This he did in signature style or a version of it. The very vocal locals soon objected. They said there was too much glass. It was too tall. Too dense. Too modern. Charles stepped in and had his pet architect, Quinlan Terry, propose an alternative. And this is where the debate gets muddied. The levels of meaning are like an architectural mille-feuille.

Terry, who says his inspiration comes from God, is a specialist in architectural pastiche. His modesty and art are in inverse proportion. He has already built a "new" extension to Wren's Royal Hospital, just opposite the contentious Chelsea site. With maximum irony, this is plainly visible from Richard Rogers's home. If you needed a metaphor of Rogers's sensibility, all I need to tell you is that this home is a pair of fine, early Victorian townhouses which he has emptied of all clutter (including floors and walls). That Rogers has to look on Terry's ridiculously gauche new building while sipping his breakfast espresso from a cantilevered metal deck is evidence, surely, that God has a mischievous sense of humour.

Maybe Rogers designed the Chelsea ­Barracks extension as a sort of revenge. Unfortunately, while he has been responsible for some of the most challenging and rewarding of all modern buildings, this is not one of them.

It is clumsy, ham-fisted, inappropriate and the wrong scale. But that does not make Terry's replacement any better; it is a preposterously ugly exercise in what used to be known as "pseudish". That the Prince Of Wales advances Terry (a tweedy hack) over Rogers (a cosmopolitan dressed in Jil Sander) is a public provocation that relaunches the 19th-century debate that was called "the battle of the styles". The Chelsea conflict has forced people to adopt extreme positions. And that extreme demands to know if we would prefer a world designed by Prince Charles's acolytes or by Richard Rogers's studio?

The Prince of Wales's vision may be intuited from Highgrove, a miasma of upper-middle-class over-stuffing. Or you could try Poundbury, his experiment in architectural determinism in Dorset. Poundbury is a de luxe version of the gross and insensitive "executive" homes that so despoil Britain. It is a deadly place, making up for what it lacks in soul and vitality with ogee curves and pop antiquarianism. By way of contrast, what is Rogers's view of the world? Well, one thing is certain – he hates suburbs. He invokes "Croydon" as if it were a circle of the Inferno. He is a boulevardier-urbanist. Rogers loves city spaces. Unfortunately, he has not created a very good one in Chelsea.

I hope Charles will reflect on all of this in his speech to Riba. I hope, perhaps, that he may consider what is highbrow and what is lowbrow. Given his taste for legible reference to the past and his distaste for abstract thought, Charles is doggedly lowbrow. The architecture of Quinlan Terry, which he so restlessly advances, is bogus, lazy, depressing and inept. Richard Rogers has many faults, too, but all are relieved by a sense, no matter how misplaced, of optimism. As a patron, Charles promotes an eco-friendly feudalism. Rogers's architecture has its absurdities, but will be remembered when Terry's name appears only in footnotes appearing in books about kitsch.

The prince is correct to criticise bad buildings. But they are bad because they are inept and ill-considered, not bad because they are new. The same principles of criticism apply to buildings as to literature: who wants pastiche and doggerel?

We must struggle to make things new. And sometimes Richard Rogers must try harder, but the past is what we build on, not where we go to hide. That is surely a proposition that no reasonably civilised person could deny? Perpetual historical reference is an insult to creativity. And creativity defines humanity. Please note that Prince Charles does not visit his future subjects in an 18th-century helicopter.