Newark has flushed out the real meaning of Nigel Farage

By dodging the Newark byelection, the Ukip leader has opted to lead an anti-politics party. It is a far-reaching decision

Nigel Farage in the Volunteer Rifleman's Arms, in Bath. 'He knows that he heads the biggest mould-breaking force since the Social Democratic party in the early 1980s.' Photograph: Matt Cardy/Getty Images

What is it about the prospect of standing for Newark that makes an ambitious politician look into his soul? In 1832 the young William Gladstone, then a Tory, was offered a run at Newark by the Duke of Newcastle. "This stunning and overpowering proposal naturally left me the whole of the evening on which I received it in a flutter of confusion," he wrote to his father. It would be, Gladstone foresaw, "a bold and terrible experiment".

The political class is sure Farage has done the right thing. They have many reasons. Ukip is not a force in that part of Nottinghamshire. Farage is not local. The Conservatives are well dug in. Labour's strength would ensure a three-horse race. And Farage says he wants to do nothing that would distract from Ukip's chances of victory in the European elections on 22 May. Those chances are looking extremely good. Yesterday a ComRes poll for ITN put Farage's party 11 points ahead of Labour with the Tories a distant third. If he had run in Newark and lost, Farage candidly admitted to the BBC yesterday, "the bubble would be burst".

The political class is both right and wrong. By not running in Newark, Farage has acted like a conventional politician. Yet in so doing he has paradoxically proved that he is not a conventional politician at all. Newark was a fork-in-the-road moment for Ukip. At Newark, Ukip could have chosen to be a parliamentary political party. Or it could have chosen to remain as an anti-politics party. It chose the latter. It is a far-reaching decision.

Farage knows his political history and this was not a lightly taken decision. Excepting the Scottish National party, Farage knows that he heads the biggest mould-breaking force since the Social Democratic party in the early 1980s.

He studies the SDP's rise and fall. He has talked to veterans of those times. Farage once told me that the great question in British politics was whether he could engineer an "SDP moment" for Ukip, in which a massive chunk of the Tory party breaks away from the mother party and into his – and possibly Boris Johnson's – embrace.

SDP veterans are said to have told Farage this week to go for it. That was what the SDP did in 1981 when it threw its biggest names at byelection contests in places where it had no prior base. That approach took Roy Jenkins to Warrington and then to Glasgow Hillhead, and Shirley Williams to Crosby, relying on a nationwide wave of support to carry off spectacular byelection wins. The strategy very nearly worked. Within two years Jenkins came within an ace of becoming leader of the opposition and leading the SDP to replace Labour as the main challenger to the Conservatives.

But there was a crucial difference between the SDP and Ukip – and it wasn't just the two parties' diametrically opposed positions on Europe. The SDP saw itself as a party of government. It had a mass of detailed policies for the administration it hoped to form. Ukip, in contrast, is not a party of government at all. It is a party of anti-government. It has no ministerial experience and very few policies. On Radio 4 yesterday, the Ukip spokesman said he had no idea where the party stood on Royal Mail privatisation. It doesn't matter. Ukip is a party of protest, and Farage underlined this by deciding not to run in Newark.

This is in no way to suggest that Ukip has taken a wrong turning by its own lights. A Ukip victory in the European elections (and a good showing in the locals the same day) will renew the agonies in the Tory party, set off a bout of heart-searching in Labour, cause fresh leadership speculation in the Lib Dems, and play into the SNP's hands by allowing Alex Salmond to frame the English as a nation of rightwing Europhobes with whom Scots have nothing in common. And it is by no means out of the question that Ukip could capture Newark, too.

But Ukip's purposes are highlighted by Farage's turn away from Newark. Farage's Ukip is a party that channels resentments of various kinds – many of them understandable at some level – towards ends that it has little or no ability of securing. It is a party that stands against what is happening here and now, not for anything that it has any practical hope of achieving or for which it will accept responsibility. It is a plague-on-all-your-houses party. And while Ukip is in no sense a fascist or communist party, the rest of us should be just as parsimonious with our embrace of the feelings that Ukip channels as we would have been towards populist parties in the 1930s.

The rise of Ukip is a verdict on the existing parties, as the rise of the SDP was 35 years ago. But because Ukip is not and does not aim to be a party of government, the existing parties will always struggle to respond to it. In the 1980s the parties responded to the SDP – which by contrast was a party of government – by creating New Labour, which led for a time to a better system of politics than the one whose failures had produced the SDP.

Today's parties, on the other hand, have much less to offer Ukip voters. As a result, it could be dangerous, as well as unprincipled, for either Labour or the Tories to suppose that they can respond to Ukip by trying to embrace Ukip ideas. That lesson applies just as much to rightwing Tory Eurosceptics, who long to pull up the drawbridge against the world to create a Dad's Army England of white people, as it does to leftwing Labour (or some Scottish yes-to-independence voting) globosceptics, who dream of pulling up their own drawbridges and creating social democracy in one country.

It is one thing to try to understand Ukip; it is another thing to outbid it in its own terms. Modern politics is difficult, not easy. It needs to be defended and improved, not denounced and abandoned. And it was part of his greatness that Gladstone always understood that far better than most.