Ed Miliband … 'as pure a product of the political system as any yet discovered'. Photograph: Christopher Furlong/Getty Images

'Half-baked", "bereft of ideas", "doesn't have the X factor", "the single worst political speech I've heard in 20 years". At first glance, Ed Miliband's conference speech deserved the opprobrium which many commentators instantly heaped upon it.

The Blairite sentences, with no main verbs; the cheesy use of his family; the cliches about putting politics aside; the easy resort to caricature about the Tories; the faux-apologies and semi-recognition of the achievements of the 1980s; the straw man of Sir Fred Goodwin, a man ennobled and enriched by Labour – all the standard moves were there.

Above all, there was the praise for hard work and producers, from a man who is as pure a product of the political system as any yet discovered. Lest we forget, the Labour leader worked briefly as a journalist before entering politics at the age of 23 – 23, mark you – as an adviser to Harriet Harman. As the son of Ralph Miliband, he is Labour aristocracy. Not for him the tedious process of actually getting out and doing a day's work for a day's pay in the world of business. His official Labour party CV covers the first 13 years of his working life in one sentence of 18 words.

Where were you, Ed, in the 1990s? Which companies did you work for? How many CVs did you send in and have rejected? Which products did you create, how many business presentations have you made, which grumpy buyers did you have to persuade to stop what they're doing and take a look at your product? The answers are none, none, none, none and none. Read in this light, Miliband's conference speech is not just breathtakingly hypocritical. It is a sad and ironic apologia pro vita sua.

And yet … and yet. The event was a dud, and Miliband's persona is unpersuasive. But it would be a colossal mistake for the Conservatives to underrate what he is trying to do. For Labour itself, its most toxic inheritance is not the present colossal financial mess; it is the intellectual cul de sac in which its own history, not exempting the contribution of Marxists such as Ralph Miliband, has landed it. The post-world war Labour party is the creation of the Fabian tradition, in which the job of politics is to shape society by means of scientifically-minded intellectuals working through the state.

This dogma shaped British politics for two generations, and its results have been disastrous: a calamitous loss of economic competitiveness in the 1960s and 1970s; a social culture of entitlement and dependency; an arrogant and self-satisfied politics; and a state itself that has increasingly struggled to deliver public services to a standard accepted as routine in many other industrialised countries.

So Miliband is not simply trying to position himself with the ordinary man; he is trying to reorient his party and to redefine the terms of political debate. And specifically he is seeking to contest the political ground now occupied by the "big society". For it is the state-first Fabian dogma that is the specific political target of the big society – the idea, not the label – which emphasises the renewal of social capital, localism and greater personal empowerment.

However, the big society also contains a robust critique of neoliberalism, and the rigor mortis economics that is ultimately responsible for the financial crash. It is deeply critical of the fundamentalism of free market uber-libertarians, who see no role for the state at all. And it clips the wings of technocrats and ideologues who would reduce all political or social questions to economic ones, or indeed substitute economics for politics as such.

The big society is ultimately derived from the ideas of Edmund Burke and Adam Smith, who emphasised not rampant but limited markets; not the over-mighty state but free and independent institutions; not personal greed but trust and sympathy. So the last thing the Conservatives should do is to allow Miliband a free rein here. They should be calling for real capitalism, not crony capitalism; for entrepreneurship and creativity, not profitable paper-shuffling; for better corporate governance and support for long-term investment, not stratospheric executive pay unrelated to performance. For the Conservatives, not Labour, are the party of small business and the working taxpayer.