Eric Pickles has an office in the Commons up a winding staircase above St Stephen's entrance. It's one of the oldest parts of the Palace of Westminster and he says the office next door was where they signed Charles I's death warrant, something to bear in mind as he plots the downfall of Gordon Brown and the Labour government.

Since he got the chairman's job in January, things have not gone too badly. The European and local elections went well, the poll leads have been consistently high and last month the Tories won the Norwich North byelection by a much wider margin than expected.

But Pickles says he is still nervous.

"I must be the only person in the party that's worried about 16% or 18% opinion poll ratings. I don't think it's anywhere close to being in the bag.

"It's going to require a lot of graft from associations, a lot of graft from candidates and unremitting pressure between now and next May because you can't rule out the possibility of something in the later part of the year or next spring."

By "something", he means Brown standing down.

"If Mr Brown decides to go at his conference? Or decides, once he's opened his last present from Santa Claus, that he would like to spend the new year in the bosom of his family? I don't think that's likely. But I think we need to be ready."

Pickles says that winning the election "wouldn't be as simple" if Alan Johnson were leading Labour. Having Johnson as leader might help Labour gain "a few percentage points" in the polls. But he quickly adds a caveat.

"I don't think changing leader would fundamentally change the picture ... because if you look at what Mr Johnson stands for, in terms of economic policy, and social policy and unemployment, there's not a hap worth of difference between him and Mr Brown. So it's all about style, not substance. And I think people are cheesed off – I won't use anything stronger – with Labour and with what they've done. I think it's Labour, not Brown."

In the Labour party MPs are "cheesed off" about Pickles's colleague Lord Ashcroft spending thousands of pounds to help Tory candidates campaign in marginal constituencies.

Under pressure from Labour backbenchers, the government agreed to include a clause in the Political Parties and Elections Act designed to make Ashcroft clarify his tax status (about which he's notoriously secretive) by making it illegal for people who do not pay tax in the UK to donate to political parties.

Pickles insists that, although this law "might well cause problems for the Labour party", it will not have any significant impact on Tory fundraising. And he thinks Labour MPs are missing the point about Ashcroft.

"Labour is very obsessed by the money and they are idiots because it's nice, but it's not vital. The difference that Ashcroft has made is the way he has focused our attention on target seats. It's Ashcroft's intellectual drive that's been important, not his money."

Pickles describes the pamphlet written by Ashcroft after the 2005 election called Smell the Coffee as a "massive" influence on the Conservative party.

Using polling evidence to back his case, Ashcroft argued that the party should resist the temptation to tack to the right. He said instead that it would only win power by attracting "the real core vote – the election-winning coalition of professionals, women and aspirational voters without whom the party risks becoming a rump."

Cameron adopted this approach and Pickles is now using the candidate selection process to give the Conservative party a facelift that might make it more attractive to the swing-vote "coalition" described Ashcroft.

He expects around 17 more Tory MPs to stand down before the general election and he predicts that many of them to be replaced by some of the people with little or no background in politics who have been applying to the party since Cameron declared he wanted to "refresh" the Conservatives with new blood earlier this year.

The Tories have also been using open primaries – final selection meetings open to all voters, not just party members – to select candidates in more than 100 constituencies. This has been taken one stage further in Totnes, where all 69,000 voters in the constituency were sent a postal vote inviting them to pick one of the three Tories on the shortlist. The result will be announced on Tuesday.

Pickles thinks that over the next 10 years open primary selection meetings will become the norm in the Tory party because they will give it an advantage over other parties who restrict candidate selection to the membership.

Why? "Because, one, you get buy in from the electorate. Two, you demonstrate that as a political party you trust the people. And, three, democracy's rather a good thing."