The Conservative peer and former deputy chairman on his fascination with polling and why Tory modernisation must continue.

Lord Ashcroft attends a rally in support of Boris Johnson on the second day of the Conservative Party conference in the International Convention Centre on October 8, 2012 in Birmingham. Photograph: Getty Images.

1.You’ve written that in November 2004: "Almost overnight I became fascinated by polling and by what could and could not be achieved by the process." For the benefit of people who are interested in politics, but fail to understand the contribution that polling can make, could you try to explain your fascination with polling?

I think the single most important contribution polling can make is to be a reality check. In fact that was the purpose of those first political polls I conducted in the run-up to the 2005 election. Although Labour were well ahead in the national polls, Maurice Saatchi, the Tory co-chairman at the time, claimed that the party’s private polling showed a Conservative lead in our target seats. That sounded so unlikely to me that I decided to find out for myself. And sure enough, things were even worse in the target seats than they were nationally.

That led me to look in more detail at why exactly the Tories kept doing so disastrously badly in general elections, which came to fruition in Smell The Coffee.

2.And what in your opinion it can achieve?

If they are prepared to take it seriously, it can be incredibly useful in showing politicians where they really are in relation to the voters. Smell The Coffee, for example, was a pretty uncompromising assessment of how people really saw the Conservative Party – it showed that we couldn’t hope to win another election until we changed quite radically. You might say that was blindingly obvious to anyone who was paying attention, but the polling put things in black and white, it made the facts inescapable.

Of course, polling only achieves anything if people are willing to learn from it. You have to be prepared to let it challenge your assumptions. Politicians are prone to assuming – or at least claiming – that people share their preoccupations, and sometimes people mistake the contents of their mailbag for broader public opinion. And there is always a temptation for political parties to talk about the things that they know people already agree with them on. The problem with that is that they might not be the things that people care most about, or that will ultimately move votes. In the 2005 election, for example, the Tories had a huge lead on immigration, so that is what they tried to make the election about. The fact that we were ahead on immigration but behind in voting intention ought to have made it obvious that that wasn’t going to work. The polling can show the things on which you need to convince people – whether it’s the economy or public services or the idea that you’re not on the side of people like them. It can help politicians discipline themselves to talk about the voters’ priorities rather than their own.

Cameron, to his credit, did read and absorb Smell The Coffee. Whatever you think about how he has gone about it, there is no doubt that he recognized that the Conservative Party needed to change. Some politicians are pretty good at staying in denial, though. I remember in the years after the 1997 election, shadow cabinet members would dismiss poll findings on the grounds that they heard a different message on the doorstep– well of course they did; if the rest of Britain were like their constituencies, we’d have more than 165 MPs, wouldn’t we?

People sometimes argue that polls can say whatever you want them to say, which is true up to a point. It’s funny how often lobby groups produce polls that seem to show the public agree with them about everything. The key is transparency. You have to be able to look at a poll in full, all the numbers, how the questions were asked, and judge for themselves what it says. All my data is published on my website. People can disagree with my conclusions – though it’s striking how many do agree, in the media and in every party – but I don’t think anyone has seriously questioned the quality of the research, or accused me of trying to engineer particular results.

3.…and what in your opinion it cannot achieve?

Research can tell you exactly where you are, which is essential, and it can help tell you where you need to be – it can tell you what your strategy needs to achieve, the kind of people you most need to win over, the perceptions or assumptions you need to change. What it can’t always do is tell you how to do it. That’s one of the reasons why pollsters don’t always make good strategists. Henry Ford famously said that if he’d started by asking his customers what they wanted, they’d have said "a faster horse".

4.In Project Blueprint, one of the four tests you set for all Conservative activity is that it "demonstrates leadership". What is the role in this of being in front of the polls, or indeed of getting credit for sticking to your guns when what you are saying or doing is unpopular? Is one of the reasons why so many politicians sound so "inauthentic" that they appear unable to say what they really think about anything, for fear of upsetting people?

People sometimes complain that polling has become a substitute for leadership, but I don’t think that’s the case. At least, that’s not what it’s for. If you went into politics with no idea what you thought about anything, and just relied on polls to tell you what to say, you wouldn’t get very far. People can spot inauthenticity at a thousand paces.

Polls can’t tell you what to say or do. Or perhaps more to the point, politicians don’t have to act in the way polls suggest will be the most popular. There might sincerely think the national interest is better served by going against popular opinion. But when they do, they need to know what they up against, rather than deluding themselves that the people are on their side.

There’s no doubt that voters give some credit to politicians who stick to their guns when they’re doing something unpopular. In fact it remains one of the biggest attractions of David Cameron for people who are sticking with the Tories or considering switching to them. But it’s always a conundrum – people say they like outspoken politicians, but they like them all the more when they’re saying something they agree with.

5.Smell the Coffee ends with the words: "The problem was not that millions of people in Britain thought the Conservative Party wasn’t like them and didn’t understand them; the problem was that they were right." This is such a serious problem that it is hardly surprising it could not be resolved overnight. How far has the party got in addressing it now and what more needs to be done? Have the lessons contained in your analyses of the 2005 and 2010 campaigns been learned?

Some more than others. One of the big failures of the 2005 campaign was a completely unrealistic approach to targeting, where we spread what resources we had so thinly in the pretence that we could win the election that we ended up with even fewer seats than we could have done. I think that lesson has been learned, and the party now has a much more professional approach to picking the seats we want to target and the particular voters we need to reach.

The other one was that we wanted to talk to the voters about what we cared about, rather than what they cared about. I think that point has been grasped but it is easy to lose that discipline so it is a constant battle. There are still some MPs who would happily talk about Europe all day every day.

Last time round, a big problem was the opportunity cost of talking about Gordon Brown all the time, which meant that although most people thought it was "time for change" we hadn’t convinced them that we were the change they wanted. I suppose we’ll see when the campaign heats up whether the party is going to fall back on attacking Miliband.

As for the overall brand problem, that remains a struggle. As I said earlier, Cameron got the point that the party needed to change pretty radically, and he did make some progress. You hear people say that modernisation failed, as though it was an experiment that didn’t work. But the point is that it isn’t finished. I’m not talking about the business with the huskies. In its true sense modernisation is about whether people think you’ve changed enough to understand them and have their interests at heart, and on that score I think there’s clearly still a way to go.

6.Could you say a word about whether it is necessary to insult traditional Tories, including the Ukip people, in order to prove to a wider audience that the party has modernised? It seems to me that the Tories need to become once more the broad church that they were under Harold Macmillan, a leader who at his peak – the 1959 election – managed to be more progressive than Labour, while at the same time offering reassurance to old-fashioned Tories. Macmillan was a businessman who married into the aristocracy, and is remembered for his "grouse moor" image. But in his prime, he was brilliant at attracting working-class support. What if anything can we learn from him?

There are several points there. First of all it’s misleading to assume "the Ukip people" are all traditional Tories; many of them are protest voters as much as anything. Second, I don’t think insulting people is necessary in politics, and is nearly always counterproductive. People notice the way politicians conduct themselves at least as much as what they do. But to show that the party has changed for the better you do sometimes end up making the point firmly that you disagree with some people on your own side. People need to be clear about where you stand.

The Macmillan case is interesting.It’s often said that part of the Tories’ problem is the idea that we’re the party for the rich, but I think it’s slightly more nuanced than that. What Macmillan shows is that you can be the party for successful, comfortably off people as long as you’re not only for them. One of the most damaging perceptions people have of the Conservatives today is not that we are on the side of the rich, it’s that we’re not on the side of people like them. At its best, and most successful electorally, the Tory party has been about spreading opportunity.

7.In Minority Verdict, you write: "When I presented our poll findings to David Cameron early in his leadership he would often say yes, this was all very interesting, but where was our next five per cent coming from?" This remains a good question. Where in your view is the Tories’ next five per cent coming from?

This is what Project Blueprint has been all about. Those who have been attracted to the Tories since the last election – there are some, believe it or not – have been largely from the Lib Dems. There are another group of people, not a very big group but enough to make a difference, who didn’t vote Tory in 2010 and wouldn’t do so tomorrow, but who might consider it some time in the future; two thirds of them are Lib Dems too. For those people, it’s all about the economy. Then you have the much bigger group, about a third of the 2010 Tory vote, who have "defected". Some have gone to Ukip, some to Labour and about 40 per cent of them don’t know how they will vote. There is no single reason why they have gone, but interestingly most of them would rather see a Conservative government with an overall majority than anything else. So for the ones who are in play, I think what they need to see is an overall sense of grip and direction.

The arresting fact I always emphasise when talking to MPs is that there are as many people who voted Lib Dem in 2010 but would consider the Tories as there are Tory voters from 2010 who would consider Ukip. To get a majority we need them both.

8.I was especially struck by your recent identification of "Suspicious Strivers" who "are not sure their efforts will bring the rewards they should". These people are not "ruggedly individualistic" and feel their lives are precarious. Can you say a bit more about how the Conservatives can win the support of these people? Could the right to buy a plot of ground and build one’s own house be as successful as the sale of council houses once was in reassuring people of modest means that the Conservatives are on their side and want to help them to make their way in life?

Certainly the cost of housing is a big issue, especially for young people paying rent or stuck with their parents. I don’t usually get involved with proposing policies. And there is rarely a single answer to these things. For the people you mention we need to show we’re delivering our promises on immigration, that welfare reform is fair, and they need to feel that what we are doing on the economy is on the right track.

9.Charles Moore has suggested in the Spectator that by buying ConservativeHome you have become "a more politically influential ‘press’ proprietor with the Tories than Lord Rothermere or Rupert Murdoch, at a tiny fraction of the cost." Do you think this is true?

I think Charles was being playful. Certainly ConHome has become the place to go for news and opinion about the Conservative Party, and I’m proud of that. But that doesn’t quite make me a Rothermere, in the sense of being able to push the government around on the great questions of the day with the threat of bad headlines. The only agenda we have on ConHome is that we all want to see a Conservative majority at the next election, and different contributors have different ideas about how to achieve it. There isn’t a monolithic line, which I think is one of ConHome’s strengths.

10.Peter Oborne suggested in June 2012 that "Lord Ashcroft and his small but beautifully placed army of editors, columnists and pollsters have started to push David Cameron around". You naturally rejected this. While you conceded that "Some of my polling work has probably made uncomfortable reading for the Conservative leadership," you insisted that "My polling is designed to convey the voters’ views, not mine". It strikes me that it is much better for the Conservative leadership to know what voters think, than to succumb to optimistic illusions. But is the price of being the messenger bound to be that people sometimes want to shoot you?

Not as often as you’d think. I’m sure there are sometimes headlines from the research that the leadership would prefer not to appear, but I think that’s outweighed by the usefulness of what’s actually in it. The feedback I get from MPs is actually overwhelmingly positive – they find it useful to have another source of high quality research so they can sense-check what they hear from the party itself.

11.You have won the respect of your fellow pollsters: Peter Kellner told me, "I think the polling he does is terrific, really good, intelligent, open-minded." But does the Ashcroft brand get the wider recognition it deserves?

I think the research is becoming a fixture in the political world – people know it is worth looking at and it is referred to more and more by commentators, which has obviously been gratifying. I think it’s recognised that what I am offering is not just punditry, it has an evidential base behind it. In fact it has opened some interesting doors. I’ve had some very stimulating meetings with shadow cabinet ministers and other Labour MPs, and some Lib Dems, who see the value in what we’re doing. One Lib Dem blogger was so impressed by Minority Verdict [the review of the 2010 election] that he put it on his list of recommended books for Christmas.

12.On a different subject, I was shocked by your poll about how much children know, or don’t know, about the Second World War. My own children (aged 17, 13 and 11) and their friends bear out the truth of your finding that most people in this age group do not even know whether the Battle of Britain was fought on land, sea or in the air. What can be done about this?

Yes, that survey was done to help publicise the launch of the Bomber Command Memorial in Green Park. The results actually helped to underline why things like the Memorial are so vital. Obviously history must be taught properly in schools but I think there is a wider responsibility to ensure that things are remembered, that people tell their children and grandchildren about our history. Having tangible things like the Bomber Command Memorial, and the Imperial War Museum which I am also involved with and does a magnificent job, are an important part of helping to ensure these stories are remembered and passed on.

Andrew Gimson is the author of “Boris: the Rise of Boris Johnson” (Simon & Schuster, £7.99)