Tuesday 20 October 2009 06.30 EDT
First published on Tuesday 20 October 2009 06.30 EDT

The Conservative party's open primary for the seat of Bracknell, which Philip Lee won on Saturday, was determined by hundreds of voters. A similar contest in Totnes in August, which was decided by a postal vote rather than an evening of deliberation in a golf club, attracted one in four of the local electorate. By contrast, the Labour party is attracting a median average of just 40 people to its selection meetings.

Feni Ajumogobia and I recently interviewed 101 of Labour's 294 prospective parliamentary candidates (PPCs) by phone and email. Our findings, published by Progress, unearthed a wealth of information about their social and professional background, and the nature of their selection contests. Because we only spoke to a third of selected PPCs, our findings are less complete than other recent surveys by, for example, Madano and the New Local Government Network. But none of these other surveys asked specific questions about the nature of PPC selection.

Consistent with other studies, our research found that the overwhelming majority of Labour's PPCs are still white men. But so long as there is not a landslide victory for the Conservative party – seen as unlikely by respected psephologists – there are likely to be advances in the number of women and non-white Labour MPs. Women make up 27.2% of Labour's 294 PPCs but 20 of the 43 candidates selected to contest seats currently held by Labour are women. Meanwhile, consistent with national percentages, 8% of the Labour candidates we interviewed self-identify as coming from black and minority ethnic (BME) groups. At least four of these candidates are contesting "safe" seats currently held by Labour. This should therefore increase BME representation within the parliamentary Labour party.

Our evidence on the selection process revealed that PPCs faced a range of different circumstances. The 101 PPCs we interviewed included 14 contesting seats held by Labour and 87 contesting seats held by other parties. This is roughly proportional to the Labour selection contests that have already taken place but the small sample means that it is hard to make generalisations. Nonetheless, we can report that between eight and 570 Labour members took part in these selection contests but the median average was just 40. Unsurprisingly, the contests attracted more members in seats that are likely to return a Labour MP.

We also asked how much money candidates had spent. Some candidates will have included more items than others in their submissions. The 26 people who said they spent nothing must have excluded costs like transport and cups of tea. But one candidate spent £4,000 and nine others spent £1,000 or more. The median average for our total group was £90, although only one candidate in a seat currently held by Labour spent that little. As well as spending cash, union support was important, with 71% of successful candidates in our sample enjoying union support.

Although there are some honourable exceptions where more than a handful of people turned out to vote, our findings reveal that the selection of Labour candidates is a minority activity which, on average, attracts fewer than one in four Labour members. There is a growing grassroots movement for the Labour party to adopt primaries, which is now supported by left and right alike, including former London mayor Ken Livingstone, Compass candidate Chuka Umunna, "Red" Ed Miliband, David Lammy, and New Labour stalwarts such as David Miliband, Peter Mandelson and Tessa Jowell.

Our findings support their campaign. Although primaries should not be seen as a panacea – a more proportional system of voting and more democratic policy-making structures within the Labour party are two critically needed reforms – they can enhance the democratic credibility of local candidates and help re-engage voters at the local level, as the Tories have found where they have experimented. The idea that they would lead to a more homogenous parliament does not stack up to scrutiny, Conservative primaries have resulted in a number of female and BME candidates.

Critics rightly ask how to keep costs for the candidates and local parties down to a minimum. The answer is stringent spending caps, which don't currently exist, and holding the contests on the same day as local elections so that there is already an infrastructure in place. If politicians take the argument to the public they may even persuade them of the case for taxpayer support.

The Labour party must now decide whether it wants to see this dwindling number at the local level determining Labour's future candidates or whether to open up the process to a wider group.