James Ball sent me the data for the Russian election vote counts this morning and asked me to test whether it deviates from Benford’s law, a test that can give a hint at whether numbers are the product of fraud. Posted below is my analysis, and also a check for last digit preference, which is another method for spotting sneakiness.

You might remember Benford’s Law from this post, it’s a good way to check if data has been faked, and was used on shady Greek economic data submitted to the EU:

Imagine you have the data on, say, the population of every country in the world. Now, take only the “leading digit” from each number: the first number in the number, if you like. For the UK population, which was 61,838,154 in 2009, that leading digit would be “six”. Andorra’s was 85,168, so that’s “eight”. And so on.

If you take all those leading digits, from all the countries, then overall, you might naively expect to see the same number of ones, fours, nines, and so on. But in fact, for naturally occurring data, you get more ones than twos, more twos than threes, and so on, all the way down to nine. This is Benford’s law: the distribution of leading digits follows a logarithmic distribution, so you get a “one” most commonly, appearing as first digit around 30% of the time, and a nine as first digit only 5% of the time.

Benford’s law doesn’t work perfectly: it only works when you’re examining groups of numbers that span several orders of magnitude, for example, so for age, in years, of the graduate working population, which goes from around 20 to 70, it wouldn’t be much good, but for personal savings, from nothing to millions, it should work fine. And of course, Benford’s law works in other counting systems, so if three-fingered sloths ever develop numeracy, and count in base-6, or maybe base-12, the law would still hold.

I analysed for election fraud using the stats package Stata, beloved of economists and epidemiologists. James has posted graphs from the analysis I did on his blog, where you can also (I assume!) download the data he sent me:

The bottom line is this: the data massively do not conform to Benford’s Law, which would make you think that fraud is at work; but I think we shouldn’t expect the data to conform to Benford’s law, as the numbers don’t span several orders of magnitude. Essentially, this tool won’t work on these numbers. Anyway, here are the graphs and tables. Firstly, a graph showing the leading digit for all vote counts, in blue, against Benford’s distribution in red.

But here’s the distribution of the vote counts, not spanning multiple orders of magnitude:

Here are the tables with the Benford figures, in case you want them, and the second table reports a chi-squared test for whether the real figures deviate from Benford’s distribution (they do, but we could have guessed that from eyeballing the data!):

Lastly, I looked at last digit preference, which is a crude tool for looking at whether someone has made up figures. If they’re all made up by the same human, you might expect them to have some preference for specific digits, as humans are quite bad at generating random numbers. There’s no evidence of digit preference in the data. You might say that’s not surprising, as if there was fraud it would not have been centralised. In any case, I’m only really posting this as an illustration of how you can take data and play around with it.

Lastly, bear in mind that all these tests are only to check if people have made up numbers after the votes have been counted, and won’t detect people stuffing dodgy voting slips into ballot boxes.

For completeness, here’s the Stata code I used. Opensource dorks: in the future, I’m planning to do more public data analyses and detailed walk-throughs, for which I’ll use R, the open source statistics package, but because I did this in a hurry between work this morning I had to use Stata (I know it well so can code without thinking too hard).

++++++++++++++++++++++++++++++++++++++++++
If you like what I do, and you want me to do more, you can: buy my books Bad Science and Bad Pharma, give them to your friends, put them on your reading list, employ me to do a talk, or tweet this article to your friends. Thanks!
++++++++++++++++++++++++++++++++++++++++++

27 Responses

chrisbeeleyimh said,

digitrev said,

Hmm. You’ve shown that the data does not follow the distribution expected by Benford’s law. But what if you ran the same analysis on what one can assume to be non-rigged elections. Say analyse the most recent national level elections in several countries. If none of them follow Benford’s law, then it’s probably safe to assume that election data cannot be expected to follow Benford’s law.

yeah, the data don’t follow the distribution of benford’s law because the numbers generated by this election don’t span several orders of magnitude. if other elections don’t produce numbers that span several orders of magnitude, then i wouldn’t expect them to conform to benford’s law distribution either.

soulinafishbowl said,

@digitrev:
thanks to google, it is easy to discover that the idea of matching vote data with Benford’s law is already tested and discussed. Some people at the Voting Technology Project (Caltech/MIT) has even done some simulation tests (vote.caltech.edu/drupal/node/327):

Abstract:
With increasing frequency websites appear to argue that the application of Benford’s Law – a prediction as to the observed frequency of numbers in the first and second digits of official election returns — establishes fraud in this or that election. However, looking at data from Ohio, Massachusetts and Ukraine, as well as data artificially generated by a series of simulations, we argue here that Benford’s Law is essentially useless as a forensic indicator of fraud. Deviations from either the first or second digit version of that law can arise regardless of whether an election is free and fair. In fact, fraud can move data in the direction of satisfying that law and thereby occasion wholly erroneous conclusions.

Doire said,

What happens if you convert the data to different bases?
I suppose you gain in orders of magnitude but lose sensitivity with fewer points to compare.

For a country wide single-question-election polling stations don’t have to be similar sizes, but might show size chunking based on physical needs, say, 1 box per 300 registered voters. I wonder if there is more information in the mis-match of voters registered and votes posted. It might have a wider range.

pwaller said,

This Arxiv paper covers the topic and shows voter fraud in the 2011 Duma election. They can also purportedly detect ballot stuffing. They make a 2D plot of the voter turnout versus the fraction of votes for a winner over all districts. Fair elections typically have a smooth-ish continuous distribution. The differences in these distributions are quite stark, and there are comparisons of the distribution from a few different countries.

aram said,

bsmith89: That would only work if the people committing fraud were 3-toed sloths, or for some reason preferred the base you transformed to.

Ben: Can you correct for the limited range of the numbers as follows? Discard the bottom and top 5%, take the log, and compare with a uniform distribution on that interval. That seems preferable to saying that it sometimes work and sometime doesn’t.

gorshkov said,

Don’t know if I’m missing anything, but I guess the only way to find out is to ask.

You can’t use Benford’s law because it doesn’t cover orders of magnitude. But – if you used a base other than 10 – say, octal, or something even smaller, so that you *did* have the required changes in magnitude – would I be right in thinking that it *would* work then?

Eponymouswhitey said,

Is it just me, or don’t the Russian boast some of the finest mathematicians around ? That being the case, I would have thought they were more than capable of producing data undetectable by any of the possible fraudulent detection methodologies ?

AndrewKoster said,

I’m not a statistician, but isn’t it quite simply that Bedford’s law works for data that you expect to draw from a uniform distribution and people who try to fiddle the books usually do this in a manner that messes up this distribution?

In that case, it only works if the distribution is also messed up in whatever base you’re changing to. When you reduce the base, you also reduce the sample size for testing whether it follows a power law. Instead of having 9 starting numbers to work with, you have less. For instance, if you reduce it to binary, your starting number is ALWAYS a 1 and Bedford’s law is useless.

I expect that in the case of this data, which only barely spans 3 orders of magnitude according to the graph, the base would have to be so low that testing Bedford’s law becomes impossible.

susu.exp said,

Benfords Law doesn´t work for data drawn from a uniform distribution. It does apply for data sets where the logs of the data are uniformly distributed (over some interval). The more orders of magnitude you span, the less difference the size of the interval makes and for empirical data with more than 5 orders of magnitude you generally recover Benford. Or -to make that a bit more precise – if the logs of the data are uniformly distributed on an interval of integer lenght, then Benfords law holds precisely. If that condition isn´t met then the relative error you make in that assumption is |RND(x)-x|/RND(x), where x is the lenght of the interval and RND rounds to the nearest integer. RND(x) thus corresponds to the number of orders of magnitude the original data span and that error goes down as RND(x) goes up (since the dividend is <1).

But apart from that, your point is valide Andrew: You move to a lower base and will recover benford, but get less data points for comparison (for base 2 the probability for getting a starting 1 is the base two log of 2, which is 1. It´s worth noting that there is one integer that doesn´t start with a 1 though… It´s only almost surely the case that each number starts with a 1).

GrahamB said,

I haven’t tried it on these data, but with the disputed Ukrainian presidential election a few years ago, I assumed that most rigging tactics will increase apparent turnout, so I plotted declared turnout against the percentage for each candidate. It showed that there was a strong relationship between areas of high turnout and a higher vote share for one candidate. Of course, you could argue that his voters were more motivated to get out and vote. It was just a bit suspicious when turnout at a polling station was around 15% higher than any nearby station. It would be interesting to know if Mr Putin did better where turnout was highest.

OperationBarbapapa said,

How about we statistically analyse the geopolitical leanings of the researchers concerned, and see if there is some kind of correlation to the countries chosen to analyse? Or possibly the susceptibility of the researchers to mass-media opinion-forming, advertising, etc. if we are being slightly more kind.
Let’s face it, elections are normally are said to be “disputed” when other countries don’t get the outcome they wanted, even after vigorously funding the usually hopelessly unpopular opposition (hint-that’s why they’ŗe the opposition). Then you call their elected leader a “strongman” until the next election comes around, What does that even mean, visions of handlebar moustaches, stripey victorian gym outfits and moulded black iron dumbells? I notice this didn’t ever happen to Sarkozy 😉

annoporci said,

How about taking a sample of election results with known fraud, if available, and analyzing that to see what sort of distribution emerges, and see if that can be used. We know elections in many countries in Africa are rigged. Chechnya is another example that comes to mind. Not sure if the data is available though.