Experts agree new method to assess camera effectiveness

The DfT and the RAC Foundation have endorsed a new method for measuring the effect of speed cameras on road casualties, according to an article in Local Transport Today (LTT).

LTT says the ‘four-time period’ method, which aims to isolate the effect of the cameras from random variations, has been used by Slough-based engineer Dave Finney and Professor Richard Allsop of University College London in reports examining camera effects.

Dave Finney explained his method in a report analysing data from mobile speed camera sites in the Thames Valley Safer Roads Partnership area, in which he separated the casualty data into four time periods:
• A pre-Site Selection Period when no cameras were present
• The Site Selection Period (SSP) – the period of time in which accident/casualty levels influenced the decision that a camera should be sited at the specific location
• ASBiC (After site selection but before installation of the camera)
• After camera installation

Mr Finney ignored the SSP period data because this would contain the influence of selection bias – i.e. an abnormally high numbers of accidents (the ‘regression to the mean’ – RTM – effect). He therefore combined casualty rates in the pre-SSP and ASBiC periods to determine the mean collision rate before the start of speed camera operations. This was then compared with the rate after camera installation.

Mr Finney claimed the method proved “beyond reasonable doubt that the entire reduction in killed and serious injury casualties at mobile speed camera sites actually occurred due to ‘regression to the mean’, and not as a result of mobile camera operations”.

Professor Allsop also used a slightly different four-time period method in a report giving advice about how to interpret casualty data at speed camera sites, published by the RAC Foundation earlier this year and revised earlier this week.

Talking to LTT, Professor Allsop said: “The main change is in the recommendation I make (to users of data who do not have knowledge of the SSPs for the cameras they are investigating) about the assumption they need to make about the SSP to apply my method.

“I recommended previously that they assume the SSP to be the last three calendar years before establishment of the camera. In the light of helpful comments and analysis by Professor Mike Maher [of the University of Leeds], fuller consideration of Dave Finney’s work and sets of recorded SSP for several partnership areas, I now recommend that they assume the SSP to be the first three of the last four calendar years before establishment of the camera.”

Professor Allsop has reworked his calculations with the revised assumption. His estimate of the reduction in personal injury collisions attributable to cameras across the nine partnership areas studied (previously 15%) is now 14%, and the estimate for fatal and serious collisions (previously 27%) is now 22%.

Explaining that the changes bring his method closer to Dave Finney’s, Professor Allsop told LTT: “I stand by my assessment that my method largely allows for regression to the mean. I recognise that some of the reduction my method attributes to the effect of cameras may still stem from RTM, but I don’t believe that this can be more than a small proportion.”

Tim Stamp, the DfT’s head of statistics, also endorses the four-time period method in a letter to retired engineer and speed camera critic Idris Francis in which he says: “The basic conceptual framework for taking account of regression to the mean (RTM) in safety camera analysis, as used by both Professor Allsop and Mr Finney, strikes me as straightforward and logical."

Mr Stamp goes on to say that a large-scale study of speed camera effects could be invaluable: “As site-level camera data continues to be made publicly available, it’s easy to see real value in a robust analysis that encompasses a wider range of camera sites and geographic areas, and that takes account of RTM and other relevant factors such as trend.”

Duncan, good question: answer – not directly! My FTP method can only separate RTM effects from scheme effects, it cannot separate the effects of the interventions within schemes (theoretically it could if many schemes had combinations of interventions but this data probably does not exist). Effects within schemes can theoretically be separated using scientific trials but in practice there generally needs to be greater numbers of intervention sites within scientific trials for results to be significant.

I consider the FTP method to be relatively simple, obvious even, but I'd like to know if RSPs also view it this way? Does the method shed light on RTM, does it make clear how and why it occurs and, therefore, why the method is capable of fully excluding RTM effects from results?
Dave Finney, Slough

Does this new method take into account other interventions that happened similtaneously? A good example is the A452 between Balsall Common and the Stonebridge roundabout which was a fairly lethal stretch of road due mainly to the number of crossing gaps in the central reservation. These gaps were closed at the same time as the speed limit was reduced and cameras installed. The safety partnership went on to trumpet the success of the cameras and the lower limit when it was clearly the closing of the gaps that were entirely responsible for the improvements.
Duncan MacKillop, Stratford on Avon

Fair enough, but the headline does say "...new method to assess camera effectiveness" so the DfT should qualify it by emphasising that it is not a wholly comprehensive assessment of their effectiveness as most of their beneficial efffects can't actually be measured, only a relatively narrow focus. I know the authorities from the outset did push the site specific benefits too much and in doing so they made a rod for their own back.
Hugh Jones, Cheshire

The primary focus of the authorities (and of publicity) has been on the site effects of speed cameras, largely because cause and effect can be established. This is where my FTP method is capable of producing vastly improved quality of results.

As you say, Hugh, speed cameras may also have area-wide effects, both positive and negative, but these are harder to determine because there are so many other factors to consider. I have a report on area-wide effects on my website, along with a suggested method to determine the magnitude of these effects:
http://speedcamerareport.co.uk/05_gb_road_safety.htm

The government’s largest speed camera report (4YE) states (p33) “Whilst it would be desirable to include … some explicit allowance for regression-to-mean, no reliable method has yet been established for doing so.” The FTP method may finally be the “reliable method” not available back then, although it is still only 2nd best to running scientific trials.
Dave Finney, Slough

Sorry to spoil the party, but going beyond the sphere of influence of individual camera sites, what about the accidents that have been prevented from happening somewhere else, due to subsequent driving bans, speed awareness courses and drivers simply learning from their mistakes and improving? I’ve known drivers honest enough to self-report that they have tightened-up on their driving due to the prevalence of speed cameras. These wider and long term effects are like a lot of road safety interventions – they may not be measurable, but they can still work.
Hugh Jones, Cheshire

This is great news, and completely unexpected. The DfT haven't contacted me so perhaps my FTP (Four Time Periods) method is easily understood just from my website:
http://speedcamerareport.co.uk/04_rtm.htm

It has taken a long time to get to this point. I developed the FTP method in order to fully exclude RTM effects from results at intervention sites (not just speed cameras) and it was on my website in 2008. I received the verified database of collisions at Thames Valley speed camera sites the following year (2009) but I didn't publish my report using the method until last year (2012). It was only after Professor Allsop used the method in his 2013 RAC report that I gave it the name “the Four Time Periods method”.

The number of people who understand the method, its strengths and weaknesses (and how to apply it accurately) is small but, it seems, growing. The problem of how to exclude RTM effects has taken over 20 years to get this far but it's now moving in the right direction.

I would like to thank everyone involved so far and hope we can contribute to improving the quality of data analysis so that limited resources can make the greatest difference in saving lives.
Dave Finney, Slough

When the DfT told me RTM was ignored in camera analysis because no reliable method had yet been established for doing so, I secured a meeting where I showed graphs of results from many thousands of qualifying sites taken from my database of 220,000 such examples, based on several million accidents. They consistently confirm Dave Finney's findings at 75 sites that average accident numbers (after trend) invariably revert immediately after the site selection period accurately to pre-site-selection levels and do so at least a year before cameras are normally installed.

There is no need to estimate RTM percentages. Instead camera effect (if any) is identifiable as any subsequent change after the RTM fall, necessarily after camera installation in the form of a step function with time constant of weeks or a few months as awareness increases. No such function immediately after installation means no camera effect regardless of what happens in the longer term.

Problem solved - though only if applied to reliable data. I am in no doubt that when this is done results will be closer to the Finney 0% than the RACF 22% but will in any case be far smaller than previously claimed (and massively less in cost effectiveness terms than vehicle activated signs).

That the DfT, it seems, proposes to do nothing unless I submit a Peer Reviewed paper (on nothing more than simple arithmetic) suggests that they prefer to kick inconvenient facts into the long grass.
Idris Francis Fight Back with Facts Petersfield