REF correspondence re DECC's use of statistics

Friday, 14 September 2012

In the publication Shortfall, Rebound, Backfire REF raised the issue of misleading statistical information from the Department of Energy and Climate Change, specifically that the headline statements, including those of the then Secretary of State, Mr Huhne, to the House of Commons, were misleading. It was stated that the net effect of the UK's climate change policies would be to reduce the average household energy bill in 2020, whereas close reading of the Department's own models showed clearly that 65% of households were expected to see their bills increase. REF raised this issue in correspondence with the UK Statistics Authority. The correspondence is as follows.

Misleading Statistical Information from the Department of Energy and Climate Change

1. The Renewable Energy Foundation is a registered charity promoting sustainable development for the benefit of the public by means of energy conservation and the use of renewable energy. REF is supported by private donation and has no political affiliation or corporate membership. We place a strong emphasis on the quality of public data in the energy sector, and are one of the leading publishers in this field (see www.ref.org.uk)

2. In the course of work undertaken for Calor Gas Ltd, and published as Shortfall, Rebound, Backfire on the 21st of May, our wholly owned subsidiary consultancy discovered that the headline message of DECC’s Estimated Impacts of Energy and Climate Change Policies on Energy Prices and Bills (2011), part of the department’s Annual Energy Statement 2011, was based on a misleading statistical technique.

3. Specifically, the department inappropriately used a mean to derive its claim that the average household would see a 7% reduction (£94) in its energy bills, compared to the bill in the absence of policies.

4. We examined a chart in Estimated Impacts in the light of remarks on a subsequent page and found that in fact the Department’s own model predicts that 65% of households would be worse off, and only 35% better off, which is not consistent with the headline message as reported in the Secretary of State’s introduction to Estimated Impacts and in his statement to the House of Commons.

5. While arithmetically correct, the mean produces an extremely misleading description of the Department’s modeled distribution, and should not have been used.

6. We cannot avoid the conclusion that the Department’s headline statement in the Estimated Impacts, and in associated publicity, was not only misleading, but was intended to mislead.

7. Consequently, and because the material misrepresented in the headline finding of Estimated Impacts is of such intrinsic importance, we feel that this matter deserves your attention, and that of the National Statistician. (To save time I have copied this letter to Jil Matheson.)

8. A number of other statistical and methodological problems have been discovered in DECC’s work. To give one example, we found that a key infographic from Estimated Impacts, and prominent in DECC’s discussions, is not drawn to scale, and while numerical values are given on the graphic the practical effect is to conceal the department’s extreme dependence on one set of policy measures to protect consumers from the costs of its policies. This is extremely unsatisfactory.

9. DECC has been unable to defend itself, either in the person of Mr David Purdy, who attended the launch of our study at the Athenaeum, or subsequently. However, it seems that DECC has no intention of correcting their misleading information, and I am therefore writing to you to make a formal complaint and to ask that you make an investigation of this matter, if only to prevent such misleading statements appearing in future issues of the Annual Energy Statement.

10. Together with this document I am enclosing a copy of our study, Shortfall, Rebound, Backfire, and we wish you to regard the detailed discussion in paragraphs 54 to 75, as being supplementary to this letter of complaint concerning the inappropriate averaging technique.

11. Please also see paragraphs 77 (on lack of base data behind charts, and over-rounding resulting in zero in some table cells) and 244 (on the misleading infographic) for a description of further problems in DECC’s work.

12. You may wish to note that Mr Purdy of DECC suggested that we visit the department to discuss this matter, but our efforts to secure a meeting have so far been unsuccessful, due to holidays I understand, and I am reluctant to delay this letter to you any longer.

13. I look forward to your reply, and trust that you will be able to give me some indication of what course of action you intend to take.

Department of Energy & Climate Change Duncan Millard Head of Statistics & Head of Profession for Statisticians Area 6D, 3 Whitehall Place London, SW1A 2AW 4 July 2012

Dear Mr Constable

I am writing in response to your letter to the UK Statistics Authority dated 31 May 2012, which was forwarded to me via Andrew Dilnot’s response to you on 22 June 2012, regarding your concerns about the analysis presented in DECC’s report Estimated Impacts of Energy and Climate Change Policies on Energy Prices and Bills published in November 2011 and the subsequent presentation of its results.

The November report represented the third annual iteration of this analysis, the second alongside the Department’s Annual Energy Statement. The analysis is produced by economists in DECC and is based on policy evidence which is published and scrutinised in various impact assessments (IAs) and fossil fuel price and carbon price assumptions that are published on DECC’s website.1 While produced by the economics team in DECC, it nonetheless aims to use the best statistical practices. It is aimed at promoting transparency and improving the public understanding of the distributional impacts of policies on energy bills. The Department welcomes external feedback on this analysis and how it can be improved and this feeds into considerations for improvements in subsequent publications.

I have considered the points on statistics that you have raised and worked with economist and policy colleagues to also respond to the key policy points in your letter including further points you raise from your report Shortfall, Rebound, Backfire. These responses are in the Annex to this letter.

We do not believe we have provided misleading information nor have we intended to mislead. However, we are open to feedback on our analysis from external stakeholders and will take this feedback into consideration for our next publication. In particular, we will:

• Be looking to set up a DECC mailbox for this price and bill impacts analysis to make it easier for external parties to contact the relevant analytical team with enquiries;

• Take on board feedback we’ve received over the past year in the development and presentation of new infographics for the future;

• Consider further ways in which to present the distributional analysis based on feedback we have received over the past year; and

I hope this letter, including the Annex, provides a satisfactory response to the points you have raised. If you would like to meet with our analysts to have a working level discussion, please let me know and this can be arranged.

I am copying this letter to Andrew Dilnot, Chair of the UK Statistics Authority, Jil Matheson, the National Statistician and Richard Alldritt, the Authority’s Head of Assessment.

Yours sincerely

Duncan Millard

ANNEX A

“The Department inappropriately used a mean to derive its claim that the average households would see a 7% reduction (£94) in its energy bills, compared to the bill in the absence of policies” – Letter, point 3, and “The Department’s headline statement…was intended to mislead” – Letter, point 6.

In our November report, we state that “By 2020 households will, on average, save £94 (7%) on their energy bills compared to what they would have paid in the absence of policies.” This captures the combined effect of policy costs and policy savings averaged over all households to create a single indicator and captures the fact that households which do benefit from at least one policy measure2 stand to save more than the increase in bills faced by households which do not. The use of a mean is consistent with how domestic energy bills are calculated and presented in DECC’s Quarterly Energy Prices publications3 and relevant Energy Trends articles.4

The November report acknowledged that the impacts of policies will differ across household and industrial users as well as within the household sector. To that end a section of analysis (Section 4.4. and Annex G) is dedicated to looking at the impact of policies on energy bills across the household distribution including between those that do and do not take up measures. In addition, we provided an accompanying explanation via the DECC blog5 to explain each component of the 7% figure and the key infographic , as well as identifying where impacts are expected to affect all or some households. At the end of the blog we also provided a simple-to-follow estimate of the impact of policies on average household energy bills where no measure is taken up and then showed the scale of savings that might be achieved if a household takes up a range of insulation measures.

The headline indicator was not intended to mislead. It provides a single weighted average number accompanied by a range reflecting sensitivity analysis around fossil fuel prices. This indicator can be tracked over time as a single projection with key sensitivities. The November report and accompanying blog also provide clear transparency of the distributional impacts of policies on energy bills.

An average (mean) indicator has also been used by the independent Committee on Climate Change (CCC) in their publication Household Energy Bills.6 Accounting for the fact that the CCC’s analysis begins at 2010 when household energy bills were lower on average than in 2011 (the starting year for DECC’s analysis), the CCC’s conclusions are consistent with DECC’s – that successful implementation of energy efficiency measures would largely offset costs of financing low-carbon investments and increased gas prices, resulting in average energy bills in 2020 broadly at their 2010 levels.

“We decoded a chart in Estimated impacts...and found that…the Department’s own model predicts that 65% of households would be worse off, and only 35% better off, which is not consistent with the headline message…” – Letter, point 4.

No decoding was necessary. We state below the aforementioned Chart 11, in paragraph 56, that “the modelling suggests that around 40% of households in the bottom three deciles could benefit from at least one of these measures. This is greater than in the other deciles , where just over a third of households are expected to benefit”.

Because the net saving for households that benefit from at least one measure in 2020 is larger than the net cost incurred by households that do not, the impact on average energy bills is lower than in the scenario without policies. However, the savings are concentrated in those households that benefit from at least one measure.

“A key infographic is not drawn to scale…to conceal the Department’s extreme dependence on one set of policy measures to protect consumers from the costs of its policies” – Letter, point 8.

The infographic is an alternative and simplified graphical representation of Chart 9 in our November report where appropriate scaling has been applied. The aim of the infographic was not to mislead, which is why we have included both the monetary values of each policy contribution within it as well as the aforementioned accompanying blog post.

November was the first time we have used infographics in such a way for this work and these are currently being reviewed, taking into consideration feedback we have received on their ease of interpretation. We will also take your feedback into consideration in developing and presenting any new infographics .

We have made a conscious effort to source all our assumptions and data. Our policy specific assumptions are based on analysis carried out for individual policy IAs and these are referenced with hyperlinks in turn in the policy annex (Annex B) of the November report. The tables in Annex E and F of the November report form the base data behind the majority of the charts. Where base data has not been included in the report (as a conscious effort to not adversely affect readability, understanding and length) we can provide it on request, except where the data is only available to DECC under a licence agreement.

“The data [in the tables] has been over-rounded” – SRB, para 77.

In order to avoid spurious accuracy, we round results for households to the nearest £1/MWh for prices and £1 for bills. By implication, any policy impact which represented between +/-50p/MWh on price or +/-50p on bill (equivalent to less than 0.1% of an average household gas or electricity bill) was labelled as zero in the report. It is worth noting some of the impacts which were rounded to zero were actually negative, i.e. estimated bill/price savings from policies. In addition, any policy that had no impact at all in a given year was labelled as N/A to distinguish between the very small non-zero impacts. Any statistical transformations (e.g. summations and divisions) presented were calculated using the unrounded numbers. Each table also included a footnote to indicate the use of rounding.

We will, however, consider the use of rounding to 1 decimal place where the existing rounding policy yields a value of 0. However, this will need to be considered in conjunction with the need to avoid spurious accuracy. It is also worth noting that some of the policy impacts could still round to 0.0 even at 1 decimal place.

“The bulk of savings assumed by DECC are in relation to electricity bills…This is strangely unbalanced…implausible and arbitrary” – SRB, para 80.

We have sourced all the evidence from which our efficiency savings assumptions are based. ( see Annex B).

In addition, it’s worth noting that a given MWh electricity saving will lead to a significantly larger bill saving than a given MWh gas saving simply on the basis that each energy unit of electricity costs 3-4 times more than the same energy unit of gas. Many of the electricity savings are also universal savings meaning all households stand to benefit compared with savings related to the take-up of specific measures.

“The government’s proposals for protecting consumers against policy-induced bill increases are fragile and subject to underperformance in only one policy area” – SRB, para 81.

Our efficiency savings assessments included evidence-based assessments of savings per measure netting off the estimated effect of comfort taking and underperformance. Further information on the assumptions used can be found in the evidence base of each policy’s IA (which have all been sourced in the November report in the policy annex – Annex B).

“DECC’s assumed bill without policies is grounded in assumptions with regard to fossil fuel prices that may well be wrong” – SRB, para 84.

As your report acknowledges, there are uncertainties around policy costs relating to fossil fuel prices. As such, we have undertaken and included in the November report (Section 6 and Annex H) sensitivity analysis around a range of fossil fuel price assumptions. The assumptions themselves are in Annex A and published on the DECC website .7 DECC’s fossil fuel price scenarios are within the range of other independent forecasts.

“We can calculate that on the government’s own view roughly 65% of households will see an energy bill increase of about £47 a year…” – SRB, para 85.

This is consistent with analysis we have put on the DECC website and consistent with the response to the second point above. In our aforementioned blog, we show that, for households which do not take up a policy measure and are not eligible for a Warm Home Discount, energy and climate change policies are, on average, estimated to lead to energy bills which are £44 higher in 2020 than they would have been in the absence of these policies. With savings being greater than additional costs, this is also consistent with the statement that, on average, household energy bills will be lower than compared to a no policy scenario.

“DECC’s claim [on electricity savings from Products Policy in 2020]…relies on an inflated savings value, namely the assumption that the value of a saved kWh in 2020 is 18p, whereas the cost of a kWh including the Government’s climate change policies is 14p” – SRB,

In the methodological Annex (Annex C), page 60, we clearly state that “energy efficiency savings are valued at final (after all policies) energy prices and the cost impact of policies is estimated using baseline (before all policies) energy consumption”. This is an algebraically correct formula to split out the costs and savings of policies on bills such that the final bill equates to the final price multiplied by the final quantity. By “before policy energy consumption” it means we multiply the price impact of policies, such as the RO, by consumption before the effect of efficiency savings, i.e. a higher consumption value. This has the effect of overstating the marginal cost impact of policies on bills but avoiding any issue of double counting the value of efficiency savings in the arithmetic.

A simple algebraic representation of this is provided below:

C0 = base consumption (no policies)

C1 = consumption impact of policy 1 (generally < 0)

P0 = base price (no policies)

P1 = price impact of policy 1 (generally >0)

Final bill = (P0+P1+P2 )( C0+C1+C2)

This is equivalent to:

P0 C0 + P1C0 + (P0+P1+P2 )C1 + P2C0 + (P0+P1+P2)C2 or

Base bill + Impact of policy 1 + Impact of policy 2

In the example where policy 1 is the RO (so C1 =0) and policy 2 is Products Policy (so P2 = 0), this simplifies to:

“[Products policy saving] implies appliance replacement on a very large scale in a short time” – SRB, para 286.

This is not the case. Defra’s analysis does not assume increased replacement rates in order to achieve the savings. The analysis assumes that the products are replaced at the end of their technological life.

“The expense of appliance replacement is not taken into account in DECC’s assessment of policy costs to the consumer” – SRB, para 287.

This is correct. We have not and we have been explicit about this in the main body of the November report (see paragraph 29e) as well as the policy annex (Annex B). These costs are not incurred via the energy bill and therefore are not reflected. The DECC report explicitly focuses on costs and benefits affecting energy bills and acknowledges that policies deliver wider costs and benefits beyond their effects on energy bills.

The Products Policy savings represent two tranches of measures. The first tranche are already in place, the second are still to be finalised. As such a conservative estimate of the savings of the second tranche was provided to DECC by Defra . In other words, where Defra highlight uncertainty around the savings delivered by the second tranche of measures, they have made cautionary adjustments to the savings they provided DECC to account for the fact that the scope, stringency and timing of these measures is still being finalised.

Thank you for your letter of the 22nd of June, and for forwarding my points to Mr Millard, Head of Statistics at DECC. I have now received Mr Millard’s response, which I understand has also been sent to you, to Jil Matheson, and to Richard Aldritt.

1. Overall, we are pleased that DECC has engaged with the substantive issues raised in our original letter, and in the relevant pages of our study Shortfall, Rebound, Backfire.

2. Mr Millard’s letter gives us reassurance that several of these points, for example the over-rounding in some tables and the scaling faults in the graphic, will be reconsidered and rectified in future issues of the Annual Energy Statement. This is positive and to be welcomed. We are also encouraged to see that DECC will be creating a dedicated email channel for inquiries relating to the AES.

3. However, Mr Millard’s letter does not, in our view, satisfactorily address our principal concern, namely that there is a serious and misleading inconsistency between the underlying statistical content of the Annual Energy Statement (2011) and the headline summary as presented to Parliament and to the media by the then Secretary of State.

4. Speaking to the House of Commons, Mr Huhne said, “By 2020, we expect household bills to be 7% – or £94 – lower than they would otherwise be without our policies” (Hansard, 23.11.11, Columns 300–301).

5. In the introduction to part of the AES, Estimated Impacts (2011), p. 3, Mr Huhne wrote:

A net saving on average from policies on household energy bills is expected from around 2013 and, over the remaining lifetime of this parliament (2012­-2015), households are estimated to be saving on average on their energy bills compared with what they would have had to pay if we did not pursue these policies. By 2020, households are estimated to be spending, on average, 7% less to heat and power their homes compared to what they would be paying in the absence of policies.

6. As noted in our original letter, and in Shortfall, Rebound, Backfire, this misrepresents the fact that access to the predicted savings is not equal for all households. The distributional impact on households as modeled by DECC itself is revealed only by careful reading of Estimated Impacts and further calculation from its data: namely, that 65% of households would in fact see an increase (i.e. those households with access only to universal policies), and only 35% see a decrease in bills (i.e. those households with access to both universal policies and at least one of the selective policies).

7. We imagine that the majority of the public would be surprised to hear that DECC believes the headline statements quoted above are a reasonable representation of this underlying statistical distribution. On the contrary, they would conclude that the headline statement is misleading.

8. In his response to us Mr Millard writes that the headline statement “captures the combined effect of policy costs and policy savings averaged over all households to create a single indicator”. Mr Millard continues in a subsequent paragraph, “The headline indicator was not intended to mislead. It provides a single weighted average number”.

9. However, Mr Millard does not justify the use of a single indicator to describe the complex distributional effect revealed by analysis of Estimated Impacts. We remain of the view that such a single indicator should not have been used since it misrepresents the simple fact that most households are predicted to be worse off.

10. It would be valuable to know the view of the National Statistician and the analysts in the UKSA on how such a distributional impact should best be represented to the public.

11. Furthermore, we are concerned to note that Mr Millard confirms our suspicion, expressed in Shortfall, Rebound, Backfire (para 74) that DECC’s single indicator is a “weighted average”. We are also concerned that “average” and “weighted average” are used synonymously.

12. The use of a weighted average is not made clear in Estimated Impacts and the procedure for weighting is not disclosed, both of which matters are extremely unsatisfactory, in our view.

13. Mr Millard defends DECC’s practice by suggesting that the Committee on Climate Change has also used an “average (mean) indicator” in its study Household Energy Bills. (It is not clear whether this is a weighted average.) However, far from being reassured by this observation, we feel that the practice of the CCC should also be examined with a view to ensuring that it is not misleading in the way that DECC’s use of a single weighted average indicator surely is.

STATISTICAL INFORMATION FROM THE DEPARTMENT OF ENERGY AND CLIMATE CHANGE

Thank you for your further letter of 10 July concerning statistical analysis produced by the Department of Energy and Climate Change (DECC).

Following on from my earlier reply to you, I was interested to read the letter from Duncan Millard, Head of Profession for Statistics at DECC, and I hope this provided reassurance on a number of points in your letter and report.

I have asked Authority officials to look closely at the statistical statements made in the Annual Energy Statement to which you refer, as well as the Ministerial statements made in the House of Commons in November 2011. We note that the DECC report, and the related Ministerial statements, were the product of economic modelling and forecasting based on different sets of assumptions and, as such, seem to fall outside the remit of the Authority in respect of official statistics, as it is defined in legislation. However, we have checked some of the points you make in your letter and agree that the modelled data suggest that 65 per cent of households may expect higher energy bills in 2020. We understand that DECC has recognised in the Estimated Impacts document that the distribution of savings across households will be uneven.

We also note the point made in paragraphs 12 and 13 of your letter of 10 July, in respect of the use of a “weighted average”. We have asked DECC officials to respond to you directly on this point.

I am copying this letter to Jil Matheson, the National Statistician, Duncan Millard, Head of Profession for Statistics at DECC, and Richard Alldritt, the Authority’s Head of Assessment.

Subject: Fw: Reply from Andrew Dilnot to John Constable - 3 August 2012

Date: 8 August 2012 10:24:44 GMT+01:00

To:
This e-mail address is being protected from spambots. You need JavaScript enabled to view it

Dear Mr Constable

Thank you for your phone call this morning. You asked for clarification on whether the Authority was still examining the Annual Energy Statement, or whether its investigation had concluded. I have checked with colleagues here and can confirm that the Authority's investigation has concluded, and that its findings are reflected in the third paragraph of Andrew Dilnot's letter to you of 3 August.

From: "Energy Price and Bill Impacts"
This e-mail address is being protected from spambots. You need JavaScript enabled to view it

Date: 22 August 2012 14:38:47 GMT+01:00

To:
This e-mail address is being protected from spambots. You need JavaScript enabled to view it

Subject: Response from DECC on policy impact on bills analysis

Dear Mr Constable,

As per Andrew Dilnot’s letter of 3 August 2012, I am writing to respond to your points made in paragraphs 12 and 13 of your letter of 10 July 2012, in respect of the use of a “weighted average”.

Our headline figure relates to the average (mean) impact of policies on household energy bills. This is calculated as follows:

a) The average impact on energy prices is calculated by dividing the total cost of policies by the total volume of energy consumed (after energy savings).

b) The average impact on energy consumption is calculated by dividing total energy savings from policies by total energy consumption (before energy savings).

c) These impacts are then applied to our baseline (no policy) prices and consumption levels for a household with a representative average (mean) consumption level to calculate an average impact on energy bills.

In Duncan Millard’s reply of 4 July, he mentioned that the result was implicitly weighted because, although the majority of households did not receive a measure, the estimated increase in their energy bill as a result of policies was smaller than the estimated reduction in energy bills for those households which did receive a measure. This difference was significant enough to result in a negative average (mean) impact. To illustrate this with an example, consider three households. One household receives a measure and the net impact of policies in 2020 is to reduce their bill by £200. The other two households do not receive a measure and the net impact of policies is to increase each of their energy bills by £40. The resulting average impact is (-£200 + £40 +£40) / 3 = -£40.

The estimated headline average impact on household energy bills is calculated by evaluating these energy price and energy savings impacts at the average levels of energy price and energy use. The calculation ignores the second-order effects for simplicity. Also, no equity weighting is applied, but if such weighting were applied it would show a larger negative average impact on bills because lower income groups on average benefit more from energy bill saving measures than higher income groups.

The Committee on Climate Change have used a similar methodological approach in determining the average impact in their analysis.

If you would like to meet to have a working level discussion about this analysis, please let us know and this can be arranged.

All further queries regarding this analysis can be sent to the email address:
This e-mail address is being protected from spambots. You need JavaScript enabled to view it
.