Low-income programs can benefit from better tracking data.

The Weatherization Assistance program (WAP) helps low-income households reduce their energy costs by making their homes more energy efficient. During the last 30 years, WAP has provided weatherization services to more than 5.5 million families. A DOE program, WAP also receives funding from various other sources, including the Department of Health and Human Services Low-Income Home Energy Assistance program (LIHEAP), and local utilities. WAP provides services that are in great demand, and it has performed extremely well in providing these services. However, the program would benefit greatly from better data tracking and measurement.

In describing the efficacy of low-income programs, DOE specifies that “the single most important metric is the number of homes weatherized.” While the reasons for using this metric are understandable, it has often produced mediocre results.

In general, the aim of low-income programs is to reduce the household’s energy burden, defined as energy cost divided by income, where energy cost is equal to energy consumption multiplied by the price of energy. Reducing the cost of energy to the consumer can be achieved in one or more of three ways. The first is to reduce energy consumption by installing weatherization measures or by educating the consumer on ways to save energy. The second is to reduce the price of energy by discounting rates to low-income households. The third is to reduce the consumer’s energy bills by paying a portion of the bill through cash energy assistance. In other words, low-income programs are not limited to weatherization, and there are measurement metrics appropriate to each program type (see Table 1). The measurement and evaluation process converts raw data collected in the course of implementing programs into information that can be used to improve the management and control of the program. You cannot manage what you cannot control and you cannot control what you cannot manage. As economist John Kenneth Galbraith once said, “things that are measured tend to improve.” Weatherization programs are no exception.

While WAP uses various sources of funding to install energy efficiency and conservation measures in low-income housing, the goals of these various sources are not necessarily in agreement. Some programs have cost-effective energy savings as their primary goal. Utility funding tends to be mostly in this camp. WAP also includes health and safety, affordability, and a reduction in the energy burden among their goals.

Some weatherization programs are comprehensive, including energy education and low-cost measures. Some use deemed savings while others require detailed audit estimates. Many programs require that the savings be validated against actual consumption from the year before.

Impact evaluation is the most common method used to assess program performance. This method examines the effects of the program on key indicators, such as energy savings.

Accurate Results

The main objective of weatherization is to save energy. Changes in energy consumption from one period to the next can be caused by a variety of factors. Those factors include, but are not limited to,

• weather; • changes in prices; • changes in the economy; • changes in household characteristics (e.g., number and age of people in the home); and finally • the weatherization program itself.

The challenge is to isolate the impact of weatherization (program impact) from the other, nonprogram, factors. Weather, for example, is a significant determinant of energy consumption. Changes in the weather from one period to the next can cause significant changes in energy use. Therefore, a direct comparison of utility bills before and after weatherization may be misleading. The observed difference in utility bills may be caused by any of the above factors. For example, if energy consumption decreases by 2,000 kWh postweatherization, but the weather postweatherization was significantly hotter, then at least part of the decrease is probably due to the change in the weather.

The impact of weather can be isolated through weather normalization. The most widely used tool for this purpose is the Princeton Scorekeeping Method (PRISM). PRISM requires utility data on energy consumption and meter read data. Meter read data are used to calculate the number of days covered by the bill. Meters are not always read on the same schedule. Sometimes the billing period may cover 30 days, sometimes 35, and so forth. Average daily consumption is then computed and the data are arranged in 12 average daily consumption values representing each of the 12 months of the year. The idea behind PRISM is rather simple. Energy consumption has two components: baseload daily consumption and temperature-sensitive daily load. PRISM runs statistical models to estimate the impact of the weather on energy consumption. The model may contain heating degree-days (HDD), cooling degree-days (CDD), or both (depending on the type of fuel used and the presence of heating or cooling equipment). Once the best explanatory model is selected, average weather data are plugged into that model to estimate normalized annual consumption (NAC). NAC corrects for weather changes as well as misalignment of billing cycles.

To isolate the impact of other factors (economic conditions, rate changes, and so forth) the impact evaluation approach often identifies a comparison group—that is, a sample of households that are similar to those receiving the treatment. The observed consumption changes in this group is then used as a proxy to show how the participating households would have performed had they not been weatherized. If, for example, weatherized homes reduced consumption by an average of 2,000 kWh while at the same time the comparison group reduced consumption by 500 kWh, the program net impact is estimated as 1,500 kWh.

Finding the right group (in this case, low-income households that have not been weatherized, in similar homes and with similar energy consumption levels) may prove challenging. Recipients of energy assistance that have not been weatherized, and households on the waiting list to be weatherized, are good populations from which to draw samples in order to create the comparison group. Net savings are a measure of the program savings net of weather and all other non-program-related factors. In a PRISM-based evaluation of a weatherization program in the Northwest, the average participant reduced normalized consumption from 15,343 kWh to 13,891 kWh per year (see Table 2). This represents a reduction of 1,452 kWh. This gross estimate does not indicate what would have happened in the absence of the program. To find out, we employed a comparison group of 587 clients who had received some form of energy assistance, but who had not participated in the program. During the same year, these nonparticipants increased their consumption by 409 kWh, or 2.5% of their preprogram consumption. We assume that, had our participants not been through the program, they would have increased their consumption by the same amount. By combining the results of the gross estimate with the results of the comparison study, we arrived at a net energy savings of 1,860 kWh.

PRISM is now extremely easy to use and fairly inexpensive. Any agency that delivers weatherization services can obtain a copy and quickly learn how to use it.

Estimating Audit-Based Savings

Across weatherization programs, auditors conduct an initial inspection to determine whether the home can proceed toward final weatherization. They do so by identifying energy-saving opportunities in the home. In most cases, audit software is used to estimate the potential savings to be realized by installing specific measures. When a measure is expected to produce sufficient savings (as indicated by a savings-investment ratio of 1 or more), the measure is recommended for installation.

Not all agencies use software tools. Some use look-up tables based on a list of average conditions and on simple engineering algorithms. Some agencies use software tools, but do not utilize one of their most powerful features—estimating savings based on actual existing conditions as measured by one full year of consumption data. When the actual energy consumption figure during the year preceding the audit is not used, the audit software will, in most cases, significantly overestimate the savings. It can, in fact, estimate savings that exceed consumption.

To obtain the most complete information, agencies should use an audit tool accompanied by one full year of energy consumption data. When this approach is not followed, extremely misleading measurements can result. With bad measurement come bad controls, and with bad controls come bad programs. The most obvious potential outcome is that programs may install measures that will realize very little in the way of savings. With proper measurement, this problem can be avoided. The funding can then be used to install measures that will realize greater savings, in the same home or in another home. Consider, for example, a program that is currently in the Northwest. In this program, there is no relationship between the audit-based predictions of savings and the actual savings experienced at the individual homes (see Figure 1). One would expect some positive relationship between the expected and the actual savings (that is, as predicted savings increase, one would expect the actual savings to increase). In fact, the relationship can never be predicted exactly, owing to behavioral factors that are beyond the capability of the auditor to predict. In this case, however, there is no positive relationship at all between expected and actual savings. This is a clear indication that the expected savings are invalid and should not be used to gauge energy-saving opportunities.

To further illustrate the point, we present three rather extreme cases drawn from client files (see Figure 2). We do not suggest that these cases are in any way representative; rather, they serve as examples of what can go wrong without the use of proper tools. Case A is a manufactured home with 10,645 kWh consumed during the 12 months preceding weatherization. Based on installation of ceiling insulation (7,514 kWh estimated savings), floor insulation (2,184 kWh), window replacement (93 kWh), and an insulated door (446 kWh), total savings were estimated at 11,170 kWh annually (105% of total preconsumption). While this savings is obviously not possible, it was recorded as such, and the measures were installed. In this case, the total project cost was nearly $8,000. When actual energy savings are considered, the owner can expect to save about $120 annually. Therefore, this project has a simple payback of nearly 68 years! The negative program impact in this instance is a result of the failure to use the proper tools; in particular, of the failure to compare savings estimates to preconsumption data.

Case B is a single-family home with 5,358 kWh consumed during the 12 months preceding weatherization. The auditor estimated total savings of 4,821 kWh (a 90% reduction in consumption). Measures installed included ceiling insulation, wall insulation, window replacement, some infiltration measures, and insulated doors. Total cost was over $4,750. We suspect that simple savings-investment ratio analysis would have shown that the doors and windows would not be cost-effective.

Case C is a single-family home with 10,530 kWh consumed during the 12 months preceding weatherization. The auditor estimated total savings of 6,863 kWh (a 65% reduction in consumption). Measures installed included ceiling insulation, wall insulation, window replacement, some infiltration measures, and insulated doors. Total cost was over $6,700.

Again, we suspect that simple savings-investment ratio analysis would have shown that the doors and windows would not be cost-effective. Savings was estimated at 4,333 kWh. Given actual savings and the cost of installation, this project has a simple payback of over 25 years.

Recommendations for Agencies

As the preceding examples show, agencies that implement weatherization programs can benefit greatly from proper tracking. Without proper tracking, measures are installed when they should not have been, predicted savings are exaggerated, and the programs do not recognize their full potential.

Again, unless data are collected, they cannot be analyzed, and if they are not analyzed, they cannot provide information that can be used to improve delivery and provide better service to clients. If the goal of the program is to provide the most cost-effective service to the most clients, tracking the projected savings and the associated costs can greatly enhance its ability to meet that goal. It enables programs to assess effectiveness along the way, and to make course corrections if and when they are needed.

We recommend that every agency that delivers weatherization services do five things:

1, The agency should track for all participating homes the amount spent by funding source.

2. The agency should estimate savings at the individual home level, using a DOE-approved audit tool and utilizing the actual consumption data from the previous year.

3. The agency should install measures for which the savings-investment ratio is 1 or better. Exceptions may be made when the auditor believes that the household’s energy consumption is low because they are not living in healthy and comfortable conditions, and so the audit tool is estimating low potential savings.

4. The agency should ensure that the best tool will always be the judgment of trained auditors. No audit tool will completely replace that. Agencies should always have enough leeway to make exceptions as needed. However, with better tracking, they will know the effect that these exceptions will have on the overall efficacy of the program.

5. The agency should keep a running total of amounts and it should accurately estimate savings. When the overall numbers indicate that the savings-investment ratio is exceeding 1, the program is well operated. When the ratio falls below 1, corrections are in order. This can be done by bringing larger homes into the program. Owners of small homes still need to be helped, but agencies need to know that they lower the overall potential of the program.

Better tracking allows agencies to make more intelligent decisions at the individual home level and at the overall program level. When agency representatives decide to weatherize a home knowing that to do so will not be cost-effective, they should have that option. However, they need to know the impact that their decision will have on the efficacy of the program overall.

M. Sami Khawaja is the president of Quantec, a consulting company in Portland, Oregon. Patricia Koss is an associate professor of economics at Portland State University, in Portland, Oregon.

Comments

Enter your comments in the box below:

(Please note that all comments are subject to review prior to posting.)

Posted By:

Email:

Your Comment:

While we will do our best to monitor all comments and blog posts for accuracy and relevancy, Home Energy is not responsible for content posted by our readers or third parties. Home Energy reserves the right to edit or remove comments or blog posts that do not meet our community guidelines.

With headquarters in Boise, Idaho, and an employee base of approximately 2,000 people, Idaho Power is an electric utility engaged in the generation, transmission, distribution, sale, and purchase of electric energy. We run a weatherization program ...