I was recently invited to a clash of titans at the University of Chicago. This gathering of some of the top economists, efficiency policy experts, and measurement professionals, (held under Chatham House rules -- no names to protect the innocent), called into question many of the core methods currently used to evaluate efficiency. There were clear tensions in the room between academic approaches to research and the hard realities of the marketplace, as well as between the multiple and complicated factors that drive policy, and the need to manage risk for markets and finance.

Randomized trials are the gold standard for many economists, and on small scale in a clinical and research environment, trials can certainly provide a high level assurance of efficacy. However, it’s simply not feasible or economical to measure or assess energy efficiency programs at scale using Randomized Control Trials (RCT) and Random Encouragement Designs (RED).

This is not to say that current EM&V methods shouldn't be improved and standardized. In fact, much of my professional life has been devoted to this effort. However, while perfect accuracy is a lofty goal, the attempt to achieve it must also support the market need for transparency, predictability, and cost efficiency.

Where possible, small-scale studies using randomized encouragement design and randomized control trials should absolutely be applied to ground truth transparent quasi-experimental results and quantify uncertainty. However, these clinical approaches are fundamentally inappropriate beyond a research setting.

Randomized control trials typically divide participants into two groups, with one group receiving the prescribed "treatment" and the other getting nothing or a placebo. Doing this with self-selecting efficiency programs would require private companies to not treat a subset of their expensive-to-acquire customers, which is simply not feasible. If a customer wants a new furnace you can't stop them, nor should you.

An alternative to the RCT is a randomized encouragement design (RED). In this approach, all customers are treated, but a defined subset is provided additional encouragement to participate (imagine a salesperson who gets an extra $250 for making a sale off the list – but all customers all get the same deal). With a large enough sample and high enough participation, it is possible to calculate savings that are additional in the encouraged group.

At face value, REDs are technically possible in energy efficiency. However, in practice there are a host of barriers that call its feasibility and efficacy into question.

Unlike a drug test or a top down weatherization program, the market-based EE landscape has a huge diversity of business models, customers, and technologies being deployed. You can’t measure once and extrapolate a given finding into the heterogeneous world of efficiency programs and business models.

Sending a strong enough signal to drive the amount of additional adoption required for the statistical power necessary to draw conclusions will be incredibly difficult and expensive. Many channels manage hundreds or even thousands of independent contractors, who in turn have many additional thousands of salespeople. Motivating both contractors and salespeople to provide this extra encouragement would be both a logistical challenge, and costly to accomplish.

Initial experience in running a RED in energy efficiency was on fully subsidized low income weatherization in a small market. Unlike highly diverse, much larger private markets, weatherization is centrally controlled, has a very defined customer, and one can apply a top down engagement strategy. In this first RED test, door knockers were paid specifically to target just the “encouragement” customers and highly motivated to get them to sign up for a free product. This very intensive and expensive approach still only resulted in a 6% encouragement rate, which was only borderline significant and insufficient to provide confidence in savings.

In the energy efficiency marketplace, customers come from all corners (but mostly from referrals) and everyone is already extremely motivated to sell. Contractors spend thousands of dollars acquiring each customer, and sales people are often offered commissions as high as 10%. So the question is, how much extra money would it take to give them enough additional encouragement to see a statically significant jump in close rate when their very livelihood requires they sell every customer all the time? The next question is, where would this money come from to encourage this to happen?

Almost as a side note. These funds would, for tax, logistical, and business reasons almost certainly have to flow through contractors to their employees, not as taxable income directly to employees from a program or government body.

So this brings up an issue of fungibility. If there is a new revenue source, and contractors are competing for customers, it is going to be next to impossible to ensure that contractors don’t improve the customer's offer to win deals. If the encouraged customer gets a different offer than those in the control group, this would undermine the validity of the trial.

But wait, there’s more… and this is the real issue.

In our pursuit of the perfect measure of savings, we forgot the most important part, which is where the money comes from to get to scale.

Ratepayer surcharge based programs only deliver about $8 billion a year into the market, resulting in only $16B (give or take) of total investment. While this sounds like a lot, it pales in comparison to the technical potential of efficiency investments at a lower marginal cost than alternative distributed energy resources. For efficiency to scale we must move beyond current consumer financing options and instead finance the upfront costs through project finance -- infrastructure investors who finance power plants, rather than credit cards.

​In order to attract this infrastructure investment, we need manageable risk, not uncertainty. Relying on randomized studies that can’t be replicated by market participants and can’t be quantified until well into a contract period, creates uncertainty that makes private investment more costly or potentially impossible. When you can’t estimate what a finding will be until after the fact, you turn investment into a guessing game.

Allowing the measurement tail to wag the efficiency dog creates an existential dilemma where perfect measurement could mean we can no longer manage risk and attract the capital we need to scale efficiency -- meaning that more power will need to be generated, with the cost of this additional generation recovered thru rates.

However there is good news. There have been massive advances in measurement that are apparently not well understood.

At Open Energy Efficiency, we are now metering energy efficiency, which means we are automatically calculating monthly, daily, and hourly resource curves across utility portfolios, and for participants in the market in near-realtime directly from normalized smart meter data. This open source platform (based on a process in California called CalTRACK) is helping some of the largest utility programs and energy efficiency companies in the United States track their savings, and conduct analytics in real-time for the first time ever.

We combine that with a distributed cloud based platform that can do similar realtime normalizations for the full population usage and meta data. Entire states are now implementing the ability to track and analyze meter data at both a project and population level.

Open source analytics tools allow regulators, utilities, and measurement professionals to filter the population by attributes such as location, sector, climate zone, and load shape (or any other meta data at our disposal). These results then generate quasi-control groups that can be tested based on historical data, allowing experts to analyze their options and negotiate a reasonable control group that fairly represents the naturally occurring population level exogenous effects.

The parameters of that quasi-control can be locked in prior to starting a given program or procurement. Historical trends can be published, and the exogenous effect can be tracked on a longitudinal basis. Now we can not only track net impact, but also we can manage the risk.

Are these quasi-controls as statically powerful as is theoretically possible in a properly conducted RCT or a RED? The answer is clearly no. However, early REDs that have been conducted in efficiency do not differ substantially from the findings of current EM&V methods.

Random encouragement trials are another tool in the box to tune and improve the quasi-experimental approach – but should only be used as research in cases where it is likely they can encourage enough adoption to be valid while still being economically feasible to conduct. The results should not replace current methods, but instead be used to improve them.

While both energy efficiency and EM&V need disruption and innovation, market practitioners in energy efficiency have different constraints and benefits than those focused on pure research and energy outcomes. I hope that we can refocus our discussions with the academic economics community to help research and address the much more pressing questions associated with the valuing of time and locational efficiency impacts, accurate calculation of avoided carbon, and the development of open source, transparent and replicable approaches to measurement that can support markets and investment.

There is a lot of concern about the impacts of a Trump presidency on energy efficiency, and the likely demise of key climate drivers we have been working on such as the Clean Power Plan, and the Paris Climate Accords.

There has also been some wishful thinking that the new administration’s infrastructure and jobs agenda could include energy efficiency work. I hate to break it to everyone, but when Trump says infrastructure and energy, he means coal and fracking, not cash-for-caulkers.

But while we can expect a stiff headwind from Washington DC for the next few years, the good news for energy efficiency is that the federal government has never had a lot of energy efficiency public policy levers at its disposal. Our earlier failure to get meaningful national policy in place for EE (e.g. HomeStar, Homes Act, Save Act, etc.), now means the industry is, thankfully, not addicted to the federal juice.

The real action in energy efficiency is, and always has been, at with state policy makers, utilities and in the hundreds of Public Utility Commissions that make up the energy regulatory landscape.

So the silver lining for energy efficiency during this next administration is that what was in the past a systemwide bug that made it hard to bring helpful federal EE policies to the states, is now a feature that can insulate the industry from the climate-unfriendly policies of a new administration.

The good news for efficiency specifically is that it's really hard for federal officials to affect policy at the state level. This fundamental principle of American government and federalism will not change for the next administration. State-based energy efficiency innovation can and will continue to flourish despite the onslaught of revanchist policies from the federal government.

As energy efficiency moves from a deemed rebate approach into a performance-based marketplace where efficiency is measured as a demand-side capacity resource, the value of efficiency is going to increase, as will the incentive for bad actors with the intention of gaming the system. Making sure there are systems in place to regulate and police this market to ensure it is fair and transparent is essential. The good news is that there are many tools available to us today to manage this risk and make sure only real results are rewarded.There have always been opportunities and incentives to game markets, and the emerging market for metered energy efficiency will be no different. In reality, a market with properly aligned incentives will drastically reduce the industry-wide bias toward over prediction and make the whole system more accurate, transparent, and enforceable. As in other markets, it will the regulator's role to put in place and enforce rules that ensure a fair and transparent marketplace.In modeled performance programs, where rebates are paid in advance based on the results of engineering models or deemed predictions, there exist substantial incentive to inflating savings predictions. Even without counting the bias of the auditor or market tipping the scales (and there is little evidence that in the current market there is a concerted effort to do so), the engineering tools regulators require have a massive bias towards overestimation. In California, the Title24 HERSII based energy model required until earlier this year, overpredicted energy savings by nearly 300 percent (CA solution: www.CalTRACK.org) while New York and other states overestimation ran closer to 150 percent.In pay-in-advance model of energy efficiency, we don’t measure results in a usable way and instead attempted to ensure good outcomes by micromanaging every step of the retrofit progress with an avalanche of regulation. A CPUC ruling in August summed it up, stating that “Program Administrator expenditures on costs other than customer rebates appear excessive, as they have come to represent approximately half of portfolio expenditures.” In other words, we are spending $1 on administrative costs to try and make sure the other $1 is spent wisely -- and then failing.Some who argue for continuing the current programmatic approach to efficiency and point to the potential for gaming in a metered market as a reason to stick with rebates and program implementers. One anecdote that is repeated over and over again in energy efficiency circles as an example of potential gaming, tells the story of a contractor who targeted families with graduating seniors, knowing that when they went off to college, bills would go down, creating “savings.”While this type of gaming is certainly possible, it would actually be much easier to regulate and police in a pay-for-performance model than in our current system.Just as insider trading is illegal in financial markets, the gaming of energy efficiency savings by explicitly targeting populations based on customer attributes that will lead to decreases in use without intervention or investment, should also be considered gaming and fraud. Just like insider trading and other market regulations are illegal and prosecuting, fraud in measuring energy savings should be made illegal, aggressively prosecuted, and be grounds for debarment from participating in the market.Fortunately, it turns out that just as with insider trading or even credit card fraud, it is harder than one thinks to game the energy efficiency markets without leaving fingerprints in the data. By looking at data in the aggregate, regulators can spot anomalies, investigate, and catch and prosecute illegal activity.Like all markets, it is possible for bad actors to game in energy efficiency, and this is true both in today’s programs and in future metered performance markets. However, this potential should be used as an excuse to stick with the status quo. Regulating these new efficiency markets will require focus by regulators, however, we have the data and tools to ensure a fair and open market. We can learn for experiences in the past, such as standard offer programs from the 90s, as well as what has worked in other markets to ensure trust.In this new world of measured efficiency, regulators can focus on protecting customers and regulating the marketplace to ensure savings are real and trustworthy. We should not let the perfect be the enemy of the good in energy efficiency and focus on learning from the past and developing data driven solutions so that efficiency can take its place as carbon free low-cost source of demand-side capacity.

Last week, the California legislature passed SB-350 Clean Energy and Pollution Reduction Act of 2015, a bill that set the state on a path to at achieving Governor Brown's ambitious clean energy goals by 2030. The Governor’s “50/50/50” plan aims to increase electricity from renewable sources by 50 percent, reduce petroleum consumption by 50 percent, and increase building efficiency 50 percent by 2030. While most media reports focused on the audacity of trying to increase the renewable portfolio standard and energy efficiency goals, and some observers expressed justified concern about items left on the cutting room floor, there was little discussion of some of the bill’s most important provisions, specifically those that address the details about how energy efficiency will be measured and delivered going forward.

Not only does the bill essentially double California's energy efficiency goals, it does so in by making a number of very important changes in how we approach energy efficiency in the state. These changes, if implemented, represent the beginnings of a major paradigm shift.

First, SB-350 specifically changes the way that energy efficiency is counted. Rather than rely on a series of ex-post studies, including a stack of regulation that includes considerations for energy code and attempts to account for “free ridership” and other subjective impacts, SB-350 is clear that “energy efficiency savings and demand reduction reported for the purposes of achieving the targets established pursuant to paragraph (1) shall be measured taking into consideration the overall reduction in normalized metered electricity and natural gas consumption where these measurement techniques are feasible and cost effective.”

Additionally, this new law defines “energy efficiency savings” as “reducing the quantity of baseline energy services demanded” and includes both the adoption of efficiency measures and practices (such as behavior). The law also directs that the CPUC “achieve greater energy efficiency in existing residential and nonresidential structures that fall significantly below the current standards in Title 24 of the California Code of Regulations.” In essence, this means that what’s important is results at the meter, not how one gets there, and that those results are the difference between the buildings’ baseline use before interventions and consumption levels during the performance period.

While this may seem like commons sense to many people, in reality, it is a huge shift from current practices, one that promises to standardize how efficiency is measured based on meter data, and evaluate it based on straight reductions in demand. More importantly, it has the potential to finally put in place a standardized and replicable system for measuring efficiency as capacity in a way that markets can treat as a reliable demand-side energy commodity. These new directives are also closely aligned with the process underway to implement the residential CalTRACK system based on the Open EE Meter.

Building on this data-driven approach, SB-350 goes on to “Authorize pay for performance programs that link incentives directly to measured energy savings. As part of pay for performance programs authorized by the commission, customers should be reasonably compensated for developing and implementing an energy efficiency plan, with a portion of their incentive reserved pending post project measurement results.” Later SB-350 goes on to specifically state that “Incentive payments shall be based on measured results.”

This approach, is similar to a pay-for-metered performance pilot that was proposed to the CPUC by NRDC and TURN, and supported as well by PG&E, which said in their CPUC filing that "PG&E supports a residential pay-for-performance pilot that we understand NRDC will propose in its workshop comments. This pilot design has the potential to facilitate comprehensive upgrades while simultaneously minimizing implementation costs through leveraging private capital."

Though it received less attention, another bill that passed last week was AB-802, which also specifically moves the State towards meter-based energy efficiency, in addition to its primary goal of implementing benchmarking across the State. This law directs the California Public Utilities Commission (CPUC) “determine how to incorporate meter-based performance into determinations of goals, portfolio cost-effectiveness, and authorized budgets, the commission, in a separate or existing proceeding, shall, by September 1, 2016, authorize electrical corporations or gas corporations to provide financial incentives, rebates, technical assistance, and support to their customers to increase the energy efficiency of existing buildings based on all estimated energy savings and energy usage reductions, taking into consideration the overall reduction in normalized metered energy consumption as a measure of energy savings.”

In addition AB-802 addresses the festering issue of Code Baseline in California, which has meant that the CPUC can only pay utility incentives for above CEC Title24 Energy Code, which represent an existential problem as code is increased to net zero energy, directing that “programs shall include energy usage reductions resulting from the adoption of a measure or installation of equipment required for modifications to existing buildings to bring them into conformity.”

Taken as a whole, these changes represent a fundamental shift in California’s approach to energy efficiency. As these changes come into effect, we will be moving from a programmatic regulated approach to efficiency to markets that treat energy efficiency as a capacity resource and rely on private capital and innovation to create the business models necessary to achieve the scale required to hit Governor Brown’s goals.

While largely overlooked, these rather weedy changes to how energy efficiency is measured, and the directive to the CEC and CPUC to begin piloting pay-for-performance approaches to measured savings represent a true paradigm shift. Supported by an unusually diverse group of stakeholders, including environmental advocates, local governments, industry, utilities, and ratepayer advocates, these landmark bills represents a major advance and new opportunity for energy efficiency in California.

Looking ahead to Phase III of this proceeding, many important policy issues remain before us. Energy savings goals continue to go up, while we are to some extent a victim of our own success: the low-hanging fruit has largely been harvested. Energy efficiency portfolios as we know them are on the verge of no longer being cost effective. Program Administrator expenditures on costs other than customer rebates appear excessive, as they have come to represent approximately half of portfolio expenditures. The rate of observed savings compared to forecast savings is distressingly low in some market sectors. Ex ante review continues to be a source of controversy.

However, when you dig a bit deeper it seems like after calling out these very important strategic issues, the actual decision is very close to business as usual and drives even more power to CPUC staff. It does little to promote use of data from the multi-billion dollar investment we have all made in smart-metering, and does not seem to recognize the growing consensus that we need a paradigm shift in how California approaches energy efficiency, not just a shuffling of the deck chairs.

You can read the full text of this decision, and please comment if you see anything in there that represents the kind of substantive and strategic changes that hold the promise of changing our trajectory.

If energy efficiency is going to deliver on its great promise as a low carbon capacity resource we need to fundamentally rethink the approach. Its time to move from reguelated programs to markets that can drive private investment and encourage real innovation. This process is underway in a number of key states including CA and NY, but progress is slow and made less certain by regulators and the weight of large entrenched interests in current programs -- program administration costs are clearly overhead when it comes to delivering efficiency, but also the core business model for many well funded firms.

Unfortunately, I don’t see ideas in this latest opus that stray from far from the status quo. We need big bold and aggressive change that is frankly unlikely to emerge from the current process without political leadership.

For ideas on a different path, check out my GreenTechMedia article on moving from Programs to Markets.

While there have been many studies that document the value of solar systems at time of sale, there has been little analysis to date on the impact of 3rd party owned solar system (leases or PPA) on home valuation or marketability. Given the enormous growth in 3rd party financed solar market, which currently represents more than 75% of the market in California, and is increasing market share in just about every viable solar market, this question is of great importance.

In terms of the direct impact on values, they found that solar systems that are owned did in fact have a positive impact on home value, adding between $1,400 to $2,600 per kW on the roof.

However, the qualitative part of this study turned up some interesting and somewhat concerning trends when one looks at the differences between 3rd party owned solar (Leases and PPAs) versus owned systems.

First off, there has long been an economic argument that given the falling cost of solar, and the zero upfront nature of third party leasing, truthfully a "solar ready" house should be more valuable than one that already has a 3rd party owned system on it, as there is no upfront costs to putting a system on, and the price keeps dropping.

However consumers are not always rational economic actors, so the fact that people values solar was not a complete surprise.

When one reads the qualitative results of this report, and looks at the effect of Leased vs. Owned solar systems on the value of houses and transactional friction, there appears to be a rather clear trend that 3rd party owned systems are not having the same positive impact on home values, and in some cases are adding significant friction.

Of the 39 systems studied, 72% were owned systems, and overall, 59% of the time solar was rated as adding market value or marketability to the transaction. However, of the 22 homes where solar systems were rated as a positive, only two (2) of the homes were leased, while of the 14 neutral impacts six (6) were Leased, and in the three (3) cases of negative impacts on the transaction 100% were Leased systems.

I would caveat all of these conclusions and concerns as extremely preliminary. This is a small dataset and these are qualitative results. But when you read the comments in the table on the right there is a clear difference in the feedback when you compare those systems that are owned with those that were leased.

There is a lot more to this study, and to the value of solar in general, this blog is merely highlighting one interest trend, that given the growth in 3rd party ownership of solar, is something worthy of further discussion and research.

The entire study is worth looking at, and the table on the right can be found on page 46.

What Realtors have to say about the impact Solar leases versus owned systems on market value and marketability:

Four years ago Property Assessed Clean Energy (PACE) was set to roll-out to communities across the country and was a key element of our State and Federal strategy to stand up a energy efficiency industry and consumer value proposition. However, In July 2010, FHFA released a Statement on Certain Energy Retrofit Loan Programs, which essentially killed residential PACE by advising Fannie Mae and Freddie Mac to avoid buying mortgages with PACE assessments, leading many PACE administrators to suspend their residential programs.

In the meantime, a number of pirate PACE programs stayed in business and proved that PACE can drive significant demand and volume. In particular, the HERO program in Southern California has done over $100M in residential PACE loans, and just today released a preliminary rating for the first PACE Securitization.

Based on now proven success, and a continued belief by many that PACE is not the threat FHFA made it out to be, and that it is a critical component to meeting our energy and climate goals, many have continued to fight for residential PACE. In California that fight is about to bear fruit.The California Alternative Energy and Advanced Transportation Financing Authority (CAEATFA), with the backing of the California Governor Brown, is about to roll-out a first of its kind PACE Loss Reserve Program. CAEATFA recently published a final version of its PACE Loss Reserve Program which should be officially approved based on their website, by the second week of March - only a few weeks away.

It would appear that we may have a second chance to make Residential PACE happen in California, and if we can make it work here, perhaps the rest of the country as well. A big thank you to all the people who continued fighting for PACE and to CAEATFA and Governor Brown for keeping PACE and residential energy efficiency a priority.

The last remaining question is really FHFA and how they will react to California's new plan. With their primary concern dealt with through the State reserve fund covering any potential losses, I hope this question will be put to rest so we can get on with the business of creating American jobs and saving the planet!

I recently came accross this video from a UC Berkeley panel from back in 2011 entitled Sustainable Residential Energy Use, Design, Feasibility, Performance. Given the current state of residential energy efficiency in California and across the country, the points being made are perhaps even more relevant today, as many of the problems outlined can now be proven based on actual programmatic results. Residential energy efficiency is at a crossroads where we will either transition towards real markets, or become a footnote. However, since 30% of California electrical energy is used by existing homes, we can't afford to keep failing. It is time to change the model away from our programmatic roots and towards market based solutions that trade in energy efficiency as a resource.

Energy Efficiency Mortgages (EEM) enable homeowners to add energy efficiency to their mortgage at time of sale, taking advantage of long-term and very low rate financing. To qualify a purchaser must undertake an HERS rating to assess the work scope and calculate payback.

It seems like a no-brainer. Target homeowners during a transaction and during a period in which they traditionally spend substantially more on home improvements. One NAHB Study showed that a buyer of an existing single-family detached home tends to spend about $4,000 more than a similar non-moving home owner, including $3,600 during the first year.

While focusing on a home purchase a trigger for an energy efficiency upgrade seems extremely logical, it has not been working in the market. This “trigger point” continues to be the focus of government programs, whitepapers, and both State and National legislation, yet EEMs have not achieved more than a toe hold in the market, and the numbers continue to fall.

Total US Energy Efficiency Mortgage (EEM) Loans Per Year

So the question is, why is something that makes so much sense on paper failing to such a degree in the marketplace?

1) People buy solutions to problems

Unlike kitchen, bathroom remodels, and new flooring, energy efficiency and comfort are not visible to a homeowner that has not yet paid a bill or spent a cold winter (or hot summer) in their new home. As the trigger to get people to invest in energy efficiency and comfort solutions for their home tends to be based on solving a pain point, it is hard to motivate a new buyer to take action on issues where they have yet to experience the problems.

2) Buying a house is stressful enough

The process of buying a house is full of stressful decisions, complexity, and risk - and for most people constitute their biggest single investment. Adding another moving part to this process, one that will require more inspections and general make transactions take longer is just not a priority for most home buyers.

3) There's no money in EEMs

Nationally, Energy Efficiency Mortgages add and average of only $7,500 for energy efficiency upgrades onto the base mortgage (based on data from HUD). When you consider that time is money, and that the brokers and realtors who are shepherding the transaction through the process get paid when it is completed, there is no wonder we don’t see realtors jumping up and down excited to promote EEMs.

On the typical EEM, realtors make no money, as their commission is based on the sale price, not the added value of the EEM. Realtors make no more money on an EEM than a traditional mortgage yet they incur risk, extra time, and expense.

Over the last four years, California Energy Commission has rolled out its HERSII rating system, which was designed as a way to provide a score to homeowners and prospective buyers that compares the energy performance of a house to similar homes (without occupant behavior). The HERSII system is based on the same engine that drives California Title 24 Energy Code and was identified as a key part of the State’s approach to driving energy efficiency in existing buildings.

The goal of the HERSII system is to label all California homes in order to equate energy performance with building value at time of sale, and as a way for a homeowner to get third-party recommendations as to the most cost effective options to save energy.

HERSII has been a centerpiece of the CEC’s thinking about how to implement Assembly Bill 758 (AB758), which is legislation that gives the CEC regulatory authority over existing buildings. While not explicitly called out as the solution in the legislation, the CEC has laid out a plan of attack that uses the HERSII rating as the driver to eventually mandate improvements. While there is a desire by many to consider alternatives, the HERSII policy has yet to officially change.

Before we move forward with this strategy, it is critical that we evaluate all of our options and the data from early HERSII pilots. This is particularly true in light of phase three of AB758 which includes the potential of mandating ratings and cost effective energy efficiency improvements on existing buildings.

Potential Mandatory Approaches could include:

• Requiring disclosure of energy performance and the completion of the most cost‐effective energy efficiency upgrades at appropriate trigger points in the life of all buildings, such as transactions that result in change of ownership, occupancy, or financing; replacement of major equipment; or building remodeling or renovation.

It was brought to my attention, that while the AB758 Action Plan did not discuss HERSII issue, there was an on the record discussion on HERSII at the AB758 Fresno Workshop where Commissioner McAllister and also Bill Pennington of the CEC, at which time both expressed a willingness to at least revisit or maybe even "reinvent" the HERSII program. I would like to commend them for being willing to go on the record, but I will also implore them that we need real resolution and clarity on this issue and that waiting for the next AB758 Report to emerge and then another proceeding is prolonging uncertainty that is affecting the market.

HERSII sounds great and asset ratings and third-party raters make for really nice policy white papers, but when put into practice on existing buildings, the results speak for themselves ― and don't necessarily agree with the theory. The HERSII ratings system is extremely costly, has low consumer demand, and does not result in significant conversion to energy efficiency projects.

In an attempt to develop a one-size-fits-all solution to provide an asset rating for use at time of sale and in the appraisal process and provide an actionable workscope and savings projection all in one system, we have instead created a solution that does not work well for either use.

So the big question is, will the HERSII strategy work?

The good news is that we now have results back from our initial tests of the system. However, the unfortunate reality is that the test phase has demonstrated serious shortcomings in the current theory, which should at a minimum call the HERSII system into question and hopefully lead to thoughtful evaluation of the shortcomings that emerge to avoid costly missteps for the marketplace and government programs.

Based on results from HERSII tests over the last four years, the HERSII rating system does not appear to be catching on with homeowners, nor does it seems to be driving market transformation in terms of conversion to energy savings or discernible asset value improvements (though the latter is harder to know with the current dataset). Admittedly, some of our data is a little sparse, but that is a function of how hard it is to get real numbers out of these pilots.

Can you guess when the incentives ran out?

What can we learn from the Sonoma County HERSII Pilot?

Sonoma County ran the largest HERSII pilot in the state, paying a subsidy of $700 per HERSII rating, which resulted in 100 percent free audits and ratings to customers, and unsustainably high margins for raters.

In Sonoma, we see a clear jump in the number of HERSII ratings while this lucrative incentive was on the table. However, shortly after the massive subsidy went away, the number of ratings drop immediately back to virtually the same number of ratings as were occurring before the incentive program. It would appear that even after this very expensive attempt to seed the market, there is little to no actual consumer demand in the market for ratings and one is left to wonder what happened to all the HERSII raters who got trained in Sonoma for this brief pilot, now that there is no more work.

Clearly, California does not have the funds to pay $700 or more per rating out of public funds on every home in California, so the fact that Sonoma’s 100 percent rebate pilot seemingly did nothing to jumpstart actual consumer demand is very concerning, and means an attempt to roll this system out would force California homeowners to shell out for an expensive service they are not valuing.

The fact that the Sonoma pilot program did not result in any discernable market transformation, either in terms of sustainable businesses conducting ratings or demand for ratings by homeowners, should be of serious concern.

Do homeowners value HERSII ratings?

When we look at the adoption curve of HERSII ratings during the recovery act period, we find that ratings are highly correlated to rating incentive programs and appear to have little organic consumer demand.

In fact, as is apparent in the chart on the right, it appears that these incentive programs had little lasting impact on demand in the marketplace.

When you couple this with the emerging facts related to the propensity of the HERSII system to overpredict potential savings by as much as a factor of 3x (see: Ex Ante Tool Review Findings Disposition for Energy Upgrade California Custom (‘Advanced’) Measure Savings, 1 Mar 2013), one realizes that not only is this system not working in terms of driving customers to take action, but it is failing to do so while massively inflating the level of savings and ROI being sold to the customer.

The propensity of the HERSII system to project significantly higher savings than what is delivered means that if HERSII in its present form was applied as a mandate, there would be many millions of California homeowners who would see only a small fraction of the savings they were told to expect.

This fundamental issue is based on the CEC’s attempt to create a system that is based on code, and then apply it to operational predictions of savings. These two uses are nearly opposite in terms of how a model is constructed, and speaks to the need for systems designed for the specific purpose being asked - one size fits all is not cutting it (learn more about this issue here).

The CEC may have the legal authority to regulate ratings in California, and AB758 may give the Commission the legal basis to attempt to implement this approach, however it is highly unlikely that they have the political capital to make it stick once we start subjecting real people to these outcomes.

Asset ratings may in fact have a place, but they are really not compatible with delivering actionable workscopes in the real world, and the one size fits all approach has resulted in a system that really fits nobody very well. HERSII is an expensive and complex system to deliver ratings that have significant accuracy issues, don't work for industry, and consumers don't seem to value.

Is HERSII worth the cost?

Each HERSII rating costs at least $500 and often more, based on the fact that conducting a rating consumes half a day in the field and then more hours back in the office inputting data into the California Energy Commission (CEC) mandated software tool. There is little room for substantial economies of scale.

When you consider that we have upward of ten million homes in California, the cost of having a HERSII rating done for all California homes will require California homeowners to invest $5 billion dollars in ratings - and that is just for ratings, not actual energy efficiency.

Do HERSII Ratings turn into energy efficiency projects?

California home performance contractors have a very hard time converting HERSII third-party audits into viable leads that result in customers who are ready to make energy efficiency upgrades.

HERSII Ratings conducted in Sonoma were cross-referenced with Energy Upgrade California™ (EUC) projects and it was found that conversion from rating to energy efficiency retrofit was under 10 percent (this analysis was conducted prior to 100 percent completion of the program, and until it is redone more accurately should be considered a clear directional indicator). This is especially surprising given that in the Sonoma program, unlike elsewhere in the state, home performance contractors were allowed to provide the rebate to customers for their diagnostic test-in as long as the project included a HERSII rating upon completion. Given that contractor close rates are typically 30 percent or greater, their participation in the Sonoma pilot would have been expected to inflate the actual conversion rates when compared to HERSII on the open market by third-party raters.

At a 10 percent conversion rate, a $700 rating incentive translates into $7,000 of public funds in rating incentive per closed retrofit in addition to Utility incentives, and other Energy Upgrade California program spending. Clearly this was not a cost effective way to drive demand for retrofitting.

This trend is borne out in PG&E figures for its entire territory, which shows that of the 582 homeowners who obtained HERSII ratings, only 59 projects have been completed energy upgrades through Energy Upgrade California, which pencils out to a similar 10% conversion rate. This of course compared to conversion rates of audits to retrofits that are typical of integrated home performance contractors of over 30%.

Why is it so hard to get homeowners to act on their ratings?

The HERSII process is convoluted, complex, and expensive. From a homeowner perspective, it goes something link this:

First, a third-party rater does a HERSII rating for a homeowner who shells out something like $500 for the audit and report. If that customer is interested in doing work, they bring the rating report to a home performance contractor to get the upgrades implemented. However, here is where it starts to gets tricky and the real world collides with theory.

Contractors are ultimately responsible for a project’s outcome, including its energy performance, and cost. When presented with a third-party HERSII rating, a contractor must determine if the report recommendations match the conditions he/she finds in the home. This means the contractor always must go back to the customer and do what amounts to another audit of the building to confirm recomendations, code compliance, and pricing, but — because the customer has already paid for a rating audit — the contractor has to conduct this work for free.

On a very large percentage of these third-party HERSII ratings, the contractor will end up not agreeing with either the estimated price coded into the rating (which drives the Return on Investment and ranking in the software), or they will disagree with the solution being recommended once someone with construction experience visits the site and creates an actionable workscope.

This puts the homeowner in a bind. On one hand, they have the third-party rater sporting a CEC logo on their report telling them one thing, and a contractor telling them another thing, and far too often the homeowner becomes uncertain as to how to proceed and who to trust, so no work gets done. The third-party model ends up with very low conversion rates, and is generally not a profitable source of customers for home performance contractors.

How do we move forward from here?

It is time that we focus our energies in California on moving to a market that relies on actual performance, not complex regulatory structures translated into software and a numerical scales. The web of regulation we have created has many unintended consequences, and with the advent of smart meters and big data, we have the opportunity to move beyond regulatory proxies and instead harness private markets and sources of capital. We should embrace this change as a great success of the California system, not a failure. It is time for State policy to advance and lead the nation towards a smarter more efficient model to deliver the deep energy efficiency required if we are going to hit our climate, energy, and economic goals.

In light of the results from HERSII to date, the California Energy Commission strongly considers adopting simpler approaches to providing homeowners with ratings, that are substantially lower in cost to homeowners and do not pretend to be of investment grade accuracy. One such solution is the national DOE Home Energy Score. A very simple model can be input through an API from a variety of software tools, and is simple enough to be part of a home inspection, resulting in a 1 to 10 score for the homeowner.

These simpler systems work as a way for homeowners to gauge a home's performance but will not be confused with real energy audits capable of delivering solutions to homeowner that contractors can actually build and that will deliver reliable results. We should separate upfront simple asset ratings from the more complex needs of a diverse contractor marketplace who can deliver the level of quality auditing necessary to develop an actionable workscope and a real price. Instead of applying an expensive and inaccurate comprehensive HERSII rating on every house in the State, we instead would only do an investment grade audit when there is pathway to getting a project built.We need to do more than tweak the knobs on the current model. We need to be open to new ideas and substantial change in our approach. Solutions are out there, but we need to be willing to admit that many current ideas are not working, and minor changes will not be sufficient to reverse the trend. It is time for California to step back into the lead and help the country move towards a sustainable market for energy efficiency.This process should start today by admitting that the HERSII system needs to be reevaluated based on evidence and feedback from the market. It does not matter how much we have invested, we need to look at empirical results and change course. We cannot afford to keep marching forward with the same basic theories, while hoping for different results.