You are here

Mis—LEED—ing

The reason we have lots of Greek symbols associated with statistics is that the ancient Greeks had figured out a lot of statistics and other sciences, including means and medians. Statistics really took off in 1600s England. Four hundred years ago an English statistician would have immediately recognized that it is really stupid to compare the median of one set of things to the average of another set of things. Of course if you were interested in trying to hide stuff you could try that approach and hope that no one noticed. Well, a bunch of folks noticed and put the US Green Building Council (USGBC) on notice.i

Let’s start with a basic discussion of statistics and then progress to a more complex discussion of politics.

Let’s say you have a collection of things—a “distribution”. The medieval English found that there are many useful values within a distribution. Some of these would be the “minimum,” the “first quartile” (i.e. 25th percentile), the “second quartile” (i.e. 50th percentile), the “third quartile” (i.e. 75th percentile), and the maximum. It is important to note that none of these values relate directly to the total of all of the numbers, or to the sample size. Now pay attention here, the second quartile has a special name; we call it “the median.”

The medieval English and others also went on and defined a bunch of different “means.” One of the best known is the “arithmetic mean.” Most of us call this the “average.” It is the value that when multiplied by the number of “things” (i.e. the sample size) gives you the total sum of the value of all of the “things.” Civilians, and most of us, relate to “averages”—the “average” of something resonates with people. Let me put it more bluntly, people are really interested in “averages” as in “the average energy consumption of a bunch of buildings is this.” Our children and grandchildren, for example, are much more interested in our means, and won’t give a damn about our medians.

The median and the mean both have the property that they will be somewhere between the minimum and maximum values of a distribution. Beyond that they have nothing to do with each other. Let me repeat the “they have nothing to do with each other” part. It will be important later on.

For hundreds of years it has been known that some distributions are better characterized by medians rather than means. Fair enough. However, given that the two statistics have nothing to do with one another, when comparing one distribution to another it is not possible to make meaningful comparisons using the median of one and the mean of another. In a comparison of distributions you either have to use the mean or use the median as the basis of comparison. If you have a problem with this take it up with the ancient Greeks and the medieval English and good luck to you in trying to change several hundred years of fundamental statistics.

Now to the politics; the USGBC wanted to see how well Leadership in Energy and Environmental Design (LEED) buildings were doing energy wise compared to regular buildings. This could be important given the claims about how wonderful LEED buildings were supposed to be according to the USGBC.ii The New Buildings Institute (NBI) did the looking for the USGBC. Information on regular buildings came from Commercial Building Energy Consumption Survey (CBECS).

The findings were presented in a March 4, 2008 report “Energy Performance of LEED for New Construction Buildings.” The trouble started with the following quote from the report: “For all 121 LEED buildings, the median measured Energy Use Intensity (EUI) was 69 kBtu/sf, 24 % below (better than) the CBECS national average for all commercial building stock. Comparisons by building activity type showed similar relationships. For offices, the single most common type, LEED EUIs averaged 33% below CBECS.”

A civilian reading this would conclude, hot damn, LEED rocks. A long dead Greek or medieval Englishman would not conclude that, but who cares as the Greek and medieval Englishman are both dead and can’t cause any trouble. But more troubling to the USGBC, a few very much alive folks who know a little bit about statistics and buildings said wait a minute, you can’t say that because what you said makes no sense. A few even had the audacity to suggest that maybe someone was trying to pull a fast one.

So what do the NBI-LEED and CBECS statistics really show? Well the first thing we have to do is decide what we want to compare to. Most folks think we should compare the NBI-LEED buildings to recently constructed CBECS buildings, not all CBECS buildings. Why? The comparison buildings should be buildings constructed at the same time the NBI-LEED buildings were constructed. Apples to apples, right? The CBECS comparison distribution should be the CBECS 2000-2003 data. It wasn’t and that’s where lots of folks started to scratch their heads and wonder what was going on. The next thing we have to do is make sure stupid stuff is not included in the CBECS 2000-2003 data—such as warehouses and unoccupied buildings which skew the results (they make the CBECS buildings look more energy efficient then that actually are—memo to the USGBC, this helps your argument). Okay, that pares the CBECS distribution down to n=334 (5 vacant buildings and 56 non-refrigerated warehouses are no longer included). We have to do the same to the NBI-LEED data set. We should drop data centers as none are included in the CBECS data (this helps the efficiency of the data set as these are the highest energy use buildings). That pares the NBI-LEED distribution down to n=115.

The NBI-LEED data that does not include the high use data centers buildings is plotted against the CBECS 2000-200 data that does not include the vacant buildings and non-refrigerated warehouses. The two distributions look pretty much the same don’t they? They are not statistically different, by t-test, by mean-to-mean and quartile-to-quartile results.

NBI-LEED mean (n=115) is 96, compared to the CBECS mean (n=334) of 111

NBI-LEED median (n=115) is 67, compared to the CBECS median (n=334) of 81

NBI-LEED median is 72 % of the NBI-LEED mean

CBECS median is 73 % of the CBECS mean

NBI compared the LEED median to the CBECS mean. Big, giant mistake, one that will haunt the report authors for a long time. If you compared means alone (i.e. averages) you could say LEED buildings performed about 15 percent better than typical buildings constructed at the same time. But that is misleading considering the scatter of the data. Let me repeat, LEED buildings are not statistically different than typical buildings, even though their mean is around 15 percent better (kind of like how a political candidate can be 3 points ahead but have it be a statistical dead heat). Aren’t statistics great? Anyway, the number is certainly not 24-to-33 percent better. And even if NBI’s claims for LEED were true, 30 percent energy savings for what is supposed to be the vanguard green program in the US is not very inspiring. Come on folks, we have to do better.

Someone had to play with the numbers to make the storyline work and that is just plain misleading. And, surprise, surprise the guy who blew the whistle is getting trashed.

So what does this mean? Let us translate—the LEED buildings did not conclusively save any energy compared to typical buildings built at the same time.iv This is not good.

LEED needs to be fixed. Manipulating a bunch of statistics to hide behind does not save any real energy. Let’s fix the problem and save some energy

Where to start? Easy. Ask a few simple questions. How big is my building? Where is it? What is going on inside of it? How much energy did it use compared to a similar sized building in a similar location with a similar occupancy built to standard practice? If you can’t show any energy savings for gods sake shut up and take your points and stick them where the sun doesn’t shine. Okay, that is a little bit harsh. So what do we need to do to make the energy savings real? We have to start making the right design decisions at the front end, but we also have to be keeping track of how well we are doing on the back end so that we can continue to improve. Right now we are doing neither.

i Henry Gifford of New York City looked at the reported results and started asking questions. Hard questions. And the predictable response? A not so quiet campaign to discredit the messenger rather than address the questions raised. Questioning the orthodoxy of the Green movement is not a particularly smart career move. Not too many principled men and women around anymore. Well done, Henry.

ii Google “LEED” and you get: “Build green with LEED, www.usgbc.org. Sustainable building saves energy & money. Learn how with USGBC.” Apparently LEED buildings do neither. They are certainly not cheaper.

iii The plot was created from data provided to Bill Rose by Cathy Turner of NBI with the permission of the USGBC. The USGBC says publicly they have nothing to hide. Great start to resolving the problem. A lot of us are pretty peeved (not Bill Rose, he doesn’t get peeved) at the attitude from the USGBC so we developed our own attitude. This release of data goes a long way to ratcheting down the tension. After our side vents a little bit we both should get on with the business of getting better buildings. The statistical analysis was done by Paul Francisco.

iv Think about what is happening behind all of the numbers. The building codes use ASHRAE Standard 90.1 to establish a “floor” or minimum for energy performance. Very few buildings, if any, are built to go beyond the building code minimums so the CBECS plot is really a plot of ASHRAE 90.1. LEED uses ASHRAE Standard 90.1 to establish a target. Guess what? The target appears to have been met. The “target” resembles the “floor.” There should be no surprise that the two data sets are pretty much the same. So how to fix this? Many folks, including the ones who helped me with this column feel that the problem is only partially LEED—they feel the real problem is ASHRAE Standard 90.1. I am not completely there yet. But the folks at Standard 90.1 are getting pretty hard to defend when they go out and say that airtight building enclosures do not save energy and airtightness standards have no place in 90.1. Fixing LEED might best happen while also fixing ASHRAE 90.1.