An evaluation of accuracy is best done in terms of bias:
bias: The average value of errors in a measurement. Lower bias measure higher accuracy.

Biases in Calorie Counting

Calorie counting systems possess several biases. They still help people lose weight, but they do so in spite of the biases.

Calorie counting systems usually have two parts: (i) a measurement of the calories you consume in a day, and (ii) a limit on the number of calories you consume in a day. The limit is derived from a model like this:

Calories In - Calories Out < 0

where Calories In is the number of Calories you measures and Calories Out is given by

Bias: EER

EER is defined in report Institute of Medicine Dietary Reference Intakes (DRI) as "the average dietary energy intake that is predicted to maintain energy balance in a healthy adult of a defined age, gender, weight, height, and level of physical activity, consistent with good health". The DRI presents a statistical model of EER based on age, gender, etc. Like any model there is some uncertainty (or spread as defined previously). The spread in the estimate of EER is about 200 Calories for males and 180 Calories for females Table 5-14.

When applying the EER estimate to calorie counting we'll get a number for Calories Out and try to eat less than that number for many days in a row. Imprecision in the estimate of that number thus appears as a persistent bias in our calorie counting system. The DRI describes the situation this way: "By definition, the estimate would be expected to underestimate the true energy expenditure 50 percent of the time and to overestimate it 50 percent of the time, leading to corresponding changes in body weight."

EER cross-sectional imprecision leads to a persistent bias in a calorie-counting system -- in Calories Out -- of around +/- 200 Calories.

Bias: Calories In

If you've ever dieted you'll be unsurprised to know that people underreport the amount of calories they've eaten. They even do it when they're not dieting. They even do it if they're dieticians (see below).

Underreporting give a persistent bias to Calories In of around 400 Calories.

Bias: Activity

Activity level is specified categorically as Sedentary, Low Active, Active, or Very Active. These categories appear in the estimation formulas for EER given on p. 185 of the DRI.

Each category corresponds to a range of calories expended on physical activity. We can use the formulas to estimate the span of calories covered for a category. For example, for a man weighing 180 lb. (81.6 kg) and standing 5' 10" (1.78m) tall, we have

If we change the gender, height, or weight in the calculations we'll get different answers. Also, it is unclear what the span of "Very Active" is, since there is no specified or obvious value for the upper limit on the amount of calories a person can expend through physical activity.

The point, however, is that there is some uncertainty in the specification of activity level. If the example individual correctly chose "Low Active" as his activity level he would have an uncertainty in the calories expended of 317 Calories/2 = 158.5 Calories. This discrepancy is already included in the EER bias discussed above.

If he chose the wrong category he'd introduce a new, persistent bias that is not part of the EER bias. Since overreporting of physical activity has been observed, there might also exist bias in a person's selection physical activity category.

Dealing with Bias

These biases affect the quality of your calorie counting by creating a diet that, on average, has more calories than expected. Additionally the biases are persistent which means you should expect to be biased by the same amount every day. In other words, you can expect to overeat relative to your goal every day if you use the formulas directly.

Fortunately, the DRI also provides methods for dealing with bias.

Method I (Dynamic):

"This indicates that monitoring of body weight would be required when implementing intakes based on the equations that predict individual energy requirements. For example, if subjects were enrolled in a study in which it was important to maintain body weight, each individual would be fed the amount of energy estimated to be needed based on the EER equation. Body weight would be closely monitored over time, and the amount of energy provided to each individual would be adjusted up or down from the EER (or TEE) as required to maintain body weight."

Method II (Static):

"If the goal of planning is to prevent weight gain in an individual with specified characteristics, the appropriate EER equation could be used to derive the aver- age energy expenditure for the individual, and then subtract an amount equal to two times the SD. This would lead to an intake that would be expected to fall below the actual energy requirements of all but 2.5 per- cent of the individuals with similar characteristics. Using the above example for the 33-year-old, low-active woman, the energy requirement would be 2,028 – (2 × 160) kcal, or 1,708 kcal. This intake would prevent weight gain in almost all individuals with similar characteristics. Of course, this level of intake would lead to weight loss in most of these individuals."

Method I says to start with the amount recommended by the EER formula, monitor a person's weight, and adjust the calorie intake accordingly if weight is not maintained. Method II says to set the calorie intake so low that you overcome the "spread" (or imprecision) of the estimate and increase the probability of avoid weight gain or inducing weight loss.

Method I addresses weight maintenance rather than weight loss, but one could modify the suggestion to say, "If weight loss is not achieved, then adjust the energy provided down." In other words, if you're not losing weight, trying eating less. In fact, if the purpose is solely to lose weight, that adaptive strategy should work from any starting calorie intake -- not just from the EER estimate.

Pertinacity adopts Method I with the modification for weight loss just described. Additionally, Pertinacity uses as a starting point (instead of the EER formula) a measurement of your current caloric intake. In short, it says, "Try eating less than you have been and see if you lose weight." And it keeps saying that every day.

Specifically, Pertinacity recommends a maximum of one fist-size portion (FSP) reduction in your calorie intake relative to your current level of intake. Based on the data collected from Amazon's Mechanical Turk (see previous blog post for a description of the data), this reduction should be equivalent to 11% on average, and about 6%-13% for most people. Since the recommended limit on FSP is always based on the trailing two weeks of measurements this number will change slowly. (It won't decrease by 11% every day, for example. It'll start at an 11%-ish reduction then when you are successful at eating at the reduced level for a couple of weeks it'll make another reduction. The process is gradual.)

Notice that using this moving average method for recommending a limit on intake obviates the need to explicitly model calories in terms of FSP. That process would require more data from the user, so it represents another simplification and, thus, another improvement in ease-of-use. Additionally, due to the use of this method, Pertinacity does not need to collect data on activity or the other user characteristics needed to estimate required energy intake.

To deal with biases in calorie counting systems and avoid having to model calories in terms of FSP, Pertinacity employs a dynamic algorithm that works like this: Reduce your FSP intake and that will reduce your calorie intake which will, in turn, reduce your weight.

Conclusion

Calorie counting systems have several biases embedded in them, including (i) statistical error in the EER estimate of calories required (+/- 200 Calories), (ii) underreporting calories consumed (400-500 Calories), and (iii) overreporting of activity. Nevertheless, the systems work for their users, in part, because of the use of a bias-defeating method like (I) dynamically decreasing the calorie intake recommendation until the desired result is achieved, or (II) recommending a large, static, change in calorie intake.

Pertinacity uses a gradual, dynamically-changing limit on the number of fist-sized portions the user may consume. This makes the system accurate and simplifies it dramatically.