A Gentle Introduction to Statistical Tolerance Intervals in Machine Learning

These bounds can be used to help identify anomalies and set expectations for what to expect. A bound on observations from a population is called a tolerance interval.

A tolerance interval is different from a prediction interval that quantifies the uncertainty for a single predicted value. It is also different from a confidence interval that quantifies the uncertainty of a population parameter such as a mean. Instead, a tolerance interval covers a proportion of the population distribution.

In this tutorial, you will discover statistical tolerance intervals and how to calculate a tolerance interval for Gaussian data.

After completing this tutorial, you will know:

That statistical tolerance intervals provide a bounds on observations from a population.

That a tolerance interval requires that both a coverage proportion and confidence be specified.

That the tolerance interval for a data sample with a Gaussian distribution can be easily calculated.

Let’s get started.

A Gentle Introduction to Statistical Tolerance Intervals in Machine LearningPhoto by Paul Hood, some rights reserved.

The interval is limited by the sampling error and by the variance of the population distribution. Given the law of large numbers, as the sample size is increased, the probabilities will better match the underlying population distribution.

Below is an example of a stated tolerance interval:

The range from x to y covers 95% of the data with a confidence of 99%.

If the data is Gaussian, the interval can be expressed in the context of the mean value; for example:

x +/- y covers 95% of the data with a confidence of 99%.

We refer to these intervals as statistical tolerance intervals, to differentiate them from tolerance intervals in engineering that describe limits of acceptability, such as for a design or of a material. Generally, we will describe them as simply “tolerance intervals” for convenience.

A tolerance interval is defined in terms of two quantities:

Coverage: The proportion of the population covered by the interval.

Confidence: The probabilistic confidence that the interval covers the proportion of the population.

The tolerance interval is constructed from data using two coefficients, the coverage and the tolerance coefficient. The coverage is the proportion of the population (p) that the interval is supposed to contain. The tolerance coefficient is the degree of confidence with which the interval reaches the specified coverage. A tolerance interval with coverage of 95% and a tolerance coefficient of 90% will contain 95% of the population distribution with a confidence of 90%.

How to Calculate Tolerance Intervals

The size of a tolerance interval is proportional to the size of the data sample from the population and the variance of the population.

There are two main methods for calculating tolerance intervals depending on the distribution of data: parametric and nonparametric methods.

Parametric Tolerance Interval: Use knowledge of the population distribution in specifying both the coverage and confidence. Often used to refer to a Gaussian distribution.

Nonparametric Tolerance Interval: Use rank statistics to estimate the coverage and confidence, often resulting less precision (wider intervals) given the lack of information about the distribution.

Tolerance intervals are relatively straightforward to calculate for a sample of independent observations drawn from a Gaussian distribution. We will demonstrate this calculation in the next section.

Tolerance Interval for Gaussian Distribution

In this section, we will work through an example of calculating the tolerance intervals on a data sample.

First, let’s define our data sample. We will create a sample of 100 observations drawn from a Gaussian distribution with a mean of 50 and a standard deviation of 5.

1

2

# generate dataset

data=5*randn(100)+50

During the example, we will assume that we are unaware of the true population mean and standard deviation, and that these values must be estimated.

Because the population parameters have to be estimated, there is additional uncertainty. For example, for a 95% coverage, we could use 1.96 (or 2) standard deviations from the estimated mean as the tolerance interval. We must estimate the mean and standard deviation from the sample and take this uncertainty into account, therefore the calculation of the interval is slightly more complex.

Next, we must specify the number of degrees of freedom. This will be used in the calculation of critical values and in the calculation of the interval. Specifically, it is used in the calculation of the standard deviation.

Remember that the degrees of freedom are the number of values in the calculation that can vary. Here, we have 100 observations, therefore 100 degrees of freedom. We do not know the standard deviation, therefore it must be estimated using the mean. This means our degrees of freedom will be (N – 1) or 99.

1

2

3

# specify degrees of freedom

n=len(data)

dof=n-1

Next, we must specify the proportional coverage of the data. In this example, we are interested in the middle 95% of the data. The proportion is 95. We must shift this proportion so that it covers the middle 95%, that is from 2.5th percentile to the 97.5th percentile.

We know that the critical value for 95% is 1.96 given that we use it so often; nevertheless, we can calculate it directly in Python given the percentage 2.5% of the inverse survival function. This can be calculated using the norm.isf() SciPy function.

1

2

3

4

# specify data coverage

prop=0.95

prop_inv=(1.0-prop)/2.0

gauss_critical=norm.isf(prop_inv)

Next, we need to calculate the confidence of the coverage. We can do this by retrieving the critical value from the Chi Squared distribution for the given number of degrees of freedom and desired probability. We can use the chi2.isf() SciPy function.

1

2

3

# specify confidence

prob=0.99

chi_critical=chi2.isf(q=prob,df=dof)

We now have all of the pieces to calculate the Gaussian tolerance interval. The calculation is as follows:

1

interval=sqrt((dof *(1+(1/n))*gauss_critical^2)/chi_critical)

Where dof is the number of degrees of freedom, n is the size of the data sample, gauss_critical is the critical value, such as 1.96 for 95% coverage of the population, and chi_critical is the Chi Squared critical value for the desired confidence and degrees of freedom.

1

interval=sqrt((dof *(1+(1/n))*gauss_critical**2)/chi_critical)

We can tie all of this together and calculate the Gaussian tolerance interval for our data sample.

10 Responses to A Gentle Introduction to Statistical Tolerance Intervals in Machine Learning

Dear Dr Jason,
(1)
Could you please refer me to where the formula for the interval in line 25 of the first example code.came from:

1

interval=sqrt((dof *(1+(1/n))*gauss_critical**2)/chi_critical)

(2) Related to (1) is the output:

1

47.95to52.66covers95%of data withaconfidence of99%

My question is this:
I understand the idea of the confidence values of mean +- 2*std_dev. That is there is a 95% probability that the mean is within these values.
What I don’t understand is the output 95% of data with a confidence of 99%. What is the meaning of 95% with 99% confidence.

The distinction between a confidence interval and a tolerance interval is “Confidence limits are limits within which we expect a given population parameter, such as the mean, to lie. Statistical tolerance limits are limits within which we expect a stated proportion of the population to lie.” 2nd para of the webpage.

The key distinction is limits within a population parameter (eg mean) and limits of a stated proportion of a popultaion.

Hi Jason,
Isn’t a multiplication by the sample standard deviation needed in the formula for interval calculation?
“The size of a tolerance interval is (reversely ?) proportional to the size of the data sample from the population and proportional the variance of the population.”

Jason, to follow on this – increasing the standard deviation in your code example has no effect on the tolerance limits calculated. How can this be? If the standard deviation is increased then surely the tolerance limits must increase as well.

i dont quite understand. There is no difference between the cases when I have sample mean of 1 and sample mean of 1000. I get the same tolerance interval. Since the codes here do not use the mean of my samples at all expect for the last line. So say i want to compute 0.99 portion and 0.95 confidence. I get my tolerance interval to be 3.355 and 102.355 with the same number of sample.