Evaluation of State Quality Assurance Program Effectiveness

APPENDIX B

SPECIFICATION 2

Per FHWA request, appendix B was added to this report. This appendix is a repetition of
the initial analysis of the HMA acceptance plan for specification 2 except that the thickness
quality characteristic has been excluded, leaving only density and air voids. Appendix B was
added because the specification is frequently applied without including thickness as part of
the pay-adjustment acceptance procedure. Repetition was used to make it easier for readers
to understand.

Acceptance Procedure Synopsis

Type of Specification

HMA pavement.

Quality Characteristics

Density (one-sided, field test).

Air voids (two-sided, plant test).

Correlation of Quality Characteristics

No information available; assume uncorrelated.

Lot Size Definitions

Density:. Day’s production.

Air voids:

Four contiguous air tests taken at the plant.

Acknowledged to be different from density lots.

Applies to tonnage represented.

Sample Sizes

Density, n = 10.

Air voids, n = 4.

Statistical Quality Measure

PWL.

Individual Pay Equations (PA as Decimal)

Surface, base, and pavements.

Density: Figure 103 is used to calculate the pay equation for density as follows:

If either PWLLSL or PWLUSL < 50, or if PWLTOTAL < 50, agency will evaluate. If left
in place, PA = -0.12 (value for PWLTOTAL = 50).

RQL consequence is not specified in a form that can be analyzed by SPECRISK.

An additional t-test requirement leads to actions that cannot be accounted for by
SPECRISK.

Composite Pay Equation

Not explicitly stated but assumed directly additive.

AQL/RQL

AQL of PWL = 90 is implied for all characteristics.

RQL consequence is not specified in a form that can be analyzed by SPECRISK but the
RQL can be set at PWL = 50 for counting purposes.

Pay Equation Transformations and Assumptions

To analyze this specification with SPECRISK, transformations and assumptions must be made.
For example, the specification implies that the individual PFs are directly additive to obtain the
overall PA for the project. This is the logical approach, and no other approach is explicitly stated.
Although neither of the individual PAs may match up exactly with any other in terms of precise
pavement locations, they collectively account for the entire paved area of the project. SPECRISK
accommodates this procedure, but the actual analysis assumes that each set of tests for different
quality characteristics applies to identically the same lot. It is believed, but has not been
demonstrated, that this assumption of common lot sizes will have little if any effect on the
final results.

Density

The equation for density in figure 103 is already in a form that can be accommodated by
SPECRISK, but it must be noted that this applies for 50 < PWL ≤ 100. For PWL < 50,
PA = -0.16 (constant). Both segments of this compound linear pay equation will be entered
into SPECRISK, as will be explained in the following sections.

Although an RQL consequence is not specified as such, SPECRISK requires that a value is
entered into the program. Entering PWL = 50 provides a frequency count of estimates for which
PWL < 50, which may trigger reevaluation by the SHA.

Air Voids

The equation for air voids in figure 104 must be modified slightly. By using the equation in
figure 105, figure 104 can be rewritten as the equation in figure 106, which is in the same form
as figure 103.

Figure 105. Equation. Fundamental PWL relationship.

Figure 106. Equation. Modified individual air voids pay equation.

Again, this applies specifically for 50 < PWL ≤ 100. For PWL < 50, a second segment described
by PA = -0.12 is required, both of which will be entered into SPECRISK.

As previously noted, PWLLSL or PWLUSL < 50 or PWLVOIDS < 50 will trigger a reevaluation by
the SHA, which might lead to a requirement to remove and replace the affected material, or if
left in place, the minimum PA = -0.12 is applied. Unfortunately, SPECRISK does not have the capability to perform the first of these two tests, so only the second one can be included in these
analyses. It is believed, but cannot be known for sure, that this will have little effect on the
final results.

There is also a special t-test requirement for air void tests. This is another refinement that
SPECRISK cannot duplicate, so it is necessary to assume that the effect will be minor.

Entering Quality Characteristics in SPECRISK

A key step of the setup procedure in SPECRISK is to make sure that the necessary quality
characteristics have been listed for later selection as the specification profile is filled out. If
SPECRISK has been used frequently for previous analyses, these may already exist. Figure 107
shows that several characteristics had been used previously, but, three new characteristics
pertaining to this specific example were added. (The new characteristics begin at the highlighted
line with the exception of the generic characteristic for smoothness.)

The next step is to make sure the necessary pay equations are included as part of the setup
procedure. In this case, none of the above pay equations had been previously entered into this
particular installation of SPECRISK. However, the pay equation wizard makes it easy to enter
these three pay equations as linear compound equations.

Density

Figure 103, the density pay equation, is shown in figure 108 as it appears after entry into the pay
equation wizard.

Following the input of the equation coefficients, it can be helpful to plot the resulting pay
equation to check for potential errors and discontinuities. This is accomplished by clicking the
“Graph” tab in figure 108, which displays the plot shown in figure 109 in which the green and
red segments represent the equation segments defined in figure 108.

Initial setup for simulation profile follows the same steps described in the "Entering Data
into Simulation Profile" section for specification 1. In this example, the initial profile with all
three quality characteristics can be found with the name "TOPR Number 11 — Specification
Number 2 — HMA." The user clicks on it to select it, then right-clicks to select "Insert New
Record Like This" in the drop-down window, as shown in figure 111.

Figure 111. Screenshot. Entering a new simulation profile.

This action causes a new simulation profile line to be added at the end of the list. The new profile
is given a name, and a value of 1,000 is entered for the number of iterations.

Next, the information from the profile that has been copied is checked to match the "Acceptance
Procedure Synopsis" section. This will consist primarily of removing any reference to the
thickness quality characteristic, which has been eliminated from this analysis. To begin, the
user selects "Quality Characteristic Data Entry" and deletes thickness by highlighting the
appropriate row with a left click and then right-clicking to bring up the dropdown menu shown in
figure 111. To delete thickness, the user clicks on "Delete." The user moves successively to the
right across the screen to select the remaining yellow buttons and continues to delete all
references to thickness.

Figure 112 shows the "Quality Characteristic Data Entry" screen after the thickness
characteristic has been deleted.

Since no information regarding correlation of the quality characteristics was provided for this
example, it was assumed for the original run that all correlation values are zero (the default
condition for SPECRISK) and no changes are required to the "Correlation Data Entry" tab.

The individual PAs are directly additive for this example. In the "Pay Relationship Data Entry"
tab, the appropriate selection of "Sum" in the "Pay Relationship" column had been made
initially, and no change is required.

The "OC Curves Data Entry" tab allows the input of user-selected levels of PA for which OC
curves will provide the likelihood of achieving this level of adjustment, or higher, at various
levels of as-constructed PWL. This result will be accessible by selection of the "Multiple OCs
Graph" in the lower section of this screen after the analysis has been run. The previously entered
levels of PA will be sufficient for this analysis.

The remaining input tab on the profile screen is "Simulation Points Data Entry." By default,
SPECRISK will analyze all possible combinations of quality levels among the selected quality
characteristics in PWL increments of 10. This usually provides a fairly complete picture of the
performance of the acceptance procedure over the entire range of possible quality-levels, so no
additional increment levels need to be added.

Preliminary Analyses

Following the input of essential entries necessary to analyze the specification, it is customary to
do a preliminary analysis by entering a few key quality level combinations rather than a
performing a complete analysis. After entering a few quality level combinations in the lower
table, analysis is initiated by clicking "Analyze Selected" near the lower right corner of the
profile screen. The results of one of these runs are shown in figure 113. This analysis can provide
almost immediate results at quality levels of interest and may also allow the user to spot obvious
input errors.

Figure 113. Screenshot. Preliminary run using "Analyze Selected."

One technique that is frequently used when running "Analyze Selected" is to run duplicates of
some combinations of quality levels to get an impression of how repeatable (e.g., reliable) the
results are. Since the analysis technique used in SPECRISK is computer simulation, it is normal
for successive runs (i.e., using automatic random seed numbers) to give slightly different results,
and the results will be more variable the smaller the number of iterations that are used. This
particular simulation, using two quality characteristics, is considerably less computationally
intensive than simulations using four or five characteristics. Thus, it was possible to obtain
results very quickly using 1,000 iterations for these trial runs (as indicated in the upper table in
figure 113). With 1,000 iterations, it is common for the results to be in good agreement.

One combination of particular interest occurs at the formal AQL when each individual quality
characteristic is precisely at its respective AQL. Since both individual pay equations produce a
PA of zero, (i.e., 100 percent payment) at PWL = 90, this is the inferred AQL. This level of
quality has been duplicated in the lower table in figure 113, as indicated by the two highlighted
rows. The first run produced exactly the desired average PA of zero, while the second one was
very close. Therefore, this run has provided two important pieces of information—the acceptance
procedure properly accepts AQL quality at full payment, and this result appears to be repeatable.

A similar test was run in the fourth and fifth analysis rows of figure 113. In this case, both
characteristics have been set at PWL = 50, which would generally be regarded as a poor quality
lot. Again, the results are very consistent, both being close to a PA of -0.25. The SHA must
judge if the amount of payment withheld is reasonably appropriate for this level of quality.

Another common type of test includes setting the independent variables at their extremes, where
it is often possible to deduce what the correct result should be. In the first row in figure 113, both
quality characteristics have been set at their maximum possible value of PWL = 100. Figure 103
and figure 106 produce incentive (bonus) PAs of +0.04 and +0.03, respectively, when all
characteristics are at PWL = 100. Since these PAs are additive, the net overall PA is +0.07, as
correctly reported in the first row of figure 113.

At the other extreme, all quality characteristics have been set at the minimum value of PWL = 0
in the last row in figure 113. Again by inspection, it can be seen that density and air voids
receive minimum PAs of -0.16 and -0.12, respectively, at PWL = 0. Therefore, when both
quality characteristics are at PWL = 0, the overall PA is calculated as -0.16 + (-0.12) = -0.28, as
borne out in the last row in figure 113.

These results are a strong indication that the many inputs have been entered properly, and it is
reasonable to conclude that the software is performing the analyses correctly. At this point, it is
appropriate to proceed with some additional preliminary analyses of interest.

This specification has virtual RQL provisions in that when PWL < 50, the SHA has option to
reevaluate either density and air voids, or both. It is of interest to know what the typical PA will
be when the quality levels are poor but not poor enough to trigger these provisions.

To answer this question, the run shown in figure 114 is made. For this test, each characteristic is
set at a low quality level of PWL = 50, while the other is set at the maximum quality level of PWL = 100. As in the previous test, the quality level combinations have been duplicated to get a
reading on repeatability. All the results appear to be very repeatable, and the PAs when
individual characteristics are set at PWL = 50 are approximately -0.07 and -0.11 for air voids and
density, respectively. The SHA will have to decide if this provides adequate protection against
these low levels of quality, but SPECRISK has provided information upon which these decisions
can be made.

An additional series of tests will be useful to provide a reasonableness check of the preliminary
results. For these tests, each individual quality characteristic is varied throughout the complete
range of possible quality levels, while the other is held at the AQL of PWL = 90. These results
are presented in figure 115 and figure 116.

Figure 115. Screenshot. Extended test of air voids PAs.

Figure 116. Screenshot. Extended test of density PAs.

Several interesting observations can be made from this last group of test runs. As each quality
characteristic is decremented by PWL = 10 while the other characteristic is held constant at the
AQL of PWL = 90, the expected PA (average) decreases steadily until its individual minimum
PA is approached. Beyond that, any decrease is very small and is due partly to the small
variability of the sum of the other adjustment (which randomly varies about zero, the expected
value at the AQL). As a result, it is possible on any particular run for the lot PA to reach a
minimum just slightly below the minimum PA for the characteristic under test, and the last two or three rows can go out of order due to random chance, as seen in figure 115 and figure 116.

These two figures also show that at the point at which both characteristics are at the AQL level
of PWL = 90, the resultant expectant PA (average) is almost exactly zero (full payment). This
reinforces the conclusion that this is an unbiased pay schedule that will award 100 percent
payment in the long run at the AQL.

Ambiguity at the AQL

Before proceeding with the complete analysis, there is an additional topic that warrants
discussion. As noted previously, the condition at which all quality characteristics are
simultaneously at their respective AQL values can be regarded as a formal definition of an AQL
lot, when full payment should be awarded. But when analyzing multicharacteristic acceptance
procedures, it is apparent that there are many other combinations of quality levels that will also
produce full payment. Furthermore, when an attempt is made to develop performance models of
expected pavement life as a function of as-constructed quality, it is discovered that this may be a
very appropriate feature of multicharacteristic acceptance procedures. In other words, it is not
uncommon to expect a slight surplus of quality in one quality characteristic to offset a slight
deficiency in another quality characteristic (within reasonable limits).

To study the performance of the specification at hand in terms of this offsetting property, a series
of tests can be run for which each quality characteristic in turn is cycled from its AQL downward as the other characteristic is held at the maximum quality level of PWL = 100. In this manner, it
is possible to discover just how low in quality each characteristic can go with the lot as a whole
still receiving full pavement. The first such test is for air voids, as shown in figure 117.

In figure 117, the air voids characteristic is tested over a sufficient range of PWL (90 to 50) to
observe that somewhere between PWL = 80 and 70, it will produce an expected PA of zero
(100 percent payment) and thus be equivalent to an AQL combination. A second run is shown in
figure 118 to determine this critical value more precisely.

Figure 118 shows that full payment (PA = 0) occurs at a point very close to an air voids quality
level of PWL = 77. The SHA must decide whether or not full payment is acceptable for this
combined level of lot quality in the two characteristics. The same series of tests was run for
density, and the final result is shown in figure 119.

Figure 119 indicates that full payment (essentially PA = 0) can occur when density quality is as
low as PWL = 83. Provided the SHA is satisfied with this possibility, this may be regarded as a
suitable acceptance procedure, especially if it has had a history of encouraging good quality on
the part of the construction industry.

Time-Saving Procedure

SPECRISK is capable of performing a full factorial experiment by simulating all combinations
of quality levels in all of up to five quality characteristics, typically in PWL increments of 10
(or PD = 10) and graphing most of these results as selected by the user. With a relatively large
number of simulation replications, a complete analysis such as this may require several hours of
computational time, depending on the speed of the computer on which it is run. However, it is
often possible to obtain certain key results without such a time-consuming effort.

A useful series of quality levels for analysis consists of all possible combinations of AQL
and RQL values for all quality characteristics. This subset of data can be selected upon
completion of the complete analysis, but it is also relatively easy to set up for fast results with
"Analyze Selected."

For the full complement of five quality characteristics, this would require entry of 25 = 32 rows
of quality combinations. For the specification currently under analysis, there are only two quality
characteristics, which require only 22 = 4 lines of data. The AQL has been implied as PWL = 90
for all quality characteristics because that equates to a PA of 0. For air voids and density, the
RQL has been taken to be PWL = 50 because that is the level of quality below which the agency
could decide to reject the work. The experimental design and the results of this analysis are
shown in figure 120, where the results for the AQL/RQL combinations are presented.

Figure 120. Screenshot. Test with all AQL/RQL combinations.

Figure 120 shows that when both quality characteristics are at the AQL of PWL = 90, the
resulting average PA is essentially zero, representing full payment for quality precisely at the
level specified. The remaining three lines represent successively poorer combinations of quality
and demonstrate that the associated expected pay levels decline accordingly.

In order to address the positive side of the pay schedules, figure 121 shows the level of incentive
PA the quality-conscious contractor can receive by controlling production at or near the
maximum quality level of PWL = 100. Although in theory no lot can achieve complete
perfection as indicated by a PWL level of 100 percent, it is possible for good quality lots to test
out at PWL = 100 because of random sampling variability, and they often do. Whenever this occurs, the result in figure 121 indicates the contractor will receive an expected (average)
PA = 0.07, which is a positive incentive (bonus) payment of 7 percent.

For many agencies, the preliminary analyses may be sufficient to confirm that the acceptance
procedure is working satisfactorily or demonstrate that modifications need to be made. For those
desiring more detailed information, a full analysis can be run. One major advantage of a full
analysis is that it provides access to the many graphical and tabular displays of data that
SPECRISK can provide.

The most important graph for SHAs and contractors is the EP graph, which illustrates how
payment is related to quality delivered. The EP graph produced by the full analysis is shown in
figure 122.

Figure 122. Graph. EP graph available with full analysis.

The red line in figure 122 represents the average (or expected) PA, and the figure represents
the composite, or overall, combined PA resulting from the PAs for the individual quality
characteristics. The red line shows the gradual increase from the minimum PA of -0.28 at
PWL = 0 up to the maximum PA of 0.07 at PWL = 100. This graph allows both the SHA and its contractors to see exactly how the acceptance procedure will perform and what the consequences
of both good and poor quality control are likely to be.

For example, highway specification engineers can see that this specification awards 100 percent
payment at the AQL of PWL = 90. Figure 122 also illustrates the hover feature of SPECRISK. A
quantitative reading can be obtained from the graph by hovering the cursor close to the point of
interest. In this case, a PA of 0.0023 (essentially zero) is expected at the AQL of PWL = 90.

With this information, SHAs can judge if the acceptance procedure is both effective and fair, and
contractors can see the benefits of targeting quality levels at or above the AQL. The highway
agency has an abiding interest in confirming that the rate at which payment declines as the
quality drops below the design quality level is both effective and appropriate. This graph also
communicates to the contractor the seriousness of the specifications and that poor quality likely
will be detected and assessed appropriate pay reductions. However, this also illustrates the
potential monetary benefits available to the contractor who exercises good quality control and
produces work at or above the AQL.

Figure 123 shows how the "Multiple OCs Graph" can provide still more information on this
subject. Like figure 122, the user has again selected the composite analysis. The legend at the
right of the figure lists several PA levels previously entered by the user, and the blue line
represents full payment (PA = 0).

Figure 123. Screenshot. Display of multiple OCs graph.

In figure 123, the hover feature shows the probability of achieving full payment (or greater)
when the work is controlled precisely at the AQL of PWL = 90 to be 55.9 percent (0.559),
meaning that there will be slightly more positive PAs than negative PAs. In this case, the
contractor who can successfully produce at the AQL can expect to average somewhat better than full payment in the long run because the positive PAs will outweigh the negative PAs to some
extent (consistent with the result in figure 122).

This graph can also provide guidance for the contractor who might prefer to control production
so that there will more “bonuses” than reductions throughout the course of a project. For
example, if a contractor wished to obtain bonuses about 80 percent of the time, a production
level of quality of about PWL = 95 would be required. If it were relatively easy for a contractor
to achieve this level of quality, this might well be a profitable strategic decision. In any case,
SPECRISK has provided information to help in making this decision.

Analysis Summary and Conclusions

Significant findings unique to the analysis in this appendix were as follows (additional
conclusions for the full scope of this specification are listed in the main body of the report):

One of the important features of these analyses is to confirm that the pay schedule is
capable of paying 100 percent as a long-term average (i.e., PA = 0) when the quality of
production is at the level that is defined as acceptable in the specification. In this
example, the tests confirmed that the overall combined PA was essentially zero.

The formal definition of an AQL for a multicharacteristic acceptance procedure should be
that combination of quality levels for which each quality characteristic is precisely at its
respective AQL value. There are essentially an unlimited number of equivalent AQL
combinations for which the resultant composite PF will turn out to be PA = 0. The
important thing to check is whether one or more such combinations might exist for which
the agency would consider it clearly inappropriate for full payment. Example test runs
were presented in figure 117 through figure 119. The SHA using the specification is
required to make the final decision.

The "Acceptance Procedure Synopsis" section for this specification indicates that for
both density and air voids, the resulting PAs (in decimal form) are applied to a flat rate of
$40 per ton. In the original analysis that included thickness, the dollar value of the PA
was obtained by multiplying the decimal adjustment by $1.90 by the area (square yards)
by the thickness (inches), which works out to be very close to $40 per ton. It is presumed
that the specification designers took this approach as a countermeasure to possible
temptation on the part of contractors to unbalance their bids for strategic advantage. As
such, it should be an effective deterrent.

Figure 115 and figure 116 included duplicate runs to demonstrate how repeatable
the independent simulation results would be with 1,000 iterations. For preliminary
exploratory work, the goal is to strike a reasonable balance between precision of
results and speed of response. The checks shown in these two figures illustrate that
1,000 iterations will usually be satisfactory. In general, it is desirable to work at the
highest practical level of precision, and it is usually desirable to use an even greater
number of iterations for final results.

Figure 122 and figure 123 show graphs describing payment results to be expected with
this acceptance procedure. This information will be of particular interest to both the SHA
and the construction industry.