Additional Materials:

Contact:

Sig Sauer, Inc., of Exeter, New Hampshire, protests the rejection of its proposal under request for proposals (RFP) No. DJA09S000028, issued by the Bureau of Alcohol, Tobacco, Firearms and Explosives (ATF), for semi-automatic handguns.

We deny the protest.

B-402339.3, Sig Sauer, Inc., July 23, 2010

DOCUMENT FOR PUBLIC RELEASEThe decision issued on the date below was subject to a GAO Protective Order. This redacted version has been approved for public release.

In a negotiated procurement for a handgun system that provided for a three'phased evaluation, the rejection of the protester's proposal after phase II was unobjectionable where the agency reasonably assessed the protester's handgun to be unsatisfactory following live-fire testing and did not treat offerors unequally in conducting the testing.

DECISION

Sig Sauer, Inc., of Exeter, New Hampshire, protests the rejection of its proposal under request for proposals (RFP) No. DJA09S000028, issued by the Bureau of Alcohol, Tobacco, Firearms and Explosives (ATF), for semi-automatic handguns.

We deny the protest.

BACKGROUND

ATF, a principal law enforcement agency within the Department of Justice, seeks to replace its current metal-frame handguns with a new .40 caliber, polymer-frame handgun system.[1] This system consists of two handguns, which are issued to each ATF agent: a standard-size duty weapon for enforcement operations and a compact model for backup and auxiliary needs; the two models are in all other respects required to be identical in function and design. RFP, Statement of Work (SOW), at 22. The SOW stated detailed design and performance specifications, describing the exact features and performance characteristics required.

Offerors were instructed to submit written proposals and sample handguns, in both standard and compact sizes, to be evaluated in three phases. RFP at 58-61. Under phase I, the handguns were to be evaluated on a pass/fail basis for compliance with the RFP's specifications; all submissions assessed as satisfying all specifications moved to the phase II evaluation. Under phase II, ATF planned to perform a live-fire assessment of the guns, consisting of the following four parts: an accuracy test, a sight evaluation test, an evaluation by a cross-section of ATF agents, and an assessment of the interchangeability of parts. Id. at 62. Offerors were informed that offers "deemed most suitable for performance" would continue to phase III.[2]Id. The RFP provided for the award of a single or multiple indefinite-delivery, indefinite'quantity (IDIQ) contract(s) to the offeror(s) whose proposal(s) were found to represent the best value after the phase III evaluation. Id. at 66-67.

At issue here is the live-fire testing by a cross-section of ATF agents, which was identified in the RFP as the most important among the four parts of the phase II testing, described above. RFP at 63. Under this test, 20 ATF agents with diverse skill levels, physical size, and right- or left-handedness were to evaluate each of the handguns that were submitted. Each agent evaluated the sample handguns first by firing 100 rounds of familiarity shooting, then twice firing 50 rounds for the pistol qualification course (PQC) test, and finally field stripping, cleaning and reassembling each type of handgun. For each handgun, the agent completed a questionnaire assigning adjectival ratings (excellent, very good, good, fair, or unsatisfactory) and providing narrative under the following categories: safety, reliability, accuracy, ergonomics and adaptability, suitability of size or weight, ease of cleaning and assembly, and overall assessment as a pistol for general issue to ATF agents.

The offerors' handgun systems received the following number of adjectival ratings in the areas of reliability and general assessment from the ATF agents who fired them:

In addition to the agents' rating sheets, a record was kept of any stoppages or malfunctions that occurred during the live fire testing.[3] These records show that ATF's agents recorded 58 stoppages with Sig Sauer's full-size and compact pistols, 13 of which were considered to be gun-induced and 45 shooter-induced.[4]Id., Tab 2, Competitive Range Determination, at 3.[5] In contrast, the agents recorded a total of 16 shooter'induced stoppages for Smith & Wesson's guns and 7 shooter-induced stoppages for Glock's guns. There were no gun-induced stoppages recorded for the Smith & Wesson or Glock guns. Id., Tab 8, Contracting Officer's Smith & Wesson Evaluation Summary, and Tab 9, Contracting Officer's Glock Evaluation Summary.

The shooters' phase II evaluations of the offerors' handgun systems were provided to the agency's source selection board (SSB).[6]See Declaration of SSB Chair, Feb. 2, 2010, at 2. According to the SSB chair, "[i]t was determined that any weapon that displayed numerous gun-induced or shooter-induced malfunctions was not suitable for agents to depend on to protect their lives." Id. at 3. Based on the results of the shooting tests, the SSB recommended to the contracting officer that Sig Sauer's handgun be excluded from further consideration as unacceptable with respect to reliability. The contracting officer reviewed the SSB's consensus reports and agreed with the recommendation that Sig Sauer's proposal should be excluded from phase III, based upon the number of stoppages recorded for Sig Sauer's handguns. AR, B-402339.3, Tab 2, Competitive Range Determination, at 1.

Following notification of the rejection of its proposal and a debriefing, Sig Sauer protested to our Office, challenging the agency's phase II live-fire assessment and complaining that Smith & Wesson's and Glock's handguns should have been rejected during phase I for failing to comply with one aspect of the specifications. ATF decided to take corrective action in response to this earlier protest. Specifically, ATF stated that it would reconsider its phase II evaluations, including reviewing the live-fire assessment results, consensus summary reports, and individual ratings, and make a new decision about which offers should progress from phase II to the final phase of the evaluation. In response, we dismissed Sig Sauer's protest earlier protest as academic.

During the course of its corrective action, ATF did not reconvene the SSB or re'evaluate proposals. Instead, the contracting officer reviewed the live-fire assessment record; the phase II incident reports; and the declarations made during the original protest, concerning how stoppages were classified as either shooter'induced or gun-induced. At the end of this review, the contracting officer created new spreadsheets to organize the information generated during the various stages of the phase II assessment "for ease of review."[7] Contracting Officer's Statement, B-402339.3, at 2. The contracting officer and SSA reviewed the spreadsheets and concluded that Sig Sauer's proposal was appropriately excluded from the final phase of the procurement. AR, B-402339.3, Tab 2, Competitive Range Determination. Specifically, the contracting officer and SSA noted that the incident reports confirmed that the reliability of Sig Sauer's handguns was a major concern, given that Sig Sauer's full and compact models experienced a total of 13 gun-induced stoppages and 45 shooter-induced stoppages during the phase II testing. It was determined that "the significant number of malfunctions was . . . unacceptable." Id. at 2.

Following a written debriefing, Sig Sauer again protested to our Office, arguing that the evaluation methodology that ATF used in the phase II assessments deviated from the procedures announced in the RFP and that the evaluation reflected unequal treatment of offerors.[8]

DISCUSSION

Sig Sauer challenges the ATF's live-fire testing assessment, arguing that the SSB substituted its own judgment for that of the agents who performed the testing. The crux of Sig Sauer's argument is that the SSB members focused on the number of stoppages the shooters encountered, instead of the actual ratings and comments provided by the shooters, and equated a high number of stoppages with unreliability''even where the shooters had not made this connection.

The protester argues that this was inconsistent with the evaluation methodology "established in the solicitation and the underlying Source Selection Plan," because that plan explicitly provided that the SSB would only tally the ratings on the agents' questionnaires, but would base its assessment upon the questionnaires' ratings and narrative comments. Protest at 4, 15. Sig Sauer notes that, notwithstanding the number of stoppages, only two agents assigned unsatisfactory ratings for reliability for Sig Sauer's handguns, whereas three agents rated Smith & Wesson's handguns unsatisfactory in this category. Similarly, only one agent rated Sig Sauer's handguns as unsatisfactory overall under the general assessment, while Smith & Wesson's handguns received two such ratings. Sig Sauer contends that, contrary to the evaluators' adjectival ratings, the SSB improperly rated Sig Sauer's handguns as unsatisfactory overall, and, in contrast, rated Smith & Wesson's handguns as excellent overall.

In reviewing protests challenging the evaluation of proposals, we do not conduct a new evaluation or substitute our judgment for that of the agency but examine the record to determine whether the agency's judgment was reasonable and in accord with the RFP evaluation criteria. Abt Assocs., Inc., B-237060.2, Feb. 26, 1990, 90-1 CPD para. 223 at 4. Our Office affords particular deference to the technical expertise of agency personnel regarding judgments that involve matters of human life and safety. PEMCO World Air Servs., B-284240.3 et al., Mar. 27, 2000, 2000 CPD para. 71 at 7. A protester's mere disagreement with the agency's judgment concerning the adequacy of its proposal is not sufficient to establish that the agency acted unreasonably. Realty Executives, B-237537, Feb. 16, 1990, 90-1 CPD para. 288 at 3.

Here, as we explain below, we find that the agency's assessment of Sig Sauer's proposal was reasonable and consistent with the RFP's evaluation criteria. In this regard, we do not agree with Sig Sauer that the SSB, contracting officer, and SSA were required to follow the shooters' judgments with respect to the reliability and acceptability of the offerors' handguns.[9] Source selection officials and higher-level agency evaluators may reasonably disagree with the evaluation ratings and results of lower-level evaluators. See, e.g., Verify, Inc., B-244401.2, Jan. 24, 1992, 92-1 CPD para. 107 at 6-8. In this regard, an SSA has broad discretion in determining the manner and extent to which technical and cost evaluation results are used, is permitted to make an independent evaluation of offerors' proposals, and may disagree with or expand upon the findings of lower-level evaluators provided the basis for the evaluation is reasonable and documented in the contemporaneous record. KPMG Consulting LLP, B-290716, B-290716.2, Sept. 23, 2002, 2002 CPD para. 196 at 13-14;Brisk Waterproofing Co., Inc., B-276247, May 27, 1997, 97-1 CPD para. 195 at 2 n.1. The issue is not whether the agency's final assessments are consistent with earlier assessments, but whether they reasonably reflect the relative merits of the proposals, consistent with the solicitation.

The RFP provided that "[r]atings on the proposed PQC [the shooters' tests], any stoppages or parts breakages during testing, and the written evaluations will be recorded and used to evaluate each model." RFP at 62-63. The record here supports ATF's assertion that it considered these things when it affirmed the SSB's earlier determination to exclude Sig Sauer. While each shooter completed a questionnaire on the basis of his or her own individual experience, the SSB conducted a broader review of the cumulative effect of all the stoppages recorded. The shooters' questionnaires, as a whole, show that 11 of the 20 shooters experienced stoppages with Sig Sauer's handguns--in some cases, as many as 10 stoppages. In total, the shooters experienced 58 stoppages with Sig Sauer's handguns, 13 of which were considered to be gun-induced. Given the relatively high number of stoppages by Sig Sauer's guns as compared to Smith & Wesson's and Glock's guns, we think that the SSB reasonably concluded that the stoppages provided a more significant measure of the guns' reliability than the adjectival ratings for reliability assigned by individual shooters.[10] In addition, notwithstanding the low number of "unsatisfactory" ratings for reliability recorded on the shooters' scoresheets, the questionnaires included a number of negative comments that the SSB reasonably found questioned the reliability of Sig Sauer's handguns.

Sig Sauer also contends that ATF placed too great an emphasis upon reliability in determining which offers should continue to phase III. In this regard, Sig Sauer argues that reliability was only one of a number of elements to be considered in the live-fire assessment, and notes that reliability was not identified as having any more importance than the other elements.[11] We disagree.

As noted above, the RFP provided that only those offers which were "deemed most suitable for performance to the Government" would be included in the phase III evaluation. See RFP at 63. An agency is accorded considerable discretion in making such a subjective judgment as to what best meets its needs, seeHandheld Sys., Inc., B-288036, Aug. 10, 2001, 2001 CPD para. 142 at 3, particularly where, as here, the judgment involves matters of human life and safety. PEMCO World Air Servs., supra. In making this judgment, the solicitation allowed the agency to assess a number of considerations, including reliability. We find that the agency could reasonably rely upon its concerns about the reliability of Sig Sauer's handguns to exclude Sig Sauer's proposal from the final phase of the procurement, even though there were other elements of Sig Sauer's proposed handgun system that were found to be attractive.

Sig Sauer also argues that the ratings assigned by the shooters resulted from disparate treatment that occurred during the live-fire assessments. Specifically, the protester contends that, in incidents involving two separate shooters, when Smith & Wesson's handguns encountered a stoppage during the live fire evaluation, the agency intervened to identify the cause of the stoppage and to correct it. In contrast, the protester complains the agency did not take similar steps when stoppages occurred during Sig Sauer's live-fire assessment.

The ATF denies that it treated offerors unequally in its live-fire testing. With respect to one of the two shooters, the agency's lead proctor states that he and another special agent had installed optional extended base pads that Smith & Wesson had supplied for use with their handguns. In documenting the stoppage of this shooter's handgun, the proctor discovered he had improperly installed the optional extended base pad, which caused the stoppage. Because the stoppage was "proctor-induced," and not gun-induced or shooter-induced, the proctor allowed correction of this problem. Declaration of Live-Fire Lead Proctor, B-402339, Jan. 2, 2010, 4-5.

With respect to the second shooter, for whom the record contains multiple incident reports, the lead proctor states that he did not intervene; rather, the shooter herself requested a change in grip size after experiencing "a high volume of stoppages." Id. at 5. Sig Sauer challenges this account of events, noting that after an initial entry by a proctor on two of the incident reports for this shooter, the lead proctor's handwriting appears on the report, commenting on the stoppage and its possible cause. Sig Sauer deduces from these report entries that the lead proctor intervened and attempted to influence this shooter to make a change, and that she then did so and experienced no further stoppages.

In our view, the record does not support Sig Sauer's conjecture in this regard. The lead proctor explained in a sworn statement that he only responded to the shooter's request for a grip change and the shooter's written comments.[12] We find that the contemporaneous record shows that the shooter herself recognized the cause of the slide problem, and that the proctor's actions involved only a change in grip. We do not see support for the protester's theory that the proctor noticed the problem and then intervened.

In sum, the record shows that ATF reasonably evaluated the firms' proposals in accordance with the solicitation's evaluation criteria and concluded that Sig Sauer's offer was not one of the "most suitable for performance to the Government" to continue to phase III.

[2] Although not at issue in this protest, Phase III consisted of four test segments which were designed to further test the reliability of the handguns.

[3] The agency's incident report form defines a "stoppage" as an incident that a shooter can solve in less than 30 seconds without using tools, and a "malfunction" as an incident that takes longer than 30 seconds to resolve or requires the use of tools, but not replacement parts. AR, B-402339, Tab 13a, Phase II and Phase III Pistol Incident Forms. Although the protest filings refer occasionally to malfunctions, the contemporaneous evaluation record shows only stoppages. We therefore refer in our decision to any incidents as stoppages.

[4] The agency describes gun-induced stoppages to be stoppages where the shooter's use of the weapon did not play a role in the stoppage. Legal Memorandum, B-402339, at 6.

[5] ATF describes its judgment as to which offers would continue to phase III to be a competitive range determination.

[7] The purpose of the spreadsheets was to organize the phase II evaluation data to permit the contracting officer and source selection authority (SSA) to determine whether the shooters' evaluations and comments and the SSB's individual evaluations and consensus summaries were consistent. Contracting Officer's Statement, B-402339.3, at 2.

[9] Sig Sauer's arguments in this area arise from its reading of the source selection plan, rather than the solicitation's evaluation criteria. An agency's source selection plan, however, is an internal guide that does not give rights to parties; it is the RFP's evaluation scheme, not internal agency documents such as source selection plans, to which an agency is required to adhere in evaluating proposals and making the award selection. Sayres & Assocs. Corp., B-295946, B-295946.2, Apr. 25, 2005, 2005 CPD para. 90 at 6 n.9. In any event, we do not find that the SSB's review was inconsistent with the spirit of the approach set out in the source selection plan. Although the agency did not limit itself to a mechanical tallying of scores, as Sig Sauer suggests, it drew conclusions based on its overview of the shooters' assessments. For example, the SSB concluded that the shooter-induced stoppages appeared to be exacerbated by the ergonomic design of Sig Sauer's gun.

[10] It is true, as Sig Sauer notes, that the RFP did not define what constituted a stoppage; establish any basis for categorizing stoppages as either shooter-induced or gun-induced; establish any standards for an acceptable level of stoppages; or define reliability in terms of stoppages. Protest at 17. This does not demonstrate, however, that these observations about the testing results led to an inconsistency between the evaluation and the RFP's evaluation criteria.

[11] The RFP provided that the live-fire assessment would consider accuracy, reliability, comfort, ergonomics, controllability, maintenance, and suitability and provided for an overall assessment of the handgun as a "potential new service handgun for ATF." RFP at 63.

[12] These comments indicated that the shooter "[s]tarted with medium size palm swell and had problems with slide not locking back. Switched to large and fit (with my grip) was perfect. No further problems and fit was excellent." Protest, B'402339.3, Tab 15, Shooter #3's Technical Rating Sheet for Smith & Wesson full-size weapon, at 1.