June 06, 2008

Grimm Prognosis for ESI Search

There’s a double standard in e-discovery. Keyword search is deemed "good enough" for identifying responsive ESI; yet when privilege is on the line, lawyers insist on page-by-page review. It’s a tacit recognition that keyword search is a blunt instrument--a point artfully made earlier this year by Magistrate Judge John Facciola in U.S v. O’Keefe and Equity Analytics v. Lundin and emphatically underscored lately by Magistrate Judge Paul Grimm in Victor Stanley, Inc. v. Creative Pipe, Inc.

It’s assumed that lawyers are qualified to review documents and decide their relevance, responsiveness and privileged character. But are we qualified to craft proxies for our judgment in the form of keyword searches?

In Victor Stanley, 165 documents slipped by a
privilege review employing keyword search and a cursory-sounding "title
page" analysis for non-searchable items. Defendants had unwisely
abandoned efforts to secure a clawback agreement. Plaintiff’s counsel
spotted the documents and dutifully reported their potentially
privileged character, but argued defendants waived privilege by using a
faulty review process. The court agreed, pointing to defendants’
failure to provide information regarding keywords used, how they were
selected, steps taken to assess the reliability of the outcome and the
qualifications of the attorneys to design an effective and reliable
search.

Do lawyers have so insightful a grasp of the words and semantic relationships behind our relevance and privilege decisions that we can distill the je ne sais quoi of our well-honed legal minds into quotidian keyword spotting?

Language is deceptively complex, and meaning is an elusive, protean quarry. We depend upon context for meaning, but keyword search ignores context entirely. Boolean search is only marginally better at gleaning context.

That leaves lawyers in a tough spot. Mushrooming volumes of ESI require us to rely more on automated search tools at the same time courts and opposing counsel are less willing to indulge the fiction that these tools perform in unskilled hands. The jig is up, and lawyers are now obliged to prove these proxies really work.

How do we meet that burden of proof? Judge Facciola deems both lawyers and judges keyword naifs, instead summoninga phalanx of linguists, statisticians and computer experts. Though expecting searches to be designed by qualified persons, Judge Grimm leaves the door open to lawyer-initiated keyword search when counsel can demonstrate adequate quality assurance and quality control.

This is a subtle but important distinction. Lawyers can become "qualified persons," though they may never be linguists, statisticians or computer experts. Still, Judge Grimm sets the bar high:

Use of search and information retrieval methodology, for the purpose of identifying and withholding privileged or work product protected information from production, requires the utmost care in selecting methodology that is appropriate for the task … [and] careful advance planning by persons qualified to design effective search methodology. The implementation of the methodology selected should be tested for quality assurance; and the party selecting the methodology must be prepared to explain the rationale for the method chosen to the court, demonstrate that it is appropriate for the task, and show that it was properly implemented.

Victor Stanley departs from O’Keefe in another subtle way. By emphasizing collaboration, Judge Grimm preserves counsel’s ability to negotiate and agree upon search methods. Judge Facciola is no less a proponent of collaboration and transparency in e-discovery, but declaring both counsel and courts unequipped to oversee keyword search without expert assistance imperils the parties’ freedom to agree on search methods and the court’s authority to ratify such agreements. What court, admittedly unqualified to weigh such matters, could endorse a search protocol framed by those equally unequal to the task? Thus Victor Stanley preserves the litigants’ inalienable right to be wrong, so long as everyone agrees that wrong is right. It’s a Faustian bargain, but one permitting cases to move forward by simply ignoring pesky questions concerning the integrity and completeness of electronic discovery.

The Victor Stanley decision gives teeth to the duty to use better search techniques. Avoiding privilege waiver is a powerful incentive to: