Many lawyers and technologists like predictive coding and recommend it to their colleagues. They have good reasons. It has worked for them. It has allowed them to do e-discovery reviews in an effective, cost efficient manner, especially the big projects. That is true for me too, but that is not why I love predictive coding. My feelings come from the excitement, fun, and amazement that often arise from seeing it in action, first hand. I love watching the predictive coding features in my software find documents that I could never have found on my own. I love the way the AI in the software helps me to do the impossible. I really love how it makes me far smarter and skilled than I really am.

I have been getting those kinds of positive feelings consistently by using the latest Predictive Coding 4.0 methodology (shown right) and KrolLDiscovery’s latest eDiscovery.com Review software (“EDR”). So too have my e-Discovery Team members who helped me to participate in TREC 2015 and 2016 (the great science experiment for the latest text search techniques sponsored by the National Institute of Standards and Technology). During our grueling forty-five days of experiments in 2015, and again for sixty days in 2016, we came to admire the intelligence of the new EDR software so much that we decided to personalize the AI as a robot. We named him Mr. EDR out of respect. He even has his own website now, MrEDR.com, where he explains how he helped my e-Discovery Team in the 2015 and 2015 TREC Total Recall Track experiments.

Bottom line for us from this research was to prove and improve our methods. Our latest version 4.0 of Predictive Coding, Hybrid Multimodal IST Method is the result. We have even open-sourced this method, well most of it, and teach it in a free seventeen-class online program: TARcourse.com. Aside from testing and improving our methods, another, perhaps even more important result of TREC for us was our rediscovery that with good teamwork, and good software like Mr. EDR at your side, document review need never be boring again. The documents themselves may well be boring as hell, that’s another matter, but the search for them need not be.

How and Why Predictive Coding is Fun

Steps Four, Five and Six of the standard eight-step workflow for Predictive Coding 4.0 is where we work with the active machine-learning features of Mr. EDR. These are its predictive coding features, a type of artificial intelligence. We train the computer on our conception of relevance by showing it relevant and irrelevant documents that we have found. The software is designed to then go out and find all other relevant documents in the total dataset. One of the skills we learn is when we have taught enough and can stop the training and complete the document review. At TREC we call that the Stop decision. It is important to keep down the costs of document review.

We use a multimodal approach to find training documents, meaning we use all of the other search features of Mr. EDR to find relevant ESI, such as keyword searches, similarity and concept. We iterate the training by sample documents, both relevant and irrelevant, until the computer starts to understand the scope of relevance we have in mind. It is a training exercise to make our AI smart, to get it to understand the basic ideas of relevance for that case. It usually takes multiple rounds of training for Mr. EDR to understand what we have in mind. But he is a fast learner, and by using the latest hybrid multimodalIST (“intelligently spaced learning“) techniques, we can usually complete his training in a few days. At TREC, where we were moving fast after hours with the Ã-Team, we completed some of the training experiments in just a few hours.

After a while Mr. EDR starts to “get it,” he starts to really understand what we are after, what we think is relevant in the case. That is when a happy shock and awe type moment can happen. That is when Mr. EDR’s intelligence and search abilities start to exceed our own. Yes. It happens. The pupil then starts to evolve beyond his teachers. The smart algorithms start to see patterns and find evidence invisible to us. At that point we sometimes even let him train himself by automatically accepting his top-ranked predicted relevant documents without even looking at them. Our main role then is to determine a good range for the automatic acceptance and do some spot-checking. We are, in effect, allowing Mr. EDR to take over the review. Oh what a feeling to then watch what happens, to see him keep finding new relevant documents and keep getting smarter and smarter by his own self-programming. That is the special AI-high that makes it so much fun to work with Predictive Coding 4.0 and Mr. EDR.

It does not happen in every project, but with the new Predictive Coding 4.0 methods and the latest Mr. EDR, we are seeing this kind of transformation happen more and more often. It is a tipping point in the review when we see Mr. EDR go beyond us. He starts to unearth relevant documents that my team would never even have thought to look for. The relevant documents he finds are sometimes completely dissimilar to any others we found before. They do not have the same keywords, or even the same known concepts. Still, Mr. EDR sees patterns in these documents that we do not. He can find the hidden gems of relevance, even outliers and black swans, if they exist. When he starts to train himself, that is the point in the review when we think of Mr. EDR as going into superhero mode. At least, that is the way my young e-Discovery Team members likes to talk about him.

By the end of many projects the algorithmic functions of Mr. EDR have attained a higher intelligence and skill level than our own (at least on the task of finding the relevant evidence in the document collection). He is always lighting fast and inexhaustible, even untrained, but by the end of his training, he becomes a search genius. Watching Mr. EDR in that kind of superhero mode is what makes Predictive Coding 4.0 a pleasure.

The Empowerment of AI Augmented Search

It is hard to describe the combination of pride and excitement you feel when Mr. EDR, your student, takes your training and then goes beyond you. More than that, the super-AI you created then empowersyou to do things that would have been impossible before, absurd even. That feels pretty good too. You may not be Iron Man, or look like Robert Downey, but you will be capable of remarkable feats of legal search strength.

For instance, using Mr. EDR as our Iron Man-like suits, my e-discovery Ã-Team of three attorneys was able to do thirty different review projects and classify 17,014,085 documents in 45 days. See 2015 TREC experiment summary at Mr. EDR. We did these projects mostly at nights, and on weekends, while holding down our regular jobs. What makes this crazy impossible, is that we were able to accomplish this by only personally reviewing 32,916 documents. That is less than 0.2% of the total collection. That means we relied on predictive coding to do 99.8% of our review work. Incredible, but true.

Using traditional linear review methods it would have taken us 45 years to review that many documents! Instead, we did it in 45 days. Plus our recall and precision rates were insanely good. We even scored 100% precision and 100% recall in one TREC project in 2015 and two more in 2016. You read that right. Perfection. Many of our other projects attained scores in the high and mid nineties. We are not saying you will get results like that. Every project is different, and some are much more difficult than others. But we are saying that this kind of AI-enhanced review is not only fast and efficient, it is effective.

Yes, it’s pretty cool when your little AI creation does all the work for you and makes you look good. Still, no robot could do this without your training and supervision. We are a team, which is why we call it hybrid multimodal, man and machine.

Having Fun with Scientific Research at TREC 2015 and 2016

During the 2015 TREC Total Recall Track experiments my team would sometimes get totally lost on a few of the really hard Topics. We were not given legal issues to search, as usual. They were arcane technical hacker issues, political issues, or local news stories. Not only were we in new fields, the scope of relevance of the thirty Topics was never really explained. (We were given one to three word explanations in 2015, in 2016 we got a whole sentence!) We had to figure out intended relevance during the project based on feedback from the automated TREC document adjudication system. We would have some limited understanding of relevance based on our suppositions of the initial keyword hints, and so we could begin to train Mr. EDR with that. But, in several Topics, we never had any real understanding of exactly what TREC thought was relevant.

This was a very frustrating situation at first, but, and here is the cool thing, even though we did not know, Mr. EDR knew. That’s right. He saw the TREC patterns of relevance hidden to us mere mortals. In many of the thirty Topics we would just sit back and let him do all of the driving, like a Google car. We would often just cheer him on (and each other) as the TREC systems kept saying Mr. EDR was right, the documents he selected were relevant. The truth is, during much of the 45 days of TREC we were like kids in a candy store having a great time. That is when we decided to give Mr. EDR a cape and superhero status. He never let us down. It is a great feeling to create an AI with greater intelligence than your own and then see it augment and improve your legal work. It is truly a hybrid human-machine partnership at its best.

I hope you get the opportunity to experience this for yourself someday. The TREC experiments in 2015 and 2016 on recall in predictive coding are over, but the search for truth and justice goes on in lawsuits across the country. Try it on your next document review project.

Do What You Love and Love What You Do

Mr. EDR, and other good predictive coding software like it, can augment our own abilities and make us incredibly productive. This is why I love predictive coding and would not trade it for any other legal activity I have ever done (although I have had similar highs from oral arguments that went great, or the rush that comes from winning a big case).

The excitement of predictive coding comes through clearly when Mr. EDR is fully trained and able to carry on without you. It is a kind of Kurzweilian mini-singularity event. It usually happens near the end of the project, but can happen earlier when your computer catches on to what you want and starts to find the hidden gems you missed. I suggest you give Predictive Coding 4.0 and Mr. EDR a try. To make it easier I open-sourced our latest method and created an online course. TARcourse.com. It will teach anyone our method, if they have the right software. Learn the method, get the software and then you too can have fun with evidence search. You too can love what you do. Document review need never be boring again.

Caution

One note of caution: most e-discovery vendors, including the largest, do not have active machine learning features built into their document review software. Even the few that have active machine learning do not necessarily follow the HybridMultimodalIST Predictive Coding 4.0 approach that we used to attain these results. They instead rely entirely on machine-selected documents for training, or even worse, rely entirely on random selected documents to train the software, or have elaborate unnecessary secret control sets.

The algorithms used by some vendors who say they have “predictive coding” or “artificial intelligence” are not very good. Scientists tell me that some are only dressed-up concept search or unsupervised document clustering. Only bona fide active machine learning algorithms create the kind of AI experience that I am talking about. Software for document review that does not have any active machine learning features may be cheap, and may be popular, but they lack the power that I love. Without active machine learning, which is fundamentally different from just “analytics,” it is not possible to boost your intelligence with AI. So beware of software that just says it has advanced analytics. Ask if it has “active machine learning“?

It is impossible to do the things described in this essay unless the software you are using has active machine learning features. This is clearly the way of the future. It is what makes document review enjoyable and why I love to do big projects. It turns scary to fun.

So, if you tried “predictive coding” or “advanced analytics” before, and it did not work for you, it could well be the software’s fault, not yours. Or it could be the poor method you were following. The method that we developed in Da Silva Moore, where my firm represented the defense, was a version 1.0 method. Da Silva Moore v. Publicis Groupe, 287 F.R.D. 182, 183 (S.D.N.Y. 2012). We have come a long way since then. We have eliminated unnecessary random control sets and gone to continuous training, instead of train then review. This is spelled out in the TARcourse.com that teaches our latest version 4.0 techniques.

The new 4.0 methods are not hard to follow. The TARcourse.com puts our methods online and even teaches the theory and practice. And the 4.0 methods certainly will work. We have proven that at TREC, but only if you have good software. With just a little training, and some help at first from consultants (most vendors with bona fide active machine learning features will have good ones to help), you can have the kind of success and excitement that I am talking about.

Do not give up if it does not work for you the first time, especially in a complex project. Try another vendor instead, one that may have better software and better consultants. Also, be sure that your consultants are Predictive Coding 4.0 experts, and that you follow their advice. Finally, remember that the cheapest software is almost never the best, and, in the long run will cost you a small fortune in wasted time and frustration.

Conclusion

Love what you do. It is a great feeling and sure fire way to job satisfaction and success. With these new predictive coding technologies it is easier than ever to love e-discovery. Try them out. Treat yourself to the AI high that comes from using smart machine learning software and fast computers. There is nothing else like it. If you switch to the 4.0 methods and software, you too can know that thrill. You can watch an advanced intelligence, which you helped create, exceed your own abilities, exceed anyone’s abilities. You can sit back and watch Mr. EDR complete your search for you. You can watch him do so in record time and with record results. It is amazing to see good software find documents that you know you would never have found on your own.

Predictive coding AI in superhero mode can be exciting to watch. Why deprive yourself of that? Who says document review has to be slow and boring? Start making the practice of law fun again.

Every judge who has ever struggled with discovery issues wishes that the lawyers involved had a better understanding of proportionality, that they had spent more time really thinking about how it applies to the requisites of their case. So too does every lawyer who, like me, specializes in electronic discovery. As Chief Justice Roberts explained in his 2015 Year-End Report on the Federal Judiciary on the new rules on proportionality:

The amended rule states, as a fundamental principle, that lawyers must size and shape their discovery requests to the requisites of a case. Specifically, the pretrial process must provide parties with efficient access to what is needed to prove a claim or defense, but eliminate unnecessary or wasteful discovery. The key here is careful and realistic assessment of actual need.

Proportionality and reasonableness arise from conscious efforts to realistically assess actual need. What is the right balance in a particular situation? What are the actual benefits and burdens involved? How can you size and shape your discovery requests to the requisites of a case?

There is more to proportionality than knowing the rules and case law, although they are a good place to start. Proportionality is a deep subject and deserves more than black letter law treatment. 2015 e-Discovery Rule Amendments: Dawning of the “Goldilocks Era” (e-discoveryteam.com, 11/11/15) (wherein I discuss proportionality, the Golden Ratio or perfect proportionality, aka Φ, which is shown in this graphic and much more, including the spooky “coincidence” at a CLE with Judge Facciola and the audience vote). Also see: Giulio Tononi, PhiΦ, a Voyage from the Brain to the Soul (Pantheon Books, 2012) (book I’m rereading now on consciousness and integrated information theory, another take on Phi Φ).

We want everyone in the field to think about proportionality. To be conscious of it, not just have information about it. What does proportionality really mean? How does it apply to the e-discovery tasks that you carry out every day? How much is enough? Too much? Too burdensome? Too little? Not enough? Why?

What is a reasonable effort? How do you know? Is there perfect proportionality? One that expresses itself in varying ways according to the facts and circumstances? Does Law follow Art? Is Law an Art? Or is it a Science? Is there Beauty in Law? In Reason? There is more to proportionality than meets the eye. Or is there?

This week’s blog continues that intention of getting lawyers to think about proportionality and the requisites of their case. It concludes with a word document designed to make it easier to play along with your own group, class or CLE event. What discovery activities required in a Big Case are not necessary in a Small Case, or even a Medium Sized case? That is what requires thought and is the basis of the game.

Rules of Federal Procedure

Proportionality is key to all discovery, to knowing the appropriate size and shape of discovery requests in order to fit the requisites of a case. Reading the rules that embody the doctrine of proportionality is a good start, but just a start. The primary rule to understand is how proportionality effects the scope of relevance as set forth in Rule 26(b)(1), FRCP:

Parties may obtain discovery regarding any nonprivileged matter that is relevant to any party’s claim or defense and proportional to the needs of the case, considering the importance of the issues at stake in the action, the amount in controversy, the parties’ relative access to relevant information, the parties’ resources, the importance of the discovery in resolving the issues, and whether the burden or expense of the proposed discovery outweighs its likely benefit.

But you also need to understand how it impacts a lawyer’s overall duty to supervise a discovery request and response as set forth in Rule 26(g). See Rule 26(g)(1)(B)(iii), FRCP:

neither unreasonable nor unduly burdensome or expensive, considering the needs of the case, prior discovery in the case, the amount in controversy, and the importance of the issues at stake in the action.

In re Bard IVC Filters Prods. Liab. Litig., D. Ariz., No. MDL 15-02641-PHX DGC, 2016 U.S. Dist. LEXIS 126448 (D. Ariz. Sept. 16, 2016). In this must-read opinion District Judge David G. Campbell, who was the chair of the Rules Committee when the 2015 amendments were passed, takes both lawyers and judges to task for not following the new rules on proportionality. He then lays it all out in a definitive manner.

Hyles v. New York City, No. 10 Civ. 3119 (AT)(AJP), 2016 WL 4077114 (S.D.N.Y. Aug. 1, 2016) (Judge Peck: “While Hyles may well be correct that production using keywords may not be as complete as it would be if TAR were used, the standard is not perfection, or using the “best” tool, but whether the search results are reasonable and proportional. Cf. Fed. R. Civ. P. 26(g)(1)(B)”)

Johnson v Serenity Transportation, Case No. 15-cv-02004-JSC (N.D. Cal. October 28, 2016) (“… a defendant does not have discretion to decide to withhold relevant, responsive documents absent some showing that producing the document is not proportional to the needs of the case.”)

Apple Inc. v. Samsung Elecs. Co., No. 12-CV-0630-LHK (PSG), 2013 WL 4426512, 2013 U.S. Dist. LEXIS 116493 (N.D. Cal. Aug. 14, 2013) (“But there is an additional, more persuasive reason to limit Apple’s production — the court is required to limit discovery if “the burden or expense of the proposed discovery outweighs its likely benefit.” This is the essence of proportionality — an all-to-often ignored discovery principle. Because the parties have already submitted their expert damages reports, the financial documents would be of limited value to Samsung at this point. Although counsel was not able to shed light on exactly what was done, Samsung’s experts were clearly somehow able to apportion the worldwide, product line inclusive data to estimate U.S. and product-specific damages. It seems, well, senseless to require Apple to go to great lengths to produce data that Samsung is able to do without. This the court will not do.“)

The Sedona Conference Commentary on Proportionality_May 2017 is more than a collection of case law. It includes commentary hashed out between competing camps over many years. The latest 2017 version includes Six Principles that are worthy of study. They can certainly help you in your own analysis of proportionality. The cited case law in the Commentary is structured around these six principles.

THE SEDONA CONFERENCE PRINCIPLES OF PROPORTIONALITY

Principle 1: The burdens and costs of preserving relevant electronically stored information should be weighed against the potential value and uniqueness of the information when determining the appropriate scope of preservation.

Principle 2: Discovery should focus on the needs of the case and generally be obtained from the most convenient, least burdensome, and least expensive sources.

Principle 3: Undue burden, expense, or delay resulting from a party’s action or inaction should be weighed against that party.

Principle 4: The application of proportionality should be based on information rather than speculation.

Principle 5: Nonmonetary factors should be considered in the proportionality analysis.

Principle 6: Technologies to reduce cost and burden should be considered in the proportionality analysis.

But, you can do more. You can lead discussions at your law firm, company, class or CLE on the subject. You can become an e-discovery proportionality Game-Master. You can find out the consensus opinion of any group. You can observe and create statistics of how the initial opinions change when the other game players hear each others opinions. That kind of group interaction can create the so-called hive-effect. People often change their mind until a consensus emerges.

What is the small, medium or large proportionality consensus of your group? Even if you just determine majority opinion, and do not go through an interactive exercise, you are learning something of interest. Plus, and here is the key thing, you are giving game players a chance to exercise their analytical skills.

To help you to play this game on your own, and lead groups to play it, I created a Word Document that you are welcome to use.

Playing games is a great way to learn. That’s one reason I’ve devised a game concerning the interesting and fairly complex issues involved in trying to determine what e-discovery activities are proportional and appropriate in various sized cases. Specifically, what should you do to prepare for federal court 26(f) conferences in small and medium sized cases, versus large, complicated cases? Small, Medium or Large? is kind of a Goldilocks game of proportionality.

I have had to give these proportionality questions a lot of thought as part of my practice as a lawyer supervising hundreds of e-discovery projects at a time, projects of all different sizes. I could simply give you my answer, but after five books, I’ve already been there and done that. So I thought I’d try something new and make this learning into a game where you consider and vote on what activities you think are appropriate for a Small, Medium or Large case.

The Hive Mind is different from Crowdsourcing, but related. To do properly, a Hive Mind requires Swarming, which we cannot really do properly in this game using polls. Louis Rosenberg, Super-Intelligence and the virtues of a “Hive Mind” (Singularity, 2/10/16). One Silicon Valley startup Unanimous A.I., is developing technologies that enable sophisticated online human swarming and thus better collective intelligence. I might try their free product for social research, UNO, if I further investigate the power of the Hive Mind. Maybe at NY LegalTech? (Email me if you’re interested.) In the meantime we are going to use simple polling for the e-Discovery Hive Mind Game: Small, Medium or Large?

Let’s play the game and see what collective intelligence emerges from many individuals giving their opinion about proportionality and e-discovery. I am playing this game now with all of the litigation associates and paralegals in my law firm, which is a pretty large swarm by itself. Your responses will join in the swarm, the collective intelligence.

In January I’ll share the results of all the polling and opine away as to how well the Hive Mind performed. You can win the game in one of two ways, either by matching the most popular Hive Mind responses or by matching my responses. (I assume there will be a difference because not enough experts will be playing, but who knows, maybe not. In theory, with enough experts swarming, the group, the Hive Mind, will always have the best answer.)

Background To Play the Game

In order to play the Small, Medium or Large? game, you first need to be familiar with the checklist of the Southern District Court of Florida of all of the things that you should do to prepare for e-discovery in a large case. It is a pretty good list and I have written about it before. Good New 33-Point e-Discovery Checklist From Miami (e-Discovery Team, October 1, 2017) (a must read to fully prepare for this game). My article contains comments and explanations about all checklist items, which is the beginning of a kind of swarm interaction, that is, if you take time to ponder the signals. The Court’s checklist incorporates the new provisions in the rules on relevance and proportionality (Rule 26(b)(1)) and on specific objections (Rule 34(b)(2)).

It is not a perfect list, but it is the best one now out there with a court pedigree. It is not too long and complex, like the older lists of some courts that are very detailed, and not too short, like the easy-peasy list that Bill Hamilton and I created for the Middle District Court of Florida many years ago. (Attorneys still complained about how burdensome it was!) In sum, the 33-Point Checklist out of Miami is a good list for legal practitioners all over the country to use to prepare for e-discovery, which means it is a good basis for our Small, Medium or Large?Hive Mind Game. Come play along.

Rules of the Game

The goal of our Hive Mind Game is to determine which of the thirty-three points on the checklist are applicable to big cases only, which are applicable to medium size cases and which to small cases. You are to assume that all thirty-three points apply to large cases, but that they are not all applicable to medium and small size cases. The Hive Mind voting is used to allow the swarm – that’s you – to identify which of the thirty-three only apply to small cases, and which only apply to medium size cases. If a checklist item applies to a medium size case, it automatically also applies to a small size case.

In other words, the game is to sort the thirty-three into three piles, Small, Medium or Large? Simple, eh? Well, maybe not. This is a matter of opinion and things are pretty vague. For instance, I’m not going to define the difference between large, small and medium size case. That is part of the Hive Mind.

The game is important because proportionality in the law is important. You do not prepare for a big case the same way you prepare for a small case. You just don’t. You could, but it would be a waste of your clients money to do so. So the real trick in e-discovery, like in all other aspects of the litigation, is to determine what you should do in any size case to prepare, including even the small cases. For instance, outside of e-discovery, most people would agree that you should take the parties depositions as a minimum to prepare for a trial of any size case, including small ones.

What are the equivalent items in the 33-point checklist? Which of them should be applied to all cases, even the small ones? Which of them are too complicated and expensive to apply in a small case, but not a medium sized case? Which too complicated and expensive to apply in a small or medium sized case, but not a large one? That is where the real skill and knowledge come in. That is the essence of the game.

Assume All 33 Items Apply to Big Cases

This Small, Medium or Large?Hive Game requires you to assume that all thirty-three items on the Court’s checklist apply to big cases, but not to all cases, that there are certain checklist items that only apply to medium size cases, and others, a smaller list, that only apply to small cases. You may question the reality of this assumption. After all, the Court does not say that. It does not say, here’s a checklist we made to guide your e-discovery, but you can ignore many of the items on this list if you have a small case, or even a medium size case. Still, that’s what they mean, but they do not go on to say what’s what. They know that the Bar will figure it out themselves in due time, meaning the next several years. And right they are. But why wait? Let’s figure it out ourselves now with this Small, Medium or Large? Hive Game, and condense years to weeks.

The Hive Game will allow us to fill in the blanks of what the Court did not say. But first, lets focus again on what the Court did say. It said that the checklist may be used by members of the Bar to guide the Rule 26(f) e-discovery conferences and Case Management Reports. It did not say shall be used. It is a suggestion, not a requirement. Still, as every long-term member of the court knows, what they mean is that you damn well should follow the checklist in a big case! Woe unto the lawyers who come before the judges in a big case with an e-discovery issue where they never even bothered to go through the checklist. If your issue is on that list, and chances are it will be, then dear slacker, prepare for a Miami style bench-slap. Ouch! It is going to hurt. Not only you and your reputation, but also your client.

I feel confident in making the game assumption that all thirty-three checklist items apply to big cases. I am also confident they do not all apply to medium and small size cases and that is the real reason for the court’s use of may instead of shall.

With that background we are almost ready to start playing the game and opine away as to which of the 33 are small and medium size only. But, there is still one more thing I have found very helpful when you try to really dig into the checklist and play the game, you need to have the numbers 1-33 added to the list. The one mistake the Court made on this list was in using headings and bullet points instead of numbers. I fix that in the list that follows, so that you can, if nothing else, more easily play the game.

Preservation

The ranges of creation or receipt dates for any ESI to be preserved.

The description of ESI from sources that are not reasonably accessible because of undue burden or cost and that will not be reviewed for responsiveness or produced, but that will be preserved in accordance with Federal Rule of Civil Procedure 26(b)(2)(B).

The description of ESI from sources that: (a) the party believes could contain relevant information; but (b) has determined, under the proportionality factors, is not discoverable and should not be preserved.

Whether to continue any interdiction of any document-destruction program, such as ongoing erasures of e-mails, voicemails, and other electronically recorded material.

The number and names or general job titles or descriptions of custodians for whom ESI will be preserved (e.g., “HR head,” “scientist,” “marketing manager”).

The list of systems, if any, that contain ESI not associated with individual custodians and that will be preserved, such as enterprise databases.

Any disputes related to scope or manner of preservation.

Liaison

The identity of each party’s e-discovery liaison, who will be knowledgeable about and responsible for each party’s ESI.

Informal Discovery About Location and Types of Systems

Identification of systems from which discovery will be prioritized (e.g., e-mail, finance, HR systems).

Descriptions and location of systems in which potentially discoverable information is Stored.

How potentially discoverable information is stored.

How discoverable information can be collected from systems and media in which it is stored.

Proportionality and Costs

The amount and nature of the claims being made by either party.

The nature and scope of burdens associated with the proposed preservation and discovery of ESI.

The likely benefit of the proposed discovery.

Costs that the parties will share to reduce overall discovery expenses, such as the use of a common electronic-discovery vendor or a shared document repository, or other costsaving measures.

Limits on the scope of preservation or other cost-saving measures.

Whether there is relevant ESI that will not be preserved in accordance with Federal Rule of Civil Procedure 26(b)(1), requiring discovery to be proportionate to the needs of the case.

Search

The search method(s), including specific words or phrases or other methodology, that will be used to identify discoverable ESI and filter out ESI that is not subject to discovery.

The quality-control method(s) the producing party will use to evaluate whether a production is missing relevant ESI or contains substantial amounts of irrelevant ESI.

Phasing

Whether it is appropriate to conduct discovery of ESI in phases.

Sources of ESI most likely to contain discoverable information and that will be included in the first phases of Federal Rule of Civil Procedure 34 document discovery.

Sources of ESI less likely to contain discoverable information from which discovery will be postponed or not reviewed.

Custodians (by name or role) most likely to have discoverable information and whose ESI will be included in the first phases of document discovery.

Custodians (by name or role) less likely to have discoverable information from whom
discovery of ESI will be postponed or avoided.

The time period during which discoverable information was most likely to have been
created or received.

Production

The formats in which structured ESI (database, collaboration sites, etc.) will be produced.

The formats in which unstructured ESI (e-mail, presentations, word processing, etc.) will be produced.

The extent, if any, to which metadata will be produced and the fields of metadata to be produced.

The production format(s) that ensure(s) that any inherent searchability of ESI is not degraded when produced.

Privilege

How any production of privileged or work-product protected information will be handled.

Whether the parties can agree on alternative ways to identify documents withheld on the grounds of privilege or work product to reduce the burdens of such identification.

Whether the parties will enter into a Federal Rule of Evidence 502(d) stipulation and order that addresses inadvertent or agreed production.

One Example Before the Games Begin

We are almost ready to play the e-Discovery Small, Medium or Large? Hive Mind Game. We will do so with thirty-two polls that are presented to the player in the same order as the Court’s checklist. To make sure the rules are clear (this is, after all, a game for lawyers, not kids) we start with an example, the first of the thirty-three items on the checklist. The court’s first item is to suggest that you Determine the range of creation or receipt dates for any ESI to be preserved.

The “right answer” to this first item is that this should be done in every case, even the small ones. You should always determine the date range of data to be preserved. In most cases that is very easy to do, and, as every lawyer should know, when in doubt, when it comes to preservation, always err on the side of inclusion. That means you should check the Small Case answer as shown in the “dummy poll” graphic below.

Checklist 1

We have set these polls up so that you cannot see the results, but you can leave private comments. We may do this again later and experiment with what happens when you can see the results. We will share the results (and some comments) when the game ends on January 1, 2019.

Now for the live polls and game proper. Note that several of the checklist items, including number two and three, which are the first two polls shown below, are so long that we had to paraphrase and shorten them to fit in the space allocated in the polling software. To see the original of all thirty-three items on the checklist, go to my prior blog explaining the list (highly recommended) or the court’s page.

Let the Games Begin!

We are now ready to begin playing the e-Discovery Hive Mind Game. So get ready to plug-in. Select an answer to each of the thirty-two polls that follow. After you vote, you also have a chance to leave a private comment to each poll, but that is optional and will not impact your score.

Congratulations! You have finished the Game and made your contribution to the e-Discovery Hive Mind. Look for results sometime in early 2018. You can then determine how your answers compared with the collective Hive Mind.

I will also let you know how the Hive Mind answers compared with my own. So you will have two chances to win. Anyone who matches all of my answers wins a free lunch with me in Orlando. Other prizes have yet to be determined. Vendors care to contribute some goodies? Perhaps Elon will donate a free trip to Mars, where I for one hope we don’t run into any Borg cubes, I don’t care how good their Hive Mind is.

In the meantime, please encourage your e-discovery friends and colleagues to join in the game. Teachers and Partners are invited to require their students and associates, paralegals to play too. Resistance is futile! Digging deep into this checklist is a great way to expand your knowledge and expertise of electronic discovery law and practice.

The most complete set of AI ethics developed to date, the twenty-three Asilomar Principles, was created by the Future of Life Institute in early 2017 at their Asilomar Conference. Ninety percent or more of the attendees at the conference had to agree upon a principle for it to be accepted. The first five of the agreed-upon principles pertain to AI research issues.

Although all twenty-three principles are important, the research issues are especially time sensitive. That is because AI research is already well underway by hundreds, if not thousands of different groups. There is a current compelling need to have some general guidelines in place for this research. AI Ethics Work Should Begin Now. We still have a little time to develop guidelines for the advanced AI products and services expected in the near future, but as to research, the train has already left the station.

Asilomar Research Principles

Other groups are concerned with AI ethics and regulation, including research guidelines. See the Draft Principles page of AI-Ethics.com which lists principles from six different groups. The five draft principles developed by Asilomar are, however, a good place to start examining the regulation needed for research.

Research Issues

1) Research Goal: The goal of AI research should be to create not undirected intelligence, but beneficial intelligence.

2) Research Funding: Investments in AI should be accompanied by funding for research on ensuring its beneficial use, including thorny questions in computer science, economics, law, ethics, and social studies, such as:

How can we make future AI systems highly robust, so that they do what we want without malfunctioning or getting hacked?

How can we grow our prosperity through automation while maintaining people’s resources and purpose?

How can we update our legal systems to be more fair and efficient, to keep pace with AI, and to manage the risks associated with AI?

What set of values should AI be aligned with, and what legal and ethical status should it have?

3) Science-Policy Link: There should be constructive and healthy exchange between AI researchers and policy-makers.

4) Research Culture: A culture of cooperation, trust, and transparency should be fostered among researchers and developers of AI.

The proposed first principle is good, but the wording? Not so much. The goal of AI research should be to create not undirected intelligence, but beneficial intelligence. This is a double-negative English language mishmash that only an engineer could love. Here is one way this principle could be better articulated:

Research Goal: The goal of AI research should be the creation of beneficial intelligence, not undirected intelligence.

Researchers should develop intelligence that is beneficial for all of mankind. The Institute of Electrical and Electronics Engineers (IEEE) first general principle is entitled “Human Benefit.” The Asilomar first principle is slightly different. It does not really say human benefit. Instead it refers to beneficial intelligence. I think the intent is to be more inclusive, to include all life on earth, all of earth. Although IEEE has that covered too in their background statement of purpose to “Prioritize the maximum benefit to humanity and the natural environment.”

Pure research, where raw intelligence is created just for the hell of it, with no intended helpful “direction” of any kind, should be avoided. Because we can is not a valid goal. Pure, raw intelligence, with neither good intent, nor bad, is not the goal here. The research goal is beneficial intelligence. Asilomar is saying that Undirected intelligence is unethical and should be avoided. Social values must be built into the intelligence. This is subtle, but important.

The restriction to beneficial intelligence is somewhat controversial, but the other side of this first principle is not. Namely, that research should not be conducted to create intelligence that is hostile to humans. No one favors detrimental, evil intelligence. So, for example, the enslavement of humanity by Terminator AIs is not an acceptable research goal. I don’t care how bad you think our current political climate is.

To be slightly more realistic, if you have a secret research goal of taking over the world, such as Max Tegmark imagines in The Tale of the Omega Team in his book, Life 3.0, and we find out, we will shut you down (or try to). Even if it is all peaceful and well-meaning, and no one gets hurt, as Max visualizes, plotting world domination by machines is not a positive value. If you get caught researching how to do that, some of the more creative prosecuting lawyers around will find a way to send you to jail. We have all seen the cheesy movies, and so have the juries, so do not tempt us.

Keep a positive, pro-humans, pro-Earth, pro-freedom goal for your research. I do not doubt that we will someday have AI smarter than our existing world leaders, perhaps sooner than many expect, but that does not justify a machine take-over. Wisdom comes slowly and is different than intelligence.

Still, what about autonomous weapons? Is research into advanced AI in this area beneficial? Are military defense capabilities beneficial? Pro-security? Is the slaughter of robots not better than the slaughter of humans? Could robots be more ethical at “soldiering” than humans? As attorney Matt Scherer has noted, who is the editor of a good blog, LawAndAI.com and a Future of Life Institute member:

Autonomous weapons are going to inherently be capable of reacting on time scales that are shorter than humans’ time scales in which they can react. I can easily imagine it reaching the point very quickly where the only way that you can counteract an attack by an autonomous weapon is with another autonomous weapon. Eventually, having humans involved in the military conflict will be the equivalent of bringing bows and arrows to a battle in World War II.

At that point, you start to wonder where human decision makers can enter into the military decision making process. Right now there’s very clear, well-established laws in place about who is responsible for specific military decisions, under what circumstances a soldier is held accountable, under what circumstances their commander is held accountable, on what circumstances the nation is held accountable. That’s going to become much blurrier when the decisions are not being made by human soldiers, but rather by autonomous systems. It’s going to become even more complicated as machine learning technology is incorporated into these systems, where they learn from their observations and experiences in the field on the best way to react to different military situations.

The second principle of Funding is more than an enforcement mechanism for the first, that you should only fund beneficial AI. It is also a recognition that ethical work requires funding too. This should be every lawyer’s favorite AI ethics principle. Investments in AI should be accompanied by funding for research on ensuring its beneficial use, including thorny questions in computer science, economics, law, ethics, and social studies. The principle then adds a list of five bullet-point examples.

How can we make future AI systems highly robust, so that they do what we want without malfunctioning or getting hacked. The goal of avoiding the creation of AI systems that can be hacked, easily or not, is a good one. If a hostile power can take over and misuse an AI for evil end, then the built-in beneficence may be irrelevant. The example of a driverless car come to mind that could be hacked and crashed as a perverse joy-ride, kidnapping or terrorist act.

The economic issues raised by the second example are very important: How can we grow our prosperity through automation while maintaining people’s resources and purpose? We do not want a system that only benefits the top one percent, or top ten percent, or whatever. It needs to benefit everyone, or at least try to. Also see Asilomar Principle Fifteen: Shared Prosperity: The economic prosperity created by AI should be shared broadly, to benefit all of humanity.

I’m a very progressive person so I feel very strongly that dignity and justice mean wealth is redistributed. And I’m really concerned about AI worsening the effects and concentration of power and wealth that we’ve seen in the last 30 years. So this is pretty important for me.

I consider that one of the greatest dangers is that people either deal with AI in an irresponsible way or maliciously – I mean for their personal gain. And by having a more egalitarian society, throughout the world, I think we can reduce those dangers. In a society where there’s a lot of violence, a lot of inequality, the risk of misusing AI or having people use it irresponsibly in general is much greater. Making AI beneficial for all is very central to the safety question.

Most everyone at the Asilomar Conference agreed with that sentiment, but I do not yet see a strong consensus in AI businesses. Time will tell if profit motives and greed will at least be constrained by enlightened self-interest. Hopefully capitalist leaders will have the wisdom to share the great wealth with all of society that AI is likley to create.

How can we update our legal systems to be more fair and efficient, to keep pace with AI, and to manage the risks associated with AI? The legal example is also a good one, with the primary tension we see so far between fair versus efficient. Just policing high crime areas might well be efficient, at least for reducing some type of crime, but would it be fair? Do we want to embed racial profiling into our AI? Neighborhood slumlord profiling? Religious, ethic profiling? No. Existing law prohibits that and for good reason. Still, predictive policing is already a fact of life in many cities and we need to be sure it has proper legal, ethical regulation.

We have seen the tension between “speedy” and “inexpensive” on the one hand, and “just” on the other in Rule One of the Federal Rules of Civil Procedure and e-discovery. When applied using active machine learning a technical solution was attained to these competing goals. The predictive coding methods we developed allowed for both precision (“speedy” and “inexpensive”) and recall (“just”). Hopefully this success can be replicated in other areas of the law where machine learning is under proportional control by experienced human experts.

The final example given is much more troubling: What set of values should AI be aligned with, and what legal and ethical status should it have? Whose values? Who is to say what is right and wrong? This is easy in a dictatorship, or a uniform, monochrome culture (sea of white dudes), but it is very challenging in a diverse democracy. This may be the greatest research funding challenge of all.

Principle Three: Science-Policy Link

This principle is fairly straightforward, but will in practice require a great deal of time and effort to be done right. A constructive and healthy exchange between AI researchers and policy-makers is necessarily a two-way street. It first of all assumes that policy-makers, which in most countries includes government regulators, not just industry, have a valid place at the table. It assumes some form of government regulation. That is anathema to some in the business community who assume (falsely in our opinion) that all government is inherently bad and essentially has nothing to contribute. The countervailing view of overzealous government controllers who just want to jump in, uninformed, and legislate, is also discouraged by this principle. We are talking about a healthy exchange.

It does not take an AI to know this kind of give and take and information sharing will involve countless meetings. It will also require a positive healthy attitude between the two groups. If it gets bogged down into an adversary relationship, you can multiply the cost of compliance (and number of meetings) by two or three. If it goes to litigation, we lawyers will smile in our tears, but no one else will. So researchers, you are better off not going there. A constructive and healthy exchange is the way to go.

This tension is likley to increase as multiple parties get close to a big breakthrough. The successful efforts for open source now, before superintelligence seems imminent, may help keep the research culture positive. Time will tell, but if not there could be trouble all around and the promise of full employment for litigation attorneys.

Principle Five: Race Avoidance

The Fifth Principle is a tough one, but very important: Teams developing AI systems should actively cooperate to avoid corner-cutting on safety standards. Moving fast and breaking things may be the mantra of Silicon Valley, but the impact of bad AI could be catastrophic. Bold is one thing, but reckless is quite another. In this area of research there may not be leisure for constant improvements to make things right. HackerWay.org.

Not only will there be legal consequences, mass liability, for any group that screws up, but the PR blow alone from a bad AI mistake could destroy most companies. Loss of trust may never be regained by a wary public, even if Congress and Trial Lawyers do not overreact. Sure, move fast, but not too fast where you become unsafe. Striking the right balance is going to require an acute technical, ethical sensitivity. Keep it safe.

Last Word

AI ethics is hard work, but well worth the effort. The risks and rewards are very high. The place to start this work is to talk about the fundamental principles and try to reach consensus. Everyone involved in this work is driven by a common understanding of the power of the technology, especially artificial intelligence. We all see the great changes on the horizon and share a common vision of a better tomorrow.

In the near term I see greater prosperity and reduced mortality due to things like highway accidents and medical errors, where there’s a huge loss of life today.

In the longer term, I’m excited to create machines that can do the work that is dangerous or that people don’t find fulfilling. This should lower the costs of all services and let people be happier… by doing the things that humans do best – most of which involve social and interpersonal interaction. By automating rote work, people can focus on creative and community-oriented activities. Artificial Intelligence and robotics should provide enough prosperity for everyone to live comfortably – as long as we find a way to distribute the resulting wealth equitably.

About the Blogger

Ralph Losey is a practicing attorney and shareholder in a national law firm with 50+ offices and over 800 lawyers where he is in charge of Electronic Discovery. All opinions expressed here are his own, and not those of his firm or clients. No legal advice is provided on this web and should not be construed as such.

Ralph has long been a leader of the world's tech lawyers. He has presented at hundreds of legal conferences and CLEs around the world. Ralph has written over two million words on e-discovery and tech-law subjects, including seven books. He is also the founder of Electronic Discovery Best Practices, and e-Discovery Team Training, an online education program that arose out of his five years as an adjunct professor teaching e-Discovery and Evidence at the UF School of Law. Ralph is also publisher and principle author of this blog and many other instructional websites.

Ralph is a specialist who has limited his legal practice to electronic discovery and tech law since 2006. He has a special interest in software and the search and review of electronic evidence using artificial intelligence, and also in general AI Ethics. issues. Ralph was the only private lawyer to participate in the 2015 and 2016 TREC Recall Track of the National Institute of Standards and Technology and prior to that competed successfully in the EDI Oracle research.

Ralph has been involved with computers, software, legal hacking and the law since 1980. Ralph has the highest peer AV rating as a lawyer and was selected as a Best Lawyer in America in four categories: Commercial Litigation; E-Discovery and Information Management Law; Information Technology Law; and, Employment Law - Management. Ralph also received the "Most Trusted Legal Advisor" industry award for 2016-17 by the Masters Conference. His full biography may be found at RalphLosey.com.

Ralph is the proud father of two children, Eva Losey Grossman, and Adam Losey, a lawyer with cyber expertise (married to another cyber expert lawyer, Catherine Losey), and best of all, husband since 1973 to Molly Friedman Losey, a mental health counselor in Winter Park.

Sedona Principles 3rd Ed

1. Electronically stored information is generally subject to the same preservation and discovery requirements as other relevant information.

2. When balancing the cost, burden, and need for electronically stored information, courts and parties should apply the proportionality standard embodied in Fed. R. Civ. P. 26(b)(2)(C) and its state equivalents, which require consideration of importance of the issues at stake in the action, the amount in controversy, the parties’ relative access to relevant information, the parties’ resources, the importance of the discovery in resolving the issues, and whether the burden or expense of the proposed discovery outweighs its likely benefit.

3. As soon as practicable, parties should confer and seek to reach agreement regarding the preservation and production of electronically stored information.

4. Discovery requests for electronically stored information should be as specific as possible; responses and objections to discovery should disclose the scope and limits of the production.

5. The obligation to preserve electronically stored information requires reasonable and good faith efforts to retain information that is expected to be relevant to claims or defenses in reasonably anticipated or pending litigation. However, it is unreasonable to expect parties to take every conceivable step or disproportionate steps to preserve each instance of relevant electronically stored information.

6. Responding parties are best situated to evaluate the procedures, methodologies, and technologies appropriate for preserving and producing their own electronically stored information.

7. The requesting party has the burden on a motion to compel to show that the responding party’s steps to preserve and produce relevant electronically stored information were inadequate.

8. The primary source of electronically stored information to be preserved and produced should be those readily accessible in the ordinary course. Only when electronically stored information is not available through such primary sources should parties move down a continuum of less accessible sources until the information requested to be preserved or produced is no longer proportional.

9. Absent a showing of special need and relevance, a responding party should not be required to preserve, review, or produce deleted, shadowed, fragmented, or residual electronically stored information.

10. Parties should take reasonable steps to safeguard electronically stored information, the disclosure or dissemination of which is subject to privileges, work product protections, privacy obligations, or other legally enforceable restrictions.

11. A responding party may satisfy its good faith obligation to preserve and produce relevant electronically stored information by using technology and processes, such as data sampling, searching, or the use of selection criteria.

12. The production of electronically stored information should be made in the form or forms in which it is ordinarily maintained or in a that is reasonably usable given the nature of the electronically stored information and the proportional needs of the case.

13. The costs of preserving and producing relevant and proportionate electronically stored information ordinarily should be borne by the responding party.

14. The breach of a duty to preserve electronically stored information may be addressed by remedial measures, sanctions, or both: remedial measures are appropriate to cure prejudice; sanctions are appropriate only if a party acted with intent to deprive another party of the use of relevant electronically stored information.