Big Data

In the last few days of a year, it’s natural to review the year that is slipping away and consider plans for the year that is about to begin. If you are lucky, you will have cause for some self-congratulation and not too much regret. Inevitably, this annual review results in promises of change for the new year. And so we begin the perennial cycle of wishful thinking known as New Year’s Resolutions.

While I’m not planning to make any major resolutions for 2018, I think I might commit to picking some fights instead. For those of you who know me in person or through my writing, that statement may seem a little out of character. But please bear with me. Here are the battles in which I intend to engage:

Fight the delusion of rational decisionmaking

Having grown up in the legal industry, I am used to dealing with people who take great pride in their good judgment, critical thinking, and rational decisionmaking. So it was a revelation to read Thinking Fast and Slow by Daniel Kahneman. He opened my eyes to the ways in which our minds trick us into making questionable decisions time and time again. We are not nearly as objective or rational as we believe. For example, we too often make decisions out of emotion or instinct and then dress those decisions up with “supporting data.” At the risk of disappointing you, I must tell you that this happens in all human spheres — including the legal industry. Just because you have confirming data does not mean you are right.

To understand more about the pitfalls of human decisionmaking and how it is affected by our well-intended but sometimes unfounded belief in our own objectivity, I’d encourage you to read Thinking Fast and Slow. If you don’t have time right now for this fairly large tome, watch the following brief videos to get a sense of the scope of the issue:

Fight the opacity of numbers

As the world becomes more complex, some of us take an ostrich-like approach by burying our heads in the sand. Others seek to understand what is going on by gamely trying to master as much of the relevant data as possible. However, even this more responsible approach has its own hazards. Among the biggest are that we aren’t all sufficiently numerate to understand what those numbers are saying (or not saying) and we cannot always see what is behind those numbers. For example, when you see survey results do you also spend some time to understand the survey methodology and the implications of the choices made in how the data were collected and interpreted?

Things are complicated enough when reviewing a relatively small survey. What happens when you are dealing with thousands or millions of data points in this era of big data? Then you rely on algorithms to help you sort and interpret the data. However, we are learning that those algorithms are not necessarily neutral or objective. Rather, they encode the assumptions and biases of the people who created those algorithms. When you don’t understand those assumptions and biases, you put yourself in danger of making decisions based on algorithms that may not, in fact, serve you well.

To learn more about how to make numbers more transparent and meaningful, start by watching these two videos:

Fight the corrosiveness of certainty

Despite impressive advances in human development, individual omniscience is still not possible. However, there are lots of people in denial about this. They believe they know all the answers and, therefore, rarely ask the key questions that can upend their certainties and unlock new stores of understanding. If we had a little less certainty and bit more curiosity we would reduce the occurrence of “unintended” consequences. Ultimately, excessive certainty corrodes our critical thinking abilities.

So in 2018, I’ll be asking more of the following questions:

What don’t I know?

What is missing? What hasn’t been disclosed?

What if my understanding is not correct?

What disconfirming evidence exists? And how persuasive is it?

Would I reach a different conclusion if my goal were generosity rather than self-protection?

Fight the comfort of complacency

Humans are creatures of comfort. However, that comfort can be a dangerous gift. It provides temporary respite but may blind us to potential opportunities and dangers. Sometimes this is because we assume that the current situation will continue indefinitely or that a better (or worse) event is unlikely to occur. And what is the source of this false assumption? Often, it is our lack of understanding of the actual root cause of the current situation or our failure to observe and properly interpret ambient information or patterns of behavior that may indicate an imminent change. This blindness leads us into imprudent complacency.

So how to fight complacency? First, learn how to identify a root cause. This analysis is a staple of many business school courses. It should be a staple of every high school’s curriculum. Until we identify the root cause, any conclusions we draw and any interventions we propose will be flawed.

The second way to fight complacency is to turn up your own antennae. Start noticing what is happening around you. The old adage that we see what we are looking for is true. So start looking for more, start looking for different.

What surprises you?

What patterns do you observe? Are they similar to or different from prior patterns? Why?

What is the likely outcome?

What could change that outcome?

In these next few days, I’ll be preparing for my chosen battles. I hope you’ll consider joining me in at least one of them. At a minimum, we will have a clearer understanding of what is and what isn’t.

This is how we began a presentation today on Big Data. I had the pleasure of presenting with Maura R. Grossman, Counsel at Wachtell Lipton, Rosen & Katz, and Chad C. Ergun, Director, Global Practice Services & Business Intelligence, Gibson, Dunn & Crutcher.

[These are my notes from the 2012 Ark Group Conference: Knowledge Management in the Legal Profession. Since I’m publishing them as soon as possible after the end of a session, they may contain the occasional typographical or grammatical error. Please excuse those.]

NOTES:

What are the Hallmarks of Big Data? Some refer to the three “Vs” while others refer to the five “Vs.” All are agreed that Big Data has the attributes of huge Volume, Velocity and Variety. In addition, some would add issues of determining the Veracity and Value of the data.

What’s the Big Challenge of Big Data? We’ve been spoiled over the last few years — deluded into thinking that we can capture useful data, cram them into spreadsheets and then analyze them at our leisure. However, as data grow, that approach becomes increasingly unviable. According to the IDC, less than 10% of Big Data is structured data. That leaves over 90% that is unstructured content such as social media streams (e.g., Facebook, Twitter), video and images, mobile data (e.g., GPS), text messages, emails, documents, IT and operational data, transactional data, search engine data and sensor data. Our simple little spreadsheets can’t begin to handle this variety of data, much less make sense of it. Further, we can’t manually populate and analyze these spreadsheets quickly enough to keep us with the high speed at which the data are created.

Companies that inject big data and analytics into their operations show productivity rates and profitability that are 5% to 6% higher than those of their peers.

Our Clients are Interested in Big Data Client interest and activity in the Big Data arena is growing. You can find many examples of how organizations are using the new insights that come from Big Data analysis: (1) Walmart analyzes 267 million transaction per day in order to stock its shelves more appropriately and profitably; (2) Google tracks flu trends based on search queries relating to flu symptoms; (3) medical scientists are uncovering the genetic and environmental causes of disease; (4) thousands of sensors provide data that helps scientists under the environment and changes in weather; (5) real-time analysis by computers of video feeds helps fight crime.

What are the Law Firm Opportunities and Concerns? The most obvious way to use Big Data analysis is in competitive intelligence research. Another possibility on the horizon is to use Big Data analytics to help us understand better the staffing, pricing and profitability of matters. Ideally, this would be done by searching all the unstructured and structured content within the firm to find the patterns. Another opportunity is in eDiscovery. The problem is that while it may be cheap to store data, it’s is still expensive to conduct an eDiscovery review of that data using current methods and technology.

Mashup Magic. Starting with the provocative question “What decisions could we make if we had all the information we needed?” the audience brainstormed to create a list of new data mashups that could solve law firm business problems. Among the suggestions were: (1) analyzing unstructured content to help uncover possible client conflicts; (2) analyzing emails to identify trending topics; (3) analyzing financial data filed with the Securities and Exchange Commission to find early warning signs of financial distress (or even bankruptcy) of current or potential clients; (4) comparing government economic data with initial public offering pricing data; (5) recording, indexing and analyzing phone calls to find opportunities for cross-selling; (6) comparing pitch document contents to pitch success rates; (7) comparing the content of pleadings with matter success rates; (8) comparing weather data with sales data in your law firm cafeteria.

What’s the Future of Big Data? For those of you who thought Big Data was a passing fad or a science fiction fantasy, think again. Surveys of senior business leaders indicate that Big Data is one of their primary strategic concerns. In fact, a recent McKinsey survey found that 60% of the CEOs and CIOs contacted believe their companies should use Big Data analytics to generate insights regarding customer preferences. Meanwhile, spending for Big Data is set to rise:

Big data will drive $28 billion of worldwide IT spending in 2012, according to Gartner, Inc. In 2013, big data is forecast to drive $34 billion of IT spending.

Final Thoughts. Used creatively and thoughtfully, Big Data can provide important insights into your firm’s operations and business environment. Our clients are embracing Big Data. Can law firms afford not to?

Topics

Topics

The Long Tail

The Long Tail

Disclaimer & Copyright

This publication contains my personal views and not necessarily those of my clients. Since I am a lawyer, I do need to tell you that this publication is not intended as legal advice or as an advertisement for legal services.