Predictive Analytics—Black Box or Pandora’s Box?

Predictive analytics is infiltrating nearly every industry and is something increasingly part of the general public’s vocabulary. It has been applied to everything from terrorism risk assessment to tailoring the retail coupons we receive in the mail.

But these methods have a downside. Some individuals perceive predictive analytics as merely profiling that could be discriminatory in nature if all it does is quantify unfair biases (particularly where socioeconomics is concerned). This little article over at the Upside blog reminds us of the number of ways where predictive analytics could do harm.

In healthcare, analysts need to think carefully about how they place predictive analytics into business workflows. Here are some scenarios we have seen both in the literature and our consulting practice where predictive analytics didn’t have the intended impact. In all cases, poor operationalization was the root cause.

Predictive Analytics Case 1

Case: A large, multi-hospital health system created a risk score for sepsis based on a predictive model. This score was embedded into the EHR and would send an alert to providers once a threshold level was reached.

Unintended consequence: Alerts were sent to providers who either didn’t have enough context with how to act on that score, or they felt the alert was too sensitive. As a result, alerts were often ignored.

Potential fix: Provide targeted training to providers so that they have clarity in how to interpret the scores. Consider a process where nurses are the first recipient of alerts so that they will triage and determine if escalation is needed.

Predictive Analytics Case 2

Case: Analysts working for a cardiology department at a large academic medical center deployed a model highlighting congestive heart failure patients at high risk of readmission. Identified individuals were assigned a care manager at discharge who ensured adherence to post-operative care guidelines.

Unintended consequence: Cardiology did not realize that other participants in the care ecosystem were also monitoring high-risk patients. Concierge medicine programs were in play where a patient advocate follows up with the patient. The affiliated Medical Group had an initiative in place to monitor timely completion of post-discharge follow-up appointments. Their intervention was to call the patients with reminders. Blue Cross, being the largest private insurer in this market, used their process to follow-up with any patient who had a hospital stay of two or more days. As a result, patients could receive numerous phone calls and reminders. Many felt either patronized or harassed while others simply stopped answering the phone.

Potential fix: While the predictive model may have worked well diagnostically, the manner in which it was operationalized was sub-par. All care stakeholders need to step out of their silos and work together to develop a coordinated approach to the patient without being a nuisance. While the organization had an analytics Center of Excellence of sorts, participation was somewhat voluntary and the various analytics groups didn’t have a clear view of what other teams were doing. We would recommend some governance processes to document the ongoing analytics efforts. If these were in place, various departments could have coordinated the intervention before sinking time, effort, and money into a patient outreach approach that was more intrusive than helpful.

Predictive Analytics Case 3

Case: One East Coast health system wanted to hit the opioid crisis head-on by using behavioral data to predict those patients who are more likely to develop dependency and abuse tendencies.

Unintended consequence: While the campaign behind the model championed the use of behavioral data to craft a risk score, the model ‘lift’ was mostly driven by polypharmacy (e.g., multiple medications) and a history of back, neck, or shoulder pain. Although the risk score was meant to only guide provider decisions, certain primary care providers became overly sensitive to the values and were very reluctant to provide benzodiazepine medications (such as Xanax) even in appropriate scenarios. In other situations, individuals awaiting back surgery did not receive the medication needed and suffered unnecessary pain due to providers being overly conservative. Patient satisfaction decreased, and some individuals indicated that they would find a new provider.

Potential fix: Not all problems are amenable to predictive analytics solution. In this case, the behavioral data was fairly thin, and the providers already knew that multiple medications (particularly antidepressants) and spine-related injuries were related to opioid dependence. We would advise that the project owners consider other sorts of information that makes better use of the state’s prescription drug monitoring database.

What do These Scenarios Teach us About Predictive Analytics?

Overall, predictive methods should enhance human judgment, not replace it. Ensuring this happens means that stakeholders should be comfortable with the analytic product life cycle, analytics governance, and risk literacy—especially if they are expected to act upon the data.

Given so many of these pieces can be missing in a healthcare organization, even if it has a strong analytics team, we have created the ThotWave Healthcare Analytics Competency model that outlines the skills, capabilities, and behaviors needed to fuel a Learning Health system. This model helps organizations identify key capability gaps that may be limiting success with analytics.

About Us

Named after Thoth, the Egyptian god of wisdom, ThotWave has been a leader in healthcare analytics since 2002. Across the United States, healthcare and life sciences organizations turn to ThotWave to help them use data and analytics to improve quality of care and overall organizational efficiency.

ThotWave empowers your organization with the knowledge and tools you need to make smart, data-driven decisions that lead to better outcomes through advisory and consulting services, and specialized education and training in our Healthcare Analytics Academy.