What Les Misérables teaches us about the dangers of AI bias

Victor Hugo’s much-loved tale of redemption helps us understand the disconnect between legality at morality

It’s not unrealistic to expect any decision made by a machine to align with human moral and legal constructs. Yet when it comes to machine learning (ML), the subset of artificial intelligence (AI) that enables machines to learn from and make decisions about data, it is the bias hidden within that data that will ultimately define a machine’s ‘morals’.

It’s vital, then, that in any situation wherein a machine is given the ability to make a decision that affects a human, we understand – very clearly and transparently – the bias of both the data and the agents creating the patterns and policies the machine will use to make sense of it. If a machine’s understanding of right and wrong is based purely on what it has been fed, the responsibility for its actions must fall on the machine’s creator first and foremost – yet we should not underestimate how important it is to look at the human decisions that shaped that data in the first place.

A lesson from Les Misérables

Let me take you back
to France, 1815, and the stage that is set by the novel-cum-musical Les Misérables. Jean Valjean is released
on parole after years of hard labor: five for the theft of a loaf of bread to
feed his starving family, the rest for attempting to escape. He breaks parole
and flees, becoming successful and well-loved only for his past to catch up
with him in the form of the relentless Inspector Javert.

Javert represents the objective letter of the law, mitigating circumstances be damned; to him, Valjean is a thief and dangerous repeat offender. Yet we are asked to reach a far more compassionate conclusion: Valjean is a good man, pushed to do what he did through desperation.

It’s this story of
redemption and courage in the face of oppression, a celebration of second
chances and the triumph of human compassion over dispassionate castigation that
has filled theatres every night for over thirty years.

Yet as we queue down
Broadway, machine learning algorithms such as the Public
Safety Assessment tool, currently in use in Texas courtrooms, are crunching
historical data from thousands of court cases in order to assist judges in deciding
bail amounts, jail sentences and the likelihood defendants will reoffend.

Asking a machine to
identify patterns in court case data is likely to result in it learning from
the most objective cases, dismissing those in which human empathy changed the
outcome of an otherwise cut-and-dried sentence. Doing so may well create a
machine with the same brutal objectivity as the ‘villainous’ Inspector Javert,
making it very bad news for poor Valjean; burglary, theft, repeated prison
breaks, parole violation, identity fraud and a continued disrespect for the law
would surely present an easy decision for a Javert-bot.

In his 1976 book Computer Power and Human Reason, the late Joseph Weizenbaum makes the crucial distinction between decision and choice. A machine can only decide, never choose, says the AI pioneer; a decision is a calculation based on programmed policy while choice necessitates non-mathematical factors such as morality and empathy. But can we, and should we, strive to create machines that are capable of thinking morally in a way that at least appears to be choice? How many layers of complex policy might be required in order for a machine to empathize with Valjean’s mitigating circumstances? And could a machine ever be capable of favoring its empathic instincts over law-based policies?

The ethical maze

The relationship between law and morality is often dictated by cultural, ethical and socioeconomic variation. Laws and punishments in one culture may also be viewed as overtly oppressive or discriminatory in another – a slap on the wrist in one culture may be the death penalty in another. Valjean and Javert may be two fictional characters, but they represent archetypes that exist throughout contemporary society: those that uphold the letter of the law and those prepared to bend those laws based on moral and ethical compromise. If we were to train two machines using the personal experiences and principles of these two characters separately, we would end up with two machines with highly polarized views on ‘right’ and ‘wrong’: a perfect example of AI bias.

In the case of the machine learning being used by the US judicial system, we understand that the principles behind each decision are based on the state-sanctioned outcomes of previous trials. Yet given the history and diversity of the United States, it is reasonable to suggest that at least a portion of those outcomes may have been unfairly prejudiced by a defendant’s age, race, gender or social status. In 2016, for example, ProPublica reported that ML algorithms in U.S courts were more likely to label black defendants as having a high risk of recidivism compared to white defendants from similar backgrounds, even though the system wasn’t explicitly fed any data on defendants’ race.

With so much reliance on historical – or simply human – data, we should not be surprised when machines demonstrate AI bias, making decisions incompatible with today’s focus on social, rather than simply legal, justice. This is not an ML problem, it is a human problem; if the textbook is biased, the student will become biased as well. In order for us to develop fair and accountable AI, psychologists, sociologists, anthropologists and technologists must all come together to explore the many ways in which bias affects human life — and what we can do to ensure that such bias does not end up amplifying the prejudices this technology is often created to mitigate.

It’s on us

Very many questions
remain unanswered. Yet with machine learning already playing a role in making
decisions over the lives of humans, now is the time to ask these questions –
and it falls on us, as creators and enablers of this technology, to answer
them.

At Arm, we are already taking steps to identify how the machine learning technology we create, and the world of artificial intelligence we will empower, affects and is affected by humanity. This move not should be exclusive to Arm. I strongly believe that every single company that engages in the pursuit of enabling AI should be dedicating significant resources to the adoption of ethical frameworks for responsible innovation and the reduction of AI bias today and in the future.

Most importantly, any framework or debate must be inclusive of a broad cross-section of societies and cultures, incorporating the diverse opinions, laws, and ethics that exist throughout the world – even if we don’t agree with them. Global technology requires not only a global approach but an appreciation of what that technology might be asked to do on a local level.

For all of us, this is
only the beginning of a long journey that is likely to require us to delve deep
into our own ethics and morals before we ever get close to enabling a sense of
morality in machines. Some questions will need to be answered very soon, others
may take years for the answers to become clear. We look forward to playing our
part in ensuring that artificial intelligence is a force for good that brings
immense benefit to society.