XAI, sometimes called transparent AI, has the backing of the Defense Advanced Research Projects Agency (DARPA) an agency of the US Department of Defense, which is funding a large program develop the state of the art explainable AI techniques and modelling.

Dr Brian Ruttenberg was formerly the senior scientist at Charles River Analytics (CRA) in Cambridge, where he was the principal investigator for CRA’s effort on DARPA’s XAI program.

He argues XAI helps to identify bias or errors in algorithms and engenders trust in the technology.

“Doctors aren’t going to tell someone they have cancer because the machine learning box told them they have cancer,” Ruttenberg told Which-50.

“They need to understand the underlying factors and explain them to the patient. It’s about building trust between the doctors and the machines and transferring that trust to patients.”

Over the past two years XAI has emerged as a research field in its own right, but some AI conclusions are easier to explain than others.

Why did the driverless car cross the road?

Ruttenberg, who is now a principal scientist at NextDroid, a Boston-based autonomous vehicle and robotics company, says explainability is “very immature” in the field of driverless cars.

That’s because XAI research is largely concerned with explaining a single neural network rather than a series of systems that decide what a vehicle should do next, he said.

For example, explaining how an image classification systems built to identify pedestrians works is “straightforward,” he said. The conclusion the neural network reaches is based on the information contained in the image, ie if an object has wheels instead of legs, then is not a pedestrian.

In the world of driverless cars, where multiple systems interact and conditions change over time, explaining why a vehicle turned a corner or hit the brakes is a more complicated problem to solve.

“Autonomous vehicles have many different interacting systems,” Ruttenberg said. “They have perception, prediction and planning that are not just a single piece of software that you are trying to explain.”

Those machine learning systems are also interacting with a changing environment as the vehicle travels through the world.

“In these dynamics applications an explanation for a action could be your own action you took 20 seconds ago and that is a very difficult problem,” Ruttenberg said.

For example, a car may turn left suddenly because there is a car in front of it, but the reason the vehicle is dangerously close to another car is because 10 seconds ago it made a horrible lane change.

“That chain of understanding of how your system works could go back very far,” Ruttenberg said.

Regulators, take the wheel

In the race to build a level five autonomous car, Ruttenberg is skeptical carmakers and tech giants are spending extra resources on XAI.

“The first person that comes up with a level five autonomous system is going to have a tremendous advantage over any competitor. And I don’t think some of these big companies aren’t interested in explainability, or wouldn’t like it, but I don’t think they are going to spend resources trying to make their systems explainable or more explainable than needed to de-bug them,” he said.

That means regulation will likely be required to enforce a level of explainability not just for developers but so a car’s systems are transparent for buyers, passengers and society more broadly.

“Like in the financial industry where you have to prove to regulators [their lending process] isn’t biased, I think you are eventually going to have to prove to regulators that your system is safe and you’re going to have to do it through explanation,” Ruttenberg said.

The Author

Tess Bennett

Tess Bennett is the editor of Which-50 and is responsible for leading the publication’s daily coverage of Australia's digital businesses for C-Suite executives, strategists, founders and directors. As the former editor of Internet Retailing Australia and journalist for Inside Retail, Tess has five years experience covering retail and ecommerce. At Which-50 Tess reports on a broad range of topics including technology, the industrial internet, analytics and digital marketing.

Join the digital transformation discussion and sign up for the Which-50 Irregular Insights newsletter.

Must Reads

Disparate data systems and teams are amplifying the already considerable challenges around cross channel marketing. That is why many marketers feel their customer’s experiences are “broken”. Many, in fact believe their organisations struggle to even accurately identify customers across channels and devices. Download the ebook: Cross-Channel Marketing: A Prescriptive Guide Fortified