Machine Learning Needs A Human-In-The-Loop

Kismet is a robot head made in the late 1990s at Massachusetts Institute of Technology by Dr. Cynthia Breazeal as an experiment in affective computing; a machine that can recognize and simulate emotions. Image: Wikipedia

Artificial Intelligence (AI) has a problem -- it’s artificial. To be fair, AI and its sister disciplines of machine learning, cognitive computing, sentiment analysis and neural networking have a problem -- they’re artificially created through the power of software developers’ algorithms.

The 80:20 rule

A fresh discussion has surfaced suggesting that the answer lies in adopting the 80:20 rule so that we can ensure there is always a human-in-the-loop (HITL) factor.

CEO of CrowdFlower Lukas Biewald has tabled this proposition given the huge amount of AI and machine learning currently being pumped into driverless cars and other smart devices from ATMs to
Facebook’s photo labeling function. These technologies need a degree of human in them, says Biewald.

CrowdFlower styles itself as a ‘data enrichment company’ and its technology has even been dovetailed by
IBM Watson. This strays into the area that we call ‘active learning’ (or semi-supervised machine learning, if you prefer) where a computer program’s learning algorithm knows that it can periodically and interactively ask questions of a user (or user group) to gather desired outputs at new data points.

"In parallel [with Watson’s cognitive intelligence], data enrichment platforms have become a valuable resource for data scientists looking to automate and scale the cleaning, labeling and enrichment of data using human intelligence for machine learning -- i.e. training data creation. While Watson continuously engages in ‘active learning’, its intelligence is strengthened by the quality of the training data it takes in from crowd contributors on data enrichment platforms such as CrowdFlower,” said Seth Teicher, content and business development leader at CrowdFlower.

Biewald argues that AI models that don't have some sort of human-in-the-loop element are flawed. Why? Because the AI naysayers (or the people selling supplementary crowd-based services as in this case) say that accuracy of AI tops out at around 80%.

This trend is becoming more pronounced and even Google still uses human beings to build its ‘intelligence’ and search prowess. Pinterest has also used this type of system to help filter out certain types of content online:

Things that are inappropriate for the general public, like sexually explicit or pornographic Pins

Hateful Pins or language that attacks a protected group or individual

Anything that promotes mental, emotional or physical harm to yourself, others or animals

Content that's fraudulent, deceptive or misleading

In order to build what it calls ‘artificial’ artificial intelligence to filter this content, Pinterest’s ‘Discovery Science’ team built a library that plugs into multiple crowdsourcing services, including CrowdFlower and Mturk.

“There are some things humans tend to do better, such as evaluating content, but having Pinployees (Pinterest employees) regulate content on Pinterest is simply not scalable. For that reason, we’ve automated human evaluation for the purposes of analyzing the relevance of search results and to filter out certain types of content. We can use crowdsourcing to do everything from evaluating search relevance to comparing treatment groups of experiments. Combine this with our workflow management system… and we’ve built ‘artificial’ artificial intelligence,” said Maesen Churchill, a software engineer on the Pinterest search team.

Why we still need humans

There are all sorts of reasons why we still need human beings in Artificial Intelligence. From nuances of spoken language to unexpected typographical errors, no single computer and no single crowd-augmented system can ever be perfect -- at least not at the time of writing in 2016.

Perhaps a better variant on Vilfredo Pareto’s famous 80:20 rule would be 80% computer-driven AI balanced with 19% human input and 1% unknown random number generator variable… just to keep things organic.

CrowdFlower CEO Biewald sums up by saying, "AI algorithms can't be debugged as easily as a traditional algorithms. In fact, the best way to train them–and in turn make them more accurate – is by feeding them large datasets. For many problems, like image recognition and natural language processing, humans simply create better training data."

As Biewald has said before, computers are great at analysing tough tactical situations, but are still not as good as humans at understanding long term strategy. The planet is still safe, for now.

I am a technology journalist with over two decades of press experience. Primarily I work as a news analysis writer dedicated to a software application development ‘beat’; but, in a fluid media world, I am also an analyst, technology evangelist and content consultant. As the ...