Program learns to recognise rough sketches of objects

Researchers at Brown and the Technical University of Berlin have
produced a program that can
identify simple sketches of objects almost as well as
humans. The computer application enables "semantic
understanding" of abstract sketches as they are being drawn in real
time. The research was presented at computer graphics conference
SIGGRAPH and the paper is now available online.

Computers can already match sketches to objects provided they
are accurate representations -- eg matching police sketches to
actual faces in mug shots. However more abstract sketches -- the
more cartoonish drawings that most people can easily produce --
present a different challenge.

For example, in order to draw a rabbit, a person might draw a
cartoonish creature with big ears, buckteeth and a fluffy cotton
tail that another person would recognise despite the fact that it
bears little resemblance to an actual rabbit.

James Hays, assistant professor of computer science at Brown
explains: "It might be that we only recognise it as a rabbit
because we all grew up that way. Whoever got the ball rolling on
caricaturing rabbits like that, that's just how we all draw them
now."

Getting a computer to understand this is more challenging. In
order to create the program, Hays and his colleagues Mathias Eitz
and Marc Alexa from the Technical Univeristy in Berlin had to
create a large database of sketches to teach a computer how humans
sketch objects.

They started by coming up with a list of everyday objects from
an existing computer vision (photographic) dataset called LabelMe.
They ended up with a set of 250 object categories. They then used
Amazon's Mechanical Turk to hire people to sketch objects from each
category -- 20,000 sketches in total. This data was fed into
existing machine learning algorithms to teach the program which
sketches belong to witch categories.

From there, they team created an interface through which they
could input new sketches and the computer could try to identify
them in real time, as they are being drawn.

The program is capable of identifiying sketches correctly with
around 56 percent accuracy, as long as the object falls under one
of the 250 categories. This is not bad given that humans were able
to identify the sketches with 73 percent accuracy.

"The gap between human and computational performance is
not so big, not as big certainly as it is in other computer vision
problems," Hays said.

Hays and team plan to expand the database to include more
categories. One way of doing this is by creating a Pictionary-style
game that can collect the data that players input -- essentially
Draw Something. They have created a similar
game which is available on iTunes already. Ultimately the
program could be used to develop a better sketch-based interface
and search applications.