Two students from MIT and Stanford created an algorithm that detects human emotion

By the time Matthew Fernandez and Akash Krishnan were old enough
to drive, they had invented an algorithm that can listen to human
speech and detect dozens of emotions.

Now, the pair are using their algorithm — the centerpiece of a
company called Simple Emotion — to
revolutionize call centers around the globe.

Often, the worst part of a customer service experience is
speaking to an agent or robot that cannot empathize with a
caller about the situation that led him or her to complain.
This tension can cause hair-pulling frustration for a
customer.

As part of a high school science research project, Fernandez and
Krishnan set out to build something that could recognize emotions
in people's voices and provide real-time feedback to the
representative taking the call. The information could
be used to provide the best service experience, as well as for
quality assurance and training purposes.

The Simple Emotion algorithm works by monitoring acoustic
features in speech — such as voice frequency, volume, and changes
in tone over time — and comparing them to a library of sounds and
tones. It identifies an emotion by finding the closest match in
the catalog.

Fernandez and Krishnan culled audio files expressing specific
emotions from a database and fed those examples to the
algorithm to train it about how different emotions
sound.Today, it understands between 30 and 40
emotions, according to Fernandez.

Companies can hire Simple Emotion to do speech analysis for
them, or license its API and integrate the algorithm into
their own customer service tools.

Though he says the film is "only okay," there's a scene in
which the robot comes into a room where two people are
fighting and offers to help. "I detected elevated stress
patterns in your voice," the robot says to Will Smith.

"We were inspired enough to Google it," Fernandez says. "[Emotion
detection] is this small, niche thing, so we decided to build our
own algorithm."

Fernandez and Krishnan started to stay up
late reading research papers and blow off their other homework.
When they finally submitted their science project, it
included 30
pages of code and 60 pages of technical writing to
explain how the whole thing worked.

They later took the grand prize at the Siemens Competition
in Math, Science & Technology, the nation's most
prestigious research competition for high school
students.

Akash
Krishnan and Matthew Fernandez took home the grand prize —
including a $100,000 scholarship — at the 2010 Siemens
Competition in Math, Science & Technology.Susan Walsh/AP

Once they enrolled in college (Fernandez at Stanford and
Krishnan at MIT), the pair continued working together across the
country to hone their algorithm, spending hours talking on Skype.
Fernandez guesses he spends as many hours working on Simple
Emotion as he does on his school work.

Though theirs is not the first system that attempts to
gauge customers' emotions, Fernandez
and Krishnan argue that Simple
Emotion's algorithm is the most accurate because it scrapes
data directly from the source: voice.

Other systems convert the speech to text first, and identify
emotions by searching for certain keywords in the script. (A
customer that swore a lot is probably angry, for example.)
Companies can also improve their customer service response
by paying a human to listen to a call once it's been
recorded and make an evaluation based on quality assurance
standards, but doing so is expensive. Or
a representative can send a customer survey by email after
hanging up, but Fernandez estimates 90% to 95% of recipients
will ignore it.

Indian
employees work at a call center in the southern Indian city of
Bangalore June 26, 2003. India's call centers provide cheap
English-speaking workers and high-speed telecoms to provide
customer service helplines for companies around the world.
They're a boon for India's army of job-hunting youth, but there
is a murkier side to the industry. Picture taken June 26,
2003.Jagadeesh
NV/Reuters

"How your customer feels — that's this Holy Grail nugget of
information that a call center representative wants," Fernandez
says, suggesting that his system is roughly 50% more
accurate than comparable systems.

In the future, companies using Simple Emotion could program
the robot that greets you at the beginning of a call
(asking you to say your account number aloud, for example)
to route you to the agent best suited to handle your
needs. The algorithm might even prioritize customers
in the queue who sound irate.

Fernandez and Krishnan plan to continue growing their
company after they finish college. Simple Emotion hasraised about half a million dollars in seed funding from
the Portland Seed Fund and ZenShin Capital, and currently has a
pilot project underway at the Mizuho Financial Group,
Japan's second
largest bank by assets. Fernandez declined to give
additional details about the pilot program.

Although a computer that knows how you're feeling and
can respond accordingly sounds manipulative, Fernandez says
that can be a positive.

"It's a little bit creepy," Fernandez says. "But hey, if
you get better service that's only a win for everybody."