Bringing Sound Into the PC Foreground

Share

Bringing Sound Into the PC Foreground

You're talking to your mother on the phone about your new boyfriend. You can't see her face, and her words seem innocent enough. But when she says "He sounds nice," the tightness and drop in her tone of voice lets you know that she does not approve of your new tattooed and pierced performance-artist lover.

Sound can provide insights that visuals cannot ...

As voice recognition makes its slow, steady progression into PC applications, the related field of "sonification" is trying to make its pitch heard, too.

Sonification uses sounds to communicate, or help a user monitor data on a desktop. Researchers at IBM's Computer Music Center in Yorktown, New York, say that human ears are sensitive enough to interpret sound, just as our eyes process words and characters.

With sonification, applications could be designed to produce a background tone or tempo that increases in pitch or speed as data changes, thus freeing users to perform other, visually-oriented tasks. Sound also provides insight into patterns that may not be apparent on a purely visual interface, and one researcher noted that people already process information through noises in everyday life.

"If you're driving a stick shift, you change gears when the engine revs up too high, without thinking," said David Jameson, a project manager at IBM's Computer Music Center, and former member of the '70s Irish band Time Machine. "You do it without thinking, even while talking to a passenger. You are detecting changes in sounds."

Big Blue's researchers are currently revamping "Sonnet," the most recent incarnation of a visual programming language that allows developers to more easily experiment with adding sound to programs. Whereas speech recognition or virtual reality applications use sound mostly as a side effect to another program, sonification has a much more specific aim: to use sound as a computer interface, so that users can receive more data without adding to the visual clutter of their desktop.

With strictly visual interfaces, people's ability to multitask on a computer may soon be maxed out due to the physical limitations of our eyeballs, research shows. Soon, we may monitor data flow with our ears - a stock's rise or fall, an increase in temperature at a chemical plant, or any other type of data feed.

Sonnet allows users to link blocks of code together and quickly build user interfaces for managing data flow, and is a tool for sonification experimentation. Jameson refers to it as an "interface glue language" for connecting different subsystems together.

But since research on sonification has only really gotten serious in the past decade, there are plenty of skeptics.

"Sonnet sounds kind of gimmicky to me," said Roger Kay, an analyst at International Data Corp. Kay said that the lack of hardware will keep sonification from becoming a reality anytime soon. "Most computing being done today is character oriented, since characters are fairly lightweight, and can be compressed and compiled," he said. "When you start bringing sound into them, you are committing to systems that accommodate heavyweight data in all ways."

Eric Scheirer, a research assistant at MIT Media Lab, agreed. "It's true that today's desktop audio systems aren't ready for prime time, but they (Jameson and co.) are working on what is the next generation of audio products," said Scheirer, who also worked on the MPEG audio standard due out next year.

"There's a good chance they will find a market for Sonnet," Scheirer said. "It's nothing really new, but it's the best attempt at doing this - making it easier to the user and moving it into a broader community. It could absolutely further the field of sonification."

Jameson pointed out that much of the research on sonification focuses on determining what kinds of noises work best to indicate information flow. Thus, a tone tracking the stock market would likely be a pleasant background tone that changes pitch with new data.

Jameson said he and his team will begin revamping Sonnet in the next few weeks, and hope to have the changes completed by the end of the summer. So far, Sonnet is for in-house, experimental use only, and is not intended for commercial use. But the researchers see promise in the technology for taking advantage of people's biological processors.

"Knowing how our senses are best used, we can provide the right sensory inputs to take advantage of what each one does best," said Jameson.

He dreamed up Sonnet while writing his PhD thesis on debugging programs with music, but the ideas behind sonification go back to the room-sized mainframe computers of the 1950s.

"Old hackers at MIT used to debug programs by placing transistor radios next to the computers," Scheirer said. The radio interference noises would change as the computer carried out its programs, and people could learn to recognize the patterns of the sounds for debugging. If the noise kept repeating for example, then the computer could be stuck in a loop.