R Is for Robot

What bots can teach tots (and vice versa).

By Larry Gallagher

"Diaper Change!"

Rubi is having a bad day. The toddlers in Classroom One were excited to see her this morning, shouting her name and poking her belly. But Rubi is not her usual self. She tries to sing "The Wheels on the Bus" but stops abruptly in midsong. She's not making eye contact. And she has some kind of crick in her neck that's causing her head to jerk erratically.

Javier Movellan has seen enough. He needs to clear the room. He has to communicate to a bunch of 1- and 2-year-olds that there is a mess so urgent they'd better get out in a hurry - diaper change! Soon all the little ones have made their way to the patio, leaving Movellan and his assistants to scurry around, trying to figure out what's wrong with Rubi.

Such are the perils on the frontiers of human-robot research here at the Early Childhood Education Center, a preschool attached to UC San Diego. For the past six months, the children in Classroom One have spent half an hour of each school day interacting with one of two robots. Rubi takes on the role of a teacher, leading songs and playing games that instill basics like ABCs, shapes, and colors. She alternates days with Qrio, Sony's swanky prototype humanoid, whose role is more peer-to-peer: He spends his time dancing with the kids. The class is taking part in a project developed by Movellan, who directs the university's Machine Perception Lab. For him, the short-term goal is to watch the kids and use what he learns to develop interactive teaching tools.

But Movellan also sees this daily routine as part of an odyssey to create the world's first nondisappointing robot. Try as we might, we can't stop measuring bots against the ridiculously high standards that Hollywood has set in our minds. We expect them to be able to follow us around the house, picking up after us, chattering with us like C-3PO, reading our emotions with the accuracy, if not the intent, of HAL. While robots have proved indispensable in narrow kinds of work, like assembly lines, when it comes to interactions with unpredictable, flesh-and-blood humans, they have yet to deliver on the promise of more lifelike responses. That's why Movellan and his team have enlisted this young group of chaos specialists. The researchers videotape each session and use the footage to tweak the bots' software.

In a closet adjoining Classroom One sits Bret Fortenberry, the proverbial man behind the curtain. Peering at Rubi through a one-way mirror, Fortenberry triggers her behaviors by pecking on a laptop. Today, for the first time, he has been trying out Rubi's face tracker, which should allow her to automatically direct her attention to the children as they cross her visual path. And though he suspects this program might be the source of Rubi's breakdown, he has no time to search for bugs now. In 15 minutes, the tykes will come waddling back in. He can only reboot and hope for the best.

Fortenberry works the keyboard as the toddlers return: F7, "Hello, Tyler." F3, "Hello, Anna." Rubi chirps their names in a cloyingly cute voice. "Do you want 'Monkeys'?" This is shorthand for "Shall we sing the 'Four Little Monkeys' song yet again?" Rubi emits a prerecorded version of the song, and the kids babble and dance. When it ends, Rubi shifts into teaching mode, initiating a game in which the children identify shapes and colors that move across her touchscreen belly. "Where is the diamond? Good job. Where is the triangle? No, that's a rectangle."

Something is still not right, though. Rubi's head twitches unpredictably. Her arms quiver with a Parkinsonian palsy. Tyler, a little caveman with a bowl haircut, tries to give Rubi a hug, but she can't be consoled. Short of spewing smoke or taking hostages, Rubi couldn't be doing worse. Suddenly she stops, then slowly tilts her head upward until she is staring blankly over the heads of the toddlers and straight up at the exposed I beams supporting the roof of the Early Childhood Education Center.

Back in the lab that afternoon, Movellan upgrades his evaluation of the morning's events from total catastrophe to miserable failure. Movellan, 45, would never have made it through more than two decades of working with robots if he didn't have a high threshold for frustration. For years there was a widespread belief in AI circles that insufficient computational power was holding back the development of social robotics. But the exponential expansion of that power has not brought us any closer to the reality of a robotic Jeeves. Movellan remains confident that he will see "everyday" robots in his lifetime: teaching machines in the classroom, physical rehab systems with a "human touch," and robotic companions that can develop personal relationships. To get there, however, the field of robotics must undergo a "scien-tific shakedown," as he calls it, a reevaluation of its methodology and agenda. The Rubi project is his lab's first step in that direction.

Movellan's Machine Perception Lab sits in a corner of the Applied Physics and Mathematics Building, with a view of the Pacific so lovely it's surprising that any work gets done at all. Here, a team of grad students, postdocs, and visiting scholars are doggedly toiling away on software that will allow Rubi or some future robot to make sense of incoming visual and auditory information. Fortenberry has already found the bugs in Rubi's face tracker that caused her problems this morning and is in the process of fixing them.

Fortenberry then shows me a program called the expression recognizer that parses my facial expressions and assigns each look to one of seven categories. But the program that Movellan is most excited about is a new so-called face detector being developed by grad student Ian Fasel. A face detector allows a machine to distinguish a human face from a visual background. Until now, such programs had to be fed tens of thousands of facial images before they would work effectively, a tedious and time-consuming process. Fasel's system is based on a series of formulas that describe the borders on a human face. The program will enable the robot to learn on its own. When he works out the kinks, Fasel tells me, his program will be the first of its kind.

Of course, this is the type of thing you hear a lot in the robot biz: Right around the bend is some significant and satisfying advance. Researchers can spend years developing programs that work in the hermetic setting of the lab but fail in the unpredictable real world. Movellan has experienced this frustration himself: "We tested our expression recognition system against the standard data sets in the field, and we were getting performance on the order of 97 percent, better than anyone else in the world. We used this in Rubi in everyday life conditions and it completely sucked." Movellan believes that, painful though it can be, testing such programs outside the lab is crucial.

This kind of pragmatism has also informed Rubi's construction. Movellan and Fortenberry built Rubi for about $3,000, using spare parts whenever possible. My own sophisticated expression recognition system tells me that Movellan is a little embarrassed by the bot's humble components, but his sponsors should be happy to know that he is not wasting their grant money on vanity robotics. Rubi's thorax is constructed from a TV bench (Ikea's Ilen, $35). Her sides are made out of pegboard (Home Depot, $14). Her belly is a touchscreen (Elotouch.com's Entuitive 1247L, $783). Her head is on loan from a lab in Japan, customized with big blue eyebrows and a simple mouth capable of smiling and frowning. Her hands were originally part of one of those claw machines you find in boardwalk arcades. "I'm not saying we have the best robot out there," Fortenberry concedes. "Not even close. But we could spend thousands of dollars making the hands alone." For a brain, Rubi has three different onboard processors that handle her various perceptions, all of which are coordinated by Fortenberry and his laptop.

On the simplest level, Movellan is using Rubi to explore the viability of robots as teaching tools. The idea first occurred to him five years ago, when his lab collaborated on a National Science Foundation project to develop interactive computers for the classroom. At one point he mounted a pivoting camera on top of the computer and found that the children in his test group were responding to the camera as if it were alive. "I asked myself, 'Why don't we make the camera the character? Why don't we make the computer a robot?'"

Movellan hopes to distinguish Rubi from existing automated teaching software by adding an emotional component to the interaction between kids and the machine. "The success of Rubi as a learning system is going to depend on whether she can engage these children - make them feel good about learning," he says.

Movellan is not the first AI researcher to propose this approach. Indeed, the past decade has seen the emergence of what's called affective computing, whose proponents believe we need to build emotions into robots. This will help humans relate to them more easily, they say. It will also help the robots better organize their own behaviors, much as our emotions allow us to compress complex chains of perception and prediction into simple responses like attraction or aversion.

In the back of the lab, by a coffee table made from scrap pegboard left over from Rubi's exoskeleton, Movellan tells me in hushed tones about the epiphany that pushed him headlong into the world of affective computing. In the fall of 2002, he was working in Kyoto at ATR, the Japanese government's robot research lab, sinking deeper and deeper into the mathematics of machine perception, drifting in the intellectual tides and feeling uninspired by it all. "I was very skeptical. There was a robot there, and I didn't like it. It would say things like 'Hug me! Hug me!' It really irritated me." One day Movellan found himself using the robot to test an early version of the face-tracking program that he and Fasel developed here in La Jolla. "It worked really, really well. As I was testing it, I kept moving, and this robot kept looking at me, and his eyes moved in a particular way, and I got close, and this robot kept looking at me. And then it hugged me. And it completely got me." Movellan was shocked by the strength of his own response. "I said, 'What's happening here? I know this thing is dead. I mean, it's not alive. But I would swear that this thing is alive.'"

Movellan was too embarrassed to mention the incident to his colleagues, but later, he confided his experience to some neuro-scientists. They were not surprised, theorizing that the robot had acted realistically enough to trigger the firing of Movellan's amygdala, a part of the brain's limbic system linked with fear and pleasure.

The experience inspired Movellan to shift his focus from the cognitive power of computation to the emotional aspect of the interaction. Whereas previously he had developed the various trackers and detectors to increase the quality of a robot's perception, now he would use these same tools to foster a feeling among human subjects that the robots cared for them. The more a robot can tap into these parts of our consciousness, he came to believe, the deeper our engagement with it will be. To hone the robot's skills, he would enlist the help of test subjects who would offer honest feedback and be unlikely to hide their feelings behind words. He would need subjects who would tolerate the repetition necessary to conduct his research. And he knew where to find them.

The Early Childhood Education Center is the very model of preschool prosperity. The facilities are bright and clean, the outdoor spaces shaded with mesh nets and padded with rubber mats. On the bulletin board in Classroom One are detailed breakdowns of the daily schedule, modes of play, developmental goals, and curriculum for the 11- to 24-month-old students. Taking Apart and Putting Together Toys and Sorting Toys fall under the rubric Pre-Math. Pre-Science includes such disciplines as Sand Play, Digging, Shoveling, and Pouring.

Though its services are available to the general public, the center gives priority to children whose parents have a connection to UCSD. As part of its association with the university, the center is the site of several studies in childhood development and psychology. And, of course, it is the host of such perennial research projects as Relative Absorption Rates of a Bovine Mammary Secretion on Synthetic Versus Phytogenic Textiles and Exploration of the Nasal Canal as an Alternative Vector for Cheerio Assimilation.

So when Movellan proposed to use the kids as test subjects for robot experiments, it did not meet with the kind of resistance you might expect. His own son had gone there; he was a known quantity to the teachers and the administration. They worked out some guidelines for the classroom, limiting the time the robots are with the kids to about 30 minutes a day and allowing the children the option of wandering away from the machines if they wanted.

And when it came time to introduce the robots, Movellan's team had the perfect envoy: Qrio is a 23-inch-tall plastic humanoid with little blue eyes, opposable thumbs, and an impressive arsenal of moves - a rock star among robots. Having Qrio drop by for playtime is like having Bono drop by for a sing-along.

On my second day visiting the center, I get to witness his charisma. Luke and Anna are pushing up against the chain-link fence, gurgling "Qrio! Qrio!" as researcher Fumihide Tanaka carries the robot, swaddled in a baby blanket, through the school gate.

Qrio debuted as a corporate ambas-sador in 2003, when he and his prototype brothers were dispatched around the globe to conduct orchestras, throw out first pitches, and generally awe the world with their motor skills. Qrio can make his way through a roomful of obstacles, turn his head toward the source of a noise, and kick a soccer ball. His balance systems allow him to walk on uneven surfaces, protect himself when he falls, then climb back to his feet again. Despite his many talents, here in Classroom One Qrio focuses almost exclusively on shaking his little plastic booty (Pre-Mating?).

After laying Qrio down on the floor, Tanaka retreats behind the tinted glass, and the teachers bring in the children. To an instrumental version of "Johnny B. Goode," Qrio rises slowly to his feet and joins the kids, who are already bobbing and flapping their arms. Tanaka's laptop displays what Qrio "sees" through his stereo cameras as he bumps and grinds his way around the room, reading the motion of the kids and reacting with his own moves. Sony has loaded Qrio with some fancy liability avoidance technologies: pinch sensors under his arms and between his legs cause him to freeze if any little digits find their way in there.

"It doesn't get treated differently from the other kids," says Lydia Morrison, the main teacher in Classroom One. "It gets attention, and it gets hugs. And it gets pushed." Tanaka is amazed that, despite all the love Qrio has received, he hasn't broken yet.

As recently as a few years ago, Tanaka tells me, Sony had plans to market a humanoid robot, a logical progression from Aibo, its popular robotic dog. But despite Aibo's initial success, Sony found that people eventually got bored with its limited interactive capacity. To avoid this situation with Qrio, the company has taken it off the production schedule and put more money into longer-term research projects like Tanaka's preschool work. This allows Tanaka to focus on a subject that fascinates him: the kind of free-form movement that little kids spontaneously generate. He believes that if Qrio can learn to emulate this dance it might spark a dialog of movement between him and the preverbal youngsters.

After a few minutes, Tanaka switches the robot to preprogrammed dances. This time, Qrio generates a cheesy techno ditty that repeats, over and over, "Qrio, Qrio. We dance together." His movements are uncannily human, subtle enough to trigger in me a mild version of the "it's alive" shivers Movellan referred to. Qrio swivels and twists through a series of moves that I have deployed myself on the dance floor over the years. I'm slightly shocked by his authentic hip-thrusting gyrations. At the end of each dance he takes a bow and giggles.

Tanaka tells me that one point of his research is to chart the waxing and waning of the children's attention. By the end of the half hour, Qrio is pretty much doing the Mashed Potato in his own little orbit. Tanaka punches the keys to put the robot into shutdown mode. Qrio squats slowly and lowers his arms to catch himself, then lies down on the floor. Tyler picks up a blanket and gently covers the machine's supine body.

Larry Gallagher (takudo@hotmail.com) is a San ­Francisco-based writer.