ASU researcher creates system to control robots with the brain

ASU researcher interfaces with up to four small robots, some of which fly.

July 8, 2016

Using wireless interface, operators control multiple drones by thinking of various tasks

Editor's note: This story is being highlighted in ASU Now's year in review. To read more top stories from 2016, click here.

A researcher at Arizona State University has discovered how to control multiple robotic drones using the human brain.

A controller wears a skull cap outfitted with 128 electrodes wired to a computer. The device records electrical brain activity. If the controller moves a hand or thinks of something, certain areas light up.

“I can see that activity from outside,” said Panagiotis Artemiadis (pictured above), director of the Human-Oriented Robotics and Control Lab and an assistant professor of mechanical and aerospace engineering in the School for Engineering of Matter, Transport and Energy in the Ira A. Fulton Schools of Engineering. “Our goal is to decode that activity to control variables for the robots.”

If the user is thinking about decreasing cohesion between the drones — spreading them out, in other words — “we know what part of the brain controls that thought,” Artemiadis said.

A wireless system sends the thought to the robots. “We have a motion-capture system that knows where the quads are, and we change their distance, and that’s it,” he said.

Up to four small robots, some of which fly, can be controlled with brain interfaces. Joysticks don’t work, because they can only control one craft at a time.

“You can’t do something collectively” with a joystick, Artemiadis said. “If you want to swarm around an area and guard that area, you cannot do that.”

To make them move, the controller watches on a monitor and thinks and pictures the drones performing various tasks.

Artemiadis has been working on the brain-to-machine interface since he earned his doctorate in 2009, specifically neural interfaces with robot hands and arms.

“During the last two to three decades there has been a lot of research on single brain/machine interface, where you control a single machine,” he said.

A few years ago, he had the idea to go to a lot of machines. It’s part of a trend in robotics and space exploration: Instead of building one giant expensive machine or plane or spacecraft, researchers build a lot of little cheap ones.

He already knew what area of the brain controlled what motions. One discovery jumped out at him.

“I was surprised the brain cares about swarms and collective behaviors,” he said.

“What I didn’t know — or hypothesized — is that the brain cares about things we are not doing ourselves,” he added. “We don’t have a swarm we control. We have hands and limbs and all that stuff, but we don’t control swarms.”

In other words, our brains are not used to all of our fingers and toes running off on their own and then returning.

“I was surprised the brain cares about that, and that the brain can adapt,” he said.

He worked with Air Force pilots on this; the two-year project was funded by the Defense Advanced Research Projects Agency of the U.S. Department of Defense and the Air Force. The pilots were skeptical. Their main objection was what would happen if they thought of something else while controlling the drones.

Artemiadis said controllers have to stay focused. If it’s close to lunch and all you can think about it is pizza, it doesn’t work. Fatigue and stress also play a part. Artemiadis said he can tell when subjects are tired or need a break.

“We tell the subject to think of two things,” he said. “Focus on breathing, or we tell them to imagine closing their left hand into a fist.”

Each subject is different. The system has to be calibrated to individual controllers, and it has to be done every day, because brain signals change from day to day.

The next step in Artemiadis’ research is multiple people controlling multiple robots. He plans to move to a much larger experimental space to refine the proof of concept. In the future, he sees drone swarms performing complex operations, such as search-and-rescue missions.

Scott Seckel

Next Story

Jewell Parker Rhodes writes children’s novels about tough subjects. The best-selling author had tackled slavery, the BP oil spill and Hurricane Katrina, but there was one challenge she hadn’t taken on: the terror attacks of Sept. 11, 2001.It had been more than a decade since the tragedy when Rhodes got an idea for a 9/11 story that she said “stayed in my soul.” Jewell Parker RhodesHer ini...

Jewell Parker Rhodes writes children’s novels about tough subjects. The best-selling author had tackled slavery, the BP oil spill and Hurricane Katrina, but there was one challenge she hadn’t taken on: the terror attacks of Sept. 11, 2001.

It had been more than a decade since the tragedy when Rhodes got an idea for a 9/11 story that she said “stayed in my soul.”

Jewell Parker Rhodes

Her initial reaction, and early advice from family and friends, had been to stay away. The topic seemed too raw, intense and emotional for young readers. But on a long flight, Rhodes said she began to feel a connection to the people on the hijacked planes and developed the story that would become her latest book, “Towers Falling.”

Timed for the 15th anniversary of the attacks, the story for young readers takes a fictional fifth-grade class through lessons about one of the defining moments of modern history. Rhodes said she hopes “Towers Falling” can be a tool for educators and parents to guide discussions with children.

“Students are the citizens of tomorrow and need to be taught how 9/11 affected our world,” said Rhodes, artistic director of Arizona State University's Virginia G. Piper Center for Creative Writing and writing professor in the Department of English in the College of Liberal Arts and Sciences.

“We’re now seeing the impacts of terrorism and how it has become even more widespread since 9/11. Do we really want to wait 30 years from now to teach the people who are going to have to live with this?”

To get the book done, Rhodes had to sensitively introduce young readers to an attack that killed more than 3,000 people, including hundreds of police and firefighters, and triggered a massive counter-terror response from the U.S. government. She also had to craft a story that teachers could teach, something that conveyed both shocking devastation and the ultimate triumph of American resilience and ideals. She wanted to make the grim moment into a story that could inspire young people to become good citizens.

She said it came together when she was “cocooned on a 14-hour plane flight.”

“It was a midnight flight and everything was dark save for a reading light,” Rhodes said. “Being in that space and spiritually connecting with the people on those planes brought it into focus for me.”

A possible approach as well as the title popped inside her head. For Rhodes it was “a sign that I should try and write this book.”

Rhodes wanted input from fellow teachers, so she consulted the principal and other staffers at the Brooklyn New School, PS 146, who witnessed the two planes flying into the World Trade Center through their school windows. The educators said they were still traumatized by the crashes, which left the school coated in debris and ashes, and the sudden realization that family members and friends worked in the twin towers. Even years later, many still couldn’t discuss it with their students, some of whom asked, “What happened?” and “Where are those buildings?”

Rhodes also discovered through classroom visits around the country that lessons on 9/11 varied widely and that many teachers had avoided the topic altogether.

Part of the trepidation had to do with age: At what point is it appropriate for young people to learn about a troubled chapter of recent history?

“It can be taught at the elementary level, but it has to be done in an age appropriate way that doesn’t focus on fear but teaches about bravery, citizenship, resilience and the human spirit,” she said.

Vickery said teachers aren’t eager to venture into such territory because they want to “preserve and protect the innocence of the child.”

In Arizona, current state social studies standards do not call specifically for lessons on 9/11 or contemporary terrorism. That doesn’t mean, however, it’s not being taught, said Kenneth De Masi, past president of the Arizona Council for the Social Studies.

“Many teachers are challenged in placing Sept. 11 in the context of world history,” said De Masi, who has taught social studies for the last four decades. “It is simply a question of when to do it, how to do it, for how long and who should do it? I know some people in our society that I would not want to be teaching my grandkids 9/11 or terrorism in general.”

That doesn’t appear to be the case with Rhodes, who has received widespread support for the project.

“Jewell is a very magical person, and she has the presence of an angel,” said Sid Reischer, a fifth-grade teacher at Castleton Elementary in upstate New York. Reisher received advance copies of “Towers Falling” from publisher Little, Brown Books as part of his yearlong study of 9/11. Reisher read it to his students in April while Rhodes participated through Skype.

The reading became “an avenue for students to have a conversation with their parents about 9/11 as part of their homework,” Reischer said. He said other student outcomes included a “feeling of connection to the country as a whole, a deep appreciation for first responders and what it means to be an American.”

Reischer said “Towers Falling” affected parents, many of whom had personal connections with people who died that day.

“For the kids to see the emotional impact it had on the parents was very valuable and an important piece,” Reischer said. “It showed that history is alive and well and is about people. We had very rich conversations about the subject for the next few days.”

Reischer’s study will culminate with a visit from Rhodes. She will travel to the 9/11 Memorial to meet with about 75 Castleton students on Sept. 9, to memorialize the attacks.

“The idea that Jewell will come and meet us there makes this a once-in-a-lifetime experience for these kids,” Reischer said. “When does an author do that? It’s amazing if this all comes off.”