ASU researchers add human ingenuity to automated security tool

May 3, 2019

The world’s top chess player isn’t a human or a computer, it’s a “centaur” — a hybrid chess-playing team composed of a human and a computer.

The Defense Advanced Research Projects Agency is looking to apply the same human-computer collaborative approach to cybersecurity through its Computers and Humans Exploring Software Security (CHESS) program. ASU Assistant Professors of computer science Yan Shoshitaishvili (left) and Ruoyu “Fish” Wang in the Laboratory of Security Engineering for Future Computing, or SEFCOM. Shoshitaishvili, Wang and a multi-university team earned an $11.7 million DARPA award to develop a human-computer collaborative approach to cybersecurity. Their approach is called Cognitive Human Enhancements for Cyber Reasoning Systems. Photo by Erika Gronek/ASUDownload Full Image

A team of researchers in the Ira A. Fulton Schools of Engineering at Arizona State University is working with collaborators at the University of California, Santa Barbara; the University of Iowa; North Carolina State University; and EURECOM to make their move in this space. The team’s project is called CHECRS, or Cognitive Human Enhancements for Cyber Reasoning Systems.

The $11.7 million award supports the multi-university CHECRS team’s efforts to create a human-assisted autonomous tool for finding and analyzing software vulnerabilities that also learns from and incorporates human strengths of intuition and ingenuity. The ASU team, led by Ruoyu “Fish” Wang, an assistant professor of computer science and engineering in the Fulton Schools, received $6.6 million of the award funding.

Wang also attributes their success to the Global Security Initiative at ASU, which offered immense help in getting the DARPA award.

“Seriously, we wouldn’t have got this award without GSI’s assistance," Wang says. "Their extensive knowledge and experience on working with government agencies are crucial to securing awards like this.”

During a time when reports of nearly weekly security and data breaches occur due to software vulnerabilities — which are a natural part of coding — the success of this DARPA research program lies in the teams’ ability to build systems that will find and mitigate these costly mistakes before they can be exploited for nefarious purposes.

As in a game of checkers, defenseless pieces are inevitable, but the CHECRS team is developing a way for humans and computers to work together to identify vulnerabilities before black-hat hackers or other bad actors have the opportunity to make a play.

“It’s a lot of responsibility,” says Yan Shoshitaishvili, an assistant professor of computer science and engineering and co-principal investigator on the project. “It’s a big undertaking that the government is making, and we have a lot of responsibility to make it a success. I have no doubt we’ll be successful.”

Autonomous tools can do the job, but they’re novice players

During the 2016 DARPA Cyber Grand Challenge in Las Vegas, a packed crowd watched seven computers on a stage just sitting there, blinking. It was an exciting day.

The computers were autonomously fighting a cyber war against each other. They were executing the results of years of research by teams of computer scientists who had created autonomous cyber reasoning systems that could analyze software systems, find vulnerabilities, create proofs of vulnerabilities and fix them automatically — all without human interaction.

Wang and Shoshitaishvili, then both graduate students at the University of California, Santa Barbara, were on one of the seven finalist teams, Shellphish. Captained by Shoshitaishvili, Shellphish earned third place in the competition, but the team also left with the seed of an idea.

“Since then we’ve been thinking about the concept of human-assisted cyber autonomous systems,” Shoshitaishvili says. “We had the realization that if you have both an autonomous system and a principled way to reinject human intuition, which these machines lack, you can create something better than the sum of its parts.”

Human qualities make for the best of both worlds

It’s important for a cyber reasoning system to be able to function autonomously — it demonstrates that machines can do all of the work, if necessary. Also, computing power is cheap, can scale easily and work constantly. However, Shoshitaishvili likens these autonomous tools to 1990s chess-playing computers — able to win sometimes, but not with the frequency and skill of a chess-playing human champion like Garry Kasparov.

Unlike chess — a game with a well-defined set of rules that can be efficiently mastered by machines — software programs are a lot more complex.

“While modern automated tools run on computers that calculate billions of times faster than a human brain, human security analysts still find the majority of software vulnerabilities,” Wang says. “This is because the knowledge and intuition that humans possess outweigh the speed of calculation when facing problems with extreme complexity, for example, finding software vulnerabilities.”

Since humans are available to help with security analysis, we might as well work together.

Automated tools already exist to help expert security researchers and white-hat hackers (those working for good) detect vulnerabilities. But these tools are only useful to an elite few.

The CHECRS team wants to create an autonomous tool that can be used by a wider variety of human assistants. Software developers, quality assurance specialists and other non-security experts have human intuition and ingenuity that can meaningfully aid the automated tool.

When humans of varying skills and expertise are at work, or when the machine needs help connecting dots using intuition, the automated tool can delegate tasks it’s not good at to the humans while it switches over to other tasks computers are optimized to perform.

Not only do humans help in the moment, the automated tool will incorporate what it learns from human contributions to continuously improve upon itself — both in how it interacts with its human partners and in its own ability to accomplish tasks.

In achieving this ability to work together and learn from one another, the CHECRS team will meet the first two of the five DARPA CHESS program goals: pulling in human assistants to the autonomous cyber reasoning system (with efforts led by Shoshitaishvili) and getting machines to understand software in ways humans do (led by Wang).

“As we observe instances of humans helping the machine, can we learn from that using machine learning or by observing and trying to recreate [human capabilities] algorithmically and reproduce it in the machine itself?” Shoshitaishvili asks. “Our expectation is yes.”

Working toward a more secure future

In additional steps of the DARPA CHESS program, other research teams will evaluate whether the human-computer teams are working effectively by competing against the system to detect vulnerabilities.

Professional security analysts will form teams, called control teams, for these weeklong competitions. At this point, it remains to be seen what types of vulnerabilities the human-computer teams will be good at finding and fixing. However, CHECRS won’t feel lonely: Some technologies underlying CHECRS, such as the binary analysis platform called angr, could also be employed by control teams.

If the team is successful and vulnerabilities can be automatically detected and people can be alerted to them in useful ways, or if they can even be fixed automatically, cybersecurity breaches and politically motivated hacks could become a thing of the past. Security troubles won’t be what they are today.

“Understanding programs and finding vulnerabilities has always been an art that is only mastered by a small group of elites. But no one wants security to be an art,” Wang says. “In the CHESS program, the CHECRS team regards vulnerability discovery as a scientific problem — which it should have been — and is a steady step toward making software and our world much more secure.”

Shoshitaishvili adds, “We could look at a world where entire classes of vulnerabilities are wiped out because all of the software was analyzed by automated systems,” noting it would require advances beyond the scope of the DARPA CHESS program.

The ability to create a tool that is scalable — to analyze more and more varieties and amounts of software beyond the scope of the DARPA CHESS program — is a huge part needed to realize such a future.

“If the system could use human assistance as needed to always be functioning, always be pushing toward a goal and never get stuck due to its limitations,” Shoshitaishvili says, “it’s hard to overstate how useful that will be.”

U.S. and European researchers collaborate on CHECRS project

The Arizona State University CHECRS team includes six top researchers in cybersecurity-related fields from the School of Computing, Informatics, and Decision Systems Engineering.

Shoshitaishvili, an assistant professor of computer science, will focus on work to integrate human assistance into an autonomous cyber reasoning system in a controlled and principled way for best results.

Wang, an assistant professor of computer science, will focus on improving the performance of state-of-the-art program analysis techniques and equipping them with autonomous but human-like capabilities of solving security problems.

Tiffany Bao, an assistant professor of computer science and engineering, is an expert in game theory in cyber reasoning systems and how cyber reasoning systems plan their actions. Her work won the National Security Agency’s cybersecurity paper competition last year.

Chitta Baral, a professor of computer science and engineering, is helping bridge Shoshitaishvili and Wang’s work through machine learning research that looks at how knowledge passed back and forth between human and machine is represented.

Adam Doupé, an assistant professor of computer science and engineering, helps to expand the tool’s abilities beyond web browser vulnerability and security to webpages and mobile applications to push their capabilities to a wider scope. Doupé is also associate director of the Center for Cybersecurity and Digital Forensics.

Stephanie Forrest, a professor of computer science and engineering who is also with the ASU Biodesign Institute, conducts research that explores biological features of software and will help the CHECRS team give software the ability to mutate like a living organism to develop an immunity to vulnerabilities. This will help automatically find and fix vulnerabilities in software before release to the public.

For the first part of the DARPA CHESS program project, Shoshitaishvili is working with North Carolina State University and EURECOM, a graduate research institute in France.

Alexandros Kapravelos, an assistant professor of computer science at NCSU, brings expertise in web and browser security. This will help the team achieve their goal of analyzing real software, in particular extremely complex web browsers like Google Chrome.

At EURECOM, Yanick Fratantonio, an assistant professor of digital security, and Davide Balzarotti, a professor of digital security, will provide insight into how expert vs. nonexpert humans approach software and interact with software interfaces.

Wang is collaborating with the University of California, Santa Barbara and the University of Iowa on the second part of the DAPRA CHESS program project.

Antonio Bianchi, an assistant professor of computer science at the University of Iowa, will assist Wang through his expertise in mobile security vulnerabilities and analysis. This will allow the automated tool to tackle issues in complex software, such as fingerprint sensor applications and voice assistants.

Wang will collaborate with program analysis researchers at his alma mater, the University of California, Santa Barbara, where Shoshitaishvili and Doupé also conducted their graduate research and where the Shellphish team emerged to earn third place at the DARPA Cyber Grand Challenge.

Christopher Kruegel and Giovanni Vigna, both professors of computer science at the University of California, Santa Barbara, are leading contributors to cybersecurity research in the past decade and bring valuable experience and expertise into program analysis to complement the ASU team.

This research was developed with funding from the Defense Advanced Research Projects Agency (DARPA). The views, opinions and/or findings expressed are those of the authors and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. government.

Monique Clement

Next Story

Editor’s note: This is part of a series of profiles for spring 2019 commencement.Often students arrive at college with set plans on what path they want to take over the next four years. Abigail Johnson was one of those students, but as she prepares to graduate from The College of Liberal Arts and Sciences this spring, she advises others to not be afraid to change up their plans.“Follow what yo...

Geography student discovers passion and community at ASU

Often students arrive at college with set plans on what path they want to take over the next four years. Abigail Johnson was one of those students, but as she prepares to graduate from The College of Liberal Arts and Sciences this spring, she advises others to not be afraid to change up their plans.Abigail Johnson will graduate with her bachelor's degree in geography from The College of Liberal Arts and Sciences.Download Full Image

“Follow what you think is right. There was a time that I really thought one major (journalism) was right for me and since I wanted it so bad and did an internship in it, I thought I would disappoint my family if I switched,” explained Johnson, who is graduating with her bachelor’s degree in geography from the School of Geographical Sciences and Urban Planning. “But when something doesn’t feel right, I think you should change your mind and really think within yourself and reflect.”

As a first-generation student, Johnson said she benefited from a number of resources in high school and while at ASU. One of those resources was AVID (Advancement Via Individual Determination).

“(AVID) helps students who don’t have the tools for college, many are first-generation students. I didn’t know about the SAT, ACT and they tell you about those things and advise you to get involved,” she said.

During Johnson’s junior year at ASU, she was invited to speak to high schoolers at an AVID event.

“I was so proud to be asked to speak at the conference. I loved it, just because I know what it’s like, I’ve been there in that exact same chair. It was really cool to be able to be the one to help them.”

Helping others was a consistent theme during Johnson’s time at ASU. She worked with elementary school students through America Reads during her freshman and sophomore years, frequently volunteered for events through American Indian Student Support Services and got involved with community gardening at the Polytechnic campus, which then led to work at the nonprofit organization Native Health. As an enrolled member of the Navajo Nation, Johnson said this work, in particular, was meaningful.

“To be around my people and to teach them about what I’ve learned at ASU about plants, it’s very fun.”

Johnson answered some questions about her time at ASU and shared what she has planned next.

Question: What’s your Sun Devil story?

Answer: I’m from El Mirage, Arizona, and grew up going back and forth to the Navajo Nation. I’m a first-generation student so my whole life I’ve been excited to go to university. I worked hard every single day in high school; I did community service and became really passionate about those kinds of things. Eventually, I made it here and really enjoyed it.

Q: What was your “aha” moment, when you realized you wanted to study the field you majored in?

A: I took GPH 111 with Erin Saffell. I walked into class and saw how passionate she was about the subject and I loved being in her class. I was taking journalism classes and realized I was looking more forward to going to that class than the journalism classes. I had office hours with her and told her I really loved physical geography and she told me to think about majoring in it and that’s when I decided to switch over.

Q:What’s something you learned while at ASU — in the classroom or otherwise — that surprised you, that changed your perspective?

A:When I moved here I realized how being a first-generation student, I was always the one to lead my family. When I came here I met a lot of people and didn’t feel like I had so much weight on my shoulders. I think something surprising that I learned was that you really need people, and when you have the right people you can create better results than you could by yourself.

Q:Which professor taught you the most important lesson while at ASU?

A:Professor Saffell. She taught me how to study and how each class is different. It can be hard as a freshman to build up the confidence to walk into office hours. I got over that barrier with her, she was very calm and I felt very comfortable in her environment. I’d ask her questions and from then on I wasn’t afraid to ask questions to other professors or TAs.

Q: What’s the best piece of advice you’d give to those still in school?

A: Students come here for their major but this is an opportunity for you to know yourself. Do something out of your major, like a club. For me, I love geography and maps but I had an interest in plants so I started doing those classes and I met my best friends there. Do something out of your element and nourish that. Create a hobby for yourself.

Q:What was your favorite spot on campus, whether for studying, meeting friends or just thinking about life?

A:My favorite spot on campus is the American Indian Students Services room, it’s a very safe space for me. It’s in Discovery Hall and on the walk there, there are beautiful trees leading the way.

Q: What are your plans after graduation?

A: My husband is in medical school and starting his rotations so we’ll be traveling around the United States. I’m planning to intern at some urban planning firms or work as a GIS analyst.

Q:What would you say to someone considering ASU?

A: Do it, it’s a very great school. There are so many opportunities here, including employment opportunities. It’s very good for someone getting out of high school, especially when they don’t know anything because there’s a good community once you’re here.