Overview

Challenge

Upon arrival, I was matched up with two other finalists: Kevin Ong of Malmö Universitet in Sweden and Mélodie Jacob of the Pôle Supérieur de Design in France; together, we were Team Pirouette. During the 72-hour inclusive design competition, we were challenged to come up with a concept to teach rhythm to a population with one of four disabilities: blindness, deafness, mobile dexterity impairment, or dyscalculia.

Solution

Project keyHue teaches students with dyscalculia how to read and play music, using projection mapping to display color on a piano keyboard, and corresponding colors on digital sheet music. These visual guides, and especially the rhythm and repetition of the guides, will teach a student to play the correct keys for the right duration. The system will manage the student’s pace through each song, and gradually transition them to eventually read and play music without color guidance. At that point, the digital sheet music will look exactly like its paper version.

Project keyHue was awarded first place in the Student Design Challenge at Interaction 18.

Process

To kick off the competition, the team from Microsoft, led by Principal Design Strategist Margaret Price, conducted a workshop to share the principles of Microsoft's inclusive design practice. They include recognizing exclusion, learning from diversity, and solving for one to extend for many. Creating a persona spectrum accounts for those with permanent disabilities as well as those with temporary and situational disabilities, resulting in benefits to more people.

A persona spectrum demonstrates that designing for the permanent disability of having only one arm can be extended to those with a temporary disability of sustaining an arm injury, or a situational disability of carrying a baby in one arm.

The brief challenged us to create an interactive way to teach the concept of rhythm, while ensuring our design provided access and reduced friction for the widest number of people. We started by ideating broad concepts of rhythm, and categorizing them by scale of personal to universal.

Concepts of rhythm from personal (left) to universal (right).

We were then presented with four constraints to consider: blindness, deafness, mobile dexterity impairment, and dyscalculia. After conducting secondary research on all four disabilities, and brainstorming how each affects rhythm, our team decided to design for people with dyscalculia for two main reasons: (1) none of us had heard of dyscalculia before the challenge and wanted to learn more, and (2) we wanted to raise awareness about dyscalculia, after learning it is as prevalent as dyslexia, yet relatively unknown and an underfunded area of research.

Brainstorming how people with disabilities may experience rhythm differently.

DESK Research

Dyscalculia is a learning difference that makes it difficult to ascribe meaning to numbers. People with dyscalculia may have trouble with calculations and problem solving, but also with telling time, spatial awareness, or memorizing lists. Recommendations to help address these difficulties include: using fingers for counting, and color to help distinguish between different numbers and develop categorization skills, gradually transitioning away from color to black and white.

Primary Research

Focusing on dyscalculia early on allowed us to conduct primary research. We reached out to all of our networks using social media, and to the 1,200 conference attendees using Slack, to try to connect with people with first-hand experience with dyscalculia.

For @IxDASDC, my team is focusing on #dyscalculia, a learning disability that affects math learning, spatial awareness and telling time. If you have any experience with dyscalculia or know someone who does, we’d love to learn more. We only have 48 more hours! #inclusivedesign

Through our outreach, we were able to connect with a father of a 14-year-old girl, who hasn’t been diagnosed with dyscalculia, but exhibits many of its symptoms. Some insights gleaned from our Skype interview were:

She lives as independently as any other teenager in New York, despite having trouble with memory, telling time, and directions.

Lots of repetition and confirmation has helped her and her family daily

She was encouraged to learn music from a young age to counter her difficulties learning math. Now, she can read music and plays guitar and piano, and sings.

Conducting an interview with a parent whose daughter exhibits symptoms of dyscalculia.

Prototype

Referring back to our research was critical as we ideated. We also found value in using real content—in our case, the song “Mary Had A Little Lamb.” Not only was it a familiar and relatively simple song, but it also comprises quarter, half, and whole notes, allowing us to test and strengthen our concept.

Using projection mapping to display color on a piano keyboard and corresponding colors on digital sheet music, keyHue will guide a student to play the correct keys for the right duration.

Extending for many

Although designed for those with dyscalculia, our concept can benefit many others, including anyone learning a musical instrument for the first time, or someone who has only learned to play by ear and wants to go further.

Presentation and Judges’ Critique

We presented Project keyHue on stage on the last day of Interaction 18, and were thrilled to be awarded first place.

“Project keyHue tackled a challenge most people knew nothing about,” judge Haiyan Zhang, Director of Innovation at Microsoft Research, said. “They were able to bring us into the experience of the user, through their video and their research, and by talking to someone with dyscalculia and speaking to some of the emotional challenges they face. Their idea was very innovative, and I liked the combination of an on-screen app and the physicality of still relying on the piano, so a person can, over time, learn how to play a real piano and learn the muscle memory.”

“They had an idea to solve some of the challenges their research subject had, and you could see how it was extensible,” said judge Neil Churcher, Head of Design at Orange. “They also had a believability to it that came through in their presentation, which gave them a little edge.”

If we had more time, we would have liked to test our prototype with more users, especially those with dyscalculia, and musicians. We also want to research the technical feasibility of our concept further.

Overall, the Student Design Challenge was an incredible experience filled with learning and building meaningful connections. I was reminded of the value of being part of a great team, conducting primary research, and prototyping with real content.