Session 106WWDC 2016

When Ajit Narayanan began working on technologies for children with disabilities in India in 2008, it was the beginning of an incredible journey down a rabbit hole — one that took him and his team on a wild ride across the worlds of electrical engineering, autism, cross-cultural communication and the mathematics of language. His journey led him to build Avaz, a wheelchair-mounted tablet to help people with disabilities communicate, then even more ambitiously, to invent FreeSpeech, an entirely new visual language that could break down language barriers altogether. Ajit will talk about the series of events that led to the creation of the Avaz and FreeSpeech apps, their impact in India and around the world (earning him the National Award for Empowerment of People with Disabilities from the President of India) and his ideas on the transformative role that apps can play in education.

[ Music ]

Hi. Good morning.

Good afternoon.

My name is Ajit, and I used to live and work not very farfrom here on the East Bay.

I was an electrical engineer in the heart of the Silicon Valley.

And in 2007, I did something rather unusual.

I moved back to India from California because I wantedto be an entrepreneur.

I'm an inventor at heart, so I dabbled in a numberof different fields, a number of different devices.

But eventually, I chose an area to workon that was rather unconventional.

I decided I wanted to work with kids with disabilities.

I wanted to make assistive technology for childrenwho had difficulties with various kinds of disabilities.

Now the way I got into this particular fieldwhen I went back to India in 2008,there was a friend of mine.

And she runs a school in the city of Chennai in the southof India, and it's called Vidya Sagar.

Now, Vidya Sagar is a special school.

It's a school for kids with various kinds of disabilitiescerebral palsy, autism.

And when my friend found out that I was lookingfor instant problems to solve, she suggestedthat I visit the school and interact with the kids and seeif there was anything that struck me.

So I went to Vidya Sagar and I sat downand I met some of the kids.

Now that visit was an eye-opener for me.

You know, I saw that these kids were intelligent,they were smart, they were eager to learn,they were enthusiastic, they were social.

But the reason that they weren'tin a regular school was primarily one reason.

And that was that they had difficulty with communication.

Now, if you look at kids with cerebral palsy, the main reasonthat they can't communicate is because the part of the brainthat process muscle movements is impaired,so they have difficulty controlling allof the different muscles that are used to generate speech.

But with autism, it's actually a very different story.

The first kid that I met with autism was actuallyat Vidya Sagar during that visit.

His name was Santhosh.

Now I was sitting across the table from Santhosh,and Santhosh had a lot of difficulty with a few thingsthat we take for granted.

For example, he found it really hard to read my body language.

He would find it really hard to see my face and to makeout what kind of emotion I was feeling at that time.

The other thing that Santhosh had a lotof difficulty with was language.

Because language is fundamentally very symbolic.

You know, it's very abstract.

It's not something that we realize every day,but it's really one of the most abstract things that we use.

We have all of these different sound sequences,all of these different words, and these words are somethingthat we assign meaning to almost arbitrarily.

For example, if I said the word "dog",the word "dog" doesn't have anything to dowith the animal the way that I pronounce the word,the way that I write the word.

You would never be able to figureout that it actually refers to this particular animal.

It's very much abstract sound through which we've assigned,you know, a convention that we agree on.

And that's why someone with autism has a lotof difficulty with this.

Our brains have the ability to, you know, encode and decode tensof thousands of these arbitrary sound sequences.

But kids with autism find it a little too fuzzyfor their comfort.

So my friend who runs this school, Vidya Sagar,she wanted me to look at this particular problemand she wanted me to see if I could come up with an inventionthat could give a voice to these children.

I thought that was a fascinating problem to work on justfrom a technology perspective.

I also thought it was a huge opportunity in this casebecause there were kids all over India, and really allover the world, who needed something like this.

My starting point to provide these kindsof assistive technologies was somethingthat had actually been discovered in the 70sand the 80s and was being used quite effectively.

What people working with kids with autism had discovered wasthat many of them have extremely strong visual intelligence.

So they have a lot of difficulty with the abstractnessof language, but they feel very comfortablewith the concreteness of pictures.

They're able to remember pictures.

They're able to recall pictures much easier than they're ableto do with sound or with alphabets.

Even in India, it wasn't very uncommon to see kidswith books like these.

Picture books, picture charts, picture bracelets sometimesthat they would wear around their wrists.

And if they wanted to ask for a specific object, if they wantedto request for water for example, they would pointto a picture of water in their book and that would communicateto other people that they wanted water.

At that time in [inaudible] America, in 2007 and 2008,this concept was being used quite effectively as partof these devices that were being built.

And this is an example of one of these devices.

This is a fairly you know, these devices were pretty big.

They were pretty bulky.

They ran a special software on them.

They were often custom-built,and they were paid for by insurance.

So they were pretty expensive.

This device, for example, would have cost anywherebetween $5,000 to $10,000.

And that price point is completely out of rangefor even the most affluent of parents in India.

But what was happening with that in 2007,for example, the iPhone came out.

And that was the beginning of the smartphone revolution.

Right? So it was possible at that time, 2008,to actually buy electronic components off the shelfwhich were being used in smartphones,in these touchscreen devices all over the world.

And to assemble them together to create a touchscreen tabletfor a fraction of the price justfor a few hundred dollars instead of havingto create these things from the ground-up.

So that became my strategy going into this particular area.

I thought I would create a special softwarethat they could use on top of a device that we would buildfrom these smartphone components,and we would then use that to give these children a wayto communicate.

So I took this strategy and then I plunged into the fieldof creating assistive technology.

And I quickly came up with a very interesting talent.

You know, those of you who've worked on solving hard problemsfor a specific class of people, you know that the processof coming up with these solutions is actuallypretty standard.

You go to the field, you talk to people, and you listen to them,you see what they have to say.

You ideate to come up with these prototypesand you build these things that you taketo the people whose problems you want to solve.

And then you get their feedback, and then you iterate a numberof different times till you have somethingthat works beautifully.

You know, that process is very standardized.

In my particular case though, I had a very peculiar problem.

Because the very reason that we were building this device wasbecause these children couldn't communicate.

So it was a chicken-and-egg problem.

You know, we were very happy to goand ask them what they thoughtabout the prototypes we were creating.

But what would actually happen isthat the kids would use the prototype for maybe a minute,two minutes, and then they would walk away.

And we had no idea you know, were they tired?

Were they bored?

You know, did they not like it?

Did they think it was completely worthless?

Were we solving the wrong problem?

So that was a frustration that, you know,we lived with for several months.

You know, every developerI'm sure every one of you here has a happy-customer storythat's very close to your hearts and that you really love.

You know, in my particular case, you know, there was this boy.

His name was Rohit.

Rohit was one of the people that we were tryingout every single one of our prototypes with.

And one day, he came into the roomwhere we were trying out the prototype.

And I could see you know, on that day,I could see the determination in his face.

He was going to communicate with me.

He was completely he had that look of grim determinationas he walked into the room.

And, you know, I gave him the newest prototype.

And I sat there, waiting with anticipation.

And Rohit actually took the device that I built.

Very painstakingly, over several minutes,he tapped on various buttons.

And then the device said, "This sucks."

[ Laugther ]

[ Applause ]

It's not exactly the kind of testimonial that is putup on your website, but it was music to my ears.

Believe me.

Because from that point onwards, the going was much faster.

You know, we could convince himto tell us exactly what he liked,exactly what he didn't like.

And within a few months, we'd come up with a devicethat met Rohit's expectations.

And that was the device that we launched to the market.

We called it Avaz.

Avaz is the Hindi word for "voice".

It also means similar things in many other languages.

Let me show you the device that we actually built.

So this was the first prototype that was ready for release,the first product that we built off Avaz.

Along with its wheelchair mount, it weighed about 8 pounds,so it was pretty heavy.

It had about an hour of battery life.

You know, there was a really embarrassing thing that happenedon the day that we were supposed to launch this.

We kept it on a desk, and a kid accidentally pushed itdown with his hand.

And it fell down on the floor and it broke oneof the tiles on the floor.

[ Laugther ]

That was how heavy it was.

You know, as a concession to safety, among other things,we replaced the case with plastic.

So that's the final version of Avaz that we put out.

This device costs about $600 30,000 Indian rupees.

And at that price, it was about 1/10 of the priceof similar dedicated devices that werein the market at that time.

And it was a breakthrough in India.

It was the very first time that anyone had created a devicefor speech-assistive technology in India.

And almost immediately after we had it out there,we started getting orders from all over the country.

You know, every special school in India wrote to usand they wanted to know how they could get their hands on Avazand how they could use it.

But Avaz was still a very niche product.

I mean, it's meant for the very specific niche of peoplethat have these particular disabilities.

So we had this you know, making dedicated hardwarein these small volumes is always a challenge.

So even though the response was fantastic,even though we got a lot of interest,and even though this was really helping the kidsthat we were intending this to help, you know,the more devices we were selling,the more money we were losing just because we were spendingso much money in creating these devices in the first place.

You know, we were struggling to figureout how we could make the business sustainable.

And then in 2010, the big news happened.

Steve Jobs announced the iPad.

And that was the start of the tablet revolution.

But it also was a revolution in the fortunes of my startupbecause we saw this as a way of getting ourselvesout of this hardware corner that we painted ourselves intoand being able to make these devices reach people,making this application reach people without havingto make the devices to do that.

So, in 2010, my teamand I started learning how to code for iOS.

And in 2011, we launched the first version of the Avaz App.

Now the core functionalityof Avaz actually remains the same today, you know,compared to the first versionthat we launched way back in 2011.

And let me show you a quick demo of how it works.

So this is Avaz.

And you can see that it's essentially a bunchof different pictures.

So there's pictures in here, and there are pictures in eachof these different categories.

So there are literally thousands of pictures in the app.

And a parent, or a teacher, or a therapist could go inand add words from the child's life with photographsfrom the child's life as well.

And the way that Avaz works for a childwith a disability is very simple.

A child would be able to recall and remember these pictures.

And every time they tap the picture.

IIt will speak that out.

And they would be able to sequence pictures together.

FeelThis way, and they could go into these different categories.

Describe feelings.

And they could pick different pictures out.

ExcitedTo be able to create sentences.

I feel excited.

And then the app would speak that out.

So that's Avaz essentially, a bunch of different pictureswith a picture vocabulary that helps children communicate.

So Avaz was literally an artificial voicefor kids with disabilities.

You know, when we put Avaz out in the field in India,it was immediately a game-changerin many different ways.

For this kid, Rohit, who had worked with usto build Avaz, he grew to love Avaz.

You know, he grew to be able to customize and use itso effectively that that same year that we launched the app,he went on to use Avaz to write his exams and to finish schooland to go on to college.

You know, the media all over India picked up the storyof this kid who'd finished his examswith an assistive-technology device.

And not just this kid.

You know, they started picking up stories from allover the country about who kids who had used Avaz to do things.

And even to this day, I think Avaz has shaped the narrativearound these disabilities in Indian media.

People started concentrating lesson what these children could not do and more on the abilitiesof these children and what they could do.

So that's one of the things that Avaz has accomplished in India.

My personal favorite[ Applause ]

My personal favorite anecdoteabout Avaz users is actually something that happened backin the school where this all started,in the school of Vidya Sagar.

You know, the children at Vidya Sagar, every year they putup this play for the parentsand for the well-wishers of the school.

And it just so happens that, you know, every year,it was almost a convention that the lead roles in this play,the most important roles, would go to the kidswho could speak I mean, the verbal children.

And the presenceof the nonverbal children was very subdued in this event.

The year we put out Avaz,the principal made this incredibly bold decision.

He said, "This year, all of the parts are going to go to kidswho can't speak, and they're goingto deliver their dialogues with Avaz."

You know, I was sitting in the front row on the daythat that play was going on.

You know, I was extensively a special guest.

But really, the principal expected me to jumpup on the stage and fix things if something didn't work.

[ Laughter ]

But I was sitting in the front row right there.

And the atmosphere was incredible.

It was electric.

You know, I could see how much fun these kids were havingdelivering their dialogues and doing all of their, you know,all of their actions and everything.

And I turn back and I looked at the parents of these children,and they had tears in their eyes.

You know, this was the very first timethat these kids were able to participatein this quintessential school experienceof being a part of a school play.

And at that moment, I think it sank into many of those parentsabout how transformational this device would bein the lives of these kids.

[ Applause ]

Another really important moment in the historyof Avaz happened a little later, a few months later.

We got a call you know, our customer support personshe got a call from a parent in central India.

You know, so this parent called her up.

And this call was actually it played out very differentlyfrom most of the calls that we get,you know, for customer support.

This lady had bought Avaz for her son.

And, you know, she wantedto know a bunch of different things.

You know, she wanted to know how she could assess her sonfor whether this device was suitable for the son,and she wanted to know how to set it up for her son.

She wanted to know how to integrate it into his lessons.

She wanted to know how to set up curriculum around it.

She wanted to know how she could track progress,you know, over the months.

She essentially wanted to know howto implement the app for her son.

Now this is actually a pretty atypical question that we get.

Because in most installations of Avaz that we've seen,a speech therapist would be the person that would workwith the kid in order to implement the app for them.

So the speech therapist would meetwith the kid maybe once a week, they would do assessments,they would do all of these different things.

And so we asked this lady,"Why don't you ask your speech therapist these questions?

You know, that person is probably much more qualifiedto give you an answer to some of these things."

On that day, I foundout something very interesting about India.

This lady's son didn't have a speech therapist.

In fact, the closest speech therapistto that lady was 300 miles away.

If she had to find a speech therapist for her son,she'd have to get on a train, travel for six hours,and go to a speech therapist and come back home.

In fact, I'll tell you this.

In the entire country of India,the country of 1.3 billion people,there are 1,800 speech therapists.

You could take all of the speech therapists in Indiaand you could put them in this room,and you could fill this room three times over with the numberof speech therapists that are in India.

So this was a mind-blowing moment for us.

This was really an eye-opening moment for us.

And what we had to do eventually was that we had to take the app,Avaz, and we had to build this complete new app inside it.

So it's almost like a sub-app within Avaz.

And what that sub-app didit was a pretty new idea at that time.

What that sub-app did was that it trained a parentto be a therapist for their child.

That was the only way that we could get over this problem.

You know, as a startup,we couldn't train the 5 million therapists that India needs.

But this was actually more transformational in my opinionthan even Avaz itself because we were not just giving a devicefor the child, we were actually building up the resource basefor these children to get high-quality intervention,and from the people that these children mattered the mostto to their parents.

So this happened, you know, in 2011 shortlyafter we brought Avaz out.

And shortly after that, I won an awardfrom the President of India.

I had less facial hair back then.

[ Laughter ]

But even more rewarding than thisyou know, after five years of lobbyingthis year, at long last, one of the big states in India decidedto buy iPads, install Avaz on them, and to give them awayto every school that they supported in the state.

[ Applause ]

This is the very first timethat the Indian government has invested in assistive technologyfor kids with speech disabilities,the very first time that the Indian government has investedin this stuff.

And in fact, the Apple guy that's handling this order,the guy at Apple that was coordinating the deliveryof these devices, he told methat this was the largest-ever government orderof iPads in India.

And it'sfor an assistive-technology application.

How cool is that?

For an assistive-technology application.

[ Applause ]

Today, there are about 30,000 kidsaround the world whose lives have been touched by Avaz.

Avaz is an artificial voicefor 30,000 children around the world.

You know, at these scales 30,000 userswe could see some very interesting patternsin the way that Avaz was used.

And let me talk to you a little bitabout how Avaz was being used around the world.

What Avaz did very successfully wasto help kids have this alternate access to words.

So instead of having to speak them out, instead of havingto remember how they sounded,they could use pictures to communicate.

So, for example, if a kid wanted to say that they were hungryand they wanted something to eat, they could go into the appand find the word "eat" and tap on it.

If they wanted to say that they wanted to go to the bathroom,for example, they could go into the appand find the word "toilet".

If they were eating somethingand they found the food incredibly gross, they could gointo the app and they could find the word "yucky"and they could say that.

The problem happened when kids had to communicatewith more than one word.

So, for example, if a kid had to say, "I want water,"it was quite common that they would pick the word"water" first and then they would pick the word "want".

And they would do it right sometimes.

They would do it wrong sometimes.

But they were doing it wrong as many timesas they were doing it right.

Or, for example, if they wanted to say, "The boy is swimming,"they would just pick the word for "swim"and they would pick the word for "boy".

You know, they weren't really creating these sentences.

You see, Avaz was very successful at breakingdown this symbolism of words and putting the concretenessof pictures into that.

But this is actually a different level of symbolism at play here.

This is a symbolism of grammar.

And grammar is something that's so abstract,and there are the rules of grammar.

And those rules of grammar are so abstractthat they're even very hard for us as native speakersto explain and to understand.

You know? For example, if you take this sentence,"I want to eat," what's the meaningof the word "to" in the sentence?

You know, if you asked me, I wouldn't have an idea.

It's just a word that's therebecause it's ungrammatical if it wasn't there.

[ Laughter ]

If you take these sentences, "The window broke," and,"He broke the window," it's interesting, you know,these two sentences are both grammatically correct.

But the object that's breaking is actually the same.

Right? The window is the object that's breaking.

Why is it that it appears on one side of "broke" on one sentenceand on the other side of "broke" in the other sentence?

Or if you take this sentence, "I eat," the past tenseof this is obviously, "I ate."

But if you negate this sentence, if you say, "I didn't eat,"why is it wrong to say, "I didn't ate"?

Right? So there are these meta-rules at play here.

There are these things that go beyond the rulesof which word presents which meaning.

There are these meta-rules of grammar.

And this was actually very frustrating to us.

We went out behind the pursuit Avaz with the intentionof giving communication, full communication,equal communication to kids with disabilities.

And we knew that our mission was actually being completelyunderfulfilled if we were only giving them wordswithout giving them the ability to also create sentencesand to put these words together.

The problem was that, unlike in the case of Avaz,there was actually very little research,very little theory behind how you can teach childrenwith autism grammar, how you can teach themto put words together.

We trawled through all of the scientific journals.

We went to all of the scientific conferences.

But very little insights that people had got.

The insight that kind of triggered a lot of ideasin our head quite consistently came from the work of somebodywho lived 2,500 years ago.

He was an Indian philosopher.

I don't know if he exactly looks like this.

[ Laughter ]

The Indian government put out a stamp about him in 2004.

He has the unfortunate distinction of sharing his namewith a kind of sandwich.

It's called Panini.

[ Laughter ]

But Panini lived in it didn't mean "sandwich" back then.

[ Laugther ]

Panini lived around 500 BC.

And Panini did something which was phenomenal,which is unprecedented even today.

He took a language, the language of Sanskrit,and he created a concise, complete, consistent setof rules 4,000 rules that described the entire language.

OK, so he had this set of 4,000 rules, and those rules areof course still relevant today.

But with these 4,000 rules, if you apply these 4,000 rules,any sentence that you created would be guaranteedto be grammatically-correct Sanskrit.

And conversely, any grammatically-correct sentencein Sanskrit, any sentence, any book, any verbal reference,any grammatically-correct Sanskrit could be explainedwith these 4,000 rules.

It was almost like he had written a computer programto generate Sanskrit.

So that codification of a language and the way that he didthat was absolutely beautiful.

It was phenomenal.

But that codification of Sanskrit gave us the first cluethat maybe this is an approach that we can take.

Maybe we can codify the rules of grammar and put itinto a computer in some sense and see if we can usethat to give kids with autism an access to grammar.

But even though we had that insight,the other missing component was really the user experience.

How could we get kids to interact with grammar?

And the answer to that really appropriately came when I wasin a school for kids with autism and I happened to interactwith I happened to be present, you know,where there was this little girl, and there was her mother,and there were her teachers.

And this girl, you know, she had autismand she was almost completely nonverbal.

But once in a while, she could speak.

And that's what happened that day.

She jumped up and she said the word "eat".

And there was no context to this.

She just finished eating her lunch,so her teacher thought perhaps she wantedto talk about her lunch.

So they tried to get her to communicateabout her lunch on Avaz.

But it was very clear that, you know,that was not what she was trying to say.

So we were all standing there and trying to figureout what she meant by the word "eat".

And then her mother started asking her questions.

So her mother said, "OK, eat what?

Eat lunch?

Eat breakfast?

Eat snacks?"

And then the girl took out her iPad, and she took out Avazand she pointed to the word "ice cream".

And then her mother asked, "OK, who eats?

You eat? Someone else eats?"

You know? And then she pointed to the word "I".

And then her mother was like, "Eat when?

Eat now? Eat later?"

And it turned out that this particular girl wantedto eat ice cream on her way back home from school.

Now it's not often that I can pinpoint the momentwhen the lightbulb goes off in my head.

But in this particular case, I remember this episode almostlike it happened yesterday.

Because what struck me was that feeling of realizationthat realization that this mother had gotten this girlto communicate what she wanted without using grammar.

I mean, this girl could not put more than one word together.

She could not put multiple words together to create sentences.

She could not select different forms of words.

She did not know what the meaning of "to" was.

She did not know the way in which English grammar worked.

It's almost as if her mother had peered into her head and figuredout what the meaning was that she wanted to convey.

So this was the other part of the puzzle for me.

You know, and I took this ideathat we could really represent meaning without grammarif we joined words together using questions and answers.

And I combined that with the algorithms that camefrom my study of Panini and all of the resource in linguisticsand the resource in neuroscience that had beenover the last few years.

And I put that together to create an app.

It took me three years to actually create this app.

I started in 2012.

It was in 2015 that the pieces finally came together.

And in 2016, we put this app outthis year, just a few months ago.

It's called FreeSpeech.

And let me show you a quick demo of FreeSpeech to explainto you how we solved this problem.

OK, so this is the FreeSpeech App.

And you can see that it has pictures the same waythat Avaz has pictures.

In fact, if I tap on the More Words button hereon the bottom-left, it has all of the pictures that Avaz has.

But unlike Avaz, when you construct a sentencein FreeSpeech, you don't have to startat the beginning of the sentence.

In fact, you can start anywhere in the sentence.

So, for example, supposing I pick the word "take"and I drop it here on the screen.

Here's the interesting part.

TakeThe app actually gives you this lattice this scaffoldof questions around the word that you've just droppedso that you can add more words to create a sentence out of it.

It's very similar to what the girl's mother was doing.

For example, if I drag the word "teacher"and I put it into the word "who".

The teacher takes.

It automatically constructs a sentencewith that specific meaning.

It says, "The teacher takes."

I can take the word "book", and if I put it into the "what",it will say, "The teacher takes the book."

And if I take the word "school" and if I drop it on the word"to", it will say, "The teacher takes the book to the school."

Let me show you a slightly more complicated example.

So, supposing we pick the word "want", and let's putthat in here, it pops up all of these questions.

And I can take the word "parent" and I put it into the "who".

If I put into "want", for example,it will say, "Want the parent."

But if I took that from that location and if I put itinto the "who", it will say, "The parent wants."

And you see that every time you add a word,it actually gives you even more questions.

So more questions pop up.

So you can see that there's a "whose"which is next to "parent".

So if I tap that, it will ask you the question,"Whose parent wants?"

And then I can drag the word "she", for example,and drop that in here.

It will say, "Her parent wants."

If I tap this little button on the top-left corner of "parent",I can convert that into a plural.

So I can make it, "Her parents want."

If I add the word "see", for example, I can say,"Her parents want to see."

And I can drag the word "you" and I can make it,"Her parents want to see you."

In fact, I can do even more complicated things here.

I can say that all of this happened in the past.

And I don't have to use the rules of grammar to do this.

All I can do is I can tap the buttonat the bottom that says Past.

It will say, "Her parents wanted to see you."

And I can negate that.

So if I press the Not button at the bottom, it will say,"Her parents didn't want to see you."

I can make a question out of it.

Didn't her parents didn't want to see you.

Or I can make even more complicated questions.

[ Applause ]

Why didn't her parents want to see you.

[ Applause ]

Thank you.

So that's how FreeSpeech works.

And FreeSpeech is pretty amazingbecause I think it's the very first timethat someone has actually come up with a grammar predictor.

That's what the app is doing, right?

I mean, you have all of these different picturesand the app is putting grammar around it.

Now why is that important for kids with autism?

The best way to kind of explain that is to draw an analogyto a different disability.

I'm sure many of you here have heard of dyslexia.

Some of you may have even struggledwith it when you were kids.

Kids with dyslexia have a lot of difficulty with spelling.

You know, English spelling is horrible.

I have difficulty with English spelling.

But if you give a kid with dyslexia an iPadand you ask them to type,magically their disability goes away.

Why? Because of spell check.

Because of autocorrect.

So a kid with dyslexia who is typing on an iPad is ableto communicate, is able to type perfectly correct English eventhough they have difficulties with spelling.

Avaz does exactly that for kidswho have a language disability like autism.

It plays to their strengths.

They're able to take pictures, they're able to pickout pictures from the tablet, and they're able to put picturesin different configurations.

But every time they do that,the algorithm is automatically creating a perfectly-grammaticalsentence and is speaking that out.

Now, we did trials for about an entire year before we putFreeSpeech out on the market.

And the results with kidswith autism were dramatic truly dramatic.

There were kids who were communicating with no wordsor they were communicating with one word.

And suddenly, in minutesliterally minutes after they started using the appthey were constructing these four or five-word sentences.

In fact, there was this one particular kidI want to show you a video of this kid.

This kid was in you know, he was using, you know,single words to communicate.

And you can see that he's not just using FreeSpeech veryfluently, but he's actuallyFreeSpeech is teaching him to talk.

Let's look at the video.

Eat want.

So, the kid is trying to speak.

Drag.

Eat.

He eats.

More words.

Food and drink.

Meal.

[Inaudible]

More words.

Press meal.

Press drag.

He eats lunch.

He ate lunch.

Did he eat lunch?

Good job!

[ Applause ]

This is pretty incredible.

You know, this kid was inthis is a kid in Portland, in Oregon.

And this kid's speech therapist, you know,he was completely blown away by this.

In fact, his therapist his name was Lucas.

You know, before Lucas became a speech therapist,he was a linguist.

And so he sent this video to us.

And to say he was blown away is an understatementbecause the same day that I got this video, I got an emailfrom Lucas asking me if he could quit his job and join my team.

[ Laughter ]

And that's how Lucas became our resident expertin speech therapy in autism.

In fact, when we put FreeSpeech out, you know,Lucas also had a very interesting ideabecause Lucas had this other kid on his caseload.

And this other kid had a you know, he was an Avaz userbut he was facing a very different kind of problem.

This other kid that wason Lucas' caseload, he was bilingual.

He came from a bilingual familybilingual, English and Spanish.

So, at school, he would be learning in English.

All of his teachers would talk to him in English.

But when he went back home, his family, and all of his friends,and all of his neighbors, they would be speaking in Spanish.

Now this is a challenge that speech therapists have battledwith for many, many years.

You know, devices like Avaz, apps like Avaz do exist.

They do exist in different languages.

But the problem of how do you give a voice to somebodywho speaks two different languages has always been achallenge for kids with autismand for the therapists that work with them.

So, let me give you a short demo of why exactly that's a problem.

Let me go back into Avaz and explainto you what the issue is.

So, this is Avaz, right?

So we looked at the I'm just goingto create a very quick sentence.

I'm going to say, "I want to," and I go to Actions.

And let's say, "I want to go home."

Right? So I pick out these different words here.

And so let's say that a kid is able to make this sentence,"I want to go home," and they can speak it.

Now there is a Spanish version of Avaz available,so let me switch into that very quickly.

So yeah.

How many Spanish speakers in the audience?

OK. I don't know Spanish at all.

I know zero Spanish.

But, you know, I'm going to try to use the picturesto create a Spanish sentence of exactly the same thing.

So, let's see.

So I have the picture for the word "I".

I can recognize that.

That's here.

And then I have "want".

And this looks like the picture for the word "to".

And this is "go".

And so there's places in here.

And this is "home".

Is that the right sentence in Spanish?

If I went to Mexico and if I said this, a kid would laughat me because this is completely grammatically wrong.

You see, I've actually done a pretty good jobof picking the right words out the wordsthe words "I", "want", "go".

All of these are correct.

But in Spanish, the way that the words work is very different.

For example, the word "querer" the word that means "want"you actually have to inflect that so it becomes "quiero"when you use it for the word "I".

And similarly, you don't use the word "tu"in Spanish the same way that you use it with English.

You'd actually put that word before the word"casa" to say "tu home".

So the problem here is that when you switchbetween two different languages,the interface actually looks very similar.

And you can see that, right?

The Avaz interface looks almost identicalbetween the two languages.

But the user experience is actually completely differentbecause kids have to learn completely different patternsto be able to construct these sentencesand to be able to put them together.

So this was a problem that not just my friend, Lucas,but really everyone who was workingwith kids with disabilities.

Essentially, the child had to learn two different patternsof using the same app for them to be able to communicate.

So the idea that kind of went off in our heads was,"Can we use FreeSpeech to be able to do this?"

FreeSpeech is essentially, in some sense,a representation of meaning.

Right? So the way that you put these words together,the way that you construct these sentences,they're essentially conveying what the meaning isthat you're trying to say.

And the app is putting the words together.

So can we use that to be able to help kids construct sentences,you know, even if they have a different language?

So I'm going to go into the Settings here,and there's a Spanish setting here.

And you can see that, when you do that,the interface completely changes into Spanish.

And all the words are in Spanish.

The labels are in Spanish and everything.

And when I drag a word and I put it in here,all of the interface elements here are in Spanish as well.

The questions are in Spanish, the tenses, everything.

But here's the interesting part.

When I start putting words in,I want.

It actually gives me an English sentence as the output.

So I can say so if I want to try to make the same sentence,I can take the same wordsI want to go.

And I can put that in here.

And I'm following exactly the same user interface.

I want to go home.

But I'm creating a sentence in another language.

You know, the[ Applause ]

So we put this out, you know, only a few weeks agoand we're already getting some reports of peoplewho found this quite useful, not in a special-education classroomwith a disability, but in a special-education classroomfor ELL for English Language Learnersand kids who are in schools in Americawho are entering the American education systemwithout knowing English.

There are teachers who are experimenting to seeif they can do this with those kids to teach them English.

So could you use FreeSpeech to teach someone English?

Or, for that matter, could you use FreeSpeechto teach anyone a new language?

Let me show you something else.

You know, I'm going to go into the app now, and I'm goingto go back into the Settings and I'm going to pick Chinese.

How many Chinese-speakers do we have in the audience?

Oh, OK. All right.

Well, typically when I give these demos,I try to pick a language that nobody in the audience knows.

[ Laugther ]

Unfortunately, WWDC is a little too internationalfor that trick to work for us.

I'll just keep my fingers crossed.

You can see that the interface has changedto Chinese now the way that it changedto Spanish the last time I did that.

But there's something very interesting that's goingto happen now.

So I'm going to take words out.

So let's pick the word "want" and put it in here.

Can you see what's happening?

It's actually creating a sentence.

It's actually creating a sentence not in Englishbut in English and in Chinese.

And I can do all of the things that I did with English.

For example, I can say, "That happened in the past."

I can say, "It didn't happen."

And that's how you can actually construct sentencesin a completely different language usingFreeSpeech interface.

[ Applause ]

Could you use this to learn Chinese?

Or, vice versa, can Chinese people use thisto learn English?

That's an experiment that we did.

I was in China in May.

And I went to China and I tried to see if we could get kidsin China to start using FreeSpeechto learn English instead.

And I want to show you a short videoof an experience that we had.

So this is a school in Shanghai.

You can see the kids are playing with FreeSpeech.

[ Foreign Language ]

They're having an incredible amount of fun.

The kid loves the iPad.

[ Foreign Language ]

They're killing it, right?

[ Applause ]

So, that's the next big project that we're working on.

And we're trying to create a game that uses FreeSpeechat its heart to teach language to kids that are tryingto learn a new language, particularly kidsthat are trying to learn English.

You know, it's very interesting.

There was a study that was published, you know,about five or six years ago.

Of all people, it was published by the NSA.

So there was this spy master probably in the NSAwho did this very interesting study.

He took a bunch of the world's languages and he triedto sort them, he tried to order them in the orderof which language is the most difficultfor an English-speaker to speak.

Which one do you think came out on top?

[ Laughter ]

Well, number 3 was Arabic.

Any Arabic speakers?

Number 2 was Korean.

And number 1 was Japanese.

Any Japanese speakers here?

OK, wow. All right.

So, the native Japanese speakers here, congratulations.

The NSA thinks yours is the most difficult language to learn.

And by the time you were three years old,you were speaking Japanese fluently.

You know, why do kids learn language so easily?

It always frustrates me.

It always makes me very jealous.

You know, I have a little son.

He's one and a half years old.

And I know that in another one and a half years,he'll be speaking completely fluently.

And it's always been a source of puzzlementto me in why they do that.

What the theory says what the theoryof language acquisition saysis that kids learn a language much fasterand much more effectively than adultsbecause they're immersed in it.

You know, they hear it, they try to speak it,and they know what the meaning is of what's being said to thembecause they understand the contextin which it's being said.

What if we could simulate emotionand provide the same capability to adults and older children?

That's why I'm excited about the prospect of FreeSpeech.

Because with FreeSpeech, you're making these sentences,and you know what the meaning isbecause you're using pictures as a bridge.

And every time you make sentences in FreeSpeech,the app is telling you what the sentence meansin the language you're trying to learn.

That's the interesting thing,and that's what we're working on.

You know, language is somethingwhich is really interesting for multiple reasons.

In this connected world where anyone anywherein the world is two clicks away on Facebook or on WhatsApp,language is really the last barrierthat we have as a species.

I mean, think about it.

Right? How many of you have a friendwith whom you don't share any language?

You know, communication is the fundamental barrierthat children with disabilities are born with.

But it's also the last barrier that we have to cross as people.

And that's why the work that I do isso satisfying and exciting to me.

You know, I build apps that help kids with disabilities,kids with disadvantaged education backgrounds, refugees.

You know, I build apps that use Apple's devicesto make the lives of these people better.

But I'm hopeful that some of the workthat I'm doing could also help all of us lookwith a different light and with a different appreciationat the most beautiful, most expressiveof human inventions language.

Thank you.

[ Applause ]

Apple, Inc.AAPL1 Infinite LoopCupertinoCA95014US

ASCIIwwdc

Searchable full-text transcripts of WWDC sessions.

An NSHipster Project

Created by normalizing and indexing video transcript files provided for WWDC videos. Check out the app's source code on GitHub for additional implementation details, as well as information about the webservice APIs made available.