T166 EyeSpot

Inspiration

One day, while in my calculus class, a series of knocks on the wall grabbed my attention. Peeking out from the glass layer of the wall, it appears that a visually impaired man was walking along my class with a cane. Throughout the following weeks, there have been numerous times where he accidentally hit an object, and thinking that it was a wall, he had to detour although the path was totally clear. The American Community Survey estimates that around 700,000 Americans suffer from visual disabilities, with an additional 7 million population experience the prevalence of such disability. How would us, three individuals who have the gift of sight, help these people who are unable to sense what's in front of them?

What it does

EyeSpot can be considered as an impaired person's virtual best friend. Through a series of available commands, EyeSpot can communicate with its user conversationally and guide them using their artificial eyes -- their camera's optic lenses. Users can ask EyeSpot about their surroundings, location and whereabouts. Furthermore, by scanning through users' friend list and recognizing their friends, EyeSpot is able to detect people around them and identifies if they are the user's friends, and will inform them if they are.

How we built it

EyeSpot was built using the Ionic Framework and deployed as an hybrid app on iPhone. We utilized at least five APIs from Microsoft Azure, some intelligence and analytics libraries in order to build our product.

Challenges we ran into

Initially wanting to build it on a Gear VR, it proved to be too technically challenging as none of us had any prior experience with Unity and Android development. We had to pivot halfway through but...

Accomplishments that we're proud of

...we were able to finish most of what we initially planned to achieve. We were able to integrate machine learning, big data (albeit with a relatively small dataset), facial recognition, vision processing and speech recognition in such a short period of time!

What we learned

We learned to have some knowledge prior to working on a product. We attempted to implement some technologies that we have zero experience on. Next time, probably it will be a good idea to consult experienced people or do some research about some technologies before we work on it.

What's next for EyeSpot

As it really has a practical use case and that helping the visually impaired was part of our initial goal too, we will try to improve EyeSpot and turn it into a product that benefits the thousands or millions of people who are visually impaired. However, some of the technology that we use as part of our solution are not advanced enough to be highly accurate, but we would love to see how we could improve these technologies too.