Seeing AI: Leveraging artificial intelligence to better view the world

I’ve been writing about apps for a long time, but they are not of equal importance.

Microsoft’s free Seeing AI app may be a game changer for people with visual impairments. The research project is designed to turn “turn the visual world into an audio experience,” narrating the world for those who cannot see it, in real time using artificial intelligence.

Available at the App Store, Seeing AI, is a talking camera experience. Introduced in late 2017, it is comprised of a growing variety of channels.

Seeing AI:

describes the people around you (including their emotions)

speaks text and signs as soon as they appear in front of your camera

scans and reads documents, books and letters, including handwriting by reading OCR and understanding formatting

identifies currency

scans barcodes on objects/products and adds product details when available. Seeing AI helps you locate the barcodes with audio beeps

describes images in other apps, for instance, email, Twitter, and WhatsApp

learns and recognizes your friends and describes other people around you, including their emotions

an experimental feature offers scene descriptions. Take a picture and the app will tell you about what the camera sees around you.

the app also generates an audible tone corresponding to your surrounding brightness and can perceive color

These tutorial videos offer just a little peek into the Seeing AI experience:

On recognizing people

On hearing snippets of text

On reading documents

On locating barcodes and identifying products

On scene descriptions (an experimental channel to turn on in settings)

On currency identification (US currency is challenging because bills are all the same size!)

Though I cannot test this one out in a truly authentic way, I believe that Seeing AI is an important app for us to share. Please spread the word.

About NeverEnding Search

News, thoughts, and discoveries at the vortex of libraries, literacy, learning, discovery and play. Joyce is an Assistant Professor at Rutgers University's School of Communication and Information, an edtech Sherpa, and a connector. Her interests include: social media curation, digital/media fluency, transliteracy and youth, online communities of practice, digital storytelling and creativity, youth information-seeking behavior, social networking, online learning, and the evolving role and powers of the teacher-librarian.