In "Limitless," Bradley Cooper plays a struggling writer who takes a pill to expand his intelligence. What if VR experiences can make people smarter just by giving new physical forms to thought?

This blog is called the Extended Mind because it’s about how people use their bodies or objects in the environment as part of cognition. Now, suppose that rather than just studying the existing ways that people are using their bodies, phones, etc to think you could imagine new ways to interact with information and how those interactions would unlock new thoughts.

That’s the topic of Bret Victor’s talk “The Humane Representation of Thought” that he delivered at the closing keynote at the 2014 Splash and UIST conferences. (H/T to Leonard Lin for telling me about this talk.) Victor is a UI designer who formerly worked on tech for the iPhone, MacBook, and iWatch.

Victor claims that his talk is not about Virtual Reality, but his insights are applicable to anyone doing experience design and grappling with new ways to represent information. Below are the excerpts of Victor’s talk that I found most relevant to VR / AR designers. I would recommend watching the entire talk, but here are the highlights from the Extended Mind perspective.

Victor begins with some history and ways that human cognition expanded once new ways of representing data were invented, such as graphs.

“Representations, by which I mean the ways we externalize thought, have been responsible over the last 2,000 years in large part for the intellectual progress of humanity, enabling us to think thoughts that we couldn’t think before. But while these media that we’ve invented have empowered us in certain ways, they cripple us in other ways and we now have an opportunity to design a new medium of thought that undoes some of that damage in a way that’s both humane and empowering."

Victor doesn’t talk much about the mechanics of cognition. However, it’s worth calling out that the working memories of humans are very limited. By reducing the amount of stress on a person’s working memory by giving them new ways to externalize thought, you will expand people’s capacity for thought.

Victor later quotes from James Gleick’s book Genius: The Life and Science of Richard Feynman to support the notion that abstract thought is aided by having a physical intuition.

“[Physicists] found they needed imagery…a style of thinking based on a kind of seeing and feeling. That was what physical intuition meant.

Feynman said to Dyson, and Dyson agreed, that Einstein’s great work had sprung from physical intuition and that when Einstein stopped creating it was because ‘he stopped thinking in concrete physical images and became a manipulator of equations.’ Intuition was not just visual but also auditory and kinesthetic. Those who watched Feynman in moments of intense concentration came away with a strong, even disturbing sense of the physicality of the process, as though his brain did not stop with the grey matter, but extended through every muscle in his body.”

Then Victor goes on to review the various models of representing thought from various academic disciples.

“The point of all of this is that the space of how we understand things, the space of our cognitive capabilities is so vast and so diverse that even people who have devoted their entire careers to coming up with a neat little theory come up with different answers. There’s so much there. It’s such a rich space.

Every circle up there is a superpower. Every circle is a capability that we’ve been honing for hundreds of thousands of years that we use in innumerable ways and can combine with the other ones in really powerful ways. And what happened with the invention of the printing press and tiny rectangle based knowledge work is this”

Formulas...yuck! Yet, this is what intellectual/analytical work looks like in academic circles.

Then Victor gets into the real substance of his talk. I would recommend watching from minute 20-30 to hear the gist of his talk on importance of expanding experience design to include all of human senses and capabilities. Here is a transcript of the key ideas:

“And all this stuff drops out and we’re working with visual symbols. We’re reading with visual symbols, we’re manipulating visual symbols. That’s what it means to do intellectual work nowadays. And you might think ‘Well, maybe that was true with paper and books, but with computers, it’s getting better, right?’ And no, with computers it is getting worse. With a book, a book is at least a physical object which exists in the world with some amount of tactile response. You can hold it and move it around. With a book you can make a shelf you can make a shelf, which is a spatial representation of knowledge. It’s is not very good but it is something you can understand spatially. When you’re writing with ink on paper, you can move freely from drawing imagery and writing in language because the paper doesn’t really care what type of marks you make on it and you can move between those two modes. But then we invent these things [imac / ipad], these flat, glassy screens that have no tactile response so [tactile] drops out and they are little tiny screens that take a portion of your view of view and special stuff drops out. And they have these keyboards. The only convenient thing to do is punch in symbols especially for us anything from writing an email to writing a computer program, anything other than typing in letters is incredibly cumbersome. Whatever you are trying to express, you express it in symbols because that’s what the interface encourages you to do. So [iconic] drops out. And so this is the cage that we have trapped ourselves in. This is the way that we have constrained our range of experience, which we have created a tiny subset of our intellectual abilities and have forbidden ourselves to use our full intellect.

There’s two things wrong with this. One is that it’s Inhumane…using media that restrict our thinking to this tiny subset is inhumane because do all our thinking things. We can’t think in all the ways that humans can think so there’s a kind of moral argument there. If you don’t buy that, there’s also a practical argument, which is that it’s just wasteful. One way to think about it is as programmers to imagine that you have an eight core processor and you’re writing some code. It’s kind of a sequential program. Can’t parallelize it so you’re maxing out of the cores and the other seven cores are just idling and that hurts. As a programmer you get a feeling in your stomach, 'God this is so inefficient…if only I could parallelize this algorithm, there’s so much latent power I can draw on' and that’s exactly the emotion that I get when I’m looking at this tiny, rectangle based knowledge work that we do. There’s all these cores. We have all these capabilities and they are just sitting there idle. If only we could parallelize our representations across all of these capabilities, who knows what we would be capable of thinking."

In the last section of the talk, Victor offers an optimistic take on how designers can break out of the design paradigm of the rectangular screen.

"So the good news is that we now have an opportunity to do something about this. So we are now, I believe, at a unique moment in history where we are inventing the next medium of thought after the printing press. We are inventing the dynamic medium. So in the same way that there are certain thoughts that can be conveyed in print or in theater, I believe there’s a wide range of thoughts that can be conveyed in programs once we understand how to do that."

In an ideal world, designers will make experiences and represent information in an dynamic medium. This dynamic medium will be responsive to the user’s capabilities and connect multiple modes of understanding at once.

Main takeaways for experience designers:

1. Cognition is dependent on representations of data. At the most basic level, if your working memory is clogged up with too much data, your internal processing power decreases. 2. You have multiple sensory superpowers to help you think. However, the current dominance of rectangular screens primarily relies on vision and shuts out the others. 3. VR & AR designers have an open canvas for representing thought and they should actively look for avenues to enhance connections among human modes of understandings.

The Extended Mind is a Portland-based research firm that applies behavioral science to increase user engagement in immersive technologies. Jessica Outlaw, the founder of The Extended Mind, is also a winner of the 2017 Oculus Launch Pad competition.