A Near-Term Vision for Everyware: Synthetic Serendipity

“Let’s look forward from the present day, using stories by well-known science fiction authors as the source for vision or concept scenarios that describe some possible experiences of living and working with everyware.”

In “First Fictions and the Parable of the Palace,” I provided an overview of everyware’s roots in early depictions of ubiquitous computing by Mark Weiser and others. I also considered the critical role of user experience in the coming world of everyware and described some of the challenges we face in designing everyware / lifeware user experiences.

In this and the next installment of my column, let’s look forward from the present day, using stories by well-known science fiction authors as the source for vision or concept scenarios that describe some possible experiences of living and working with everyware. While these concepts are clearly speculative, they help define the boundaries of the potential scope of everyware—in both the near-term and distant future. Though these stories may lack the metaphoric and literary significance of Borge’s “Parable of the Palace,” their visions shed light on everyware’s likely implications for design within more concrete contexts. I’ll review only the elements of the stories that are most relevant to designing everyware. (For the curious, I recommend reading the stories in full.)

Everyware in the Near Term

Vernor Vinge’s entertaining short story “Synthetic Serendipity” [1]—first published in the IEEE magazine Spectrum in 2004—presents a near-term future that is fully permeated by everyware. The narrative is from the perspective of a high-school student named Mike, who likes to play games.

“Vernor Vinge’s entertaining short story “Synthetic Serendipity” … presents a near-term future that is fully permeated by everyware.”

Though publication of the story in Spectrum does not constitute an official endorsement of the future “Synthetic Serendipity” presents, the story’s appearance there—in combination with Vinge’s reputation, professional background, and outlook—tends to validate its premise as a possible future.

Vernor Vinge is an award-winning science fiction author and former professor of computer science who anticipated the rise of cyberspace, describing it in his 1981 story “True Names.” Vinge is also the leading exponent for the concept of technological singularity, that moment in the future at which machines eclipse human intelligence, accelerating their own evolution. Spectrum is “the flagship publication of the IEEE, the world’s largest professional technology association.” [2] While Spectrum publishes fiction on occasion, its primary audience is the professional community of engineering and business technology.

“Synthetic Serendipity” begins, “Years ago, games and movies were for indoors, for couch potatoes and kids with overtrained trigger fingers. Now they were on the outside. They were the world.” [1] With this opening, Vinge deliberately inverts two essential aspects of the world most people take for granted—that games are distinct from reality, and media is separate from the wider environment—thereby requiring readers to suspend their mental models of play and entertainment.

The remainder of “Synthetic Serendipity” elaborates on this inversion, showing a world in which alternate-reality games (ARG’s) and augmented reality (AR) have converged with ubiquitous computing to erase the boundaries between entertainment and real life.

As the story progresses, protagonist Mike Villas is playing a game with friends while walking to high school. Vinge describes the augmented landscape around the group of boys:

“The canyon was not a dead zone. Not at all. County Flood Control kept the whole area improved, and the public layer was just as fine as on city streets. As they walked along, Mike gave a shrug and a twitch just so. That was enough cue for his Epiphany wearable. Its overlay imaging shifted into classic manga / anime: the manzanita branches morphed into scaly tentacles. Now the houses that edged the canyon were heavily timbered, with pennants flying. High ahead was a castle, the home of Grand Duke Hwa Feen in fact, the local kid who did the most to maintain this belief circle. Mike tricked out the twins in manga costume, and spiky hair, and classic big-eyed, small-mouthed features. ‘Hey, Jerry, look.’ Mike radiated, and waited for the twins to slide into consensus with his view. He’d been practicing all week to get these visuals.” [1]

From the design perspective, this paragraph is dense with implications for the nature of the everyware experience. Vinge presents a world in which the average adolescent seemingly has free and ubiquitous access to a pervasive, geo-localized information and communications infrastructure layer the local government maintains. Further, Mike Villas and his friends use this pervasive network to join an active, geographically distributed community that stages complex, extensively customized alternate / augmented-reality games. Mike and the other participants play these augmented reality games using a range of sophisticated wearable computing-hardware devices.

“Shared augmented visualizations, integrated social elements, and communication capabilities such as presence, identity, and conversation let players maintain layered, nuanced dialogues with other players and people who are either nearby or linked remotely from afar.”

These wearable systems employ gesture, audio, video, haptic feedback, and possibly other sensory channels as their primary interaction and control mechanisms. This is not the world of the keyboard and mouse we know today and goes far beyond multi-touch. Synchronously shared personal displays with stylized visualization themes let game players define and exchange their individual viewpoints on the augmented-reality landscape surrounding them. Broadcasting and sharing players’ individual views facilitates cooperation and competition. Players create, manage, and enjoy games by exchanging their reference points. Shared augmented visualizations, integrated social elements, and communication capabilities such as presence, identity, and conversation let players maintain layered, nuanced dialogues with other players and people who are either nearby or linked remotely from afar.

All of these social and co-creative dimensions allow, or perhaps—given common social pressures for self-presentation and peer sociality—require players to define and manage multiple digital identities. [3] Technically, their avatars—the digital identities Mike and his friends create for playing games—persist outside game contexts, remaining present as elements of the networked public space. [3, 4]

In addition to the pervasive information layer in public areas of the landscape that is devoted to utilitarian purposes, some private-property locations serve as enhanced game environments that offer richer experiences for players, or co-creators. Vinge describes an example of this sort of public / private area as follows:

“Pyramid Hill had all the latest touchy-feely effects. These were not just phantoms painted by your contact lenses on the back of your eyeballs. On Pyramid Hill, there were games where you could kick lizard butt and steal raptor eggs, or games with warm furry creatures that danced playfully around, begging to be picked up and cuddled. If you turned off all the game views, you could see other players wandering through the woods in their own worlds. Somehow the Hill kept them from crashing into each other.” [1]

Some of the identifiable mechanisms of everyware to which Vinge refers include

three-dimensional projections of information onto contact lenses

multiple game styles and visualizations

overlapping networks of players who are either physically present or remotely and digitally present

Vinge describes a system that smoothly coordinates all of these interacting elements throughout an augmented-reality domain.

In the park, Mike and his companions encounter a large dinosaur creature of Mike’s design, which promptly ‘eats’ one of them.

“‘Ahh!’ the monster opened its mouth and vomited horror. It was scary good. Mike flicker-viewed on reality: Fred was standing in the steaming remains of his raptor. His shirt was pulled out of his pants, and he was drenched in slime, real, smelly slime. The kind you paid money for.

“The monster itself was one of the Hill’s largest robots, tricked out as a member of Mike’s new species.” [1]

Here, we can see that Pyramid Hill offers an entertainment environment populated by large, autonomous or remotely managed robots. Its multi-sensory special effects, adaptive system rules, and shared visualizations support the game experience. Altogether, Pyramid Hill sounds a bit like Disney’s trademark imagineering approach to amusement park design—though the enormous socio-technical possibilities of everyware amplify its consequences.

When Mike arrives at school, everyware is also part of his out-of-game experience:

“Indoors, the choice of view was not entirely his own. Mornings, the school administration required that the Fairmont School News appear all over the interior walls…. He followed the little green nav arrows with half his attention…and abruptly realized he had climbed two flights of stairs. School admin had rearranged everything since yesterday. Of course, they had updated his nav arrows, too. It was a good thing he hadn’t been paying attention.” [1]

Vinge’s future school directly controls some aspects of what students’ wearable systems display. The school provides personalized wayfinding, based on both Mike’s course schedule and facilities management decisions. In design terms, Mike’s wearable is coordinating multiple organizational and procedural layers of rules and information to direct him to his next assigned classroom, using time and location awareness—which are presumably supported by a mixture of sensors, mapping / cartography, and geographic coordinate systems. While this may be convenient for students who are thinking of other things on their way to class, the ability to literally control what someone else sees raises critical questions about privacy, surveillance, and control. School policy in “Synthetic Serendipity” requires students to appear in person:

“That was why the school made you show up in person for most classes; you had to pay a little bit of attention just because you were trapped in a real room with a real instructor. Chumlig’s lecture graphics floated in the air above them. She had the class’s attention; there was a minimum of insolent graffiti nibbling at the edges of her imaging.” [1]

Like the dramatic reframings that open the story, this description of school policy represents another profound inversion. In the world of everyware, physical presence in a classroom setting limits students’ range of activities—particularly, their access to social interactions, other information, and modes of communication—rather than enhancing access to knowledge, instruction, and the professional guidance of teachers. Virtual absence constitutes truancy—which merely reflects physical absence.

“Vinge suggests a mixed palette of shared visualizations, originating from the cameras and sensors of fellow students, as well as cameras with unspecified, though presumably non-human origins.”

In addition to affecting administrative policy, everyware permeates educational practices and tools in “Synthetic Serendipity”—such as the three-dimensional display teachers use to focus the class’s attention. And it seems Vinge’s future is at least partially hackable, given the mention of graffiti.

In class, Mike uses many of the same elements of everyware that support the augmented-reality games he plays in his free time, letting his attention wander “…from viewpoint to viewpoint around the room. Some were from students who’d set their viewpoints public. Most were just random cams.” Here Vinge suggests a mixed palette of shared visualizations, originating from the cameras and sensors of fellow students, as well as cameras with unspecified, though presumably non-human origins. Vinge uses the dense deployment of instrumentation and networked sensors throughout both man-made and natural environments that is stock in trade for many depictions of the future of ubiquitous computing.

As students often do, Mike has mastered skills that let him avoid the attention of teachers while in class:

“Like most students, Mike kept lots of stuff saved on his wearable. He could run a search like this very close to his vest. He didn’t route to the outside world except when he could use a site that Chumlig was talking about. She was real good at nailing the mentally truant. But Mike was good at ensemble coding, driving his wearable with little gesture cues and eye-pointer menus. As her gaze passed over him, he nodded brightly and replayed the last few seconds of her talk.” [1]

Interaction designers might note Vinge’s mention of “ensemble coding,” a carefully learned technique for using wearable computers that minimizes any signs one’s attention is not focused on the social roles and interactions that are supposed to be in the foreground—for example, the relationship between teacher and student. Ensemble coding is an advancement on contemporary usage practices that aim to hide or disguise when people are working with computing devices—the familiar crackberry—forbidden in group settings like personal and professional gatherings, classrooms, and business meetings.

Like business people who have learned the art of working during meetings, Vinge shows Mike Villas putting his attention on other goals:

“Mike sent out a couple hundred queries, mainly pounding on certificate authorities. Even if the certs were solid, people and programs often used them in stupid ways. Answers came trickling back. If this weren’t Friends of Privacy chaff, there might be some real clues here. He sent out follow-up queries….” [1]

This passage makes it clear that skilled and savvy students can discretely conduct sophisticated information retrieval exercises against large collections of locally cached data, as well as the wider cloud of the Internet. Mike is using his equipment and advanced searching skills to investigate the background of his fellow students, part of a business arrangement—called an affiliance—that relies on anonymous agents, brokered contracts, identity certificates, electronic payments, and deep reservoirs of publicly accessible, personal information. He finds “lots about Ralston Blount [an adult student] that was public record, even some legacy newsgroup correspondence. That was always good for shaking up your parents and other grown-ups….” Vinge invokes the stereotype of younger generations adapting to new technologies more quickly than their parents and grandparents and also reminds readers that a world of everyware is a world of mediated public information.

Teachers, with a good eye and their own skills at hiding interactions with computing devices, can use the same techniques to monitor and manage students, as Mike discovers here:

“Suddenly a message hung in letters of silent flame all across his vision:

“Chumlig > Villas: You’ve got all day to play games, Mike! If you won’t pay attention here, you can darn well take this course over.

“Villas > Chumlig: Sorry. Sorry!

“Most times, Chumlig just asked embarrassing questions; this was the first time she’d messaged him with a threat.

“And the amazing thing was, she’d done it in a short pause, when everyone else thought she was just reading her notes. Mike eyed her with new respect.” [1]

“Vinge shows a classroom that is thoroughly permeated by digital elements such as presence, surveillance, traffic and activity monitoring, messaging, and communication with semi-autonomous query agents.”

Vinge shows a classroom that is thoroughly permeated by digital elements such as presence, surveillance, traffic and activity monitoring, messaging, and communication with semi-autonomous query agents. All of these digital elements directly affect the social norms and dynamics of the classroom environment, which is itself repositioned not as an enhanced means of sharing information, but as a venue with increased barriers to the normal flows and interactions of ubiquitous computing.

“Synthetic Serendipity” presents a wide-ranging vision of the experience of interacting with everyware, in terms of

enabling devices and user interfaces

social norms and patterns, in settings ranging from education to business, play and cooperation

supporting public and private infrastructures

architecture and land-use practices

local and supra-local forms and mechanisms of government

economic and political structures

shifting demographic and generational power balances

The first four topics are naturally of most interest to user experience designers, but the contexts in which the everyware arises and the environments it permeates are also important.

It’s Everyware Right Now

“Many of the foundational elements of Vinge’s vision are available now in some form.”

The games-versus-reality inversion, truly pervasive network, and gesture-driven wearable computers of “Synthetic Serendipity” may seem farfetched, as do many of the other components of everyware that appear in the story. But a brief survey of existing technologies—some new since 2004—shows many of the foundational elements of Vinge’s vision are available now in some form.

Gestural Interfaces

Vinge’s ensemble coding and shared visualization spaces may be quite close at hand with the launch of technologies like g-speak. According to its creators at Oblong, g-speak is a “spatial operating environment” that is “built around free-hand, three-space gestural input. Applications are controlled by hand poses, movement and pointing. Finger and hand motion are tracked to 0.1mm at 100 Hz; pointing is pixel-accurate. Two-handed and multi-user input are fully supported.” [5]

g-speak’s creators claim their software platform is ideal for supporting spatially aware interactions among groups of people, and is already in use for

analysis of large data sets

operation of three-dimensional interfaces

construction of efficient multi-user collaborative applications

integration of large screens and multiple computers into room- and building-scale work environments

development of large-scale applications that run interactively across enterprise networks [3]

These contexts resemble many of the activities and ideas that appear in “Synthetic Serendipity,” including

Mike’s investigation of extensive background information about his fellow students

3D visualizations Mike’s wearable generates

group design activities in a shop class, which are featured in the story

a shared virtual classroom environment

Pyramid Hill’s augmented-reality game space

Figure 1 is a video showing the g-speak environment being used for film and video editing.

Figure 1—g-speak video editing

Augmented Reality

“‘Synthetic Serendipity’ assumes technologically savvy people have pervasive access to deep layers of public and private information attuned to their current location.”

“Synthetic Serendipity” assumes technologically savvy people have pervasive access to deep layers of public and private information attuned to their current location. A present-day antecedent to this sort of experience might be Wikitude, an application Mobilizy developed for the G1 Android phone shown in Figure 2, which delivers an augmented-reality experience. The application offers users geo-localized information from a variety of sources like Google Maps and Wikipedia, superimposing Internet data on live images from the phone’s camera. Wikitude uses combined location and image information to recognize elements of the landscape that the G1 camera sees, identifying landmarks and local points of interest.

Figure 2—Wikitude’s augmented view of reality

Wearable Devices

“Synthetic Serendipity” does not describe the wearable computing system Mike and other students use in any detail, leaving its size, weight, components, visibility, and other physical attributes to the reader’s imagination. But video eyeware manufacturer Vusix is chasing Vinge’s future scenario. The company sells lightweight 2D / 3D video display glasses in both military models and the civilian model shown in Figure 3. Vusix glasses visualize many kinds of input at VGA and SVGA+ resolutions, creating an experience that is similar to looking at “a 60-inch monitor as seen from 9 feet.” New models have transparent lenses, allowing wearers to maintain their awareness of the real world while using the video displays.

Figure 3—Vusix’s Wrap 920AV ambient awareness glasses

Social Norms

Creating effective and affordable wearable hardware is primarily a technical challenge. However, the broad social acceptance of people who are encrusted with information technology and entertainment devices throughout the course of their everyday lives presents a very different threshold. Yet a November 2008 survey on mobile usage habits “finds 36 per cent of Canadians spend more time with their cell phone each day than their significant other.” [6] The same survey found substantial indications that the perceived social weight of mobile devices is declining rapidly:

30 per cent of those surveyed said they sleep with their cell phone beside their bed.

30 per cent of Canadians admit to using their cell phone secretly in the bathroom.”

The Near Future: SIXth Sense

“SIXth Sense intends to ‘redefine our interaction with the digital and physical worlds.’”

Commercial technologies like g-speak, Vusix, and Wikitude have already fulfilled part of the “Synthetic Serendipity” vision, but prototype devices like SIXth Sense have greater potential for realizing the wearable systems of everyware. A project of the Fluid Interfaces Group at MIT showcased at TED 2009, SIXth Sense has capabilities that echo many aspects of the wearable systems Vinge envisions in “Synthetic Serendipity.” As described by its creators, SIXth Sense intends to “redefine our interaction with the digital and physical worlds.” [7]

SIXth Sense uses a combination of off-the-shelf devices, including a camera, mobile phone, and projector, which together act as “a wearable computing system that turns any surface into an interactive display screen. The wearer can summon virtual gadgets and Internet data at will, then dispel them like smoke when they’re done.” [8]

The inventor’s intent for SIXth Sense is pure lifeware:

“SIXth Sense frees data from its confines and releases it into the world, seamlessly integrating information and reality. SIXth Sense bridges this gap by augmenting the physical world around us with digital information and proposing natural hand gestures as the mechanism to interact with that information. SIXth Sense brings the intangible information out into the tangible world. … SIXth Sense sees what you see and visually augments any surfaces or objects you are interacting with. SIXth Sense projects information to any surface, walls, and the objects around us, and [lets us] interact with the information through natural hand gestures, arm movements, or with the object itself. SIXth Sense is having the entire world as your computer.”

The videos in Figures 4 and 5 show SIXth Sense in action.

Figure 4—SIXth Sense wearer demonstrating the prototype

Figure 5—SIXth Sense wearer shows prototype’s capabilities

Pioneers

“The researchers, entrepreneurs, inventors, and creative technologists behind SIXth Sense and its like are pioneers, working to design the ubiquitous experience of lifeware.”

The researchers, entrepreneurs, inventors, and creative technologists behind SIXth Sense and its like are pioneers, working to design the ubiquitous experience of lifeware in the absence of well-established tools, methods, and perspectives. The people designing lifeware are already confronting the challenges of creating balanced experiences within complex contexts. What do these early examples of everyware tell us?

Rob van Kranenberg of the Institute of Networked Cultures identified three implications of rising ubiquity in his pamphlet “The Internet of Things.” First, according to van Kranenberg, “When computational processes disappear, the environment becomes the interface.” [9] For example, the Vusix eyeware unit projects video-display overlays onto its lenses that its wearers see in combination with actual reality. This requires a shift in the framing of design that recognizes the locus of computing and interaction is no longer distinct from the surrounding context. Instead, this locus is immersed in the environment. For designers, the everyware interface is not a strictly bounded or controllable space.

“The designers of everyware must successfully determine how to integrate the many different layers of everyware experiences—including those that are personal / universal, local / global, synchronous / asynchronous, as well as co-created.”

Second, van Kranenberg noted, “In such an environment—where the computer has disappeared as a visible technology…human beings have become designable and designerly information spaces.” [9] From the perspective of design, this means the subject / object positioning of people changes—people themselves become objects of design. Both the first and second implications are readily apparent in the SIXth Sense demonstration video. The wearer makes a telephone call by dialing a touchpad that is projected onto his palm—human body as user interface—checks the time by drawing a circle on his wrist—human body as user interface—and projects data about someone else directly onto that person’s body during a face-to-face meeting—using another human being as a designed element of a user interface.

Third, van Kranenberg declares, “Design decisions inevitably become process decisions.” [9] Designing everyware means taking into account all relevant contexts, and as “Synthetic Serendipity” shows, the greatly increased complexity of the user experience directly affects how design is done. The designers of everyware must successfully determine how to integrate the many different layers of everyware experiences—including those that are personal / universal, local / global, synchronous / asynchronous, as well as co-created. Foregoing or incorrectly balancing these contexts can adversely affect the experience. For example, the designers of SIXth Sense discovered the sensitive implications of projecting possibly embarrassing information into the social contexts of face-to-face conversations.

As everyware continues to evolve, all three of these implications will take on greater significance and urgency. I’ll revisit these issues again in the next installment of my column, which looks at the more distant future of everyware.

The More Distant Future: Post-Humanity

“Computing power will expand beyond the tight confines of dedicated machines and permeate the wider world.”

One of the fundamental tenets of everyware is that computing power will expand beyond the tight confines of dedicated machines and permeate the wider world, becoming, as Mark Weiser said, “Machines that fit the human environment….” [10] In this view, as I’ve discussed in this column, computing is deeply integrated into the surrounding environment, yet remains external to human beings themselves.

In the next installment, I’ll review a science fiction scenario that considers what happens when the boundaries separating humans from the environment blur and dissolve—in which humanity itself becomes, as van Kranenberg foretold, an object of design.