Living La Vida Virtual: Interfaces of the Near Future

“On one
hand, we are rapidly moving into ubiquitous computing
environments that let people constantly interact with the omnipresent
network; on the other, the devices and interfaces we are using
to enter these new frontiers provide woefully inadequate user
experiences.”

Personal computing is in an awkward adolescence right now.
On one hand, we are rapidly moving into ubiquitous computing
environments that let people constantly interact with the omnipresent
network; on the other, the devices and interfaces we are using
to enter these new frontiers provide woefully inadequate user
experiences. Let’s take a look at one of the key technologies
that will take mobile user experiences to the next level: holography.

Holography and the State of Input

The primary reason why the BlackBerry® became such an enormous
success is its miniature QWERTY keyboard, which lets people
rapidly enter information and, in the process, made easy-email-while-on-the-run
a reality. Earlier devices such as cell phones and Palm® PDAs
provided a substandard means of communicating with a computing
system, but the BlackBerry took the well-established and long-practiced
QWERTY keyboard interface and employed it in a practical and
portable form. This allowed people to engage in a more natural human/computer interaction.

Unfortunately, while it is our accustomed way of entering
information into a computing system, even in its most optimal
state, using the QWERTY keyboard is an unnatural and awkward
activity. Shrinking it down to fit on a handheld device may
have allowed mobile computing to leap far forward, but this
remains a poor and ultimately temporary solution to the problem
of mobile data input. While more people—particularly people
in east Asian cultures—are embracing an active mobile-computing
lifestyle, the modes of input mobile devices offer have not advanced much.
Components that behave like a mouse and
miniature touch screens cannot replicate the speed and power
of desktop computing environments.

“Unlocking a superior user experience for mobile
computing begins with the development of more sophisticated modes of input.”

The potential of mobile computing is enormous. Indeed, despite
clumsy input devices, the mobile computing market is taking
off and will only continue to get bigger—even if input
devices do not get demonstrably better. Yet, unlocking a superior
user experience for mobile computing begins with the development
of more sophisticated modes of input. Soon, more effective voice recognition will drive input, in part. And, eventually, modes of input will evolve into what we have
come to think of as science fiction—controlled by our
eyes, our gestures and body movements, and ultimately, perhaps
the holy grail of direct mind-to-machine interaction. But in
the space between, we need a better, yet technologically practical
way of entering data.

Most people who use computers know and are accustomed to using
the QWERTY keyboard. For all of its faults, it remains both
the primary and the most rapid, yet robust way for people to
communicate with computers. That is why incorporating it into
BlackBerry devices was such a success. However, miniaturization
and thumb typing compound the issues of a full-size keyboard.
There are already reports of heavy usage of the BlackBerry
causing physical problems in users’ hands and wrists—the
new repetitive stress injury BlackBerry thumb.

The obvious opportunity might seem to be a mobile device with
a larger QWERTY keyboard. But others have done this before.
Palm has long offered a fold-up, peripheral keyboard that somewhat conveniently fits into a coat pocket. But it is just one more thing to carry,
among an ever-increasing cache of personal gadgets, and ultimately,
lags in performance compared to a standard, full-size keyboard.
At the same time, notebook computers might seem to be effective
mobile devices—most sporting a full-size QWERTY keyboard,
along with full desktop-computing power. Unfortunately, notebooks
are just too big and bulky to function as truly mobile devices.
While portable enough to carry around, they remain a far cry
from a cell phone or PDA-sized device that a user can slip
into a pocket, without being noticeably encumbered in any environment—short
of a nudist colony.

So there is a tension between our desires for usability and
mobility: the QWERTY keyboard is an effective and well-practiced
input device, but is too large and bulky for use in a truly
mobile user experience. Enter the holographic keyboard—virtual
save for a small component that is part of a mobile computing
device. A holographic projector could place a full-size keyboard anywhere, giving us a computer input experience that is similarly robust to
that of our desktop/place-bound computing stations. A holographic
keyboard synthesizes the essential usability problem of providing
desktop-quality computing with seamless and always-available
portability. Now, I’m sure the early versions of such
a technology will not deliver on the promise of the idea, but
this is the likely technology to take us from the awkward fumbling
of today into the powerful mobile computing environment of
tomorrow.

Holography and the State of Output

But input is only part of the problem. One of the biggest
challenges facing everyone in the mobile space today—on
both the hardware and software sides—are the device outputs, particularly
their displays. The screens are incredibly small. The resolution
is poor. The output of even the most remarkable devices of
today pales in comparison to the standard 14-inch desktop monitors
of some 15 years ago—a monitor that today is essentially
obsolete. Indeed, for desktop computing environments, we are
rapidly moving into the world of large LCD flat-screen displays—some
in excess of 30 inches. Why bigger? Because when it comes to
visual output, bigger is almost always better. More pixels.
Higher definition. Improved usability and a more natural interactive
environment. This is what makes the “Baby
Faces” of mobile devices—as Aaron Marcus so cutely and appropriately
calls them—so inadequate for displaying computer output.

“Holography would
enable us to project a large—even full-size—virtual
object with next to no physical space requirements.”

The primary constraint is again one of portability. Notebook
computers offer relatively effective visual output, but as
I mentioned earlier, are not truly mobile devices. They do
not integrate well into our on-the-move lifestyles. So, how
can we provide a large, high-resolution display that is truly
portable—small enough to fit in a pocket?

Nothing even remotely close to such a technology exists today,
but as with the virtual QWERTY keyboard, we could once again
use holographic technology to create a projection of a computer
display or even a virtual 3D environment. Holography would
enable us to project a large—even full-size—virtual
object with next to no physical space requirements. It might
take some time to develop very high-resolution holographic
displays, enabling this technology to achieve display quality
that is comparable to that of a desktop monitor. However, the
sheer size of such a display would, to some degree, make up
for any deficiencies in resolution. Just in their being much
larger than the mobile output devices that are now standard,
holographic displays would exceed the current mobile user experience—to
say nothing of their potential for three dimensionality, which
would open up entire new frontiers of interacting with our
machines.

Privacy Concerns, and How to Solve Them

“Implicit in next-generation interface technologies— holographic output displays as well as rich voice-enabled inputs—are problems relating to privacy. And these are significant problems.”

Implicit in next-generation interface technologies—holographic
output displays as well as rich voice-enabled inputs—are
problems relating to privacy. And these are significant problems.
For users, the confidential information that is a normal part
of living a digital lifestyle—from banking and purchases
to password entry and communicating secret personal information
for account verification purposes—becomes dangerously
public in these new contexts. This is to say nothing of the
personal privacy we prefer to have when communicating with
a lover or getting dressed-down by our boss or playing a computer
game. These more effective means of interaction bring with
them very real challenges to our personal security and comfort.

At the same time, there are also privacy concerns for non-users.
Already today, people complain bitterly about others talking
on cell phones in public spaces. What if voice inputs were
extended beyond voice communication and became the driver for
most human/computer interaction? What is now a relative annoyance
would become an unbearable cacophony. Then consider the physical
space requirements of holography: even though these are virtual
objects, the reality is they fill space just as real objects
do. Others would need to walk around a projected computer display
just as they would need to walk around a physical one. The
problem will amplify if and as more and more people adopt and
use the technology. The next time you go into your local Starbucks® during the morning rush, imagine half or more of the people
seated inside with 15-inch computer monitors on tables a few feet
in front of them. What impact would that have
on the experience? How would that change the very personality
of Starbucks? After all, if these technologies work, people
will use them. The relatively modest number of notebook users
today would undoubtedly be replaced by an ever-increasing number
of rich-mobile-device users.

“As is often the case, technology will need to solve the problems that technology creates.”

As is often the case, technology will need to solve the problems
that technology creates. I saw a great example of a possible
technology solution in practice recently, in a government R&D
environment: a device that looked something like a DirecTV®-style
satellite dish, focused and directed music in a specific and
completely unexpected way. We started in one room, where music
was very lightly playing on iPod® speakers. Then we went outside
the room and walked down a really, really, really long hallway.
Our colleague took the device, stood outside the office, and
tilted it in a way to create triangulation. Suddenly, we were
able to hear the music at approximately the same volume level
as we had while in the office! If we moved a few feet to the
right or left, we couldn’t hear it at all. But if we
stood within the correct field, we were able to hear it. No
one else outside the room in this busy work environment was
able to hear it at all. It was really an amazing experience.

Whether or not that technology in particular will ultimately
be the one to solve some of the privacy problems next-generation
interfaces create, it underscores the essential truth that
there is already technology out there that is well ahead of
anything we are seeing in mainstream use today. Historically,
technology has been successful in solving the problems it creates.
With every gain come inevitable challenges. If we are to enjoy
the power and freedom of robust mobile user interfaces, we
will need to develop technologies that can help keep our computing
experiences somewhat contained and, therefore, private.

Filling the Space Between

“The more
distant future will be one of entirely digital environments,
in which inputs and outputs are all around us.”

Yes, the more distant future will be one of entirely digital
environments, in which inputs and outputs are all around us,
and it is only a device containing our personal data—or
a key into our personal data—that serves as our mobile
computing device. But in most parts of the world, we are a
decade or more away from realizing that degree of integration.
In the meantime, we are dealing with ubiquitous computing experiences
that are painfully low resolution in every way, presenting
a real opportunity for a technology or company that can change
this very tepid paradigm. Holography is one technology that
very nimbly addresses issues of both input and output. Whether
applications of the technology will reach the point where they
are both sufficiently effective and affordable to the mass
market—as well as relatively private and secure—remains
to be seen. But this is certain: the trend that digital computing
technologies have started—transforming elements of the
physical world into a virtual existence—is one that will
continue in the decades ahead. Holographic technology is an
early technology that could serve an important role in not
only the short-term improvement of the mobile user experience,
but also in changing our very mental model of the relationship
between the real and the virtual.

5 Comments

Jonathan wrote:

Good article Dirk. We are rapidly seeking new innovative ways to interact with technology. The iPod interface is a superb example of the new breed. Simple, effective and easily learnt by anyone. A circle cannot be complicated!

You’ve solved it! Make sure to include a holographic keyboard with motion sensing devices to register keystrokes. To counter privacy concerns, create cloaking technology, perhaps using an electronic version of polarized optics. This raises some questions: “Would people ever leave their virtual pods? And what will it be like to move through 3D space?

The most promising implementation idea I’ve ever heard is Augmented Reality … Virtual reality on top of the real world.
Technically, you’d wear glasses that can overlay virtual images on the real world. They’d track your position and head orientation, so that virtual objects could be connected to real ones. This would also neatly solve the privacy issue.
A colleague and I can sit at a conference table, shuffling papers and folders, and a casual passer-by just sees two people with glasses, shuffling air.

I think you jinxed yourself with the words “Nothing even remotely close to such a technology exists today”. This was not actually true when the piece was written, although the fact that it was an error wasn’t public knowledge. All the more impressive you targeted holographics as the enabling technology that would drive the next generation.

Light Blue Optics has developed a unique,
patented laser projection technology that
exploits the principles of diffraction to project full-color, real-time video images.
Our projection technology uses laser light sources in conjunction with a phase modulating liquid crystal on silicon (LCOS) microdisplay on which a diffraction pattern, calculated from the desired image, is displayed. When the microdisplay is illuminated by coherent laser light, the desired image is projected.