Exploring the future

ON MODELINGBuilding support for use-based design into hardware products

Authors:
Tim Misner

Editor’s Note: Use-based design is a new model of product
development. It is a process of measuring user behavior and
applying the resulting data to improve the next version of a
productcreating a feedback loop between user and designer.
(We might refer to the process as data-driven design, but that
term has another meaning among software developers.)

Basing design decisions on customer behavior has roots in
mail-order catalogs of the late 19th century, such as Montgomery
Ward and Sears, Roebuck. Use-based design grew along with direct
mail in the mid-20th century. More recently, the Web created
opportunities for collecting data on user behavior. Google is
famous for driving decisions with use data. But few designers
have experience in basing decisions on data, and many are still
uncomfortable with the practice.

Now the use-based design model is being applied to
hardware. For consumer electronics and office products
especially, connecting to networks has become almost standard,
creating new opportunities to collect information about user
behavior (and about device behavior). In a sense, networked
hardware products are very much like Web servicesand in
many cases hardware products are being integrated with Web
services. For a growing class of “monitoring” hardware and
services (e.g., network management, security, sports, and health
products), collecting user information may even be the primary
mission.

Taking advantage of these capabilities requires a new model
of design as well as a plan to integrate them into the first
version of a product.Hugh Dubberly

Increasingly, hardware products (especially consumer
electronics) include computers, sensors, and connections to the
Internet. These capabilities enable changes in what products
“know,” how they are used, and how we develop them. They are
becoming more like websites. This simple fact became apparent to
me during my work at Dash Navigation as their design
director.

There, I worked on the Dash Express, a personal navigational
device (PND) that included GSM networking and exploited the
following networkedservices principles.

Networked services differ from traditional hardware products
in at least four important ways:

Networked services may also enable field upgrades of
software, continually improving the product.

At first, however, I didn’t understand how designing an
integrated system of hardware, software, and network applications
requires a new way of thinking about a product and its
development. On one hand, we design end-user applications. On the
other, we design platforms for both services and user feedback on
these services.

This allows us to adopt the essayist Clay Shirky’s theme of
building “systems where having good participants produces better
results than having good planners” [8]. Internal
discussions change from “what feature or quality status do we
think our products need?” to “what data can we collect about our
features and quality?”

The Product Development Experience

Getting from the standard 18-month hardware development cycle
to a rapid cycle of data-driven verification requires that the
product team commit itself to building the infrastructure to:

send updated software to all, or just segments, of its user
base;

obtain and analyze data from the users themselves; and

provide explicit, periodic methods for the users to submit
their opinions from the device.

Thematic features of this magnitude require strong executive
and product management sponsorship to succeed. This evolution
cannot happen without explicitly specifying these needs as a core
product feature. Back-haul data, in particular, benefits from
discussions between many departments (operations, support, QA,
and development), as well as the user experience (UE) team. In
fact, it’s quite likely that the other departments will be the
initial drivers of the feature. However, UE participation is
required to create the kind of infrastructure that allows design
iteration based on behavioral data.

Uses of Back-haul Data

In contrast to data submitted by users through emails, phone
calls, forum posts, and surveys, back-haul data enables direct
observation of user and system behavior. “Back-haul” refers to
moving data from a remote site (a networked device) to a central
site (an application server).

One great place to start prioritizing your needs is by
brainstorming with a broad range of stakeholders about what
questions they’d like to answer. Some of the most obvious will be
questions that you can’t reasonably or reliably ask users to
answer themselves. With behavioral data, we can collect answers
from the total user population, not just those who respond to
survey requests. Another good place to start is with the
statistics that measure, from a product management perspective,
whether the feature was successful against business goals.

Secondarily, questions asked in user surveys can be
crosschecked against behavioral data to assess the degree to
which users accurately answer survey questions. For example, when
asked how many times a month they use a device, the estimate from
survey respondents is often larger than the observed degree of
device usage. While this is a well-known bias of surveys,
unsophisticated product teams often believe that what a user says
is the unvarnished truth. They have either forgotten or not
learned the principle that says, “Don’t trust users; observe
them.”[9]

Common Networked Functions

Many connected devices will use the following networking
functions:

Transmission of sensing data from the device back to a
server. For Dash, the sensing data is the driver’s current
road and speed; for a medical device, the data might be the
monitored patient’s heart rate, blood oxygen level, or glucose
level.

Automated distribution of software updates to the
device. Dash advertises its auto-update feature as an
important advantage over competitors.

Ability for the manufacturer to configure and diagnose the
device. At Dash, the networking-derived functions of the
device (real-time traffic, Internet search) are disabled if users
let their service plan lapse [10].

Ability for the user to send data from the Web to the
device, thus eliminating more tedious device side entry. For
the Dash Express, a well-loved function is entering an address by
sending it from the web to the car [11].

All of these connected functions can potentially be exploited
by UE for research needs. For example, variants of the software
can be sent to target user populations for either their
subjective feedback or to generate “A/B” test cases. A
population’s configuration settings can be analyzed to determine
the most and least popular setting changes.

Most important, the software development organization can
rapidly update the product during usability testing and then
observe users for their reactions to the device and its new
software.

Many Groups Use Back-haul Data

User experience is hardly the only group motivated to scoop up
back-haul data. Back-haul data collection is essential for other
functions to fulfill their objectives.

The ability for UE to understand, leverage, and champion these
other needs will maximize its influence over the form that
back-haul infrastructure takes. In particular, UE needs to
realize its stake in the design of these infrastructure
components when initial implementations are sketched. A common
mistake for typical small UE teams is not seeing the potential at
early stages.

For example, the operations group will require usage data for
authentication and billing. IT needs aggregate usage metrics that
trigger alarms when usage levels drop, whether due to instability
in its own infrastructure or to service-provider outages. Support
needs some form of back-haul data to understand customer issues
more efficiently, such as versioning, configuration settings,
stack traces [12], and prior usage. QA values
back-haul data on common performance metrics, such as “time until
first GPS fix” and “time to first network connection” so that it
can assess release readiness. QA also values back-haul data to
help track metrics of system performance and stability. With this
data, the QA team can better evaluate release readiness. Finally,
everyone wants some method to “file a problem report” from the
device to reduce the time to file, reproduce, and fix bugs.

Issues Unique to UE Needs

The UE group has needs distinct from the other functional
groups. Both QA and operations, by and large, are more concerned
with aggregate numbers than an individual’s experience. Their
numbers may say the product is performing as specified, but they
don’t indicate if the users are actually happy with that level of
performance.

For UE design, it is beneficial to triage use cases into
buckets such as “frequent for all,” “frequent for some,” or
“infrequent for all.” In order to do this, the data source must
maintain a marker of individuality along with the data, so the
data analyst can slice and dice the data to discover such
relationships.

In addition, the UE group desires the ability to run
longitudinal queries, particularly the ability to see that new
user features lead to perceptible user benefits. This requires
the UE group itself to create and maintain over time a database
of high-level user events in a normalized format.

Infrastructure Needs

In order to exploit the back-haul data, central sampling
issues such as who, what, and when (how frequently) need
addressing.

Behavioral logging benefits from the ability to understand and
filter the results through the lens of various user-grouping
mechanisms. For example, stakeholders will question unwelcome
results because various outlier groups may have skewed the
results.

Invariably, both business and operational groups have an
interest in creating user segmentation; this functionality is
thus almost certainly available to the UE group. However, the
grouping mechanisms will be driven by customer purchasing
segments, and this in turn can constrain an experiment’s
design.

Problem reports. Since users have problems, they
need a mechanism to explain what those problems are: the problem
report. There are two different design approaches:

Mostly passive, where the problem report is created on behalf
of the users. They just need to submit the report.

Mostly active, where the user initiates the problem
report.

Both are, of course, useful. But for devices, the active
report is particularly useful, as it is often very difficult to
create bug reports independent of a complex environment that only
the user fully understands.

The work required is not just the data transmission and
recording. In implementing the problem report feature, I
recommend budgeting a fair amount of time for implementing tools
that facilitate analysis of the reports. Tools that facilitate
categorization and visualization of data pay for themselves in
short order. Otherwise, time will be spent telling the bug
reporter, “I can’t reproduce your problem.”

Server-side logging. The simplest and most
cost-effective technique is to execute server-side system
logging. Then scripts can be run to collect and analyze the
logged data.

The main advantage of this approach is that little planning is
required to generate data because there are no changes to the
device code. In the simplest case, an IT operator or engineer
adds some logging code (using the existing logging framework) to
a server-side component. These changes are usually low risk to
any release. This modification gets deployed to the main user
population at the next server-side software release.

However, back-end logging alone doesn’t provide a rich picture
of the user experience because many behavioral issues can be
resolved only via data from the device side.

In addition, tools construction is required to “extract
knowledge from data.” Mike Kuniavasky gives an overview of both
the benefits and issues with server-side logging
[13].

Device logging. Logging from the client side is
more complex than logging from the server side, because device
logging has to handle the transmission of state back to the
mother ship. In addition, the organization has to be willing to
pay the bandwidth cost required to send back the data.

The simplest code approach is to just add a logging function
to each “user event” (such as a keypress). This is analogous to a
website “click-stream.” The UI framework may have a very small
number of places where this logging code could occur. So the
implementation can be both simple and broad. The downside, of
course, is that an enormous amount of data is generated and
transmitted and most of the data is not germane to the user
experience questions at hand.

Fundamentally, this functionality is only useful for debugging
purposes when focused on a small number of users.

A more complex approach is to log, for the purposes of a
targeted experiment, only the data germane to the researcher’s
question. This requires custom code to map between various system
states and the user state. This code can be more complex because
it is integrated into the application logic as opposed to
targeted at a base-system level. In addition, you must remember
to turn it off after the experiment is complete.

My experience is that adding this instrumentation is
unnecessarily costly when done after the design and initial
implementation phase of a project. Don’t shoehorn it in during
the beta cycle.

Conclusion

From my inefficient, but valuable, initial experience, a
central theme emerges for UE professionals: Think through the
data you want to collect as a fundamental part of sketching and
specifying your design. This facilitates a cheaper engineering
implementation, but it also allows you to drive design directions
from an initial, limited implementation published to a specific
user population.

In addition, all functional groups can feel enriched by the
quality of user data derived from an Internet-connected device.
Customarily, these user research and debugging techniques were
available only to Web services. But they are now possible with
today’s connected, physical devices, provided your product team
invests in the key infrastructure items such as:

With such infrastructure, you can implement a truly iterative,
use-based (or data-driven) design approach for your device, just
as website designers do today. But to get this infrastructure
built in the first place and to exploit it fully, you’ll also
need to convince key stakeholders that your complex,
sophisticated, stand-alone product can in fact be conceived by
your product managers, your designers, and your users, as just
another very profitable website.

Tim Misner is senior director of software engineering at
Oracle. Before joining Oracle, he was director of software
engineering at Dash Navigation, makers of Dash Express, a
two-way, Internet-connected GPS-based navigation system. Misner
also served a stint as director of user experience engineering at
Sun. He has a background in product management, engineering, and
mathematics.

Google and Amazon have built big businesses by collecting and
analyzing previously unheard of amounts of data. Their businesses
are not accidents. They are not corner cases. They are signals of
an emerging future.

Several trends point to the same thing: the value of large
amounts of data and the ability of that data to support
tailoring, learning, and decision makingto enable new
categories of business, or perhaps a new model for all
businesses.

Today, these trends may still appear separate:

1. BIG DATA

Big data refers to the assembly of very large databases,
perhaps first by research in the physical sciences and
intelligence gathering by the NSA and the military; followed
closely by telephone companies, securities exchanges, credit card
companies, and consumer list consolidators (e.g., Acxiom,
ChoicePoint, Equifax, Experian, and TransUnion); more recently,
Web-based services have begun to generated huge amounts of
data.

2. CONVERSATION-BASED MARKETING

Conversation-based marketing refers to a broad shift from
one-size-fits-all broadcast advertising to real one-on-one
conversations with customers. The shift began with direct mail
that broadened into direct response and direct marketing. Direct
mail was one of the first areas to apply big data to drive design
decisions and improve customer service.

3. CRM SYSTEMS

Customer relationship management systems collect and store
information on a business’s customers (and potentially other
constituents in service delivery). CRM systems are one part of
the complex community of systems that sophisticated businesses
need to manage conversations with constituents. CRM systems and
practices are related to, but typically separate from, DM/DR
activities. A business will install a CRM system internally, but
serious players in DM/DR outsource customer database management,
because it’s a specialized skill not available from in-house IT
groups. (For example, your IT folks are not likely to know where
to begin to “de-dupe” your customer list or perform other forms
of regular “data cleansing.”)

4. CROWD SOURCING

Crowd sourcing is a system (often electronic) that uses large
groups of people to collect and organize data. We might propose
crowd sourcing as an umbrella term for a range of activities
involving large numbers of people: Open source projects that rely
on volunteers (e.g., the OED, Wikipedia, Linux); collaborative
filtering, the process of making recommendations based on the
actions of people with shared traits; Google-style search, which
uses links as one of the predictors of relevance; and flash mobs,
which quickly form, take collective action, and disperse.

5. DATA VISUALIZATION

Data visualization refers to the application of graphic design
principles to making large volumes of data easier to
understandthat is, making pictures out of lots of numbers.
Today data visualization is a subspecialty at the intersection of
the hard sciences, computing, and information design, drawing on
the disciplines of statistics, animation, and filmmaking. It will
become increasingly important to communications design,
interaction design, and service design. Closely related are
design of service dashboards and augmented realities (virtual
overlays).

6. USE-BASED DESIGN

Use-based design refers to a process by which the actions of
users are recorded, analyzed, and used to make decisions on
changes in the next version of a product or service. Use-based
design has roots in the world of direct marketing. Websites
created a new level of opportunity. Sites can log every view and
click, and enable the testing of alternatives. Google is famous
for its data-driven decision making. And now, as more products
are connected to the Internet, consumer electronics (and office
products) can also log every action users take, providing data to
drive decisions for the next version of a product or for interim
software upgrades.

7. MASSIVE CLOUD COMPUTING

Cloud computing involves Internet-based services that provide
large amounts of computing power on an as needed or lease basis,
much as utilities provide electric power. Massive cloud computing
refers to recent initiatives by IBM+Google and HP+Yahoo
supporting research and development efforts to increase the power
and speed of cloud computing systems.

8. SENSORS

London is under surveillance by 400,000 closed-circuit video
cameras. Wal-Mart has mandated that all its suppliers build RFID
chips into packages that go through its distribution system.
Every iPhone includes at least six types of sensors. We are on
the cusp of a huge wave of sensor technology. Sensors will be
embedded in everything, and they will pour out a continuous flood
of data.

9. SERVICE DESIGN/SERVICE SCIENCE

Service design and service science refer to the process of
developing and managing services. As hardware becomes
increasingly commoditized, services offer opportunity for
differentiation. A customer’s experience with a brand may extend
across a family of services, each with a collection of touch
points. These touch points are increasingly networked, and thus
customer behavior may be logged, analyzed, and used to drive
improvements.

10. SOCIAL MEDIA

The next generation of computing will assemble vast stores of
datafrom a growing array of physical and virtual sensors.
These technical and social changes will create opportunities for
a wide range of companies. Consumer electronics makers like Apple
will monitor their hardware and supporting services.
Network-tools makers like Cisco will instrument their customers’
networks. Google (and other Web-services providers) will continue
to instrument search and everything you do with their tools.
Health care device makers like Johnson & Johnson will
increasingly offer services to complement products that
continuously monitor your vital signs. Sports and apparel makers
like Nike will also build biometric sensors into the soles of
their shoes and the fabrics of their clothes, supporting another
form of continuous monitoring.

Apple, Cisco, Google, Johnson & Johnson, Nike, and others
like them will find themselves in essentially the same business,
certainly dealing with the same customer management, design
management, and information technology management questions. They
will have entered the era of customer-data-driven business.

Permission to make digital or hard copies of all or part of
this work for personal or classroom use is granted without fee
provided that copies are not made or distributed for profit or
commercial advantage and that copies bear this notice and the
full citation on the first page. To copy otherwise, to republish,
to post on servers or to redistribute to lists, requires prior
specific permission and/or a fee.