IoT played an increasingly important role in people's lives; I was excited to explore its UX aspect. In Info214, I embarked on a UX research project to evaluate and improve the usability of an early-stage IoT application called OpenBAS.

Research Process

1

Project Background

I learned about the OpenBAS project through a CS course I took on IoT development. When I approached the OpenBAS team about doing UX research for their project, they gladly led me and my teammates on board.

OpenBAS

OpenBAS is an early stage open-source building automation platform that provides tools to integrate isolated sensor networks in commercial buildings. It was created by EECS Professor David E. Culler and his team at the University of California, Berkeley.

Commercial buildings have huge sensor networks, currently composed of temperature sensors, electrical sub-meters, fans, air dampers, light relays, pumps and valves. With the proliferation of more devices such as security, electrical, appliances, entertainment, etc., collectively referred to as the “Internet of Things”, the number of sensors a building manager needs to monitor is increasing at an unprecedented rate. However, most of these sensor devices do not talk to one another. Even though modern networked devices offer web service API’s, they are all different even across models from the same vendor. OpenBAS is developed to enable compatibility and interoperability between sensor devices in commercial buildings.

Integrating sensor networks would not only make building management an easier task for building managers, but also make smart energy savings possible, as better decisions can be made with access to more building data.

In its current beta version, OpenBAS focuses on integrating the HVAC (heating, ventilation, air-conditioning) and lighting system. It was consisted of an open-source IT architecture, a plug-and-play control hardware, and a web-based user interface. Through this interface, building managers can monitor and adjust HVAC device settings remotely all in one place.

Target Users

OpenBAS' primary target users are building managers of small and medium size commercial buildings that are equipped with multiple sensor devices. One-fifth of U.S. energy use is in commercial buildings, more than half of which are under 50,000 square feet. OpenBAS hopes to deliver a readily adoptable tool to bring new energy efficiency to smaller commercial buildings.

There are currently two deployments of the system, P and C. C is the focus of our usability assessment, a medium commercial demonstration in the CIEE facility. The second floor of a downtown Berkeley historic building (1935), 7500 sqft. There are 5 HVAC zones, lighting control, environmental sensors (carbon dioxide, etc.).

Research Objectives

Understand the nature of office managers' daily work

Interview

Diary Study

Evaluate what is working/not working with the existing OpenBAS UI

Heuristic Evaluation

User Testing

Explore better UI solutions

Card Sorting

Participatory Design

2

User Research

We worked on our three research objectives roughly in the same chronological order as they were numbered above. Findings from an earlier objective would inform the work for a later one. I worked on the first two objectives with my teammates, and on the last objective individually.

User Interviews

To understand how an office manager use OpenBAS, we needed to first understand what their jobs were like. We decided to interview office managers to find out how they conducted their daily work.

Recruiting and interviews

Recruiting our target users turned out to be a challenge. Based on our search (given time-constraints), at the moment there were very few small or medium sized commercial buildings equipped with multiple sensor devices around the area. Although putting sensor devices in buildings was a macro trend, we were still at the technology's early adoption stage. At the end, six individuals agreed to talk to us:

Two current and former OpenBAS users: The OpenBAS system was deployed for pilot testing in two commercial buildings in Berkeley. Through these deployments, we got in contact with two office managers who used the OpenBAS interface.

Two proxy users: We got in contact with a manager for a building not equipped with sensor devices, and a building energy saving consultant who advised sensor devices for commercial buildings. Even though the proxy users do not fit our exact target user group, they had similar experiences or relevant knowledge that would help us better understand how our users conduct their daily work.

Two expert users: We also talked to two members of the OpenBAS development team.

Input from the current/former users and proxy users was used to formulate insights into user habits, needs, and pain points. Input from the expert users was used only for project background and context. We did not utilize a formal interview guide for our interviews but we did develop a set of questions prior to each interview based on the context of the interviewee.

Findings

These interviews helped us understand the nature of an office manager's work. Talking to the expert users also helped familiarize us with the core functionalities of the OpenBAS system.

Officer managers in small and medium sized buildings are often tasked with many responsibilities. Controling HVAC and lighting systems were low priorities in the context of their everyday jobs. They felt existing manual methods of controlling these systems adequately perform the required functions. However, interviewees indicated that the concept of a digital dashboard that could monitor and control all systems would be appealing, if the digital interface was as easy to manipulate as the existing manual solutions.

Thermostats we saw during interviews:

Diary Study

To gain a better understanding of an office manager’s daily building management activities, I decided to conduct a diary study with our user J. I would have liked to have a sample size of five people or more for the study; given our recruiting difficulties, I decided a diary study with a single participant would still provide us with valuable insights.

Planning

GoalTo better understand what the office manager currently does to maintain the office:

What building management tasks she performs?

What are the pain points?

How much of this is accomplished through openBAS?

How much is through traditional interfaces?

DurationOne week (4/10/15 - 4/17/15)FormatPaper log sheets that we provided to the manager at the beginning of the study. (Although using electronic media would allow us to view the manager’s entries in real-time, we decided against it because it would take longer for the manager to retrieve than a paper log, and as a result might decrease the manager’s logging accuracy.)

Findings

The manager logged 12 entries in total. Among them, 10 were for turning on/off the lights in the office hallway. One entry was for turning up the heat in one building zone, and another one was for turning down the temperature in the conference room. All tasks were performed on the physical interfaces (switches for lights, thermostats for temperature).

I derived three observations about the manager’s building management activities:

The manager did not perform many building management tasks over the week.

The manager opted for the physical interface to perform tasks that the web interface was capable of.

The lighting control tasks were very quick (“took a second”), but the AC adjustments took longer (“took ~5 minutes”).

Heuristic Evaluation

Now we have a better understanding of an office manager's work. We moved on to evaluate the current user interface. We thought heuristic evaluation would be a good method to start, as it would allow us to quickly spot some major UI issues, while familiarizing ourselves with the OpenBAS web interface in the process.

The Current UI

Findings

Visibility of system status: Inconsistent and confusing system response for selected operations

Match between system and the real world: Information conveyed to be read by a software engineer, not end user

User control and freedom: With no menu hierarchy, easy to get lost after clicking on links

Consistency and standard: Different icons, text and colors used throughout the UI for different operations

Error prevention / Help users recognize, diagnose, and recover from errors: Can delete crucial components with no warning and no easy recovery

Recognition rather than recall: Multiple ways to locate key operations

Flexibility and efficiency of use / Aesthetic and minimalist design: Many nested screens and use of links

Help and documentation: Very little if any help features or documentation

In summary, users are uncomfortable exploring the interface. The lack of error prevention, recovery, and little documentation makes the user uncertain what operations will damage the system. The nested screens and links with poor navigation tools make it easy to get lost and difficult to remember where you previously found a key operation.

User Testing

The second method we chose for evaluating the current UI was user testing.

Recruiting

We contacted the two OpenBAS users that we interviewed earlier:

User 1 (J): novice, rarely interacted with the product

User 2 (A): more experienced, had used the product on a daily basis over a period of several months

Both users are office managers that fit the project’s target user set. The varying levels of expertise of our two users allowed us to assess whether certain tasks become easier to perform given increased exposure to OpenBAS.

Planning

Based on our user interviews, we developed a set of five tasks that we presented to each user at the beginning of the test. For each task, we asked the user to narrate her thought process as she worked to complete it, then to rate the overall difficulty of the task upon completion.

Task 1: the office is thinking about making every Wednesday a work from home day. Adjust the thermostat schedule such that the building doesn’t heat/cool on every Wednesday from now on.

Task 2: somebody in the office just complained that the northeast corner office was too cold. We want to increase the temperature of that office.

Task 3: someone mentioned that they noticed the lights were on in the office late at night over the past couple weeks. We want to verify if that was the case.

Task 4: we installed a new thermostat a few months ago but we’re not sure if it’s actually working. In order to determine whether it is working, we want to check on the date and time that the thermostat last reported its status to the openBAS system.

Task 5: let’s go to the display tab of the website,how would you design it if we were to start from scratch? What features would you want to see?

Following the usability test, we conducted a brief interview to better understand the user’s experience with the product and how it does or does not align with her needs as an office manager.

Findings

Our usability tests confirmed many of the findings from our earlier heuristic evaluation. First, the UI occasionally hangs up for periods of up to two minutes, preventing the user from performing any actions; this issue came up several times in both usability tests. Second, the UI is unintuitive - the users had difficulty navigating the interface, could not easily distinguish interactive elements of the UI from static elements, and did not understand the purposes of many of the features.

Card Sorting

The above studies helped us understand the nature of office managers's work, and revealed usability issues present in the existing UI. At this point we had mostly fulfilled the requirements of this class project. However, as a UX design, I wanted to pursue one step further and create better UI design alternative for OpenBAS. Hence, I embarked on the third phase of our research project- explore better UI solutions. This time I worked individually.

Our previous studies showed that one major issue with the current design was it had too many features that ended up confusing the user. To explore better UI solutions, the first thing I wanted to find out was the primary features an office manager would need. To answer this question, I did a card sorting session with user J.

Planning

I designed a two-tier card system. Separating cards into two tiers would help make the task easier and more manageable for the participant, likely resulting in better data and a more comprehensive understanding of the mental model. Tier-1 is the overall feature categories:

Control Temperature

Control Lighting

Monitor Energy Usage

Set Building Schedules

View when a device was last detected

And under each category, there is a list of tier-2 features:

Control Temperature

Select a thermostat on a clickable location map

Select a thermostat from a list of thermostat locations

Set current temperature

View heat/cool setpoint

Adjust heat/cool setpoint

Make temporary temperature change

Set the length a temporary change remains in effect

Plot temperature change history

Set Building Schedules

Edit regular schedule

Edit exception dates

Monitor Device Connection Status

Control Lighting

Select a lighting zone on a clickable location map

Select a light zone from a list of thermostat locations

Turn on/off lights in an entire zone

Turn on/off lights in an entire room

Turn on/off an individual light

Monitor Energy Usage:

View monthly energy target

View real-time energy consumption data

View energy consumption history

View comparison between current consumption and monthly target

View when a device was last detected

View if a device is active/inactive

I generated this collection as a quasi-exhaustive list that covered features available on the current interface, and additional ones brought up during interviews and studies. To cover unlisted categories and features, I also prepared ample blank cards for impromptu writing during the study.

Findings

During the study, I first instructed the manager to sort the tier-1 categories in the order of preference, and then asked her to sort the tier-2 features under each tier-1 category. In the process, the manager suggested a number of additional features that I included into the cards. At the end of the session, I had a clear list of preferences.

I made the following observations:

Temperature control is the most important feature, followed by lighting control.

Monitor energy usage is desirable but is challenging to present in an understandable and actionable way for a building manager (in fact, the manager didn’t like any of the tier-2 features we provided). This seems to be an area that should be automated, either fully or to the extent that actionable items are provided to the manager.

“Set building schedule” module is important but does not need to appear on the home page.

“Monitor device connection status” module should be integrated with temperature and lighting controls.

Some Important features are currently overlooked:

Label devices in a locatable way

Inform users which devices are controllable by OpenBAS

Allow users to sync thermometer dates in an easy and reliable way

Participatory Design (Aha Moment!)

Planning

Referencing results from the card sorting, I felt ready to put designs on paper. Since OpenBAS was new system was a limited number of early adopters, I thought using a participatory design approach to involve an early adopter in the design process would help me better address user needs. J agreed to help out.

To prepare for the participatory design session, I pre-selected a list of important features from card sorting and created a collection of pencil-drawn design wireframes, and separated them into moveable module. During the design session, I showed J each design module and asked for her critique. Then I asked her to arrange the modules to form a layout she would prefer.

Findings, and epiphany

The following design challenges surfaced during the session:

Crowding caused by the large number of data to show and parameters to adjust (e.g. zone name, current status (heating/cooling/idle), current temperature, connection status, set target temperature, set time in effect, show confirmation. ).

Absence of proper naming system for locating lights (needed for controlling individual lights from the web interface.)

After the session, I experimented with various design changes in an effort to make the interface less cluttered while still maintaining the granularity of data and controls. Whatever why I tried, it was impossible. Then, while sketching yet another iteration, an idea hit me. I thought back to some comments J made during the participatory design session and had a sudden realization:

When I asked how she usually determined what temperature adjustment to make when someone complained, J said “usually a couple degrees initially, and if someone complains again, adjust one more degree.” If this is how the manager adjusts the temperature, it seems a computer algorithm could do equally well, if not better.

When I asked how she managed the lights, J said “other than turning on/off the hallway lights (the only lights that did not have sensors and not fully automated in the building), I never touch the lights.” Then when I asked if she would use the lighting zone control, she said “No.”

J's comments made me realize that granular control of the system may not be desirable to office managers. Instead, they might want to issue a simple command and let the system decide the precise parameters.

This realization led me to my findings.

Temporary temperature adjustment needs to be better automated. Implement an algorithm that automatically adjusts temperature when an adjustment request is received, and develop an interface that handles adjustment requests.

The status of user interface should be secondary to automation. User should only need the interface to supplement information to assist automation, and to do manual adjustments when automation fails to deliver.

These realized finally cleared up my design direction, and I designed a new interface mock-up soon after.

3

Reporting

Recommendations

Our recommendations are based on the research findings explained in the above section. Due to the limited scope of our research (6 interviews, 3 heuristic studies, 2 usability studies, 1 diary study, 1 card sorting, 1 participatory design), we think more research and studies will be necessary to further validate them. With this understanding in mind, here are our most important recommendations (full Finding Summary available here):

The interface for technicians can retain most of the current features.

The interface for the office manager will be simpler. The interface is designed to be a supplementary tool to automation. Mostly, the manager will not need to use the interface as openBAS automatically takes care of the daily operation. The interface is used to (1) put in temporary adjustment requests (2) adjust schedule (3) manually control when the auto operation is not working. The interface will also send push alerts to the manager (e.g. via email) if a malfunction is spotted (e.g. a device disconnects).

Please see the Interactive Prototype (explained in next section) for the proposed redesign.

Issue 4

For the typical office manager, the existing energy consumption data visualization interface is hard to interpret or translate to action. Moreover, energy consumption is not a high-priority issue in the context of the day to day job function of an office manager. (see: interviews, usability study).Solution

Proof-of-Concept Mock-Up

To help the OpenBAS team envision what an improved interface could look like, I designed a PoC mock-up that addressed the issues listed above. As part of a user research deliverable, I did not spend much time polishing and testing this mock-up, as its main purpose was not to be a final design itself, but rather to illustrate the research findings and inspire new designs. I did, however, showed the mock-up to J, and she thought it was a major improvement to the current design.

Note: To enable the mockup’s interactive features, download the linked PDF to your local computer.

Final Report

I presented our research findings and design recommendations to the OpenBAS team. Our work was well-received by the team. Going forward, OpenBAS plans to incorporate the findings and design into their new product version.

I want to thank the OpenBAS team and our users, who generously help us out of their busy schedules. I also want to thank my INFO214 course instructor, Steven Fadden. His insightful guidance and encouragement made this project a very fun and rewarding experience.