Adding Search to Jasperserver: A Cautionary Tale

Summary

Our upcoming 3.7 release of Jasperserver includes a major new feature: faceted search.

As the first major initiative I pushed for after joining Jaspersoft as User Experience Architect, search was my pet project. While I am certain the new functionality has improved the user experience for some users, I believe, due to flaws in the design process I led, we missed an opportunity to deliver an interface that made finding objects in our system dramatically simpler for all users.

This post is an illustrated attempt to break down that design process in the hopes that others may benefit from understanding the errors I made and the lessons that I have learned (or, sadly, re-learned).

Lessons (Re)learned

To help you parse this long post, I’ll begin by listing the lessons I took away from the search project:

Never assume the primary persona or the essential use cases

Mockups are not prototypes

Evolution is an easy sell; revolution requires data

Performance matters

And most importantly:

NO PET PROJECTS!

The Starting Point: An Expert Interface

In Jasperserver 3.5 the only way users can find reports and other objects contained in the system is via an interface that looks like this:

This design is obviously modeled on a file browser. As such, it suffers from the problem shared by all browse experiences, namely that it is a tool for experts, not novices.

Because it is designed to expose the file structure, a browse tool provides a good experience for the administrator who needs to create or maintain that structure. However, for the business user who knows the name of a report stored in the repository, but no idea of the logic the administrator used to construct the structure, using a browse interface is a bit like an easter egg hunt. I explain this issue in more detail in this post.

As the percentage of Jasperserver users who maintain repositories is considerably smaller than the percentage of Jasperserver users who want to find the reports contained within the repositories, my feeling was that the repository interface was optimized for the wrong persona. I believed that we could fix this usability issue by adding a search functionality to our server, and then optimizing the repository interface around the new search feature.

I made this argument in a number of product planning sessions and succeeded in getting a search project prioritized for our next development cycle. Paradoxically, this early success was the source of my biggest mistake. By focussing on functionality, I sold the idea of adding search, but, as it turned out, never actually got buy-in for the crucial underlying assumption that we should optimize our design for the repository user, rather than the repository creator.

The Mockups

In my initial storyboards for the new repository experience, I proposed a faceted search navigation combined with a federated results display:

The point of the default design was to present the objects in the system in terms of meta data most relevant to a business user. However, because we did need to support the administrative workflow for creating and maintaining the hierarchical structure, I proposed that users be able to opt-in to a view of the file structure, by pressing a button:

If exposed, in addition to various management functions, the folders could be used to control the scope of the search, and, combined with the search string and the other facets used to refine the found set. I thought this design was starting a bit complicated, but as viewing the folders was an opt-in experience, I was comfortable that by default we would be offering a simpler presentation of the repository objects.

I used images like these in storyboards detailing a variety of use cases that I shared with project stakeholders. The storyboards were met with a favorable response and we moved into production.

In retrospect I now understand that this was the point where I made a second, crucial mistake. This was that I moved directly to storyboards without first establishing with the stakeholders what the critical use cases actually were. If I had begun by asking the stakeholders to help me define the most important use cases I think we would have uncovered almost immediately the fact that many of the stakeholders did not share my main assumption that we should optimize the design for business users. Instead, I presented storyboards that embodied the business user persona assumption implicitly. As a result stakeholder approval of these storyboards really only meant that the storyboards told stories that made sense. Approval did not mean, as I assumed, that the storyboards made sense and told the stories we all thought were most important!

First Prototype and First Hiccup

This problem only began to be clear to me when we finally presented our first working prototype. It didn’t have the all the filters working and it didn’t group the results by type, but was otherwise quite faithful to the mockups:

And, as proposed, folders could be exposed and used to set the search scope:

The reaction of several project stakeholders to this prototype first alerted me to the fact that we did not all share my fundamental assumption that we were optimizing this design for repository users, rather than administrators.

The trouble began when the stakeholders finally understood that the folders were acting to set search scope, not to define the location of the panel that displayed objects. You can see this in the screenshot above. The folder ‘Ad Hoc Components’ is selected, but if you look carefully at the path shown for the displayed objects, you will see that these objects are all actually located in a child folder ‘Topics.’ From a search perspective this is correct: the search is finding all objects that match the current search string (in this case the empty string “”) located within the selected folder and all its children.

However, from a browse perspective this is an incorrect result because only results located in the currently selected folder should be visible. In this case, this would mean no results would display. Once this difference was understood, the stakeholders insisted that the administrative user could no longer use the interface effectively because it would not meet his browse expectations.

Compromise Mode

It was now clear to me that I had made strategic blunders in managing the design process for this project. Rather than leading the good fight to improve the experience for our agreed upon primary persona, I understood that I was perceived by several critical stakeholders as forcing a dramatic change in interaction model to which they had not agreed and strongly objected.

Suddenly finding myself without consensus regarding the primary design persona, and lacking data demonstrating that the change I proposed would be acceptable to the existing administrative users, I decided that the most responsible thing for me to do was find a compromise design that would quell the stakeholder rebellion and move the project forward.

The first step towards this compromise was the addition of a means by which a savvy user could switch the experience from search to browse mode. While I accepted the political need for this adding this mode, I knew it probably meant my vision of using search to simplify the repository experience was out the window.

That said, I continued to hope we could still offer a simpler default display. Here, in the next working prototype, the initial view continued to look like the storyboards, including the results grouping:

However, when folders were exposed, a mode switch was also revealed:

If the user switched to browse mode, then the results display only showed objects located within the current folder (in this case no objects):

The search field and filters would continue to work in this mode, but would be of limited utility as they could only be used to refine the display of the objects within the current folder.

This prototype was a failure for two reasons. The first was that in playing with it, it became clear that the hidden mode switch was totally confusing. The second issue was that several stakeholders continued to object strongly to the idea that one had to opt-in to see the folders because to quote one “folders are first class filters.”

After presenting this prototype I realized my belief in compromise was overly optimistic. In fact, what was required on my part, given the political situation, was complete capitulation. I needed to accept that in order to close this project I had to re-set my fundamental assumption and accept that most stakeholder believed the primary persona for this project was the administrative user, not the business user I had been designing for.

Evolution not Revolution

With this insight, it was clear to me what I needed to do to get to a design all the stakeholders would accept: treat search as an incremental evolution of the existing experience rather than a revolutionary change.

The next prototype looked like this:

The functionality was the same as the previous prototype but the presentation was now designed to promote folders as the primary organizational structure. The interaction mode was still by default search, but the mode switch was made more prominent and visible at all times.

With this presentational change, stakeholder resistance evaporated.

Performance: the unkindest cut of all

Having accepted the shift in priority, I had made my peace with the new presentation when a final problem surfaced: performance. It turned out that for technical reasons having to do with how the repository database stored object meta data, creating the object type groupings was extremely inefficient. The user experience impact of this was that a search that attempted to bring back all types of objects and group them by type would take a very long time when the repository got large.

This discovery came very late in the the production process when no appetite remained for any further design changes. As a result we settled on the expedient solution of breaking the simple faceting model for object type. While the rest of the filters defaulted to some kind of ‘all’ setting, the one for type would default to a custom group named ‘Visualizations’ that contained the object types most interesting to business users: reports, dashboards, and analyses. You can see this change if you compare the screenshot above which shows a filter set to ‘All types’ versus the one below that shows the same filter set to ‘Visualizations.’

With this change we knowingly made it harder to understand the conceptual model underlying the filters. The only other short-term alternative would have been to abandon the grouping concept entirely, but, lacking any kind of ‘popularity’ algorithm (an earlier sacrifice to project schedule, deemed acceptable because of the grouping concept) this would have meant that by default searches would return arcane object types (e.g. JDBC connections) with the same priority as frequently used types (e.g. Reports).

Conclusion

The conclusion I draw from this post mortem is that the last lesson I listed above is really the most important: no pet projects.

The bottom line here is that, because I was personally invested in this project, I skipped or at least shortcut, a number of steps that I know are part of a successful design project. Probably the most critical error I made was to confuse the process of selling the project to stakeholders with the user and stakeholder research I should have done, but didn’t, before beginning the design phase. I compounded this error by relying on mockups, rather than prototypes, to communicate status until far too late in the development phase. Finally, I did not engage deeply enough with the technical aspects of the project to avoid being surprised by last minute performance issues.

Shifting to the bright side, the early feedback I’ve gotten from users familiar with our product is that faceted search does, in fact, materially improve the experience of finding and interacting with system objects. In other words, what we did achieve with this project was a tangible improvement in user experience compared to our previous product release. While my initial design aspirations were larger than that, this positive outcome raises the perennial question of design leadership: when is the incremental, but safe, value offered by evolution a more responsible strategy than the dramatic, but risky, value offered by revolution?