Mobile Testing Toolbox, Part 2

A survey of mobile testing tools for unmoderated usability studies finds products and features that suit many different design needs.

Article No :1106 | October 7, 2013 | by Amber DeRosa

This is a continuation of a two part article on mobile testing tools. In Mobile Testing Toolbox: Part 1, I talked about tools that can be used for moderated remote usability testing and for recording in-person mobile usability studies.

In part two of this article I will share my knowledge of unmoderated mobile tools. Unmoderated tools are good for collecting behavioral data about what people are doing with a mobile website or an app.

If a researcher wants to know why people are doing certain things on a mobile site or app they should use the moderated tools discussed in my previous article.

Please note: this is not a complete list of all mobile remote usability testing tools. It is just a list of tools that I have reviewed. These tools are constantly being updated with additional features and functionalities, so please visit the tool’s website for the most up-to-date information.

An Overview Of Features By Tool

What Tools Can I Use For Unmoderated Remote Usability Testing?

Unmoderated remote usability testing tools provide users with a link to a website evaluation. The website evaluation includes set of tasks that the user can complete at any time. The task instructions serve as the usability test moderator. A survey can be shown after the tasks to gain additional information. Userzoom and userlytics are two products on the market that can provide unmoderated remote usability testing.

UserZoom

UserZoom is an enterprise level software tool that enables you to conduct unmoderated remote usability testing and integrate with a suite of other mobile tools, such as mobile Voice of Customer (VoC) and real-time recruiting from your mobile sites. Mobile VoC is a feedback feature that allows users to comment on various portions of the mobile website. The comments are connected to a page or feature and can be sorted and analyzed. Real-time recruiting allows the researcher to intercept a user who is browsing on the site and invite them to participate in a survey or an online study. UserZoom is a quantitative tool that records analytical data such as clickstreams, time-on-task, and completion rates, offering a video session replay. (The video session is a screen capture with no picture-in-picture or audio.) The user accesses the study by downloading an app and using the app to access a mobile website.

Email invitation to download app, allowing for screening of certain types of users

Clickstreams (no tagging), time-on-task, and net promoter scores

Video session replay and the ability to create video clips

Mobile surveys and voice of client

Limitations

Mobile websites only; no app capabilities

User must install UserZoom app

No picture-in-picture or audio in video

Videos stored on UserZoom servers

Userlytics

Userlytics is a more qualitative tool that allows for unmoderated remote usability testing. The tool provides video files to analyze and annotate. This tool does not provide any analytical data such as clickstreams and time-on-task, it only provides the raw video file of the session. Userlytics requires the user to use their mobile device as well as a laptop and webcams—one pointed at the mobile device and a second to record the picture-in-picture of the user’s face. Again, similar to UserTesting, this setup can get complicated, therefore Userlytics offers the use of a panel of users who have their systems configured to be able to access any remote mobile studies. If using a panel for a study is a viable option for your business, the tool can be a good fit. If not, recruiting your own users is possible, but it requires you to send your users a webcam they will have to configure to their systems in a specific way.

If you test with your own users, they must have laptop, iOS device, and webcam

Apps only, no mobile website capabilities

In order to ask follow up questions after the tasks, you must either create a zip file with a presentation or ask the questions after the recording part of the videos

The videos are stored on the Userlytics servers

What Tools can I use for Analytical Data?

Unlike research tools where you have to recruit participants, analytical data tools run in the background for everyone who accesses your site or app. All of these tools are for apps only and additional code must be built into your app’s SDK.

Appsee

Appsee (formerly UserVOD) is an analytic tool that provides both quantitative data as well as qualitative video files of the user’s interaction with your app. Reports include touch analysis heat maps, unresponsive gestures, and crash records. The unresponsive gestures report can help you understand what parts of the screen users are tapping, swiping, or pinching that are not currently active. This can help identify ways to improve usability by potentially enabling gestural interactions that users expect. In addition to its usability benefits, the app has some quality assurance features because it includes crash records. Video files are stored on Appsee’s servers, but they have the ability to set private screens that will not be recorded to ensure user privacy.

In order to preserve data useage the app has an option to select if the sessions should only be recorded and uploaded through Wi-Fi

Limitations

iPhone or iPad only; Appsee is developing an Android solution they hope to have ready soon

Apps only, no mobile website capabilities

Appsee code must be built into the app SDK

The videos are stored on Appsee servers

Delight

Delight is an analytics tool that provides qualitative video files of the user’s interaction with your app. Delight also has the capability to capture picture-in-picture and audio by having the user consent to a recording of their session. Delight does not provide quantitative reports of the data, but it provides a dashboard to watch recordings of specific user flows that can enable you to gain insights from actual user sessions. Video files are stored on Delights servers, but Delight has the ability to set private screens that will not be recorded to ensure user privacy.

An option to select if the sessions should only be recorded and uploaded through Wi-Fi or not

Ability to track events

Limitations

iPhone or iPad only, Delight is developing an Android solution they hope to have ready soon.

Apps only, no mobile website capabilities

Delight code must be built into the app SDK

The videos are stored on Delight.io servers

Swrve

Swrve is an A/B testing tool that measures the differences between two variations of an app. The tool is tailored to marketing or return on investment (ROI) tasks such as revenue, conversion, user retention, and engagement, but can also be used to determine the best designs for improving traffic or usability as well. Swrve has the ability to target specific users for A/B tests by using their interface to send the versions to specific users.

Arise

Similar to Swerve, Arise is an A/B testing tool that measures the differences between two variations of an app. For example, it provides data that makes it possible to decide between button color, font size, and text labels. It can show you which version of the app has the best results based on pre-determined goals such as return on investment, new account creation, or traffic to certain pages. Arise is still in the beta version of its product, although they claim to be production ready.

Works on iOS or Andriod; mobile app

Features/Functionality

Mobile app A/B testing tool

Simple, clear graphs allow you to instantly see which version of your app is driving results

Can export test data to other software programs for viewing

iOS and Android

Limitations

Apps only; no mobile website capabilities

Arise code must be built into the app SDK.

I hope this article was educational and helpful in distributing the knowledge of what tools are available to assist with mobile usability testing. I expect it at least serves as a good overview and starting point for researchers looking for tools in the mobile space.

About the Author(s)

Amber DeRosa is a senior usability engineer working for a large financial company. She has tested over 150 mobile users across nearly 20 mobile studies, examining native iOS and Android applications as well as responsive websites. Since 2009, she has adapted her mobile testing techniques to maximize learnings gained from every test session. To ensure usability tests positively impact the business and our users’ experience, she regularly presents test findings with executive audiences throughout the company. Beyond testing, executive stakeholders and product owners look to her for consultative guidance on mobile best practices and heuristic feedback. She was the founding Vice-Chair of PhillyCHI a local chapter of the ACM SIG CHI and has been an active member of the local Philadelphia UX community since 2004. Feel free to follow her on twitter at @ajderosa

Usertesting and Userlytics are similar products. I see there is a mistake in the limitations section of the article above. Userlytics does allow both app and website testing like Usertesting. Usertesting allows you to easily ask follow up questions and Userlytics has a bit of a roundabout way to do it. Usertesting has an hour turnaround time and Userlytics states that it is 24 - 48 hours in their FAQ. Usertesting has UX researchers available for analysis if your company needs analysis support. Hope this clarifies.

Yaw

November 8, 2013

Thank you for good information.
I have a question.
What is the different with 'Usertesting.com' with Userlytics?
First one is Moderated.
And the second one is Unmoderated.
But I don't know what is difference.

Bijal Shah

October 23, 2013

Hey there! Great article! I'm part of the UserZoom Marketing Team and there are a couple of comments (one is actually an error) in this article that I'd like to highlight, if I may:

UserZoom's VOC tool does work with native apps. Customers are able to insert the invitation layer into the application and track time and clickstreams of their native apps. See more info here:

http://www.userzoom.com/mobile-testing/ (bottom of the page)

Moreover, companies who want to perform mobile task-based usability testing on native apps can prototype and host native app interfaces for testing purposes, but we do need prototypes to be hosted to test them with this product. Some HTML5 native applications (hosted apps that use native app wrappers) can be tested with UserZoom.

Then one more comment:

"Unmoderated tools are good for collecting behavioral data about what people are doing with a mobile website or an app. If a researcher wants to know why people are doing certain things on a mobile site or app they should use the moderated tools discussed in my previous article."

We'd disagree with this statement since UserZoom allows for behavioral data collection (both quant and qual) and allows researchers to ask why participants did want they did when they were either successful or unsuccessful via our conditional survey questions.

Thanks for your comment and clarification Bijal. “If a researcher wants to know why people are doing certain things on a mobile site or app they should use the moderated tools discussed in my previous article.” Perhaps I overgeneralized here; most of the tools in part 2 do not have survey questions as a feature. You are correct you can start to get at why people are doing certain things using survey questions.

To me VOC is a separate product than the tool that I was reviewing. The ability to work with native apps is in reference to the unmoderated remote usability testing tool, not any other products in your suite of tools. One of the benefits of Userzoom is that you offer a full suite of products that is why I mentioned the other tools. Although it is a possible work around, I don't consider a prototype of a mobile app the same as the ability to test a native app.

I appreciate you commenting on the article. I want to ensure it is as accurate and helpful as possible.

Dan Turner

October 7, 2013

Thanks for the article!

Though I'd love a heads-up or best practices on usability testing for mobile apps that require two users to interact (like Bump); I know many testing companies have no protocols for that.

Gregry Anderson

October 7, 2013

Your first link in the article is wrong. It takes you back to current page instead of part 1.