end user experience

Sometimes running load, usability, functional and UI testing separately is not enough, as it operates on certain sub-set of variables, assuming the others to be static. It’s like projecting a cube in 2d. This is why one of the tests I like to do is “End User Experience” testing: simulating a real user, performing a real set of tasks.

Preparation:

1. Choose a few transactions or scenarios most commonly performed by the users. Say, if I did this type of testing for WordPress.com, I would probably choose “Add New Post”, “Search on site” and few more.

2. Define an overall goal for each transaction. It’s best if the goal is close to what typical real user would do. For example: if an average post length on WordPress is about 240 words, tested transaction “Add New Post” may have an overall goal of creating post with 240 words.

3. Break transactions into steps, and define data for each step: what exactly will you do during the transaction? How will you navigate from step to step? Which options, features, shortcuts you will use? And so on. Since there’s usually more than one way to accomplish the same task, defining those actions is very helpful for the analysis: it takes away the guessing game of “how did I actually accomplish it?” and it also allows to later concentrate on some transactions that are seems problematic. For example: in order to add new post, I may go to Dashboard, or I may just click a “New Post” button from the top menu. My final results may be different depending on how did I accomplish it, and thus it’s important to remember which way it was done.

At this step, we have something like the following table:

Transaction

Goal

Actions & Data

Add New Post

Create post with 240 words

Navigate to Dashboard

Click Add New in Left-side menu

Provide title (4 words)

Type 100 words

Provide link with 10 words

Type 130 more words

Click Publish

etc.

etc.

etc.

Testing

There are many ways to perform this test, for example:

Single “experienced” user runs the designed test in normal (not too fast, not too slow) pace, noticing time it took him or her to accomplish different steps of the testing, different inconveniences (was the scrollbar present? Was font too small?) and issues.

Same “experienced” user runs the same test, but this time automatic load test is running on background.

Same as above, but this time let “novice” user to run the test (how fast he or she will discover how to accomplish steps? How much time the mistakes this user will make will cost him or her? Will their wrongdoing cause any additional problems?)