Archives

Category: Usability Testing

Thank you to everyone who participated in the Summit on Sunday – there were several dozen people in attendance and it was a very productive session.

A transcript of proceedings can be downloaded here PDF (note: it’s about 150 pages of Skype chat, so only do this if you’re feeling brave/bored/exceptionally curious). We also tried to keep a document of record showing our agenda, some outputs and an incomplete list of participants, which you can see here. It is not entirely comprehensive and should be considered a working document only.

The discussion around the overall outcome for Structure continues and we will post an update soon.

The essential informationOur mission: to come to an agreement on whether the Site Building Tool should be included in D7 and if so, how we can best make it work for both the Drupal technical architecture and existing Drupal users AND for it’s primary target audience.

Add your name to the comments below so we can add you to shared Google documents (use your proper email address in the comment form so you can receive access notifications)

Be online at 16:00 GMT on Sunday 7 June (what time is this where you are? and with profound apologies to our Australian and NZ friends for such an unsociable time)

Spread the word! We want all kinds of people involved from Drupaller to People Who Should Use Drupal But Can’t, from developers to designers to shop owners and site administrators.

Our Moderator
I’m really excited to report that Ivanka Majic, Head of Design at Canonical, will be moderating our Summit on Sunday. Ivanka has a great appreciation for the challenges of designing for open source projects and a vested interested in D7UX being a good user experience, because the Canonical site is running on Drupal! She’s also a great moderator and I’m confident she’ll help us to ensure that we get to some firm and actionable outcomes on Sunday. Thanks Ivanka!

Homework
If you have the chance beforehand it would be great if you can check out the two most recent ‘prototype’ walkthroughs that we’ve posted (video see below), as well as the Project Framework Page for Structure.

A little background. If you’ve been playing along at home you have no doubt come across the Site Building Tool that we have proposed to live in the Structure section of the D7UX interface. We believe that the design and implementation of this tool can make the most significant difference in making it possible for non-technical users to make a reasonably sophisticated site using Drupal within a matter of hours, not the weeks or months that it often requires of new players at the moment.

The more time I spend talking to people who *should* be able to use Drupal but who can’t, and people who have had brief experiences with Drupal then ran away screaming in either frustration of fear, the more convinced I am of this. As recently as yesterday I took our latest prototype out to show some people, and despite the fact that it is still very rough and confusing, it’s potential was obvious.

We’ve been talking about this tool since early April – time is passing rapidly and if we want to make this happen we need to get on board with the concept and start working out how we can make it work. We need to do this as a matter of urgency. Our current channels are not moving us forward quickly enough, so I’m going to propose that we try something a little different.

Structure Summit Sunday

The absolute best way to get to the bottom of these complex issues is to get everyone in a room to thrash it out. Of course, we can’t all get in the same physical space together, so let’s try the next best thing and all meet this Sunday with the express purpose of coming to an agreement on what this tool can be and how we can make it happen.

I want to walk away from this session on Sunday with a clear vision for the tool and a feeling that the people who participated share this vision and our enthusiasm for the challenge ahead. (Of course, the flip side is that we decide that we can’t do it… which would be unfortunate, but I guess we have to consider it as an outcome).

I considered a bunch of ways to approach this, originally thinking that we’d do it in IRC, but with some more thought I’ve decided to create a public Skype chat that we can all join. This is a less intimidating environment for non-techy, non-Drupally people (and I’d really like to have a good mix of all of us in the discussion).

chances are you’re more likely to be in the target audience for this tool – your insight will be incredibly helpful in helping us get this right. (Especially if you can identify with the Jeremy character sketched out here)

you will be the people who will be able to keep us focused on the right user experience, so that we don’t get caught up in How Drupal Works and What Can Be Done

you can be instrumental in making Drupal an Awesome User Experience – this means putting some pretty powerful tools in the hands of people who can do amazing things with them.

you will win mine and Mark’s undying gratitude by helping us solve a really tricky design problem that will make a big difference.

As we start to get some outputs that are getting higher and higher in fidelity (that is, more like what they’ll be when they’re done), it becomes easier and easier for more people to take a look and give their feedback. This is great, but it’s also a challenge. Why so? Well, because, D7UX isn’t for everyone – and if it’s not for you, then in order to give feedback we can work with you need to be able to put yourselves in the shoes of someone it *is* meant for, or better still – find one of these people and try it out on them.

A well designed interface doesn’t work for everyone all the time, that’s just an impossible dream. It should work for it’s defined target audience most of the time though, and that’s what we’re aiming for.

So, who is D7UX designed for?

There are two main audiences that we’ve had in mind as we’ve designed this interface for D7.

Our ‘Clients’, the Content Creators – the first group we’ve broadly named ‘our clients’ because for many of the community who make sites with Drupal, these sites are then turned over to another person or group of people who use Drupal to add and maintain content on the website.Often these people share the following characteristics:
– They’re not developers. They don’t know much about code and how it works,
– They’re not in the Drupal community, they don’t love Drupal as much as we do, they don’t speak Drupal-ese,
– Updating the website is probably just one of many jobs they have to do,
– They’ve probably used other Web Content Management Tools than Drupal in the past.

These people tend to have a small number of tasks they need to do on the website, and those tasks tend to be repetitive.
The best thing we can do for these people is to make those tasks as simple, question free, and error free as possible.

Non-technical Site Builders – these guys are a whole other kettle of fish. They have a more advanced understanding of what technology can do and how code works, but they’re not developers. They are, however, considered technology or web experts in their field, which may be something like academia, or ‘social software’ or they may just be the person at their tennis club or church who ‘understands the web’. People expect them to be able to build fairly sophisticated websites and they’d rather like to be able to do it themselves too, except without the technical skills, it can be hard. You’ll probably find a bunch of these people currently using Ning.Often these people share the following characteristics:
– They’re not developers but they do understand quite a bit about how code works and may be able to write and understand a little code themselves
– They find themselves needing to create websites (often for others) that have fairly sophisticated requirements, including forms, community tools, online stores, and content publishing using a range of page templates.
– They have experience with a range of online content publishing tools.
– They have probably heard of Drupal but had little experience with it before and certainly don’t speak Drupal-ese

If you know anyone who fits one of these two descriptions then these people will be idea test candidates for the D7UX interfaces as they emerge. If you don’t then here are a couple of very sketchy ‘personas’ you should consider when evaluating the usability of D7UX.

Verity (our client, the content creator)

Verity is a part time administration assistant for a cancer support charity. She is working part time while she completes her Social Care Diploma. Before she left work to study, Verity was a Personal Assistant in the Finance Sector. Verity is in her late twenties. She is quite proficient with the MS Office Suite (Word, Excel etc.), and she uses Facebook and email but she doesn’t consider herself very good with technology.

One of many tasks Verity has to complete each week is to update the ‘news’ section of the website and any other small content updates that are given to her to make by more senior staff. She doesn’t mind this job but she is often interrupted by the phone and other staff while she is updating the website, which can make it tricky for her to remember where she is up to in the process. If she has any trouble with the website updating there is an IT consultant Verity can call, but she likes to avoid doing this unless something is very broken as the consultant is both very busy and expensive!

What Verity Wants: to be able to complete her website updates quickly, without getting confused, and feeling confident that she’s knows where she is and what she’s doing at all times.

Jeremy (non-technical site builder)

Jeremy is a social software consultant (no, he doesn’t like that job title either). He works with medium to large organisations to help them understand how social tools could help their businesses be more successful and he recommends tools that they should use and how they should be implemented.

Jeremy is often frustrated because it is difficult to customise existing tools to suit his clients. Also, he often has to use tools from many different places to put together the solution they require. He would love to have the ability to ‘build’ a site that would meet his client’s requirements, however his technical skills are not great – for example, he can edit some PHP code to get it to do what he wants, sometimes, but he can’t write much from scratch. He understands what CSS is and what it can do, but he’s never written much himself.

Jeremy has friends who are developers and professional designers and he can call on them for some assistance. He heard of Drupal a few years ago at a social networking event and tried downloading and installing it but didn’t get very far and his general reaction to Drupal now is ‘it’s scary’.

What Jeremy Wants: the ability to build sites with rich functionality and flexibility without have to write code.

Note: these are not formal personas, they are just quick ‘sketches’ drawn from the research we’ve done to date (since August 08) that we hope will assist you to evaluate D7UX interface design.

What about the developers?

Good question. We do care very much about the Developer Experience (DX) but we also know that many developers are perfectly happy with Drupal the way that is is (as long as they have the Admin menu installed) and that they, of all people, know that essentially all Mark & I are doing is ‘theming’ the admin and that they can override this with ease. We’re not doing anything to The Way Drupal Works that will inhibit their ability to do all the amazing things they do now, and more in the future. Having said that, we hope that they are pleasantly surprised by the interface and might even consider keeping it. Or, perhaps, at least part of it. And in particular, we hope this buys them more time to do what they’re great at – development work – rather than spending that time training and supporting people who know (and need to know) Drupal less well.

So, today is supposed to be our first day of Crowd Sourced Usability Testing – you might have noticed that we’re running a little behind on this, so to make up for the delay and lack of notice – and also because this is our very first go at such an activity, we’ve decided to try to make it a fairly simple task. You should be able to conduct this test in around 15-20 minutes. Please test as many or as few people as you can find the time (remember that you’ll need to allow some time for each person to share your results, which might take as much time again as your test!)

What are we testing:
If you have looked at some of the sketches that we have been working on over the past few weeks (some posted on this site and a lot on Flickr), you may have noticed that a ‘header’ has been quite a persistent feature. This is a concept that we are pursuing. We have a first draft of an ‘Information Architecture’ or navigation for that header. We would like you to help us test how well that navigation is working and where we need to make amendments. Based on these findings we will be able to further refine the navigation.

I have created a PDF document that includes the Header design we’ll be testing as well as a copy of the suggested script for your interview – you can download that here.

When are we testing:I’m going out to do my first round of testing this afternoon, but if you can find any time this week to do a test and submit your findings that would be great (that is by end of Friday 10 April) – we’ll have another task ready for you shortly after then!

Who are we testing:
For this round of testing your interview participants should have some familarity with Web Content Publishing Systems. They do not have to be familiar with Drupal, nor do they have to be Drupal developers. Anyone who publishes content to the web using some kind of content management system is appropriate for this round of testing.

Sample script for your interview:

Your interview should be divided into five parts:

Your introduction
– Start by thanking the interview participant for being involved in the project and telling them how much we appreciate their time.
– Explain to them that we are sharing the results of testing publicly, ask them for permission to record the session on video and to publish the video to the web. (Allow them to be unidentifiable/off camera if they prefer)
– Give them a brief overview of the project (we are trying to find a way to make Drupal a better user experience, especially for people who are not developers). Tell them it is very early in the process and that their feedback will play a big role in helping us get the design right.
– Reassure them that it is the design that is being tested, not them – if they don’t understand something, if something is unclear, if they don’t know the answer or feel they have made a mistake that is a problem with the design that we need to fix, not a problem with them as users!
– Encourage your participant to ‘think aloud’ – if they are considering the answer to one of your questions, to talk you through the how they are making the decision (this is what is most interesting to us – *why* people get to the decision they do, not necessarily which decision they make).

Background information of your participant – start with a simple question about your participant (if you know these already you don’t have to ask them, although it is a nice way to ease into the interview).
– What is your experience with managing content on the web?
– Do you have any experience with Drupal? (if no, what do you know about Drupal, if yes, describe experience)
– (If appropriate, Are you a Drupal developer?)

‘What If’ questions
We are going to work through the top line of the header navigation one item at a time and ask:
‘If you were to click on ‘Content’, what do you think would happen? What would you see?’
‘If you were to click on ‘Information Architecture’ what do you think would happen? What would you see?’
and so on.For each of the items, if your participant seems unclear as to what the navigation item means or refers to, be sure to ask them what the *think* it might mean, and why they think that. For example, we suspect the vast majority of people have never heard of ‘Information Architecture’ but we wonder whether they can make a reasonable guess as to what that section contains.

TasksNow we are going to run through six quick tasks. Again, pay as much attention to *why* your participant is doing / saying what they are saying as what they actually do and ask questions along the way, encourage them to think aloud. These tasks ask the participant to imagine that they have a company website that they are administering.
– Task 1. – You need to add a new staff biography to your website
– Task 2. – You need to remove an old staff biography from your website
– Task 3. – You want to add a Contact Us form to your website
– Task 4. – You want to give your new staff member access to publish content on your website.
– Task 5. – You need to make revisions to an article you published on the website this time last year.
– Task 6. – You need to make a new product category for the products listed on your website

Final feedback and thank youFinish by asking your participant to give you and further feedback they would like to on what they have seen today, then thank them again for their assistance with the project.

Some Tips for Interviewing

The best way to get good feedback from your participant is if they are relaxed and focused. It is always worth spending some time chatting at the beginning of the interview to build some rapport with the participant (that is, if you’re not already friends!), so that they feel comfortable. Even if they *are* your friend, you should make sure that they know that it is not them being tested, it is the design – never allow your participant to feel ‘stupid’ during an interview. If they are having trouble, ask them to explain what is troubling them, then help them out or move on.

Remember that the most important information we can get from these sessions is *why* people do and think the way they do, not necessarily what they do – continue to prompt them to think aloud and ask questions whenever you see them do something interesting or unusual, or if they are stuck or unsure – ask them *why* they aren’t sure what to do, and to explain what is troubling them and how they are trying to work it out. This information is gold!

Sharing and reporting your findings:

I’ve created a webform where you can submit your findings for each interview you do. You can find it here. (I had hoped to use Google Forms for this, but it turned out easier and faster to do using Survey Monkey. Hopefully in the future we can switch to Google so that the data can be shared with you all more easily.)

You will need to complete one form for each participant, although you’re welcome to just email me your findings if that is easier.

If you are able to video your interview, it would be great if you can post it to our YouTube group (or otherwise, let me know where you have posted it!). Note that YouTube only allows videos of 10mins or less, so you may have to edit your video or, easier still, break it in two as you record it.

This is experimental – your feedback welcome!

As mentioned above, this is the first time we’ve run an exercise quite like this, so there are probably a million ways we can do it better. Rather than labour over trying to get it perfect the first time (which we would never be able to do anyway!) we’re going to go for it, have a go, and see what we learn. You are our guineapigs and we appreciate no end your participation, but as a result, there may be something that don’t work as well as they should. To that end, please let us know what works well and what could be better, and how we might do things differently, and we’ll iterate our process so that the next time and the next time after that are continually improved!

Thank you again for your participation, and do let me know if you have any questions!
Leisa

If you took a look at our outline for the project process the other day, you’ll know that we’re taking an iterative and user centered approach to this project. That means that at regular intervals through out the project, as well as doing all this online stuff, we’re also going to sit down with real people in ‘real life’ and let them use our design work while we observe them and make notes about what is working well and what requires improvement. This is sometimes known as Usability Testing.

Now, Mark & I are going to be doing a bunch of this ourselves, and our friends in the Drupal Usability Group are also going to be helping out as much as they can (thank you!), and Jeff at Acquia will also be helping out – but (as we did with the d.o redesign), we’d like to invite YOU to help out with some usability testing as well!

Why on earth would you want to do this?

As I see it, there are three main reasons why you’d get involved in something like this:

you’d like to get some (or some more!) experience in usability testing. This is an opportunity get some experience under your belt, or find out what doing usability testing is like. You may have students or interns who need to get some experience, here’s a great project to keep them busy AND use their time to contribute to a great project.

you want to see for yourself how other people interact with the design. One of the main reasons that we do usability testing is that it is very easy to design for ourselves and often incredibly difficult to design for other people. People come to Drupal with all kinds of backgrounds and understandings, and very often what is clear as day for us is impossible to understand for them. (See the video we posted of Mark & I installing Drupal for some evidence to this point).

you love Drupal and this project and you desperately want to see it succeed: one of the key ways to reduce the risk of failure of this project is to get it out in front of people and see what they make of it. Not only does this ensure that we’re not designing and releasing something that is broken and unusable, it makes us feel confident that the decisions we are making are the right ones and that they will have a significant and positive impact on the user experience of Drupal.

As we move forward with this redesign and start to make some concrete decisions, we are going to have some tough discussions about ‘what users do’ – the BEST way to have these discussions is with *real* experience of our future end users, and not just sweeping statements about users in general. Help us build up that body of knowledge and inform yourself in the process.

How will it work?

As you can imagine, we don’t do this kind of thing every day. We tried this on the d.o redesign project and whilst lots of people voiced interest, the only person who actually did any testing was me! However, we still think this is a good idea and we suspect that, with a little more structure and a fixed schedule, it might work. So, here’s the current plan:

one week before Test Time we will release a description of the audience type(s) that will be appropriate for this round of testing. You should then recruit as many people who fit that description as you think you will have the time and/or inclination to interview/test. You should allow around an hour for each interview, but don’t schedule more than 4 or 5 in a day (trust me, your brain will melt and you will make a LOT of data to analyse)

a few days before Test Time we will release:

the Prototype that you will be using to test (starting off with PDF/Paper prototypes and moving onto web-based interactive prototypes as the project continues).

an Interview Guide that you will base your interview on, listing all the key tasks and issues we want you to cover in the interview

During Test Time (or as close to it as you can) you conduct your interviews. If you can (please!) record your interviews on video and post either the entire video or some highlights from it to the YouTube Group (or elsewhere if you prefer). We will also provide a place for you to log your test findings (TBC, we’re looking at a few options here, ping me if you have suggestions!)

We will then include your findings in our analysis of the design work to date and use it to inform our design decisions going forward.

The Testing Schedule:

We have penciled in some dates for ‘Testing Time’ throughout the project… these are subject to change, but if you are thinking of participating, perhaps see if one or more of these dates works for you:

6-8 April

15-16 April

27-29 April

18-20 May

1-3 June

29 June – 1 July

Need more help?

Over the coming weeks I’ll post some more information that will help provide support for those who aren’t seasoned researchers. Things I’m thinking of posting include: tips for a good interview, tips for recruiting, suggested software/hardware for recording/editing sessions… can you think of anything else that would be helpful?

Thank you!

I’m really excited about this exercise and I hope you are too. Please leave a note below if you’re interested in participating or if you have any questions, need more information, have any feedback on the proposed process. If you know of people in schools or companies who might be interested in participating, please send this on to them!