I work at a company where we design software that, in most cases, is targeted at leasing agents in the multi-family housing industry, a very specific group of people.

When I want to go about testing our UI with users, how important is it that I actually recruit leasing agents? The main issue here isn't necessarily finding them, but rather getting proper access to them (in other words, via their managers, which is slow and usually results in dead ends).

If I recruited more normal people "off the street", would the usability issues uncovered still hold? What might I be missing from a more representative group of users?

My goal is to make the recruiting effort easier if I can, because the amount of effort I'd need to put in currently is prohibitive to testing.

Just a tip - I've worked with products for estate agents myself, and I've found it a lot easier to do remote user testing (with something like GoToMeeting or a screen sharing program) than arrange physical visits. It's much easier to arrange and you allow your users to pause the sessions when calls / customers come in etc.
–
Jimmy Breck-McKyeMar 7 '13 at 10:09

3 Answers
3

If I recruited more normal people "off the street", would the
usability issues uncovered still hold? What might I be missing from a
more representative group of users?

Some would. Some would not.

There are some issues that are pretty universal. If the text size is too small everybody will have problems reading it. If phone number field forces you to remove spaces and hyphens everybody will find that annoying.

What you'll miss is any domain related knowledge. For example things like:

Yes this process is slightly opaque - but I need to do it N hundred times a day so faster is better than clearer

Everybody knows what this acronym means

Being able to take all this information in at a glance is more important than an aesthetic display with lots of whitespace

This bit of information is really important to my problem, and this other bit isn't.

... and so on...

In short - testing with people "off the street" is better than not testing at all. But you will undoubtedly miss important issues that apply to people in your specific domain.

In one of the comments you talk about "friction" you encounter in the recruitment process. I think asking a question about ways to overcome that might be a more fruitful route.

In my honest opinion, it is best to test the app on a set of actual to-be users. This must be backed by administration. On the other hand, if the corporate mechanics are in they way, I'd personally find a to-be user and befriend him/her to do it on I-scratch-your-back-you-scratch-mine basis.

In any case, the tester must be familiar with the business logic and other specifics of the company/group-definition, thus you can't just put any random person to test the app. That would work only for user friendliness when the users are general public, but on a specific group of people you really need a real to-be user.

[EDIT]
You'd be missing a number of factors if you don't use the representative group:

Business logic specific important parts of the software won't be noticeable to normal users. Some parts will be used extensively by the real users and will need simplification and additional UI elements to ease the use, regular users won't notice the needed change because they don't know how much they would use the particular interface.

For the same reason, will get better feedback from real users to change some specific parts, which in turn will let you know what you will need to focus on more.

That feedback in return will also allow you to familiarize yourself more with business logic and day to day activity from the user, which will lead to a better software in the end.

I highly doubt that any usability issues uncovered by the "off the street" user will hold much value to the real users. At most it will change some few minor issues, but they won't uncover the real headaches that will come in production. "off the street" users will uncover some facts and you'd be happy fixing them, and you'd feel accomplishment which goes to waste when it comes to production.

Yes real users would be better, but you don't answer the main part of the question: "If I recruited more normal people "off the street", would the usability issues uncovered still hold? What might I be missing from a more representative group of users?"
–
JonW♦Mar 7 '13 at 8:47

In the end, it's all about simplifying day to day processes, "off the street" users don't and won't know the day to day processes, you can't expect a gardener to tell you what's wrong or right with the new plumbing tool.
–
fuximus foeMar 7 '13 at 8:59

If you can't get people from the real estate industry to even test the app, how are you going to get them to buy it? It sounds to me that the issues with your app might start up the chain a bit, particularly in what problems your app is trying to solve for these users. If you've approached them and explained how the app can help them and they've shown little interest, perhaps they don't have the problem you think they do?

Testing an app that you can't get in the hands of the actual consumers is what you'd call wastage in the lean startup world. Not to say that the app is wastage, because I don't know what it is or much about the real estate agent world, but you might have already learned a key thing and that is that your app isn't resonating with it's intended user base in it's current form.

Sorry, I should have included a little more background. The company I work for has been established for over 10 years now, and it's used on a daily basis by several thousand users. So, finding users isn't the issue, but rather the friction I run up against in the process.
–
Dave LucianoMar 6 '13 at 23:53

Ah ok well I would probably get pretty bored with an interface where I don't understand the context of the functionality it's offering me (I.e real estate terms and processes). My opinion is that you should be trying to test with users who do have a background with the problems the app is solving. Can you offer incentives for test users, perhaps subsidised pricing for a while? Or maybe you could try the opt in model where you can indicate to current users that they can try the new interface out and it simply a case of pressing a button to switch over?
–
Chris PaynterMar 7 '13 at 0:04

I think that you can't make an assumption about whether someone will buy an application based on whether they're willing to test it. Professionals are very busy, and user research on its own doesn't have an immediate benefit to it. It's my experience that the more highly-skilled an audience is, the more that you have to pay them to take part in your user research to make it worth their time.
–
nadyneMar 8 '13 at 0:17