Products work successfully on two levels. Firstly, they function correctly, so when you perform action A expected effect B takes place every time. Anybody can test whether something is working in this way; in fact, we often use machines to do this kind of testing. The results are objective: the product either works under certain conditions, or it doesn’t. You don’t need product users to do this testing: the developers can (and should) do it. Let’s call it functional testing.

The other measure of success is whether the product helps its users achieve something (or things). This is less objective. Figuring out the user’s requirements, and whether the product helps meet them, involves judgement. The user’s experience of the product is even more subjective and difficult to measure.

Only users can perform user requirement tests as developers experience the product in a different way. Let’s call this user testing.

All obvious stuff, but in my work I’ve yet to come across a product provided by a third party that is actually tested by users. Functionally tested to various degrees, yes; but actually tested on users – no. This results in bad products:

They’re engineered so that they technically ‘work’ even though ‘working’ may involve you having to do 6 months of training, reading a 7,000 page manual and going through 673 excruciating steps that take 92 hours, when it should only be 3 steps, taking 2 minutes. Gerry McGovern, How does price affect the “user” experience?

When I asked one provider whether they’d actually tested their product on users, they replied by agreeing that yes, this sounded a good idea. When they upgraded the product they gave us a huge list of objective tests to sign off, thereby even shifting the functional testing to us.

Worst of all, product development is driven by product buyers voting on what they want developed. Actual users are never asked, or observed. We can’t raise bugs based on poor user experience.

This also results in bad products.

Us buyers can mitigate this problem by writing user testing into our tender and request for proposal documents. Don’t provide a long check list of technical requirements – any vaguely modern design process will include a functional requirements research phase, and providers with no UX expertise can easily work through a check list.

Instead, do some user research before writing your request for proposal. Find out what your users want from the product and build a loose spec around that. Let the developer establish what the product needs to do in order to help users succeed, and let them do the functional testing. If it doesn’t work functionally, file a bug. Most importantly: judge agency proposals on how well you feel they can meet your set of users’ needs.