Actual v simulated use in human factors testing of medical devices

Before
medical devices can be used by the public they need to be tested,
either with actual patients or through the use of simulation. Richard
Featherstone discusses the advantages and pitfalls of these two types of
testing.

Simulated use human factors testing is an
established requirement for many types of medical technologies, in
particular those devices that are used in higher risk scenarios such as
operating rooms and intensive care units.

However, a recent trend
is for manufacturers to be asked by the regulators to provide human
factors data based on actual use in addition to simulated use. This is a
relatively new development, and seems to be coming from reviewers
within the FDA who appear to be applying a ‘clinical trials’ mindset to
human factors study requirements. There is only passing reference to
them in guidance, and methodologies are not widely understood.

What is ‘simulated use’ human factors testing?

Testing
your product during simulated use involves recreating the scenarios in
which you anticipate that your device would be used. The closer you can
get to reality, the more naturalistic the behaviours of the study
participants. Creating a realistic environment may involve setting out a
room to simulate an operating room, including lights, drapes, visual
and auditory displays, and using a mannequin to simulate the patient.
The newer mannequins can be programmed to simulate particular medical
conditions and exhibit symptoms such as a particular breathing rate,
oxygen saturation and pulse. Users will interact with the device in a
naturalistic way and perform use scenarios that are designed to simulate
the frequent use scenarios for your device.

Simulated use testing
applies the rigours of a study methodology in a way that does not
endanger patients but allows just enough realism to generate reliable,
representative human factors data. However, it has the drawback that it
is obvious to the user that there is no actual patient involved, and
therefore no risk of harm. How much this affects user behaviour is a
matter for some discussion, but it is undeniably less realistic than
actual use.

What is ‘actual use’ human factors testing?

Actual
use involves your intended patient using the device to receive the drug
or have the procedure performed on them. Actual use of your product can
obviously be done during clinical trials before it is launched. If you
are asked to conduct an actual use human factors study as part of your
regulatory file, it will in practice mean that you will need to gather
human factors data during a clinical study. An actual study of your
product once launched usually means conducting some type of post-market
survey of your users.

How do the two types of study compare?

The
two types of human factors study generate human factors data with
distinct differences, and the methodologies involved differ too. Some of
the key differences are:

Sources of bias. Bias is introduced
into simulated use studies because the participant and the study
moderator both know that there is no real patient involved, and
therefore there is very limited potential for harm. Bias is introduced
into actual use studies when an investigator intervenes to collect human
factors data, for example by the use of diary cards recording use
difficulties, or electronic monitoring technologies to track device use
in real time.

Use scenarios. A simulated environment enables the
study moderator to simulate very precisely a use scenario and to gather
data points that lead directly from it. Because we are not putting a
human being in harm’s way, we have more scope for gathering human
factors data in ways that may be impossible during actual use. In
simulated use studies we can recreate a massive trauma scenario quite
precisely, and reproduce it exactly from one participant to the next and
thus give the data some rigour.

Linking usability to clinical
outcomes. This is clearly where actual use testing wins over simulated
scenarios. A clinical study is designed to generate data on clinical
outcomes and involves real users using the device with real patients in a
controlled way. When the product being tested has a user interface, and
therefore requires the user to perform specific tasks, there is
obviously a potential to link the use of the product to the clinical
outcomes for the patient.

Whilst actual use studies can
potentially link to clinical outcomes, simulated use studies can get
part way there too. For example, if a series of ‘surrogate markers’ of
clinical outcome can be defined, there is the potential to provide data
that provides some link to outcomes. If a simulated use study can show
that the full dose of drug was delivered in a way that could reasonably
be expected to lead to clinical benefit, then that data is of some
value. Or if you can show the time taken for a healthcare professional
to respond to a medical crisis, and if there is a proven link between
time to treat and outcome, then again simulated use human factors data
provides value.

The moderator in a human factors study has a
crucial role in avoiding bias, and there is a very fine line between
guiding the participant through the tasks and intervening and therefore
biasing the data. I have not yet met a clinician who would also regard
themselves as a human factors expert, and the most practical solution is
for both types of investigator to work together to a common agreed
protocol.

So can actual use and simulated use studies work together?

There
are some fundamental differences between human factors studies and
clinical studies that make it very difficult to see how one study can
satisfy the requirements for both clinical outcomes data and usability
data.

For example, let’s imagine you are developing a new type of
inhaler device for diabetes. You have an open label clinical study in
which patients use the inhaler daily for a month, and you then collect
data on certain clinical outcomes such as blood glucose levels.
On
enrolment into the study, you will need to teach the patient how to use
the inhaler correctly so that every patient starts off with a comparable
technique and you have a baseline against which to compare their
technique later in the study.

At the first follow up visit, you
ask your patient to demonstrate their inhaler technique. If their
technique is poor, it is going to cause bias because their technique has
deviated from the baseline you established at the start of the study,
plus it is not ethical to send your patient away knowing that their
technique is poor and that they are likely to struggle to use the
device. However, you could record their difficulties and explore the
reasons for the difficulty, but you must train these out again, and send
them away with an improved technique.

Now we come to the end of
the study, and you bring the patient back into clinic for all the
battery of clinical tests you want to run such as blood tests, quality
of life and of course, an assessment of their inhaler technique. But
since you have been correcting the use-related difficulties along the
way, how much rigour will there be in any claimed correlation between
the user’s performance with their inhaler and the clinical outcomes?

Can technology help?

For
pre-launch products, the only opportunity that manufacturers have to
gather actual use human factors data is to include human factors data
collection during clinical trials. One of the problems presented by
clinical trials is how to evaluate the use of a product when it’s being
used at home unsupervised.

What should the clinician do if the
patient has difficulties using the device? The ethical choice is to
correct the user’s technique, but this may introduce bias. If the
patient’s technique is not corrected, does this introduce bias into the
clinical outcome?

There are some technologies that may help, such
as remote monitoring of devices using inbuilt electronic monitors, small
video cameras placed in the patient’s home, and tele-health systems
that require users to record data on their product’s usage. We have used
some of these technologies in our studies. None are perfect, and none
give the rigour that a simulated use study would. However, with smaller
and more discrete sensors that may be invisible to users, it may be
possible to collect reliable data during actual use.

The best
quality human factors data can only be generated by a well-designed
simulated use human factors study. Actual use studies are probably
necessary only where a simulated use scenario is not feasible or would
not give results that could be applied to the wider user population.