Abstract

Suicide is the second most common cause of death among young men in England and Wales. Gatekeeper training aims to reduce suicide rates by teaching members of the public how to recognise individuals at risk, intervene when necessary and direct them safely to professional help. In addition, the training aims to challenge taboo, promote open discussion about suicide and improve confidence to intervene with at-risk individuals. Gatekeeper training is currently employed around the globe, but the evidence base for its use is not yet fully established. Our aim was therefore to determine, using the currently available literature, if gatekeeper training is an effective suicide prevention strategy as analysed in the context of the Kirkpatrick model. The studies analysed may suggest a positive impact of training upon both trainee knowledge and, more tentatively, behaviour. However, this is by no means a universal finding and it is difficult to conclude whether or not improvements persist over time. The nature of the training used and the demographic targeted appear to be important, but variation in the standard of available studies as well as differences in their results make it difficult to draw firm conclusions. The authors therefore conclude that the currently available literature does not enable a reliable assessment of whether or not gatekeeper training is an effective suicide prevention strategy.

Introduction

There were 5,675 reported suicides in the UK in 2009 [1]. Suicide is the second most common cause of death among young men in England and Wales [2], and although suicide risk is known to be higher among those with mental health problems it is not limited to such groups. Approximately 10% of individuals aged 16 to 74 in Britain who do not suffer from a mental health condition have had thoughts of suicide at some stage in their life [3]. With the economic cost of suicide estimated at £1.7m per life lost [4], this makes suicide prevention an important public health issue with relevance to the entire population.

It is well established that, in general, individuals at risk of suicide do not tend to seek help when they most need it [5], making it necessary for others to do so on their behalf. It is around this rationale that training programmes have been developed which aim to equip lay members of the public, as well as various professionals that may encounter at risk individuals, with the key skills needed to recognise those at risk, perform a basic intervention and then assist those they identify in seeking professional help. These programmes are known as gatekeeper training programmes and are the focus of our investigation.

The two most common gatekeeper training programmes are QPR and ASIST. QPR (Question, Persuade, Refer) teaches individuals to recognise those that might be at risk, question them openly, persuade them to seek help and refer them to a mental health professional [5]. ASIST (Applied Suicide Intervention Skills Training) is a more advanced course run over two days and involves theory and role-play based teaching. In addition to teaching the skills found in QPR, ASIST encourages participants to help at-risk subjects explore possible reasons for wanting to live and develop a “SafePlan” (a contract to protect them until professional help can be accessed) [6]. The training further aims to challenge taboo, promote open discussion about suicide and improve confidence to intervene with at-risk individuals [6].

Gatekeeper training is currently employed around the globe [6] in an attempt to prevent suicide, but the evidence base for its use is not yet fully established. Our aim was therefore to determine, using the current available literature, whether there is an effect on the Kirkpatrick model [8] outcome measures (see methods) in the general population after gatekeeper training.

Methods

In order to assess the efficacy of a suicide prevention programme, studies would ideally be randomised controlled trials comparing the incidence of suicide in closed communities over an extended period in order to demonstrate a direct reduction. There are several problems with taking such an approach. There are issues surrounding suicide itself; reporting of suicidal acts is inaccurate, “suicidal behaviour” is difficult to define and suicide is statistically rare, requiring a large population to be studied over a long period. There are issues surrounding the intervention programmes; gatekeeper training by its very nature is indirect (as it does not target the individuals at risk, but those around them), the effects will not be immediate and it is not known what proportion of a population must be trained for the intervention to have an effect. Furthermore, influences on suicidal behaviour are numerous and thus any attempt to prove a direct causative link between suicide rates and gatekeeper training would be impaired by multiple, ill-defined confounding factors [7].

Given these limitations in adopting a randomised controlled trial approach - the gold standard - our evaluation is structured instead within the Kirkpatrick model [8]. Though imperfect, this is a widely used and accepted model for evaluating training interventions, considering efficacy at four levels with respect to the learner and their environment:

A literature review of accessible evaluations of the effectiveness of gatekeeper training published within the last ten years was performed, initially using an online literature library (PubMed) to identify relevant literature in addition to assessing papers cited by the training organisations themselves. The literature pool was narrowed down by the further requirement that appropriate statistical analyses be included in order for result significance to be determined. Whilst a randomised controlled trial would be the gold standard, this was rarely employed in the studies and thus lack of randomisation is a factor we have had to be aware of throughout our evaluation. Studies were intentionally included which analysed at different follow-up times in order to gain a broader insight into the effects of training.

Several papers included an evaluation of the “reaction” level of the Kirkpatrick model and very few of the “organisational change”; in neither case were reference points set a priori, or statistical analyses performed, thereby limiting analyses of the results as either positive or negative. It is for this reason that these outcome levels are not given lengthy consideration here. A further analysis of the qualitative data for these two outcomes would be a useful addition to our evaluation.

Results

Capp et al. 2001 [9]

In this study conducted in an Aboriginal community in Australia, 53 participants attended one-day gatekeeper workshops. Learning changes pre- and immediately post-training were measured by survey; training had a significant effect on knowledge, intention to refer to professional mental health services and confidence to identify suicide risk (p<0.05). Intention to help was not affected by training (Table 1). The study is only directly applicable to a specific Aboriginal community, particularly given the small sample size and lack of randomisation in choice of participants. In addition, the interpretation of these results is limited by the lack of follow-up.

Cross et al. [10] conducted a study evaluating the effect of gatekeeper training on staff and parents in US schools. The study compared the learning outcomes (as measured by survey and simulated interviews) pre-training, post-training and after a 3 month follow-up for two training programmes (QPR with or without behavioural rehearsal). Both types of training resulted in enhanced declarative knowledge, self-perceived knowledge and self-efficacy to intervene (p<0.001) which was maintained at follow-up.
In addition, the Observational Rating Scale of Gatekeeper Skills (ORS-GS: a scoring system based on communication, direct questioning, persuasiveness and referral) was significantly higher with behavioural rehearsal compared with gatekeeper training alone (p<0.05). However, there was a decrement over the 3 month follow-up for both groups.

Wyman et al. 2008 [11]

This randomised controlled study used multiple choice questions to study the effects of QPR training on staff in secondary schools. A significant impact on learning was illustrated, including improvements in staff knowledge, perceived preparedness and perceived access to services at one year post-training compared to pre-training (p<0.001). The largest effect was on staff with the lowest baseline levels.
In contrast to the effect on learning, the impact on behaviour varied with professional role. Staff with different roles reported significantly different baseline values for communication (defined as asking students about suicide when they suspected risk) (p<0.001), and there was no significant effect of training on this outcome in staff with the lowest baseline levels.
The main limitation of this study is that whilst the authors randomly selected subjects to approach initially, those who enrolled in the study did so voluntarily (n=249). The possibility of selection bias therefore cannot be excluded.

Doley et al. 2008 [7]

This study is an evaluation of gatekeeper training (ASIST) in Scotland. Participants were evaluated by survey pre-training (n=2000), immediately post-training and at 9 to 12 months post-training, and learning outcomes were measured.
According to self-report, some increase in knowledge from pre-training baseline persisted to the 9 to 12 month follow-up, albeit with a significant decrease (p<0.05) occurring between the initial post-training knowledge level and the 9-12 month knowledge level. However, this perceived maintenance of increased knowledge was not collaborated by results from multiple choice question assessment. The real impact of training on learning after 9 to 12 months is thus uncertain.
Furthermore, there was no significant difference between the number of people who answered “yes” to the question “it is harmful to engage in open communication with an individual at risk of suicide” between pre-training and at 9 to 12 month follow-up, despite an initial decrease which persisted to 3 months. This study clearly highlights the potential efficacy decline seen over time.

Williams et al. 2006 [12]

This study compared a group of trained individuals (n=373) pre- and post-training, and then further compared them to a non-trained group (n=249) in Virginian schools.
Some data from this paper suggests a positive impact of gatekeeper training. Training was demonstrated to have an impact on both learning and organisation when assessed at post-training and compared to pre-training levels. Individuals showed a significant increase in knowledge regarding suicide risk factors (p<0.001) as measured by questionnaire, and significant changes in school partnerships and “other [suicide prevention] changes” (p<0.05) were seen after training. At the level of behaviour, trained individuals made significantly more contracts with students not to harm themselves in comparison to controls (p<0.001). An additional finding was that the number of suicide attempts in trained schools was found to be lower in comparison to control schools (p<0.001).

However, some data opposed the positive effects of training. There was no change in the number of contracts made with students or in the overall number of students questioned about suicide risk between pre- and post-training within the test group. As compared to the control group, trained individuals referred fewer students to professionals (p<0.001) and “wondered if a student might be suicidal but decided not to question that student” more often (p<0.001).

In this study the changes seen after training are limited to knowledge and organisational change with no evidence for any behavioural change despite significant behavioural differences between the trained and untrained cohorts. The data is somewhat contradictory. Several of the reported significant findings were shown to be significantly different depending on the profession of the subject (teacher/counsellor) and the study did not perform the necessary analysis to investigate this finding further. The observation that trained individuals referred fewer students to professionals and failed to question about suicide risk when it was suspected suggests the possibility of a negative impact of training. In addition, it is impossible to place a great deal of weight on the analysis between control and test groups: the changes seen cannot be attributed to the training given as the groups were not randomised and the data is not supported by evidence from the within-group analysis of the trained individuals.

Public Health Agency (Northern Ireland) 2011 [13]

The PHA in Northern Ireland conducted an evaluation of the impact of forty-two ASIST courses delivered in Northern Ireland and the Republic of Ireland over one year. Impact was measured in the context of the Kirkpatrick model [8] with learner’s reaction, knowledge and behaviour being analysed using multiple choice and open ended questioning at baseline (n=718), immediately post-training (n=779) and at three to six month follow-up (n=313). To an extent, the authors also attempted to analyse how different professional backgrounds may impact upon the effect of training.

At the level of reaction MCQ scores measuring preparedness to intervene, confidence in ability to intervene and perception of the preventability of suicide were all significantly greater at three to six month follow-up than at baseline (p≤0.001). Knowledge, as assessed by MCQs, also increased significantly between baseline and follow-up (p≤0.001). The possession of a mental health qualification was associated with significantly higher baseline scores at the levels of learning and knowledge, but this significant difference in knowledge of the suicide intervention model and what action should be taken when faced with someone at risk was not present post-training.

Analysis of baseline behaviour showed that 81.7% of participants had previously been in contact with someone who they perceived to be at risk, and of these 77.6% had intervened. Among those who were exposed to someone at risk during the follow-up period, 94.9% intervened (p≤0.001). Again, the possession of a mental health qualification was associated with significantly greater baseline scores for both exposure to individuals at risk and likelihood to intervene, but during the follow up period a previous mental health qualification had no significant impact on likelihood to intervene once exposed to an individual at risk. Gender, age, nationality and working with vulnerable groups were also found to have no significant impact on likelihood to intervene during the follow-up period.

This study appears to highlight a number of significant benefits of training. However, there are a number of limitations on the interpretation of this data. The potential for results bias at follow-up is high as only 44% of those trained returned follow-up questionnaires, and decrements in knowledge over the follow-up period cannot be assessed as comparisons between scores at follow-up and immediately post training were not performed. Furthermore, the definition of an intervention varied between individuals and not all aspects of the intervention model were applied in every case. The authors found that of those who intervened during the follow-up period 91% asked about suicidal thoughts, 86% developed a safe plan, only 63% referred on and 73% followed up with the person at risk. The impact of profession and previous qualifications on the nature of the intervention performed was not analysed.

Discussion

We assessed, using papers from the currently available literature, whether or not gatekeeper training is an effective suicide prevention strategy as analysed in the context of the Kirkpatrick model.

Learning

Studies analysing learning demonstrated significant improvements across several measures: subjects’ knowledge of suicide risk [9,10,11,12] and their confidence in identifying it [9], their perceived knowledge and ability to intervene [10,11,13], and their confidence in being able to access support services [11] all improved. These increases were demonstrated in analysis of trainees pre- and post-training by either self-report or objective assessment.

It is important to determine whether this enhanced knowledge is maintained over time, but there are contradictory findings in the literature thus far published. Knowledge gains have been shown by objective assessment to persist after 3 months [10,13] and one-year [11] in three of the studies we have analysed, albeit with some decline in certain outcome measures. However, we have discussed one study which observed only the subjective perception of enhanced knowledge to persist; objective assessment by Doley et al 2008 found that neither declarative knowledge nor knowledge of the benefits of open communication with at risk individuals was maintained at 9 to 12 month follow-up [7].

These differences could be attributed to numerous factors. The studies analysed the results of different types of gatekeeper training, and therefore there may be intrinsic differences in efficacy. For example, ability to intervene was greater [10], and more “SafePlan” contracts made [12], in those who received additional simulation training. In addition, multiple choice question assessments were not standardised across the studies and thus direct comparison is difficult. A lack of participant standardisation may also contribute to discrepancies. Further study is required to elucidate whether there is persistence of learning outcomes over time.

Behaviour

Studies analysing behavioural changes report mixed results. For one study, within-group analysis pre- and post-training failed to demonstrate any significant behavioural effect on either of the outcomes measured: number of contracts made or number of students questioned [12]. The same study does report significant behavioural differences between the trained and untrained cohort, however they appear to demonstrate the behaviour of controls as more desirable: although they made fewer contracts, when risk was suspected they referred more students and failed to question students less frequently [12]. The authors suggested that this reflects increased confidence of trainees in their ability to manage someone at risk of suicide, and that trainees may be making judgements about risk without directly questioning the student. A higher awareness of suicide risk post training is an additional possible explanation. It is questionable, however, as to whether or not these outcomes are desirable as gatekeeper training is not intended as a substitute for professional help, and training emphasises the importance of open questioning [13]. The failure to question an individual when risk is suspected, or an attempt to manage an individual at risk without professional assistance due to increased confidence by a trainee in their perceived ability to do so, should be considered to be a potentially harmful effect and at a minimum suggests the need for greater emphasis during training on the necessity of questioning and referring in all cases. Furthermore, given that this difference is not demonstrated when the same cohort is compared pre- and post-training, it is difficult to know how useful these comparisons to controls are.

Another study analysed what trainees intended to do if confronted with a person at risk of suicide, comparing the cohort pre- and post-training. They used this to predict behaviour according to the "Theory of Planned Behaviour" [14], where intention is considered a precursor to actual behaviour. They observed a significant increase in intention to refer someone at risk to a mental health professional, but no significant change in intention to intervene [9]. This was partially supported by findings of a significant increase in “perceived preparedness to intervene” [11].

Conversely, another study analysed here did appear to find significant improvements in likelihood to intervene post training, with a significant increase from baseline observed in the number of trainees who attempted an intervention when confronted with someone at risk during the follow-up period [13]. Bias in results and confounding factors cannot be excluded as an explanation for this difference. Only 44% of trainees returned follow-up questionnaires [13], and a trainee with more experience of being exposed to an individual at risk may be more likely to encounter an opportunity to intervene during the follow-up period, and this greater previous experience may increase the likelihood of their attempting an intervention. This may represent a confounding factor during the follow-up period, and whilst the authors did demonstrate that a previous mental health qualification had no significant impact on likelihood to intervene post training, they did not perform the relevant statistical analysis to determine the impact of number of previous exposures (regardless of other qualifications). Differences in the training itself may also be a contributing factor, as may self-selection of trainees for training; among this cohort, 81.7% had previously encountered someone they perceived to be at risk of suicide [13] and the authors of the study argued that targeting training to those more likely to be exposed to individuals at risk was crucial. Variability in the nature of interventions performed during the follow-up period was also found in this study, where again not all interventions included making a safe plan or referring on to professional help [13]. This difference in endpoint definition may also contribute to higher perceived behavioural improvements.
One finding of interest is the suggestion that behavioural changes are not universally observed, but instead have been shown to differ significantly based on baseline levels of communication [11] and, perhaps consequently, profession [12]. This issue was not addressed thoroughly in these studies but should be an important consideration for anyone applying these programmes: exactly who do these programmes benefit? The study carried out by the Public Health Agency in Northern Ireland [13] found that the possession of a mental health qualification or working with vulnerable groups did not impact significantly on likelihood to intervene post ASIST training, but the authors did not analyse the impact of these factors on the nature of the intervention carried out.

Organisation

There were limited significant effects found on organisational changes within a trainee’s workplace, institution or community as a result of their training, for reasons discussed previously. Only one study reported such a finding; significant increases in school partnerships and “other changes with respect to suicide prevention” were observed in the three month post-training period in schools that had sent staff to gatekeeper training [12]. However, no significant increase in the number of new school policies or procedures was observed [12] and there are multiple confounding factors to be considered.

Further outcome measures

Only one study attempted to analyse suicide rates. It compared suicide rates in schools from which staff had been provided with gatekeeper training with those in schools from which staff were not trained. They found that trained schools had a significantly lower suicide rate (p<0.001) than those in which training had not yet been provided [12]. There are, however, severe limitations on the interpretation of this observation. The test and control groups were not randomised, and the large number of potential confounding factors (such as general pastoral care differences between schools, other school policies, suicide rates in each school prior to training and other events occurring within the school) were not controlled for. The authors acknowledged that whilst this data may be indicative of a desired effect, it is not sufficient to draw conclusions about causality.

Further limitations

There are a number of additional problems with the studies analysed. One of the major problems was a lack of randomisation; only one study randomised those initially approached, but those who enrolled in the study did so voluntarily. The possibility of bias therefore cannot be excluded [11].
All of the studies analysed here had difficulty with loss to follow-up, with the majority of trainees not reporting the impact the training had on them past the initial post-training assessment. This implies that there may be a degree of reporting bias in the results discussed.

As previously discussed, in all but one study analysed here no effort was made to control for the capacity in which subjects came into contact with potentially at-risk individuals, and yet the evidence suggests that this affects the outcome significantly [11,12]. The results discussed above should be interpreted with this in mind.

Future studies aiming to establish the benefits of gatekeeper training could be informed by these observations. It is likely that the ideal environment for a randomised controlled trial would be one that consisted of a relatively closed community in which utilisation of the training, referrals to professional services and suicidal behaviour over time could be monitored more easily. Those trained in gatekeeper skills should belong to a relatively homogenous group, and where differences in profession or baseline skills exist these should be compared so as to establish the impact of that variation on outcome measures. A large university or college setting may provide such an environment, and would be likely to enable better randomisation of test and control groups as well as help to reduce loss to follow-up.

Limitations of this analysis

Results are presented from articles, accessible online via PubMed, published within the last ten years, which included statistical analysis sufficient to assess the statistical significance of any changes. As such, numerous papers which attempted to consider the efficacy of training with regard to the “reaction” or “organisational” levels of the Kirkpatrick model [8] have been excluded from this study, as they failed to perform said statistical analysis. The exclusion of this subsection of literature likely results in bias of our reported results, but also supports the conclusion that the currently available research on this topic is inadequate in assessing the efficacy of these widely implemented programmes.

Conclusion

The studies analysed may suggest a positive impact of training upon both trainee knowledge and, more tentatively, behaviour. However, this is by no means a universal finding and it is difficult to conclude whether or not improvements persist over time. It is also impossible to assign causality for these findings for the reasons discussed above. Changes in the community post- gatekeeper training may occur, but evidence for this is severely limited. In addition, more information is needed on the differential effects of training with regards to a trainee’s background, and whether or not efficacy would be greater if training was more specifically targeted.

A large number of studies had to be excluded from our evaluation due to inadequate statistical analysis of results. Of the small number that met the required standard there was a wide range of variation in study design, confounding factors and indeed findings with regards to outcome measures. The authors therefore conclude that little can be drawn with certainty from the available data, and so despite the wide use of gatekeeper training programmes the currently available literature is insufficient for a reliable assessment of their efficacy. Further investigations are therefore required in order to establish the efficacy of gatekeeper training. In particular, a randomised controlled trial would be of benefit if it enabled better follow-up of trainees, their use of gatekeeper skills, the nature of the interventions they performed and the impact of those interventions on individuals at risk. The interaction between the nature of the training used and trainee profession or role should also be addressed.