Figures

Abstract

Background

Previous reviews show that reporting guidelines have improved the quality of trial reports in medicine, yet existing guidelines may not be fully suited for social and psychological intervention trials.

Objective/Design

We conducted a two-part study that reviewed (1) reporting guidelines for and (2) the reporting quality of social and psychological intervention trials.

Results

(1) We identified 19 reporting guidelines that yielded 147 reporting standards relevant to social and psychological interventions. Social and behavioural science guidelines included 89 standards not found in CONSORT guidelines. However, CONSORT guidelines used more recommended techniques for development and dissemination compared to other guidelines. (2) Our review of trials (n = 239) revealed that many standards were poorly reported, such as identification as a randomised trial in titles (20% reported the information) and abstracts (55%); information about blinding (15%), sequence generation (23%), and allocation concealment (17%); and details about actual delivery of experimental (43%) and control interventions (34%), participant uptake (25%), and service environment (28%). Only 11 of 40 journals referenced reporting guidelines in “Instructions to Authors.”

Conclusion

Existing reporting guidelines have important limitations in content, development, and/or dissemination. Important details are routinely missing from trial publications; most leading journals in social and behavioural sciences do not ask authors to follow reporting standards. Findings demonstrate a need to develop a CONSORT extension with updated standards for social and psychological intervention trials.

Funding: SG holds a linked Clarendon Fund-Green Templeton College Annual Fund Scholarship to support his studies and research. EMW and PM have accepted a grant (no reference number assigned at time of submission) from the Economic and Social Research Council (ESRC; http://www.esrc.ac.uk/) to develop a CONSORT extension for complex psychological and social interventions. GJMT holds a Marshall Scholarship to support his studies and research. The authors thank the Centre for Evidence Based Intervention (Oxford), the Centre for Outcomes Research and Effectiveness (UCL), and the National Collaborating Centre for Mental Health (NCCMH) for internal support. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests: SG, EMW, and PM are currently involved in the development of a CONSORT extension for social and psychological interventions (http://www.tinyurl.com/CONSORT-study). This does not alter the authors' adherence to all the PLOS ONE policies on sharing data and materials.

Introduction

Research in disciplines such as public health, psychology, education, social work, and criminology often involves complex interventions to improve health and related outcomes. Randomised controlled trials are increasingly used to evaluate these interventions and to inform decision-making in evidence-based policy and practice. However, these complex interventions have several unique features, such as multiple, interacting components (see Box 1)[1] that complicate critical appraisal of trial quality (e.g. risk of bias). Moreover, these interventions are often delivered in environments that are difficult to control and to measure, which makes reporting and interpretation of external validity (i.e., generalisability) difficult.[2]

High quality reports of complex intervention trials are important to diverse groups of stakeholders, including researchers, journal editors, funding agencies, practitioners, policy-makers, and research participants. These research consumers depend on accurate, complete, and transparent reports to appraise the validity and generalisability of trials. To address these needs, researchers and journal editors have developed reporting guidelines[3] that highlight key information about internal validity, external validity, and knowledge transfer of trials (e.g., locating trials in databases, assessing conflicts of interest). Reporting guidelines should consist of reporting standards (i.e., recommendations about the content that authors should consistently and transparently report) that are based on previous research and developed via expert consensus using rigorous, systematic, and transparent methodology.[4], [5]

The Consolidated Standards of Reporting Trials (CONSORT) Statement and its extensions are the preeminent guidelines for reporting trials. CONSORT is based on empirical evidence and expert consensus about biases related to trial validity.[6] Since its launch in 1996, CONSORT has had a considerable impact in the biomedical sciences; numerous reviews in the biomedical literature have shown an association between improvements in reporting quality and these guidelines.[7], [8]

Despite improvements in the completeness of RCT reports, major deficiencies in reporting quality still exist,[8] indicating that further actions are needed. For example, while CONSORT guidelines are well-known in the social and behavioural sciences, there is less evidence of widespread uptake and implementation in these disciplines compared with biomedical disciplines. Several studies also indicate that deficiencies persist in the reporting of social and psychological intervention trials.[9], [10], [11], [12], [13] A common explanation is that current standards in prominent reporting guidelines are not adequately tailored to these trials. For example, the CONSORT Statement and its extensions have primarily focused on standards related to internal validity, but researchers are increasingly interested in the applicability of trial findings and have called for updated standards to improve the assessment of external validity.[14], [15], [16], [17], [18], [19] For example, researchers have asked for more information related to process evaluations, such as intervention theory of change, assessment of intervention mechanisms during the trial, and relevant information about the influence of trial context.[14]–[16] To determine whether a new reporting guideline is needed, it is necessary (i) to assess the suitability of current reporting guidelines for social and psychological intervention trials and (ii) to investigate the quality of reports of these trials.

Objectives

Following recommended techniques for guideline development and dissemination,[3] a structured approach to reporting guideline development should begin with a needs assessment that (i) reviews whether an adequate guideline already exists for a given research method and (ii) obtains evidence of the reporting quality of published research using that method.[4] Though highly informative, previous reviews have not investigated the characteristics and methods of development of reporting guidelines specifically for social and psychological intervention trials. Moreover, previous reviews about the reporting quality of these trials have consisted of small samples and have assessed reporting quality according to a narrow set of reporting standards.[12], [13], [20]

We conducted a two-part study that examined:

the content, development, and dissemination of current reporting guidelines; and

the current reporting quality of social and psychological intervention trials across several disciplines according to a comprehensive set of reporting standards.

Methods

Eligibility Criteria

For the first part of the study, a reporting guideline had to consist of a published, peer-reviewed article that introduced a formal, itemised checklist of reporting standards relevant to trials of social and psychological interventions. In order to identify all published and potentially relevant reporting standards, quality assessment tools (e.g., tools designed to be used for critical appraisal) were also eligible. For practical reasons, we limited the search to guidelines available in English.[5] We excluded guidelines for the design and conduct of trials rather the reporting of trials, as well as tools pertaining to a specific intervention focus that is unrelated to social and psychological interventions (e.g., acupuncture, complementary medicine).

For the second part of the study (i.e., the review of trial reporting quality), a trial report had to discuss a randomised experiment of a complex intervention with psychological, social, or health outcomes. We excluded trial reports that: (i) described process evaluations without trial outcomes, (ii) evaluated only cost-effectiveness, (iii) used randomisation to balance order of exposure to conditions that were experienced by all participants, or (iv) explicitly evaluated medical or pharmacological interventions. No other eligibility criteria were used.

Search Strategy and Study Selection

For the first part of the study, we used an adapted version of a peer-reviewed electronic search strategy[21] to identify relevant reporting guidelines (see Text S1). We also searched three registries of reporting guidelines: the EQUATOR Network library of identified health research reporting guidelines (www.equator-network.org), a recent review on the development and contents of reporting guidelines for health research,[21] and a systematic review of studies assessing the quality of conducting or reporting trials.[22] We also searched references of all eligible guidelines identified through this process.

For the second part of the study, we conducted a hand search of journals' Table of Contents throughout the year 2010. From the ISI Web of Knowledge 2010 Journal Citation Reports (JCR) for Social Sciences, we identified journals publishing trials of complex interventions in clinical psychology, criminology, education, and social work. To obtain an extensive sample of trials, we searched the 10 journals with highest impact factors in each field (40 journals total) that published trials in the year 2010.

Data Abstraction

We first examined the content of reporting guidelines by compiling reporting standards from all identified guidelines into a comprehensive, non-redundant, itemised list of standards (see Appendix S1).[6] To assess the quality of reporting guideline development, we compared the techniques used by guideline developers to recommended techniques,[3], [21] which were organised according to four phases of process: preliminary work, development of the guideline itself, publication, and dissemination activities (see Appendix S2). One reviewer (SG) assessed whether guidelines adhered to each standard.

We assessed guideline dissemination in several ways. Akin to previous studies,[4] we performed a full-text review of each journal's “Instructions to Authors” to identify references to guidelines for reporting trials (e.g., instructions on the journal webpage, mention of a reporting guideline) and whether journals required authors to register trial protocols before recruiting participants. For each guideline, we also counted citations through November 2012 using Google Scholar, which provides a wide measure of impact across most publication mediums.[23] If a guideline was published in multiple journals or included an official explanatory document detailing how to adhere to its reporting standards, we combined the citations for all documents.

To assess the reporting quality of identified trials, two reviewers (SG and GJMT) independently assessed whether trial reports adhered to each standard in our comprehensive list of relevant reporting standards (Appendix S1). As the goal was to identify potential limitations in both guidelines and reporting quality, we used a comprehensive checklist to assess trial reports according to all published and potentially relevant reporting standards rather than a single instrument (such as the CONSORT Statement). Coding rules were adapted from previous studies about trial reporting quality.[9], [24]

Before assessing the entire sample, the reviewers coded one trial report in each discipline and compared results to ensure consistent application of coding rules. The two reviewers then each coded the entire sample. Discrepancies in judgment were resolved through discussion and consensus. Using SPSS version 18, inter-rater agreement prior to discussion and consensus was calculated as κ = 0.71, indicating substantial agreement.[25] Data resolved after discussion were used for the final analyses.

Data Analysis

Similar to previous studies,[9], [24] we analysed guideline content by mapping identified reporting standards onto standards included in the CONSORT Statement in order to organise the checklist according to the common sections of a trial report (i.e. introduction, methods, results, and discussion). We also noted any reporting standards that are not in official CONSORT guidelines but were found in other guidelines. We summarised adherence to recommended techniques for reporting guideline development as frequencies,[3], [21] and we converted total citations of each guideline into median citations per year. Data about the development and dissemination of guidelines were compared by the following pre-specified types of reporting guideline: official CONSORT guidelines, non-CONSORT guidelines for medical sciences, or non-CONSORT guidelines for social and behavioural sciences.

To describe the quality of trial reports, we summarised adherence to reporting standards as frequencies.[21] We analysed compliance to reporting standards for the whole sample and by academic discipline to provide a preliminary view of differences in reporting across social and behavioural sciences. We also categorised reporting standards into a priori conceptual themes often targeted by reporting guidelines: internal validity, external validity, and other important study details (e.g., information for indexing and certain ethical concerns).

Results

Previous Guidance

Through the literature search (see Figure 1), we identified 19 unique, eligible reporting guidelines and reporting quality assessment tools (see Table 1) developed between 1980 and 2010 (median 2004).[6], [9], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42] Six were developed by the CONSORT Group for reporting RCTs; six were non-CONSORT documents for health-research trials in general; and seven were specific to research in the social and behavioural sciences, namely non-randmoised trials of public health interventions,[38] empirical research in education,[27] empirical research in psychology,[41] trials in criminal justice,[9] outcome studies of alcohol treatment,[26] trials in occupational therapy,[24] and the content of behavioural change interventions.[39]

Overall, CONSORT guidelines used recommended techniques for guideline development and dissemination more frequently than non-CONSORT guidelines in medical, social, and behavioural sciences (see Table 2). Notably, most CONSORT guidelines tended to use more rigorous consensus methods in the development stage (75% of recommended techniques) compared with medical guidelines (44%) and social and behavioural science guidelines (37%), such as formal consensus development processes (see Table S1). Most CONSORT guidelines adhered to most dissemination activities (77%), such as endorsement and adherence by journals, while most other medical guidelines (10%) and social and behavioural science guidelines (34%) did not. In addition, CONSORT guidelines were cited more often (74 citations per year) than other guidelines in medicine (10) or social and behavioural sciences (4).

Table 2. Average percentage of recommended techniques for guideline development by document type.

doi:10.1371/journal.pone.0065442.t002

The 19 included reporting guidelines included a median of 32 reporting standards (interquartile range (IQR) = 17 to 54; range = 3 to 201) From these, we developed a list of 147 non-redundant reporting standards that are relevant to social and psychological interventions (see online Appendix S1). Of these 147 reporting standards, 89 were either not included in CONSORT guidelines or were tailored versions of CONSORT standards for social and psychological interventions (see Table S2 for a full list). Amongst these standards, requests for details about setting, implementation of the interventions, data collection, generalisability, and ethical concerns were common.

Assessment of Reporting Quality

Only 11 of the 40 journals referenced a published reporting guideline in their “Instructions to Authors” section (see Table 3). Two journals provided advisory text about reporting certain aspects of intervention studies without reference to any published reporting guideline; no other journals provided any textual instructions specific to reporting trials. Only 5 journals required trials to be registered in a trial registry (e.g., clinicaltrials.gov) prior to publication.

From these journals, we identified 239 eligible trials (Figure 2), including between 1 and 39 per journal (median 3). Overall, trials reported a mean of 42% of all reporting standards; there was low compliance with reporting standards related to internal validity (38%), external validity (47%), and other study details (34%). Reporting quality did not vary substantially by discipline (Table 4).

Several important aspects of trials were not consistently reported and would be easy to include in all trial reports (see Table S3; Data File S1). Only 20% of reports identified the trial as randomised in the title, and only 55% identified the trial as randomised in the abstract. Overall, 60% of reports included the trial eligibility criteria, but the majority of these reports did not explicitly list all inclusion and exclusion criteria. Trial reports adhered to only 23%, 17%, and 15% of the standards related to random sequence generation, allocation concealment, and blinding respectively. While most reports (71%) included the number of participants randomised to each condition, few reports described other aspects of participant flow through the trial, such as the number of participants: eligible for the trial (33%), receiving treatment (31%), and included in the primary analyses (38%). Less than half of the reports reported primary outcomes (27%) or secondary outcomes (45%) sufficiently to be included in meta-analyses. Very few reports (5%) indicated that the trial had been registered, and few reports included information about a trial protocol (8%) or access to a treatment manual (40%). Reports adhered to 50% of standards related to the implementation of the intervention and included a mean of 28% of standards related to the context of the wider service environment (see Figure 3).

Discussion

Overall Findings

Results establish the need for a new reporting guideline. This review identified numerous guidelines that have made useful contributions to reporting medical and social research. However, this study demonstrates that current reporting guidelines are insufficient for social and psychological intervention trials. Compared with the CONSORT Statement and its official extensions, guidelines in the social and behavioural sciences have not consistently followed recommended techniques for development and dissemination,[3] and they have not been widely utilised. If not properly developed and disseminated, these guidelines are potentially of limited use and are less likely to improve reporting of key features of trials that are important to stakeholders.[5] However, these guidelines include important, tailored standards for social and psychological interventions that are not found in CONSORT guidelines. Due to the substantial variability of recommended standards across reporting guidelines, disseminating CONSORT or another guideline would insufficiently address social and psychological intervention trials; further work is required to improve the applicability, utility, and acceptability of reporting guidelines in disciplines outside medicine.

Our analysis of trial reporting quality suggests that trial reports often fail to comply with published reporting standards, including well-established standards in the CONSORT Statement and its extensions. While reporting quality varies across standards and disciplines, this review shows that most trial reports omit information that is necessary to assess internal and external validity. This finding is consistent with previous studies of reports of social and psychological intervention trials in specific disciplines.[9], [10], [11], [12], [13] Poor reporting also has serious implications for knowledge transfer. For example, reports that are not identified as randomised trials in their titles or abstracts may not be identified in electronic literature searches and may be omitted from reviews as a result. The development and dissemination of a tailored reporting guideline could help resolve these problems.

Strengths and Limitations of the Current Study

This study is the most comprehensive review of reporting guidelines and the reporting quality of social and psychological intervention trials ever conducted. We undertook a highly sensitive search for reporting guidelines and assessed their use across numerous journals in several disciplines. We also conducted a complete assessment of all trial reports in 40 leading journals in one year, double coded their reporting quality, indicated clustering of reporting quality by journals within disciplines, and utilised a comprehensive set of standards to prevent selective assessment and reporting of quality.[8] While the reviewers weren't blind to the authors, institutions, and journals of RCT reports due to resource restraints, there is currently no evidence to suggest that such lack of blinding the validity of these reviews assessing reporting quality.[8]

It is clear that reporting guidelines for trials are not widely used outside medicine, but there may be several reasons for this. Regardless, social and behavioural scientists have been aware of the CONSORT Statement and its extensions for some time, so lack of uptake is not the result of ignorance of these guidelines.[14], [15], [43] Our correspondence with journal editors confirmed that many are familiar with CONSORT and some related guidelines.

In assessing compliance with reporting standards included in these guidelines, we found that several standards are vague and underdeveloped, particularly those related to external validity, such as theory of change.[14] When standards are imprecise, reports can be compliant without describing evaluations sufficiently to allow critical appraisal, replication, and inclusion in reviews and meta-analyses. Moreover, though inter-rater agreement in the review of reporting quality was high (κ = 0.71), it did not reach newly-developed criteria (κ≥0.80) for assessing the validity of evaluations of RCTs reporting quality.[8] Our own difficulty in applying some standards reaffirmed the need to develop clear, specific recommendations for social and psychological intervention trials based on best current evidence.[18]

Despite the difficulties in developing a comprehensive set of reporting standards, deficiencies in trial reports are both real and important. We included trial reports that are most likely to be cited (i.e., those published in high impact journals) and which may be of better quality than articles published in low impact journals.[4], [44] The reports assessed are probably representative of the best trial research in these disciplines.

Future directions

A reporting guideline designed specifically for social and psychological interventions would help improve the quality of these trial reports.[43], [45] To be acceptable and widely utilised, such a guideline should be developed using rigorous methods that engage members from all relevant stakeholder groups during development and dissemination, and its reporting standards should be based on sound empirical evidence where possible.[3], [5] Given the prominence of CONSORT internationally, the precedence of its standards, and the rigorous development and dissemination practices of the CONSORT Group, an official CONSORT extension seems the best method to facilitate better reporting of these trials.

This study identified many new and modified reporting standards that could be added to the CONSORT Statement to form an official extension. Several standards in current CONSORT guidelines could be amended to make them more applicable and acceptable for trials of social and psychological interventions. For example, modifications could attend to difficulties in: blinding participants and providers of complex interventions, participant and provider preferences, the use of multiple measurement formats (e.g., self-report, observation) within a study, and the complexity of data analysis.[9], [14], [33], [34], [41], [46] In addition, researchers are increasingly demanding better reporting standards related to external validity, theory of change, and implementation.[1], [14], [19] Standards in guidelines other than the CONSORT Statement include relatively more information about sample characteristics,[35], [47] the extent to which trials differ from usual practice,[27] details about facilitative or obstructive aspects of the trial context,[48] and contextual factors related to feasibility and coverage,[14] such as organisational resources and the wider service system structure.[49], [50] Such information is important to improve the knowledge base for effective transfer of research findings to real-world settings.[51] Details of trials not related to internal and external validity are also important, such as discussing other relevant research when interpreting trial findings,[52] referencing other reports about the trial that may have a different focus (e.g., process evaluations),[20] and issues related to conflicts of interest (e.g., researcher development of the intervention) and ethical considerations (e.g., informed consent by participants with limited mental capacity).[9]

These reporting standards should be considered through consensus methods, such as a Delphi process and formal consensus meeting.[3], [4] In addition to the standards identified in this review, there may be other factors that have not yet been included in relevant reporting guidelines that could emerge using a rigorous consensus processes. Given the plethora of possible reporting standards, a formal consensus development process would best ensure that new guidance incorporates collective wisdom while providing only the minimal, essential standards for reporting these trials.

Implications

The CONSORT Statement has been extended and modified in the past, and the CONSORT Group welcomes further extensions.[53] CONSORT guidelines have been developed and validated in the context of biomedical treatments; their applicability to other disciplines could be improved by accounting for specific methodological issues related to the assessment of social and psychological interventions. Members of previous CONSORT groups, journal editors, and researchers believe that stakeholders need to be included in guideline development to promote buy-in and to improve the relevance of CONSORT guidelines to disciplines outside medicine.[12], [54] This review demonstrates that a unified set of standards could be applied to social and psychological intervention trials. Moreover, the impact of CONSORT and the recent proliferation of publications about reporting quality in social and behavioural sciences indicate that such a CONSORT extension could be well-received by various stakeholders.

Since the conduct of this review, an international collaboration of stakeholders has convened to develop a new CONSORT extension for social and psychological interventions. This CONSORT extension has the potential to benefit this area of research in several ways. Developed and disseminated according to recommended techniques,[3] it will aim to synthesise previous work on reporting standards and methodological research about social and psychological interventions. This guideline could improve the reporting and utility of these trials for various stakeholders, including trial report authors, systematic reviewers, journal editors, peer-reviewers, funding organisations, research students, and users of research in policy and practice. While trials are not the only method for evaluating interventions, nor are they the only method that can benefit from updated reporting standards,[50] the importance of trial reports is growing. Improved reporting is needed so that judgments can be made about the validity and application of research findings.[41] A CONSORT extension for social and psychological interventions would be an important step towards improving the reporting of these trials.

Acknowledgments

Data sharing: All data from this study are available as an online supplement to this article.

Copyright: The Corresponding Author has the right to grant on behalf of all authors and does grant on behalf of all authors, a worldwide licence to the Publishers and its licensees in perpetuity, in all forms, formats and media (whether known now or created in the future), to i) publish, reproduce, distribute, display and store the Contribution, ii) translate the Contribution into other languages, create adaptations, reprints, include within collections and create summaries, extracts and/or, abstracts of the Contribution, iii) create any other derivative work(s) based on the Contribution, iv) to exploit all subsidiary rights in the Contribution, v) the inclusion of electronic links from the Contribution to third party material where-ever it may be located; and, vi) licence any third party to do any or all of the above.

Author Contributions

Conceived and designed the experiments: SG EMW PM. Performed the experiments: SG GJMT. Analyzed the data: SG EMW PM. Wrote the paper: SG EMW GJMT PM.

52.
Durlak JA, DuPre EP (2008) Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. Am J Community Psychol 41: 327–350.
doi:
10.1007/s10464-008-9165-0