Interventions Directed to Consumers

Searching for and Selecting Systematic Reviews

Searching for Systematic Reviews

One reviewer hand searches The Cochrane Collaboration’s database of systematic reviews and the Database of Abstracts of Reviews of Effects (DARE) to identify all systematic reviews of interventions designed to educate, communicate with, inform, support, involve and seek the participation of health care consumers. Cochrane reviews and DARE reviews are screened by title and by abstract, where available.*

Selection Criteria

Systematic reviews specifically related to evidence-based prescribing and drug use are selected from this larger pool of reviews and ranked for relevance using a set of criteria. These criteria were developed based on detailed consideration of interventions of obvious high relevance (e.g., interventions to influence drug adherence) and of less direct relevance to consumer drug use as well as prescribing (e.g., the role and use of medicines in broader self-management programs for specific diseases). The level of relevance is based on a ranking of the focus of the review and interventions on evidence-based prescribing and drug use, from “exclusively focused” to “could include a focus.” A distinction is also made between interventions that were aimed at consumers directly and those that affect consumers indirectly through structural, organisational or financial changes.

Relevance

Review level

Intervention level

Outcomes

LEVEL I – Direct Consumer Interventions

HIGH

Main objective of review focuses exclusively on evidence based drug use by and prescribing for consumers

Interventions in the review target consumers and exclusively focus on evidence based drug use by and prescribing for consumers

Outcomes are related to drug use by and prescribing for consumers (searched for and/or found and/or reported)

MODERATE

Main objective of review includes a focus on evidence based drug use by and prescribing for consumers and it is specifically identified (can include all types of treatments but includes drug)

Interventions in the review target consumers and include a focus on evidence based drug use by and prescribing for consumers and it is specifically identified in the review

At least one outcome is related to drug use by and prescribing for consumers

(searched for and/or found and/or reported)

LOW or VERY LOW

Main objective focuses on all aspects of care which could include evidence based drug use by and prescribing for consumers

Interventions in the review target consumers and could focus on medication issues for consumers but it is not specifically identified in the review

At least one outcome is related to drug use by and prescribing for consumers

Main objective of review focuses exclusively on evidence based drug use by and prescribing for consumers

Interventions in the review target consumers and exclusively focus on evidence based drug use by and prescribing for consumers

Outcomes are related to drug use by and prescribing for consumers (searched for and/or found and/or reported)

MODERATE

Main objective of review includes a focus on evidence based drug use by and prescribing for consumers and it is specifically identified (can include all types of treatments but includes drug)

Interventions in the review target consumers and include a focus on evidence based drug use by and prescribing for consumers and it is specifically identified in the review

At least one outcome is related to drug use by and prescribing for consumers

(searched for and/or found and/or reported)

LOW or VERY LOW

Main objective focuses on all aspects of care which could include evidence based drug use by and prescribing for consumers

Interventions in the review target consumers and could focus on medication issues for consumers but it is not specifically identified in the review

At least one outcome is related to drug use by and prescribing for consumers

(searched for and/or found and/or reported)

OR

No outcomes are related to drug use by and prescribing for consumers

Using these criteria, systematic reviews are screened by title and abstract and categorised as high, moderate, low or very low relevance, or no relevance to prescribing and drug use. Reviews identified as being of no relevance are excluded.

Reviews identified as being of some relevance to prescribing and drug use are then categorised according to the criteria in two steps. In step 1, all Cochrane reviews in the subset that are identified as having some relevance to prescribing and drug use are obtained in full text. The reviews are then independently categorised by two reviewers as either “direct to consumer” or “indirect to consumer,”. Differences in categorisation were resolved by discussion. Cochrane reviews of high and moderate relevance are included in CADTH’s Rx for Change database.

In step 2, all DARE reviews identified as having some relevance to prescribing and drug use are independently categorised by two reviewers, based on the title and abstract. Reviews are classified as high relevance to prescribing and drug use, or as of lower relevance. Lower relevance reviews are not categorised further. The full-text copies of all potentially high-relevance reviews were independently assessed by two reviewers to confirm relevance. Differences in categorisation were resolved by discussion.

To optimise the breadth and comprehensiveness of the available evidence while minimising duplication, an additional set of selection criteria were developed and applied to the DARE reviews that are identified as high relevance. A DARE review is excluded from the database if:

The review has a clear and significant overlap (in the scope of the review, and in the studies included and excluded) with Cochrane systematic reviews already included in the database

The review is of low quality or had serious methodological flaws according to the Centre for Reviews and Dissemination’s assessment of the review published as part of the DARE abstract, or as assessed by the reviewers.

For each potentially eligible review, the methodological quality is assessed using AMSTAR (A MeaSurement Tool to Assess Reviews)

The AMSTAR tool consists of an 11-item checklist where reviews score one point for each criterion met, for a maximum score of 11. Reviews that achieve high scores indicate a higher methodological quality than those that achieve low scores.

Two reviewers independently assessed each AMSTAR checklist item and total for each review. The results for each review are compared and disagreements are resolved by consensus *. Reviews rated to be of moderate to high methodological quality (AMSTAR scores of 4 or more) qualified for data abstraction and analysis, and are summarised and included in the Rx for Change database.

Reviews rated as of low quality (AMSTAR scores of 3 or less) do not qualify for data abstraction. However, the bibliographic details of these reviews are collected and made available through the Excluded Reviews page of the Rx for Change database.

Differences in categorisation are resolved by discussion between the two reviewers.

In this way reviews can be excluded based on:

relevance - where a review is of limited/insufficient relevance to the overall scope of this database either because the interventions in the review are not aimed directly at the relevant populations OR only a small proportion of the included studies in the review have relevant interventions and outcomes.

quality - where a review is of low methodological quality, with an AMSTAR score ≤3 or

overlap - where a review has been updated OR the topic is more comprehensively covered by another review.

Reviews that were excluded on the basis of relevance, overlap or quality are listed in the excluded studies page.

Limitations

By choosing high-relevance reviews in which the main objective was evidence-based prescribing and drug use, the reviewers can ensure that the effects of the interventions are focused on drug use and prescribing for consumers. Consequently, many of the reviews identified and included in this database to date focus on adherence to medicines. Whereas the reviews are likely a reflection of the clinical trial literature, it should be recognised that adherence is only one component of the interventions affecting how, when, and why people take or do not take their medicines. Inclusion of moderate-relevance Cochrane reviews has helped to expand the focus of interventions included in the database and, as the number of reviews assessing interventions targeted at different aspects of prescribing and drug use for consumers increases in the future, may further expand the range of interventions.

* For the 2013 April update, a single reviewer performed the data screening, two reviewers independently applied selection criteria to identified reviews, and one reviewer undertook extraction activities, with some assistance from a second reviewer. Quality assessment was also performed by a single reviewer, with some assistance from a second reviewer. Reviews scoring AMSTAR 3+ were assessed by a second reviewer and differences in scoring between reviewers were resolved by discussion or third party adjudication if necessary. For the 2014 April update, we employed electronic search strategies to search for evidence. The titles and abstracts of all retrieved citations were screened by a single reviewer with some assistance from a second reviewer. Reviews meeting the inclusion criteria are listed on the database as pending further assessment and have been assigned to the intervention categories they address.

Synthesis, Presentation, and Rating of the Evidence

Outcomes

The prescribing- and medicines-related outcomes are categorised broadly, according to the four categories used for the overall database: concordance, appropriate use, cost containment, and other. For the consumer component of the database, the “other” category is expanded to account for unique consumer outcomes that are not adequately captured under the other three categories. This wider set of outcomes is adapted from a comprehensive taxonomy of outcomes developed by the Cochrane Consumers and Communication Review Group that is designed to map outcomes of relevance to people’s interactions with health care (available at http://www.latrobe.edu.au/chcp/cochrane/resources.html).

Statements of Effectiveness

Standardised statements rating the evidence are used to describe the results and conclusions of the reviews. These statements were adapted from those developed by the Cochrane Consumers and Communication Review Group to standardise, synthesise, and classify evidence across diverse disease groupings and across systematic reviews where interventions are complex and diverse.

Summary Evidence Statements for Rating the Evidence in Reviews

Summary Statement

Translation

Sufficient Evidence

There is sufficient evidence that [intervention] improves/increases/etc. [outcome] - it is generally effective

There is evidence to support conclusions about the effect of the intervention(s) in relation to a specific outcome(s). This includes evidence of an effect in terms of either benefit or harm.

Statistically significant results are considered to represent sufficient evidence to support conclusions, but a judgment of "sufficient evidence" is also based on the number of trials/participants included in the analysis for a particular outcome.

A rating of "sufficient evidence" is often based on meta-analysis producing a statistically significant pooled result that is based on a large number of included trials/participants. The intervention would also be "generally effective".

This judgment may also be made, based on the number of trials and/or trial participants showing a statistically significant result; e.g., (in a narrative synthesis) a result for specific outcome in which 12 studies of a total of 14 showed a statistically significant effect of an intervention would be considered to represent "sufficient evidence". The intervention would also be "generally effective".

Some Evidence

There is some evidence that [intervention] improves/increases/etc. [outcome] - it is generally effective.

OR

There is some evidence that [intervention] improves/increases/etc. [outcome] - the results for [intervention] were mixed.

There is less conclusive evidence to make a decision about the effects of a particular intervention(s) in relation to a specific outcome(s).

This may be based on narrative synthesis of review results. In this case, the result is qualified according to the findings of the review e.g. "some evidence (5 studies of 9) reported a positive effect of ..." would be classified as "some evidence" with "mixed effects".

This may also be based on a meta-analysis, with a statistically significant result obtained in a small number of trials; or, a statistically significant result obtained from trials, with a small number of participants; or, a statistically significant result obtained from studies of low quality.

Insufficient Evidence

There is insufficient evidence to support the use of [intervention] - it is generally ineffective.

OR

There is insufficient evidence to determine if [intervention] is effective.

There is not enough evidence to support conclusions about the effects of the intervention(s) on the basis of the included studies.

This should be interpreted as "no evidence of effect", rather than "evidence of no effect".

Statistically non-significant results are considered to represent insufficient evidence.

Where the number of trials is small, and/or the number of participants included in the trials is small, "insufficient evidence" might reflect under-powering of the included trials to be able to detect an effect of the intervention.

Where the number of trials is large, and/or the number of participants included these trials is large, "insufficient evidence" and "generally ineffective" may reflect underlying ineffectiveness of the intervention to affect the outcomes being examined.

Structure of Database

Systematic methods are employed to construct the database (such as selecting reviews and summarising data) to maximise transparency and minimse bias. However, summaries of systematic reviews can only present the main findings and characteristics of reviews; for example, major characteristics of populations in which interventions have been studied and limited descriptions of the interventions themselves and of other key elements of the review. Database users are strongly encouraged to consult the full reviews in addition to the summaries of the reviews included in this database.

For each review summary, both the number of included studies (as noted in the review) and the number of studies specifically related to evidence-based prescribing and drug use (and therefore relevant to the database) are indicated. Each summary also provides a link to the list of individual primary studies included in the analyses of a systematic review. The number of the studies listed and the actual number of links to studies in the list are sometimes unequal. Possible reasons for this discrepancy include:

One study can be published in multiple articles (all citations were included in the list).

Multiple studies or comparison arms can be described in one article.

Citations were not provided for studies not available or accessible in main databases such as PubMed, EMBASE, CINAHL or PsycINFO.

In poor-quality reviews, not all studies in the analyses were clearly identified or cited in the review.