BACKGROUND: Systematic reviews are considered the best design to synthesize all existing information of a given research topic. To date, there is no study that investigated the quality of reporting of systematic reviews relevant to physical therapy published in Portuguese. Objective: To analyse the quality of reporting of systematic reviews in the field of physical therapy published in Portuguese by using the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analysis) checklist. METHOD: All systematic reviews published in Portuguese that were indexed on PEDro database up to August 2011 were included. The quality of reporting of the eligible papers was analysed by using the PRISMA checklist. Each quality assessment was performed by two independent reviewers with arbitration of a third reviewer if necessary. RESULTS: A total of 37 systematic reviews were identified. These studies were published between 2003 and 2010. Less than 30% of the PRISMA checklist items were satisfied, being most of the items related to the introduction and discussion sections. No improvements over time were observed. CONCLUSIONS: Most of the studies did not satisfy the items from the PRISMA Checklist. It seems that most of authors did not know the existence of this checklist. The implementation of reporting statements such as the PRISMA statement by Portuguese-written journals is likely to help authors to write their systematic reviews in a more transparent and clear way.

Keywords: systematic review; editorial policies; physical therapy.

Introduction

Physical therapists that aim to be updated on the effectiveness of interventions commonly deal with a challenge: how to manage the large volume of scientific articles? Currently, there are nearly 20 thousand studies related to the efficacy of physical therapy interventions1, being about 1,000 clinical practice guidelines, 3,000 are systematic reviews and 16,000 are randomized controlled trials. If the current rhythm of publications in the field of physical therapy is constant, there will be a duplication of the whole content published in three years2,3.

Physical therapists commonly need to search for high-quality scientific evidence to support their clinical decisions. This evidence should be searched in randomized controlled trials or in systematic reviews, since these experimental designs are the most adequate to measure the effects of a given intervention4. Due to the high volume of randomized controlled trials published, probably the most adequate source of information for any health-care professional are systematic reviews of randomized controlled trials.

Systematic reviews are considered the best method to synthesize all the existent information about a certain topic4. A high-quality systematic review must always summarize all evidence available, taking into account the methodological quality of each study. The results from the systematic reviews must, therefore, consider either the sample size as well as the methodological quality of each individual trial using, whenever possible, statistical methods such as meta-analysis5. Three characteristics are essential for a high-quality systematic review: 1) to synthesize all evidence available; 2) to evaluate the methodological quality of each individual study; 3) to summarize the results from the eligible studies adequately (either by meta-analysis or descriptively). If the systematic review does not have such characteristics, caution is needed to interpret the results, since they could not represent the real evidence of the intervention reviewed.

To guarantee that the readers will be able to adequately judge the information of a systematic review, it is necessary that systematic reviews are clearly presented6. Only reviews with adequate reporting of methods, results and conclusions allow the adequate critical appraisal of the study and consequently, allow if the information is reliable enough to be used to support clinical practice as well as research.

A group of methodologists aimed to create guidelines on how to report a systematic review in a clear way developed in 1999 a checklist of essential items to be included in any systematic review. This guideline is known as the QUORUM statement (Quality of Reporting of Meta-analyses). The QUORUM7 recommendations had been updated in 2009 and is now named as PRISMA Statement8,9 (Preferred Reporting Items for Systematic Reviews and Meta-Analysis). The PRISMA statement includes a checklist of 27 items properly described and exemplified and a four-phase flow diagram8,9. The 27 items guide the authors of systematic reviews on the information that must be clearly described in the manuscript, including specific instructions for title, abstract, methods, results and financial support (Appendix 1). Among the items from the PRISMA Statement, there are the international registration of systematic review, the summary of the major findings of the review and the description of the limitations and results of the articles. These items reduce redundancy, increase transparency and facilitate the interpretation of the results of systematic reviews10.

The PRISMA Statement was used to evaluate the characteristics of Chinese traditional medicine systematic reviews11. This study concluded that the adherence of Chinese publications to the PRISMA recommendations was poor in the studies published up to 200911. The authors of this study reinforced that the use of PRISMA recommendations can improve the quality of presentation of systematic reviews. In addition, there are no similar studies that had evaluated the reporting of systematic reviews published in Portuguese. Therefore, the aim of this study was to analyse the reporting of systematic reviews in physical therapy published in Portuguese using the PRISMA Statement. It is important to highlight that the actual analysis involve manuscripts published prior to the publication of PRISMA Statement, and therefore, these studies could not benefit from this guidelines.

Method

This is a bibliometric analysis of systematic reviews and/or meta-analysis related to physical therapy interventions published in Portuguese. Eligible reviews were retrieved from Physiotherapy Evidence Database (PEDro). PEDro was chosen for this study as it is the most comprehensive database in indexing studies related to effects of physical therapy interventions and because PEDro is freely available on the internet (www.pedro.org.au)12, 13.

All systematic reviews published in Portuguese indexed on PEDro up to August 2011 were included. The search was performed using the "advanced search" option of the database, in which we typed the search term 'Portuguese' and limited the results for systematic reviews only.

Six previously trained raters for the use of the PRISMA checklist participated of the analysis of the eligible systematic reviews. Each article had been randomly allocated for two of five team of raters. These ratings were performed independently. In the case of disagreement between raters, a final arbitration was performed by a sixth, more experienced rater. The instrument used by the raters in the analyses of the eligible systematic review was the PRISMA checklist. The 27 items evaluated by this checklist are described in Appendix 1.

All systematic reviews were also classified according to the subdisciplines standardized by the PEDro database. For each item of the checklist, it was established to consider as satisfied only those that fully contemplated recommendations from the PRISMA Statement, being rated as 'yes'. When the rater considered the description of the item incomplete, inexistent or doubtful, the item was rated as 'no'. Such dichotomous criteria of rating were chosen to avoid bias due to different interpretation of the information from the raters. The ratings were organized in independent forms by the raters and were collated in a single document for further consensus if needed. The satisfied items were then summed in a score ranging from zero (no item satisfied) and 27 (all the items satisfied). Moreover, we summed the number of articles that contemplated each one of the checklist items (this value ranged from zero to 37 articles). We also performed frequency distribution analysis of the PRISMA total score by year of publication.

Results

Our search retrieved 41 systematic reviews; however four articles were excluded because they were not published in Portuguese (two were published in English and two in Italian). Of the 37 eligible studies included, the subdisciplines musculoskeletal and cardiothoracics showed the larger number of systematic reviews. In contrast, no systematic reviews in sports physical therapy were found (Table 1). The analysis of the individual articles, according to the items of the PRISMA checklist can be observed in Table 2.

In the analysis of the total number of items satisfied by year of publication, we observed that, on average, less than a half of the items from the PRISMA Statement were satisfied. Moreover, our data reveal that there was no improvement on the quality of reporting over time (Figure 1). The proportion of items satisfying the recommendations of PRISMA was 29.83%.

In the classification by category, the most satisfied items are the ones related to introduction and discussion sections. On the other hand, most of the items that compose the methods and results sections were not satisfied. The items 5 and 15 (Methods) have not been satisfied in none of the 37 eligible articles while the items 14 (Methods), 21, 22, and 23 (Results) were satisfied in just few studies (Figure 2).

Discussion

Our results indicate that the adherence to the PRISMA statement recommendations8 for most of systematic reviews published in Portuguese in the field of physical therapy was lower than 30%. We also observed a large variability in the fulfillment of the PRISMA items by year of publication; which shows a great potential for improvement in the reporting of systematic reviews published in Portuguese in the future. In spite of the low adeherence to the recommendations, it is important to point out that such results do not mean that the methodological quality of these reviews are low, since PRISMA was not developed for such purpose. The adequate tool for measuring the quality of the systematic reviews is called AMSTAR (Assessment of Multiple Systematic Reviews)14.

Considering that systematic reviews are probably the most important type of study to guide the clinical decision in physical therapy, the number of systematic reviews published in Portuguese is still very low if compared to the 3,057 reviews registered in PEDro1 database, which equals to only 1.21% of all available reviews. This number indicates a gap that should be filled in order to help a large number of Portuguese-speaking physical therapists that could benefit with a higher number of systematic reviews published in Portuguese.

The occurrence of a large number of systematic reviews in the subdisciplines musculoskeletal and cardiothoracics in Portuguese analyzed in the present study followed the same trend of distribution presented in other languages. These subdisciplines are also the most prevalent among the 19,729 studies indexed on PEDro1. This large number of systematic reviews in the subdisciplines of musculoskeletal and cardiothoracics ultimately reflects the areas with a largest number of professionals working in these areas.

The first systematic review in Portuguese was published in 2003, three years after the development of the QUORUM recommendations7,15, and therefore this guidelines seems to have not influenced our results. Similarly the use of PRISMA was not observed in most of the eligible articles, which might mean that there was a small influence of these guidelines in the current systematic reviews published in Portuguese.

The 27 items of the PRISMA checklist had not been fulfilled consistently, and some items were satisfied in most of the eligible reviews (for example reaching 100% for the item 3). On the other hand, items 5 and 15 had not been satisfied in none of the eligible reviews. Item 5 refers to the international registration of systematic review. In a study developed with the same purpose of our study for Chinese reviews11, the authors reported that none of the articles analyzed by them quoted the registration number, as well as the reviews published in Portuguese. It is known the problems of selection bias (i.e. publishing only positive results) of systematic reviews in international journals. The registration of reviews favors the good practice and transparency of the process of review and publication16. Considering this, the practice of registration must be encouraged, although there are few sites available for such procedure (see http://www.ncddr.org/cgi-bin/systematicreview_submit.cgi, for registration).

Item 15 concerns to the risk of bias of results, which adherence was null in the present study. In the study of Ma et al.11, 53% of the Chinese articles satisfied this item related to the risk of bias. The items that were attended by a larger number of authors in our study (3 and 26) corresponded to those items of PRISMA whose criteria of analyses are more subjective. Item 3 refers to the rationale of the study, and the 26 refers to the general interpretation of the results, in other words, to the conclusions of the study. Most part of the other items of the PRISMA recommendations requires a direct answer. These items were able to be more clearly evaluated in the present study, since, for the analysis of the reviews; it was enough to search in the text if the information was available or not (for example, if the title indicates the study as a systematic review  item 1).

It needs to be highlighted that, in the study of Ma et al.11, none of the articles analyzed in Chinese showed a structured abstract and a summary of the main results in the discussion, items contemplated by 40 and 55% of the articles in Portuguese, respectively. The structure of writing of the abstracts seem to be critically dependent on the "instructions to the authors" section of the journals rather than from the knowledge of the author. The publication rules of some scientific journals in Portuguese language require a different manuscript format from that recommended by the check list of PRISMA, which suggests a structured format. Knowing this, in the present study it was considered as fulfilled the item that contemplated all the aspects indicated in PRISMA recommendations, even if the abstract was not structured, however, the same criteria might not have been observed by Ma et al.11.

Most of the systematic reviews evaluated in this study did not satisfied to the criteria of the PRISMA checklist. This results does not necessarily indicate the non-observance of the item by the authors during the execution of their study, but, the absence of a clear reporting. The suggestion that the authors must use the PRISMA statement while writing their reviews can avoid this problem in the future.

The aim of the PRISMA Statement is to allow authors to perform a verification of the items that compose it before the submission of articles of systematic review. The results from this study suggest that the adherence of the authors to PRISMA recommendations is low, possibly because it is still new. This adherence could improve the reporting of future systematic reviews.

An analysis of journals related to free access Pediatrics17, showed that 19.5% of 41 the journals had the indication to use the PRISMA (or QUORUM) in the instruction to authors section. It is possible that the suggestion of such recommendations by the editorial board of the journals published in Portuguese, together with the other guidelines for authors, is likely to increase the reporting of systematic reviews published in Portuguese6.

A possible limitation of this study would be that some of the articles analyzed were published prior to dissemination of the PRISMA Statement8. On the other hand, all eligible reviews were published after the publication of the QUOROM15. It is important to note that even if the QUOROM recommendations were used in this study, the results would be very similar. This fact can be inferred by Figure 2, which shows that the reporting of the reviews did not improve over time.

We concluded that most of the authors did not adhere to the criteria recommended by the PRISMA recommendations. Our results may serve as an alert to the scientific community that aims to publish systematic reviews in Portuguese, encouraging future systematic reviews to be reported more clearly and transparently. The implementation of these recommendations requires extensive endorsement by Portuguese-written journals. We also suggest extensive training of the use of these recommendations by journal editors, reviewers and authors. Certainly this is not a simple task, but this can improve the reporting quality of systematic reviews in Portuguese language.