Login using

You can login by using one of your existing accounts.

We will be provided with an authorization token (please note: passwords are not shared with us) and will sync your accounts for you. This means that you will not need to remember your user name and password in the future and you will be able to login with the account you choose to sync, with the click of a button.

Musculoskeletal injury confers an enormous burden of preventable disability and mortality in low- and moderate-income countries (LMICs). Appropriate orthopedic and trauma care services are lacking. Leading international health agencies emphasize the critical need to create and sustain research capacity in the developing world as a strategic factor in the establishment of functional, independent health systems. One aspect of building research capacity is partnership between developing and developed countries, and knowledge sharing via these collaborations. This study evaluated the efficacy of a short, intensive course designed to educate surgeons on fundamental aspects of clinical research using evidence-based medicine (EBM) principles. Orthopedic surgeons from the United States and Canada presented a one-day course on the fundamentals of clinical research in Havana, Cuba. Knowledge acquisition was assessed on the part of course participants and surveyed current involvement with and attitudes toward clinical research. Questionnaires were presented to participants immediately preceding and following the course. The mean pre-test score was 43.9% (95% CI: 41.1–46.6%). The mean post-test score was 59.3% (95% CI: 56.5–62.1%). There were relative score increases in each subgroup based on professional level, subjective level of familiarity with EBM concepts, and subjective level of experience in research. This study establishes the short-term efficacy of an intensive course designed to impart knowledge in EBM and clinical research. Further study is necessary to determine the long-term benefits of this type of course. This may be a useful part of an overall strategy to build health research capacity in LMICs, ultimately contributing to improved access to high-quality surgical care.

Introduction

A fundamental understanding of the concepts behind research design, implementation, and analysis is increasingly necessary on the part of physicians and surgeons in order to incorporate rapidly developing biomedical knowledge into practice and deliver the highest quality of patient care. Evidence-based medicine [EBM; or practice, as opposed to opinion-based medicine (Straus et al., 2004)] is the judicious and intentional application of the highest quality available medical knowledge, including epidemiology and clinical research, in conjunction with clinical acumen and experience to the treatment of an individual patient (Evidence Based Medicine Working Group, 1992; Sackett et al., 1996; CEBM, 2013).

Clinical research is intimately connected to EBM. The former provides material for the latter, and the practice of seeking data to answer clinical queries reveals when gaps in that material exist, provoking additional relevant research. Certain underlying knowledge and skills are necessary to both appraise and conduct such work (Evidence Based Medicine Working Group, 1992). Working knowledge of study design types, levels of evidence, power calculation, statistical significance, and outcome selection are all relevant to the critical interpretation of literature as well as to study design. Formulation of a clinical question to answer via literature review or for the purpose of study genesis is the same skill applied to two related purposes. The course examined in this study took as its purpose the instruction of health professionals in the conduct of clinical research. Doing so required introducing and emphasizing the aforementioned underlying common principles of EBM. The instruction did not extend to the interpretation and application of evidence-based guidelines to clinical care.

From an orthopedic perspective, it is clear that musculoskeletal injury is a major contributor to disability and mortality in the developing world. Rates of extremity injury from falls and road traffic crashes are two to five times higher in LMICs than in high-income countries (HICs) (Mock and Cherian, 2008). Disability data consistently demonstrate that LMICs suffer an enormous burden of long-term injury-related disability. One study in Ghana found a 0.83% population prevalence of injury-related disability; 80% of this disability was due to extremity injuries (Mock et al., 2003; Mock and Cherian, 2008). Serious injuries are twice as likely to result in death in the developing world (Mock and Cherian, 2008). These numbers represent avoidable death and disability. Resolving this disparity is a complex problem involving training, resources, infrastructure, and political will. Supporting research capacity is an important part of developing robust global health systems and empowering local health professionals to examine and solve the healthcare challenges that their countries face (Global Forum for Health Research et al, 2009). Given the massive impact of orthopedic injury on the global burden of disease, this is particularly important for the orthopedic community to recognize and act upon at this time.

An ongoing collaboration exists between the Orthopaedic Trauma Institute at the University of California, San Francisco (UCSF) and the Centro de Investigaciones Medico Quirúrgicas (CIMEQ) hospital in Havana, Cuba. Via this collaboration, surgeons determined that despite some degree of infrastructure in Cuba for clinical research and professional interest in carrying it out, there was an inadequate understanding in the Orthopedic Surgical community on how to engage in clinical studies. Although Cuba has medical institutions (i.e., hospitals and clinics) with the capacity to support clinical research and ancillary workers who can assist with study design, data collection, and statistics, their system is not ideal because Cuban orthopedic surgeons do not receive sufficient training to engage clinical research.

In response, faculty from UCSF, CIMEQ, and the Mexican Orthopaedics and Trauma Association [Asociación Mexicana de Ortopedia y Traumatología (AMOT)] collaborated on the design of a one-day clinical research course, to be taught in conjunction with a national course in orthopedic trauma presented at CIMEQ in Havana. The course was designed to impart fundamental skills and knowledge to surgeons on the design, implementation, and publication of original research, employing basic EBM principles. Teaching attendees how to use and interpret EBM in their practice was not a specific course goal, but knowledge of EBM principles will arguably influence clinical practice. The aim of the present study was to evaluate and quantify the efficacy of this course.

Materials and Methods

Course Presentation and Content

During the clinical research course, surgeon researchers from the United States, Canada, Mexico, and Cuba presented lectures on study protocol, surgical research methodology, and navigation of the publication process (Figure 1). Surgeons presented the one-day training immediately prior to a course on orthopedic trauma. All slides were in Spanish, and live interpreters translated for those presenters who lectured in English. In addition, each participant received a 60-page syllabus, also translated into Spanish.

FIGURE 1

Figure 1. Plenary session topics.

Course Evaluation

No existing and validated tool specifically addressed the content of our course, which included instruction on fundamental elements of EBM incorporated into instruction on practical aspects of carrying out clinical research and preparing work for publication. Therefore, established quality guidelines (Shaneyfelt et al., 2006) were used to design our own assessment tool.

Evaluation instruments for education in EBM are diverse. Multiple authors conclude that no one tool would be appropriate to evaluate any proposed EBM training intervention—rather a tool designed to evaluate knowledge acquisition in the domains of intended knowledge gain should be thoughtfully selected (Straus et al., 2004; Shaneyfelt et al., 2006). Straus et al. proposed a conceptual framework for the evaluation of EBM that advised considering a physician’s “mode of practice” of EBM and choosing an evaluation instrument that reflects the different learners, interventions, and outcomes expected (Straus et al., 2004).

A recent review of instruments available for the evaluation of evidence-based practice suggested that the highest quality instruments were characterized by “interrater reliability (if applicable), objective (non-self-reported) outcome measures, and multiple (> = 3) types of established validity evidence (including evidence for discriminative validity)” (Shaneyfelt et al., 2006). Ultimately, an instrument was designed specific to the course, using the quality guidelines established by Shaneyfelt et al. and focusing on fundamental knowledge acquisition, appropriate for a one-day educational event. This was adapted in part from the questionnaire designed by Sprague and colleagues, who described the development and evaluation of a short training course in EBM for a surgeon audience in North America. Using a pre- and post-course questionnaire scheme, they reported a statistically significant increase in knowledge acquired based on relative increase in score as well as participant satisfaction with the training (Sprague et al., 2012). The instrument they employed was fundamentally based on the highly reliable and validated Fresno Test (Ramos et al., 2003; Thomas and Kreptul, 2015). The Fresno Test is a performance-based measure that was designed to evaluate a broad range of EBM knowledge and skills. Sprague and colleagues adapted it to their group’s more narrow goals and requirements, and the instrument was altered to apply to our training event.

By defining the learners (surgeons, physicians, etc.), intervention (education on clinical research and EBM), and the outcome (objective questionnaire results), our tool was consistent with the Straus conceptual model (Straus et al., 2004). A demographic survey, which specifies gender, age, and profession was also included. Once participants were separated into subgroups, it could then be determined which subgroups entered the course more knowledgeable in clinical research and EBM principles than others, and which subgroups benefited the most from the one-day course (as indicated by an increase in test scores). Pre-course and post-course questionnaires were intended to highlight objective (and not self-reported) measures, and the self-reported levels of knowledge were only present on the pre-test. In total, there were 18 multiple choice questions, eight of which referred to a specific clinical scenario case study, and 10 relating to clinical research design and methodology.

Participants received a printed pre-test upon entering the conference area, and these were collected before commencement of proceedings. Post-tests were handed out at the conclusion of the course and collected as participants exited the conference area. The outcome measure was defined as change in number of questions answered correctly on the post-test compared with the number answered correctly on the pre-test; thus, it was objective. Responsive validity was ensured by comparing individual participants’ scores from before and after the intervention. Discriminative validity was maintained by our ability to stratify participants based on their reported level of expertise in EBM and clinical research.

Statistical Analysis

Descriptive statistics were performed using means and proportions. The distribution of scores pre- and post-course were assessed for normality. The paired Student’s t-test was used to assess the difference in means of pre- and post-course scores for statistical significance, assuming an assumption of normal distribution of scores. The data were further analyzed for further trends and hypothesis setting. The data were stratified based on professional level (surgeon, physician, and other health professional) as well as by experience with EBM and by experience in clinical research, rated by participants based on six-point scales. All analyses were conducted using STATA version 12.0 (College Station, TX, USA).

Ethics Statement

This study was submitted to the University of California, San Francisco’s Human Research Protection Program (IRB number: 11-06489) and was certified as exempt from Institutional Review Board consideration under Exempt Category 2: Use of educational tests, surveys, interviews, or observations of public behavior.

As required by this body, participants were provided a document with details regarding risks and benefits of the study. This document informed them that completion of the survey was considered consent to participate. The Human Research Protection Program did not require a signature on this document.

Results

Participant Characteristics

Ninety six of 102 registered attendees (94.1%) completed both the pre-course and post-course questionnaires. Eight participants completed only one of the two instruments and were, therefore, dropped from the analysis. Attendees were asked beforehand to complete the questionnaires only if they planned on staying throughout the entirety of the course. Surveys were collected from 104 total attendees, indicating that at least two other non-registered attendees elected to participate in the questionnaires.

Though the course was aimed at a surgeon audience, professionals from a number of other healthcare disciplines also attended and completed the surveys. Due to the high participation rate among attendees, it is improbable that certain demographic groups were more likely to complete the survey than others. Thus, in terms of who completed the survey, there was little to no selection bias. The mean age of participants was 44.6 years (SD: 11.8 years) and 52% were male. 46% identified themselves as surgeons, 17% as physicians, 23% as other professionals (nurses, physical therapists, etc.), and 15% did not respond to this question (Figure 1; Table 1). Self-reported exposure to EBM and clinical research on the part of this group was low to moderate; 46% of participants judged their EBM experience to be “poor” or “very poor,” and 38% described it as “good” or “average.” Similarly, 30% of participants judged their clinical research knowledge to be “poor” or “very poor,” while 53% described it as “good” or “average.”

TABLE 1

Table 1. Demographics of course participants.

The mean pre-test score was 43.9% (95% CI: 41.1–46.6%). The mean post-test score was 59.3% (95% CI: 56.5–62.1%). Despite the relatively low mean post-test score of 59.3%, there were no clear trends in what questions participants answered correctly or incorrectly. Overall, participants demonstrated a significant increase in post-course questionnaire scores compared with pre-course scores. 68% of participants improved by at least 20% relative their pre-test scores, and 35% of participants improved by at least 50% relative to their pre-test scores.

Figure 4. Relative increase in scores from pre-course to post-course questionnaire by level of self-reported clinical research knowledge.

TABLE 4

Table 4. Change in scores from pre-course to post-course questionnaire by level of clinical research knowledge, participant reported.

Discussion

This study of 96 participants in a short course on clinical research in an international setting demonstrated a significant improvement in test of knowledge acquisition. These participants reprsented a range of healthcare disciplines and endorsed a spectrum of levels of experience with EBM and clinical research. The magnitude of increase in score on our evaluation instrument was consistent with significant improvement (Sibley et al., 1982; Sprague et al., 2012). The results of this study suggest that a brief, intensive course presented by surgical faculty on the fundamentals of clinical research using EBM principles can be an effective way to support clinical research capacity in a LMIC setting.

Intended as a pilot project for future courses in Latin America, the aim of quantifying the value of this course was achieved in terms of knowledge acquisition on the subject of clinical research. This finding is consistent with previous work. Sprague et al. demonstrated that participants who took a short course in clinical research skills taught by surgical faculty significantly increased their scores on a test of knowledge acquisition (Sprague et al., 2012). They used a pre- and post-test format to evaluate the efficacy of the Principles and Practice of Family Research Course, a two and one-half day course that was offered to medical students, medical residents, physicians, surgeons, and research coordinators. Medical students and residents in particular appeared to benefit from the course. Argimon-Pallàs et al. (2011) found a significant increase in EBM knowledge and skills among family medicine residents who underwent a short training course on evidence-based practice. Nadler et al. found that a career and research development course presented to Nigerian surgeons had at least short-term efficacy and observed that potential cultural differences were not an impediment to knowledge acquisition (Nadler et al., 2010). Dodani and LaPorte (2008) showed short- and long-term benefits of a relatively more extensive course on research methodology presented in Pakistan.

Strengths and Limitations

The strengths of this study included a design that utilized a pre- and post-course questionnaire. As these were matched for each participant, a more robust statistical analysis was possible. Published guidelines were engaged to develop a high-quality instrument specific to the course content, participants, and structure. Potential contamination was limited by implementing both parts of our instrument on the same day as the course—immediately prior to its start and before participants departed from the final lecture.

Though our intended audience was orthopedic surgeons, many professionals from other health disciplines also registered for and benefited from the course. It would seem that non-physician, non-surgeon participants benefited the most, although their relatively higher change in score is the result of a lower pre-test mean. The intervention appears on the whole to have raised all participants to a common, higher level of understanding of the fundamentals of clinical research as presented in the course.

Limitations to our study include the possibility that an unknown proportion of participants were unable to be present for all portions of the course. In addition, the degree of effort exercised by participants in completing the questionnaires could not be assessed. Also, because of the nature of this course, with distance and logistics as significant barriers, an instrument was not designed to test the knowledge retention of the attendees over time. However, other similarly structured 1-day courses in pediatric life-support and partner abuse suggest that trainees retain increased knowledge for at least 3 to 4 months after the course (Durojaiye and O’Meara, 2002; Madden et al., 2015). Finally, it is unknown whether the gain in knowledge and skills that were measured will translate into an increase in clinical research or improvement in patient outcomes. Other studies have shown that teaching EBM knowledge and research skills can change health care professional behavior (Straus et al., 2005), and international health organizations, including the World Health Organization (WHO), the Council on Health Research for Development, and the Global Forum on Health Research, have consistently emphasized the critical importance of health research capacity development for the success of health systems in LMICs (Commission on Health Research for Development, 1990; Global Forum for Health Research and Davey, 2004; Lansang and Dennis, 2004; Changing Mindsets, 2008).

Application to Global Orthopedic Practice

The lack of quality data on the burden of global orthopedic trauma and the availability of care to manage trauma-related injuries impede the establishment of policies that prevent avoidable morbidity and mortality (Elliott et al., 2015). It is most appropriate that those who live and practice in the affected communities conduct the research, collect the data, and apply the results to their practice (Gosselin et al., 2009).

A 2004 Bulletin of the WHO states: “A combination of short-term and long-term strategies, directed at individual, institutional, and country levels are necessary to develop a sustainable system of health research.” (Lansang and Dennis, 2004). The authors believe that directed coursework is an effective short-term strategy to improve clinical investigation. Following this course, academic and clinical research collaborations developed between the course supporters, including UCSF, CIMEQ, AMOT, and the Osteosynthesis and Trauma Care Foundation (OTCF). It was felt that the course enabled this collaboration, most importantly by improving the understanding of EBM and the ability of the Cuban orthopedic surgeons to access more established and resourced international partners. Subsequent to this course, there are plans to include Cuban surgeons in large prospective studies through McMaster University and the University of California, San Francisco. While not a direct indication of the long-term efficacy of the course, Cuban surgeons who participated in this work have become active members of the recently established international research consortium, Associación de Cirujanos Traumatologicos en las Americas (ACTUAR) that supports clinical research across the Americas (Chomsky-Higgins et al., 2017).

Based on the results of this work and successful experience with this course, the authors plan to offer additional clinical research courses to medical professionals in other countries in Central and South America. Similar results are anticipated, and the attendees of these courses are expected to not only improve their ability to better interpret the existing literature but also engage with local, regional, and international clinical research efforts. Ultimately, the authors hope to empower surgeons to create their own an international Latin American network, where the most relevant questions for their patient populations can be addressed.

Conclusion

Despite the favorable short-term impact of this course, future work will be needed to establish the long-term content retention of this type of course, as well as the impact the information has on the development of clinical research capacity. It would be informative, for example, to know whether attendees of such courses publish more papers, pursue additional further education in clinical research, develop practice patterns that incorporate EBM, or have improved patient outcomes.

Ethics Statement

This study was submitted to the University of California, San Francisco’s Human Research Protection Program (IRB number: 11-06489) and was certified as exempt from Institutional Review Board consideration under Exempt Category 2: Use of educational tests, surveys, interviews, or observations of public behavior. As required by this body, participants were provided a document with details regarding risks and benefits of the study. This document informed them that completion of the survey was considered consent to participate. The Human Research Protection Program did not require a signature on this document.

Author Contributions

TAM and KC-H contributed equally to the development of this article. All authors participated in the discussion and intellectual contribution that was the foundation of the report. TAM, KC-H, and TM wrote the draft manuscript. Each of the authors critically reviewed the content, approved the final version to be published, and agreed to be accountable for all aspects of the work.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Funding

The authors gratefully acknowledge Dr. Nancy Hills of the UCSF Department of Neurology for her assistance with statistical analysis, as well as funding from the Osteosynthesis and Trauma Care (OTC) and Atlantic Philanthropies Foundations to TM to support course content development, faculty travel, and course logistics.

Changing Mindsets. (2008). Research Capacity Strengthening in Low and Middle-Income Countries. Geneva: COHRED, Global Forum for Health Research and UNICEF/UNDP/World Bank/WHO Special Programme for Research and Training in Tropical Diseases (TDR).

Dodani, S., and LaPorte, R. E. (2008). Ways to strengthen research capacity in developing countries: effectiveness of a research training workshop in Pakistan. Public Health 122, 578–587. doi:10.1016/j.puhe.2007.09.003