Transcription

1 Justifying the adequacy of samples in qualitative interview-based studies: Differences between and within journals Prof Julie Barnett a, Konstantina Vasileiou a, Dr Susan Thorpe b, Prof Terry Young c a University of Bath, Department of Psychology b Newcastle University, School of Psychology c Brunel University London, College of Engineering, Design and Physical Sciences Symposium: Quality in qualitative research and enduring problematics Qualitative Methodology Forum 27 January 2015 Faculty of Humanities and Social Sciences, University of Bath

2 Experiential triggers for this project (1) Editor First of all, most of the articles published in the journal are not qualitative in protocol. So we need to enter this manuscript gently as has been done with some others Third, I don t have a good understanding of the representativeness of the sample. Only 30% of winning organizations were represented. Of them, the key informants are not at all well-described. How do we know that these 15 people best represent their organizations? Who are they? What are their characteristics. The sampling issue (at both of two levels, organization selection and individual selection within organizations) is a critical issue I thought for a topic like this you may need to increase the number of participants to at least 25 (50%). This was not a random selection. 15 is respectable. However, can you prove they are representative of the 51 winners? Secondly what are these winners of? What are the broader implications of using such subjects? Who can they speak for?

3 Experiential triggers for this project (2) Reviewers There remains an issue of how useful this information would be. The poor response rate is certainly something that gives the reader pause, and the results of the work cannot be seen as generalizable. The generalizability of findings is what makes science different from faith-based study. If the purpose of qualitative research is not to produce findings that are capable of generalizing beyond the data gathered, then why should a reader care about the results? We only care when we think the results may generalize to our own studies, theories, situations, etc. While no decisions regarding clinical practice should ever be based on such a small sample size, in the end I believe it makes a contribution solely in the way we conceptualize evidence. Your sample size of 15 out of 51 organizations is very small.however, the issue of sampling error is just as pertinent to qualitative researchers as it is to quantitative researchers. Three statements are just not enough for me to believe that the result will generalize to the larger population. In other words, how do I know this is a real signal rather than noise? The generalizability of the results, perhaps, might be more believable if a large proportion of the interviewees concurred with the quoted statements.

4 How many qualitative interviews are enough? Purposeful sampling Informationally representative samples (vs statistically representative) Tension between achieving informational redundancy and be able to conduct in-depth, case-oriented analysis (Sandelowski, 1995) So how many interviews are enough? Experts in the field tend to concur with the answer that It depends (Baker & Edwards, 2012) Research objective(s) Epistemological and theoretical underpinnings Type of analysis Epistemic community (i.e. rules, norms of the scientific community one belongs to) Practical considerations (e.g. hard to access participants; resources; researcher s career stage; institutional constraints) The criteria of data/empirical saturation or theoretical saturation (i.e. when no new insights about the phenomenon/theoretical category are found by adding new cases) remain useful in determining qualitative sample sizes.

5 Our Research Questions To what extent are arguments to justify the sample size of qualitative interview-based research employed? What are these arguments? Do the presence and/or the nature of justifications differ across journals from different disciplines? How might the justifications relate to other characteristics such as the type of analysis? What justifications, other than sample size, are used to defend the adequacy of the sample? Do the various justifications change over time?

6 How do we do science: Scrutinising published research A few examples How is theory used in qualitative research? (Bradbury-Jones, C., Taylor, J., Oliver Herber, O. (2014). How theory is used and articulated in qualitative research: Development of a new typology. Social Science & Medicine,120, ) How are research questions constructed in social scientific work? (Alvesson, M., & Sandberg, J. (2013). Constructing research questions: doing interesting research. London: Sage) Are participant recruitment and retention in RCTs adequately reported? (Toerien et al. (2009). A review of reporting of participant recruitment and retention in RCTs in six major journals. Trials, 10, 52.)

10 Data analysis We used both qualitative and quantitative analysis of data

11 Some preliminary results: British Medical Journal (BMJ) Search keywords: interview* AND qualitative Identification 132 results were obtained from the search Screening All 132 results were screened as to whether they met the criteria Eligibility Data were extracted from the 23 articles that met the eligibility criteria Included 22 eligible articles were included in the review One paper was excluded due to longitudinal design

14 BMJ: Qualitative analysis of sample adequacy argumentation (1) Diversity/Variation Aspects of the phenomenon of interest An additional strength is its focus on reactions to intermediate results as well as positive and negative diagnostic results (BMJ07) Participant Demographic characteristics One strength of our study was the diverse range of respondents in terms of age, socioeconomic group, location, and household smoking profile.(bmj10) Study Populations We included the views not only of patients and GPs but also of practice nurses and receptionists, who have generally been excluded from previous studies (BMJ02) Participants were purposely sampled to represent a wide range in medical specialties, age, and sex to reflect the possible diversity of opinions. (BMJ19) The argument of diversity counterbalanced the lack of sample representativeness As with any qualitative study aiming for a maximum variation sample, the findings are not intended to be numerically representative the sampling method is intended to show the diversity in responses, including those that are less usual. (BMJ17) These tables were, however, derived from a purposive sample and should not be taken to represent the population; rather, we aimed to capture the range and diversity of experience, beliefs, and opinions instead of providing a quantitative summary of findings. (BMJ20)

15 BMJ: Qualitative analysis of sample adequacy argumentation (2) Sample particularity: Constructed negatively as it undermines the potential of representativeness of the sample and thus limits the generalizability of findings The generalisability of our findings is limited because the sample was drawn from a prison in southwest England that predominantly holds white British offenders with sentences of less than one year; to what extent our findings might relate to long term offenders, those from black and ethnic minorities, or women is therefore unclear. (BMJ09) One limitation of our study is that we looked only at cases in which a request for euthanasia had not been granted or granted but not performed (about two thirds of all requests), and the perspectives of patients and physicians with regard to unbearable suffering might be different in cases where euthanasia was performed for instance, showing more agreement between patients and physicians. (BMJ15) But because of the particular nature and characteristics pertinent to older people and patients with chronic pain the results presented may not be generalisable to other drugs or different age groups. (BMJ11) Finally, within the sub-saharan African population, the participants in this study are a relatively unusual group in that all were receiving palliative care. Most people dying with advanced illness in South Africa and Uganda, and indeed in the rest of sub- Saharan Africa, may have even less access to information and care than this sample. (BMJ16) "Another limitation of our study is that we focused on difficult and straightforward cases rather than on the type of cancer; our study may therefore not be representative for the whole cancer population. (BMJ19)

16 Some preliminary conclusions Just over half of the papers did not justify their sample size in any way Claim to data saturation was the most common justification for the sufficiency of sample size Stakeholder, demographic or phenomenal diversity/variation was the strongest line of defence of sample adequacy Sample particularity was viewed to be problematic as it undermined the scope of sample and thus the generalizability of results There was an absence of claims around theoretical generalisations Quasi-quantitative referential system of evaluation of sample adequacy

17 Thank you for listening! Any questions?

18 References Alvesson, M., & Sandberg, J. (2013). Constructing research questions: doing interesting research. London: Sage Baker, S.E., & Edwards, R. (2012). How many qualitative interviews is enough? Expert voices and early career reflections on sampling and cases in qualitative research. National Centre for Research Methods Review Paper. Bradbury-Jones, C., Taylor, J., Oliver Herber, O. (2014). How theory is used and articulated in qualitative research: Development of a new typology. Social Science & Medicine,120, Sandelowski, M. (1995). Sample size in qualitative research. Research in Nursing & Health, 18, Toerien et al. (2009). A review of reporting of participant recruitment and retention in RCTs in six major journals. Trials, 10, 52.

Research Design Service London Writing a Qualitative Research Proposal When to use qualitative methods Qualitative methods should be used when the aim is to: Investigate complex phenomena that are hard

CLINICAL EXCELLENCE AWARDS Academy of Medical Royal Colleges submission to the Review Body on Doctors and Dentists Remuneration Introduction The Academy of Medical Royal Colleges (the Academy) welcomes

Developing an implementation research proposal Session 2: Research design Learning objectives After completing this session, you will be able to: Develop a research design outlining your data collection

1 Faculty of Nursing Master s Project Manual For Faculty Supervisors and Students January 2014 2 Table of Contents Overview of the Revised MN Streams in Relation to Project 3 The Importance of Projects

Plan for Masters Degree in Social Work 009 1- General Rules and Conditions: A- This plan conforms to the regulations of programs of graduate studies. B-. Areas of specialty accepted for admission according

Courses Description Bachelor Degree in Social Work Introduction to Social Work 2701101 Understanding the history of social work profession. Other topics include the philosophy, principles and ethics of

PEER REVIEW HISTORY BMJ Open publishes all reviews undertaken for accepted manuscripts. Reviewers are asked to complete a checklist review form (http://bmjopen.bmj.com/site/about/resources/checklist.pdf)

Research and diabetes nursing. Part 3: Quantitative designs Vivien Coates This article is the third in a series that aims to assist nurses working in diabetes to understand research from a broad perspective,

PEER REVIEW HISTORY BMJ Open publishes all reviews undertaken for accepted manuscripts. Reviewers are asked to complete a checklist review form (http://bmjopen.bmj.com/site/about/resources/checklist.pdf)

Appendix 5 Systematic Review Proposal Outline Developing a Protocol for a Systematic Review Using the Joanna Briggs Institute (JBI) Method The New Jersey Centre for Evidenced Based Healthcare and Patient-Outcomes

PEER REVIEW HISTORY BMJ Open publishes all reviews undertaken for accepted manuscripts. Reviewers are asked to complete a checklist review form (http://bmjopen.bmj.com/site/about/resources/checklist.pdf)

The Literature Review in the Masters Dissertation Roberta Sammut If I have seen further it is by standing on the shoulders of giants Isaac Newton Why are reviews needed Research information is like small

For discussion on 25.9.2014 HAB-P211 Hospital Authority 2013 Hospital-based Patient Experience and Satisfaction Survey Purpose This paper reports to Members the findings of the 2013 Hospital-based Patient

PhD Project Proposal - Living well with dementia: a PhD programme to develop a complex intervention that integrates healthcare for people with dementia 1. Background to the study There are 800,000 people

Survey to Doctors in England End of Life Care Report prepared for The National Audit Office 1 2008, medeconnect Table of Contents 1 SUMMARY OF FINDINGS... 3 2 INTRODUCTION... 5 3 RESEARCH OBJECTIVES AND

FOUNDATION AND SPECIALIZATION COMPETENCIES FOR COMMUNITY HEALTH SCIENCES GRADUATE STUDENTS Updated Feb 2013 Department Mission Statement: The Department of Community Health Sciences is committed to enhancing

What is Grounded Theory? Dr Lynn Calman Research Fellow School of Nursing, Midwifery and Social Work Grounded theory The aim of grounded theory is: to generate or discover a theory (Glaser and Strauss,

PEER REVIEW HISTORY BMJ Open publishes all reviews undertaken for accepted manuscripts. Reviewers are asked to complete a checklist review form (http://bmjopen.bmj.com/site/about/resources/checklist.pdf)

Overview of mixed methods: a health services research perspective Julia Addington-Hall, Chair in End of Life Care, School of Nursing and Midwifery, University of Southampton, UK Outline of presentation:

How to Write A Research Proposal 1. Introduction: Different disciplines, organizations and institutions have different formats and requirements for writing a proposal. There are, however several key components

Graduate Program Course Descriptions The following section provides course descriptions for courses offered and required in the Research College of Nursing Graduate Programs: the Family Nurse Practitioner

PEER REVIEW HISTORY BMJ Open publishes all reviews undertaken for accepted manuscripts. Reviewers are asked to complete a checklist review form (http://bmjopen.bmj.com/site/about/resources/checklist.pdf)

13-30 14:15 Focusing the Question Andrew Booth, Reader in Evidence Based Information Practice, ScHARR, University of Sheffield, UK Importance of a focused question The review question should specify the

535 REGULATIONS FOR THE POSTGRADUATE DIPLOMA IN PSYCHO-ONCOLOGY (PDipPsycho-oncology) (See also General Regulations) M.105 Admission requirements To be eligible for admission to the programme leading to

MN-NP GRADUATE COURSES Course Descriptions & Objectives NURS 504 RESEARCH AND EVIDENCE-INFORMED PRACTICE (3) The purpose of this course is to build foundational knowledge and skills in searching the literature,

Brochure More information from http://www.researchandmarkets.com/reports/3023234/ Headaches: Current Treatments, Unmet Needs, and Patient Recruitment Description: This report identifies and explores the

PEER REVIEW HISTORY BMJ Open publishes all reviews undertaken for accepted manuscripts. Reviewers are asked to complete a checklist review form (http://bmjopen.bmj.com/site/about/resources/checklist.pdf)

Why Community Engagement? Community engagement as a community-based or driven empowerment model is seen as more effective than topdown approaches Multi-level sociological interventions with empirically

RESEARCH GRANTS PROGRAM GUIDE FOR WRITING A RESEARCH PROTOCOL Research Grants Program (RGP) Research Coordination (HDP/HDR) Division of Health and Human Development (HDP) Pan American Health Organization

PEER REVIEW HISTORY BMJ Open publishes all reviews undertaken for accepted manuscripts. Reviewers are asked to complete a checklist review form (http://bmjopen.bmj.com/site/about/resources/checklist.pdf)

PositionStatement NATIONAL PLANNING FOR HUMAN RESOURCES IN THE HEALTH SECTOR CNA POSITION CNA believes that successful human resources planning in the Canadian health sector requires a collective and integrated

Purpose Purpose Purpose Primary responsibility for implementation, coordination, evaluation, communication and/or management of research studies. May involve more than one study or multiple sites within

1 What are research, evaluation and audit? Barbara Sen, Maria J. Grant and Hannah Spring I don t do research. I don t have the time. I am too busy with the day to day running of the library. I do evaluations

PEER REVIEW HISTORY BMJ Open publishes all reviews undertaken for accepted manuscripts. Reviewers are asked to complete a checklist review form (http://bmjopen.bmj.com/site/about/resources/checklist.pdf)

MSN Course Description Revisions Necessary for Introduction of the Nurse Educator Track, September 2015 Rationale: 1. MSN courses for both tracks need to delete specific reference to the CNL or make reference

MSc International Programme in Addiction Studies Prospectus 2015-2016 The Institute of Psychiatry, Psychology and Neuroscience www.kcl.ac.uk/ioppn The Institute of Psychiatry, Psychology and Neuroscience

Critical appraisal of qualitative research Slides by Sarah Lawson 1 Learning objectives Understand the principles of critical appraisal and its role in evidence based practice Be aware of the key elements

3 June 2014 Education and Training Advisory Board 5 To consider Education surveys plan 2014-17 Issue 1 With the success of the National Training (NTS) there have been calls to expand the use of surveys

Answer keys for Assignment 7: Experimental study designs (The correct answer is underlined in bold text) 1) Which of the following interventions can be tested in a clinical trial? a) b) c) d) e) New Vaccines

Highlights From the 2012 National Sample Survey of Nurse Practitioners Health Resources and Services Administration Bureau of Health Professions National Center for Health Workforce Analysis The Health

: The Need for Career Advice in Medical Training How should the support provided to doctors and medical students to help them make career decisions during their training be improved? Experience elsewhere

PALLIATIVE CARE IN CANADA CHA S PERSPECTIVE Brief Submitted to the Parliamentary Committee on Palliative and Compassionate Care November 2010 About the Canadian Healthcare Association The Canadian Healthcare

Overall Summary CenteringParenting, a unique group post-partum care and social-support model, is ready for implementation in Calgary. Thanks to the funding provided by the Alberta Center for Child, Family,

Summary of a review of literature on the impact of working context and support on the postgraduate research student learning experience Diana Leonard Institute of Education, University of London Who is

Engaging primary care in selfmanagement support a combined presentation prepared by Carolyn Chew-Graham Professor of General Practice Research and Christos Lionis Professor of General Practice and Primary

Research Agenda for General Practice / Family Medicine and Primary Health Care in Europe Summary EGPRN EUROPEAN GENERAL PRACTICE RESEARCH NETWO RK EGPRN is a network organisation within WONCA Region Europe

When enough is enough Donna Bonde research by design About this White Paper The purpose of this white paper is to guide market research professionals on how many interviews are needed in qualitative research.

UMEÅ INTERNATIONAL SCHOOL OF PUBLIC HEALTH Master Programme in Public Health - Programme and Courses Academic year 2015-2016 Public Health and Clinical Medicine Umeå International School of Public Health

Exploring the barriers to effective communication between senior doctors and patients Developing the dance of a medical consultation Susan Salt Background Doctors are regularly criticised for not communicating

SHORT COURSES FOR 2016 Wits School of Public Health Division of Epidemiology and Biostatistics Obtain Certificate of Competence in : 1. Processing, Distribution & Archiving I 2. Programming for Research

FORMULATING THE RESEARCH QUESTION James D. Campbell, PhD Department of Family and Community Medicine University of Missouri Where do questions come from? From patient-centered questions in routine clinical

COMPREHENSIVE EXAMINATION Adopted May 31, 2005/Voted revisions in January, 2007, August, 2008, and November 2008 and adapted October, 2010 All students are required to successfully complete the Comprehensive

The Development of the Clinical Healthcare Support Worker Role: A Review of the Evidence Executive Summary The Development of the Clinical Healthcare Support Worker Role: A Review of the Evidence Executive

Analysing Interview Data Dr Maria de Hoyos & Dr Sally-Anne Barnes Warwick Institute for Employment Research 15 February 2012 Show of hands Aims of the session To reflect on the nature and purpose of interviews

Biost/Stat 578 B: Data Analysis Emerson, September 29, 2003 Handout #1 Organizing Your Approach to a Data Analysis The general theme should be to maximize thinking about the data analysis and to minimize

Research & Development Guidance for Students 2 Contents Introduction 3 Understanding the Research Approval Process 3 Is My Project Audit, Research or Something Else 4 What Next? 4 R&D Step by step Guide

GRADUATE EDUCATION AT USM: A STATEMENT OF THE GRADUATE COUNCIL March 2, 2009 Graduate education is an essential part of the University of Southern Maine. As one of only two universities within the University

Chapter 7: Nursing Organization Teaching Goals After finishing this chapter, every student should have a firm concept of both what nursing contributes to high quality, safe, efficient, and effective patient