Although serving a utilitarian purpose, transcripts effectively convert the all-too-human research experience that defines qualitative inquiry to the relatively emotionless drab confines of black-on-white text. Gone is the profound mood swing that descended over the participant when the interviewer asked about his elderly mother. Yes, there is text in the transcript that conveys some aspect of this mood but only to the extent that the participant is able to articulate it. Gone is the tone of voice that fluctuated depending on what aspect of the participant’s hospital visit was being discussed. Yes, the transcriptionist noted a change in voice but it is the significance and predictability of these voice changes that the interviewer grew to know over time that is missing from the transcript. Gone is an understanding of the lopsided interaction in the focus group discussion among teenagers. Yes, the analyst can ascertain from the transcript that a few in the group talked more than others but what is missing is the near-indescribable sounds dominant participants made to stifle other participants and the choked atmosphere that pervaded the discussion along with the entire group environment.

Missing from this article is an explicit discussion of the central role audio and/or video recordings – that accompany verbal qualitative research modes, e.g., face-to-face and telephone group discussions and in-depth interviews (IDIs) – play in the analysis of qualitative data. Researchers who routinely utilize recordings during analysis are more likely to derive valid interpretations of the data while also staying connected to the fundamental goal – the raison d’être – of qualitative research, i.e., to embrace the complicated realm of the lived experience to gain an in-depth understanding of people in relationship to the research question(s).

In this regard, there are at least two key advantages to conducting a careful examination of the recordings, advantages that are missing when solely relying on transcripts. A review of the recordings

Aids in recalling peripheral but critical content. This is content that is typically deemed outside the scope of interest by the transcriptionist, such as the “mood swing” mentioned in the above excerpt. In that case, a review of the recording allows the researcher to hear (and see in a video recording) the energy in the participant’s voice when talking about his mother’s illness and reminds the researcher of how this energy ebbed and flowed, bouncing from rapid-fire gleeful enthusiasm to barely audible doubt and despair spoken in unusual voice variations and accompanied by fully engaged eye contact or distracted attention depending on the direction of his mood.

Clarifies meaning by way of a broader context. As the excerpt above suggests, it is only by re-living the focus group discussion with teenagers through the recording that the researcher begins to gain an understanding of the profundity of the “choked atmosphere” in the group and its impact on the outcomes. Unlike the transcript, the recording reminds the researcher of how and when the atmosphere in the group environment shifted from being open and friendly to quiet and inhibited; and how the particular seating arrangement, coupled with incompatible personality types, inflamed the atmosphere and seriously colored participants’ words, engagement, and way of thinking. The discussion content and derived meaning gathered within this context will clearly be at odds with the content and meaning derived from a separate focus group discussion consisting of teenagers with similar characteristics, discussing responses to the same discussion guide, but with personalities that foster a supportive group dynamic environment.

Qualitative researchers owe it to their participants to think carefully about the nuance and complexities of their lives as shared in a focus group discussion or IDI. Not unlike note taking (discussed here), developing a standard practice of reviewing recordings “helps to maintain the all-important participant-researcher relationship” by preserving the integrity of the qualitative event and retaining the essence of what it means to conduct qualitative research.

Filed under: Audio/video Recordings, Qualitative Research, Research Analysis Tagged: Audio/video Recordings, qualitative analysis, research analysis]]>https://researchdesignreview.com/2017/08/15/the-virtue-of-recordings-in-qualitative-analysis/feed/0Margaret R. RollerThe Unique Quality of Qualitative Content Analysishttps://researchdesignreview.com/2017/07/29/the-unique-quality-of-qualitative-content-analysis/
https://researchdesignreview.com/2017/07/29/the-unique-quality-of-qualitative-content-analysis/#commentsSat, 29 Jul 2017 19:12:20 +0000http://researchdesignreview.com/?p=1745]]>A unique attribute of qualitative content analysis is the focus on a continual process of revising and developing meanings in the data based on new discoveries. Unlike quantitative content analysts who set their coding scheme early in the research process — typically modifying it only slightly or not at all during data collection — qualitative researchers methodically and frequently revisit the content they are studying to better understand each relevant piece as well as its relationship to the entire context from which it was chosen (sampled), thereby modifying how and what they are coding throughout the data collection period. In this way, and as Krippendorff (2013) points out, qualitative content analysis puts the analyst in a hermeneutic circle1 whereby interpretations are reformulated based on new insights related to, for example, a larger context.

This more flexible, less rigid, approach to content analysis also embraces the notion of multiple meanings derived from multiple sources. A case in point is triangulation, which is used in qualitative analysis to verify the analyst’s interpretations by considering alternative points of view or analyzing deviant cases. It is this more far-reaching consideration of the data — along with the added support of the research participants’ verbatim comments that are typically included in the final research document — that is indicative of the unique qualities of the qualitative approach.

Indeed, it is the inductive strategy in search of latent content, the use of context, the back-and-forth flexibility throughout the analytical process, and the continual questioning of preliminary interpretations that set qualitative content analysis apart from the quantitative method.

Adapted from Roller & Lavrakas, 2015, p. 233-234.

1Krippendorff (2013) uses the concept of the “hermeneutic circle” in content analysis to mean that “text is interpreted relative to an imagined context, and these interpretations in turn reconstruct the context for further examination of the same or subsequently available text” (p. 259).

Filed under: qualitative content analysis, Qualitative Research Tagged: content analysis, qualitative content analysis]]>https://researchdesignreview.com/2017/07/29/the-unique-quality-of-qualitative-content-analysis/feed/1Margaret R. RollerThe “Real Ethnography” of Michael Agarhttps://researchdesignreview.com/2017/07/21/the-real-ethnography-of-michael-agar/
https://researchdesignreview.com/2017/07/21/the-real-ethnography-of-michael-agar/#commentsFri, 21 Jul 2017 14:55:21 +0000http://researchdesignreview.com/?p=1729]]>Several years ago, when working on Applied Qualitative Research Design, I began reading the works of Michael Agar. To simply say that Agar was an anthropologist would be cutting him short; and, indeed, Anthropology News, in an article published shortly after Agar’s death in May 2017, described him as

“a linguistic anthropologist, a cultural anthropologist, almost an South Asianist, a drug expert, a medical anthropologist, an applied anthropologist, a practicing anthropologist, a public anthropologist, a professional anthropologist, a professional stranger, a theoretical anthropologist, an academic anthropologist, an independent consultant, a cross cultural consultant, a computer modeler, an agent-based modeler, a complexity theorist, an environmentalist, a water expert, a teacher…”

One doesn’t need to look far to be enlightened as well as entertained by Mike Agar – On the “Scribblings” page of his Ethknoworks website, he lightheartedly rants about the little money most authors make in royalties stating “If you divide money earned by time invested in writing and publishing, you’ll see that you’d do better with a paper route in Antarctica.” It may be this combined ability to enlighten and entertain that drew me to Agar and keeps me ever mindful of the words he has written and the ideas he instilled.

For some reason I come back to his 2006 article “An Ethnography By Any Other Name…”. In it, Agar explores the question “What is a real ethnography?” with discussions of debates (“tension”) between anthropologists and sociologists, and about various nuances such as whether applied anthropology is actually “real” given that “ethnography no longer meant a year or more by yourself in a village far from home” (Agar, 2006, p. 4), where ethnographers’ focus should be (the community or a particular problem), and geographical (Agar was deemed a “South Asianist”) and institutional labels. These debates have sparked many questions including “Is educational ethnography really ethnography?” (Agar, 2006, p.3) as well as the provocative, Is ethnography really “qualitative research”? These days, Agar might also wonder about modern-day “in-home ethnographies” and “video ethnography,” asking What are these approaches really, and can we really call them “ethnography”?

Of particular interest in this 2006 article is Agar’s discussion of what he considers “acceptable and unacceptable ethnography,” and specifically his focus on abductive logic along with meaning and context. The emphasis here is on the idea that any ethnography “has to produce new concepts” untethered from earlier or existing theories and instead emerging from the researcher’s embrace of “surprises” in the data and an eagerness to pursue them. This willingness to pursue revolutionary observations in the data also supports the added notion of “iterative abduction” which speaks to a flexible approach to ethnography, e.g., altering the interview guide as warranted after each set of two or three interviews. Flexibility is an important attribute to qualitative research and is actually one of 10 unique attributes discussed in an earlier article in Research Design Review.

But acceptable ethnography, according to Agar, goes beyond abductive logic to include meaning and context. Importantly, Agar is referring to the meaning and context which is derived from absorbing a different point of view while in pursuit of surprising concepts. In doing so, the ethnographer is not looking for or analyzing “variables” within an observed event but rather “patterns” of behavior or activities. Like flexibility, meaning and context are two of the 10 unique attributes associated with qualitative research as discussed in the RDR article mentioned earlier. Going one step further, I would suggest that meaning, context, as well as the participant-researcher relationship are the three unique attributes of qualitative research that underscore and serve to define the remaining seven attributes.

“I find that I am much more able to ‘do sensory ethnography’ when I slow down and take the time to properly assess people and situations. My bias and assumptions need to be set aside, and I must seek to truly sense the truth about the object that I am studying. My view must be both broad and detailed, and my account to others must embody the truest picture possible.”

As in all qualitative research, the research skills of most import in the ethnographic approach are those of patience, reflection, the ability to set aside assumptions and beliefs while also embracing the meaning and context of our participants in order to come as close as we are capable to their reality.

Filed under: Context, Ethnography, Personal Meaning, Qualitative Research, Research Design & Methods Tagged: context, ethnography, Michael Agar, qualitative research design]]>https://researchdesignreview.com/2017/07/21/the-real-ethnography-of-michael-agar/feed/1Margaret R. RollerRapport & Reflection: The Pivotal Role of Note Taking in In-depth Interview Researchhttps://researchdesignreview.com/2017/06/26/rapport-reflection-the-pivotal-role-of-note-taking-in-in-depth-interview-research/
https://researchdesignreview.com/2017/06/26/rapport-reflection-the-pivotal-role-of-note-taking-in-in-depth-interview-research/#commentsMon, 26 Jun 2017 18:38:59 +0000http://researchdesignreview.com/?p=1720]]>Note taking is fundamental to the in-depth interviewing process and an essential interviewer skill. And yet note taking – e.g., why note taking is important, how to take notes, and how to use notes from a completed interview – does not get much attention. Note taking is important – actually, critical – to the in-depth interview method because it is about much more than jotting down a participant’s comments and responses to the interviewer’s questions.

In fact, an effective note taker is a more effective interviewer. This is because

Taking notes during an interview helps to focus the interviewer’s attention on the participant’s point of view and lived experience relevant to the research question.

Taking notes helps the interviewer internalize what is being said by the participant which in turn helps the interviewer identify seemingly contradictory statements and follow up on new, insightful topic areas that may not appear on the interview guide.

The interviewer’s heightened focused attention and internalization helps to build rapport and enhances the participant-researcher relationship.

The interviewer can add sidebar notations while taking notes that add context to what is being discussed or remind the interviewer to follow up on a particular comment.

Taking notes allows the interviewer to identify and flag important quotes made by the participant in the moment when the contextual import of participant’s statements can be fully appreciated and noted.

An effective note taker is also better equipped to conduct meaningful analyses of the data, leading to useful outcomes. This is because

The notes serve as an immediate resource for reflection: 1) during the interview – when the interviewer can flip back and forth to consider the participant’s earlier comments and ask for clarification as the need arises to fully comprehend and better analyze the participant’s point of view – and 2) at the completion of the interview – when the interviewer can quietly review the interview notes and add any informative annotations that will aid analysis.

The interviewer can use the notes from each interview to record the participant’s attitudes and behavior related to each primary and secondary research question. Ideally, this should be done within an hour of the interview completion and by way of Excel, where the columns consist of key research questions and the rows contain input from each participant. This format allows the researcher to quickly capture interview data when it is fresh on the mind as well as easily review and analyze the data within and across participants.

Importantly, the note taking discussed here pertains to notes writtenby hand (pen [or Echo smartpen] on paper) in contrast to taking notes with an electronic device. Research has shown that the use of laptops (for example) is great at creating large volumes of notes (with lots of verbatims) but it also encourages a “mindless” transcription rather than a meaningful engagement with the material. Indeed, as reported in this research, individuals who wrote their notes by hand demonstrated “a stronger conceptual understanding and were more successful in applying and integrating the material” compared to those who took notes with their laptops.

Handwritten note taking compels the interviewer to fully engage with the participant and fosters highly reflective behavior in the researcher. You might say that, in this way, note taking helps to maintain the all-important participant-researcher relationship throughout data collection and analysis; a relationship that can be too easily lost when utilizing more mechanical processes such as the reliance on audio recordings and data transcripts.

Filed under: In-depth Interviewing, In-depth interviews, Note Taking, Qualitative Research, Researcher Skills Tagged: in-depth interviewing, Note Taking, participant-research relationship, qualitative analysis, qualitative research design]]>https://researchdesignreview.com/2017/06/26/rapport-reflection-the-pivotal-role-of-note-taking-in-in-depth-interview-research/feed/4Margaret R. RollerRe-considering the Question of “Why” in Qualitative Researchhttps://researchdesignreview.com/2017/06/16/re-considering-the-question-of-why-in-qualitative-research/
https://researchdesignreview.com/2017/06/16/re-considering-the-question-of-why-in-qualitative-research/#commentsFri, 16 Jun 2017 21:09:17 +0000http://researchdesignreview.com/?p=1693]]>It is easy to fall into the trap of relying on the “why” question when conducting qualitative research. After all, the use of qualitative research is often supported with the claim that qualitative methods enable the researcher to reach beyond quantitative numerical data to grasp the meaning and motivations – that is, the why – associated with particular attitudes and behavior. And it is in this spirit that researchers frequently find themselves with interview and discussion guides full of “why” questions – Why do you say you are happy? Why do you prefer one political candidate over another? Why do you diet? Why do you believe in God? Why do you use a tablet rather than a laptop computer?

Yet “why” is rarely the question worth asking. In fact, asking “why” questions can actually have a negative effect on data collection (i.e., Credibility) and contribute bias to qualitative data. This happens for many reasons, here are just four:

The “why” question potentially

• Evokes rationality. By asking the “why” question, researchers are in essence asking participants to justify their attitudes and behavior. In contemplating a justification, it is not unusual for participants to seek a response that “makes sense,” seems logical, or is otherwise deemed appropriate. This defensive reaction may go unnoticed (by the participant as well as the researcher) unless participants are asked to reflect further on their rationalizations, allowing the researcher to identify and mitigate potential bias associated with social desirability and other forms of distortions.

• Stifles the researcher-participant conversation. The “why” question potentially stifles the research interview or group discussion in at least two ways: 1) It stops the flow of conversation while the participant considers rational scenarios in response to the researcher’s question and 2) It requires a certain amount of backtracking by the participant to explain a rationalization that hopefully “makes sense” but may not be particularly relevant to the research topic or intended question.

• Clouds question meaning. Along with potentially stifling the interview or group discussion, the “why” question does little to convey the researcher’s intent or meaning of the question. As a wide-open question, the participant may struggle with its ambiguity and become frustrated in attempts to find meaning. In this regard, the “why” question potentially results in – what survey researchers call – “respondent burden.” For example, it is much easier on the participant, and more informative for the researcher, when the question is “What are the specific aspects of your life that make you happy?” compared to “Why do you say you are happy?”

• Asks a different question from the one intended by the researcher. In addition to being construed as vague or ambiguous, the “why” question might also be interpreted as asking something different than the researcher’s intent. Because of this potential for misinterpretation, the researcher needs to think carefully before asking the “why” question. For example, the question “Why do you use a tablet rather than a laptop computer?” is essentially a different question than “How does a tablet computer offer you advantages over a laptop?

With qualitative inquiry researchers gain critical insight on the lived experience. But this insight is not necessarily rooted in the why of life events as much as it is in the aspects of participants’ lives that can only be discovered by asking what, when, where, who, how – and sometimes, why.

Filed under: Credibility, Qualitative Research, Question Design, Research Design & Methods Tagged: focus group discussions, in-depth interviewing, qualitative research design, question design]]>https://researchdesignreview.com/2017/06/16/re-considering-the-question-of-why-in-qualitative-research/feed/2Margaret R. RollerMaking Connections: Practical Applications of the Total Quality Framework in Mixed Methods Researchhttps://researchdesignreview.com/2017/05/30/making-connections-practical-applications-of-the-total-quality-framework-in-mixed-methods-research/
https://researchdesignreview.com/2017/05/30/making-connections-practical-applications-of-the-total-quality-framework-in-mixed-methods-research/#respondTue, 30 May 2017 12:32:55 +0000http://researchdesignreview.com/?p=1672]]>The Total Quality Framework (TQF) (Roller & Lavrakas, 2015) offers researchers a way to think about qualitative research design from the vantage point of core principles. It is an approach that helps qualitative researchers develop critical thinking skills by giving explicit attention to the quality of the conceptualization and implementation of their qualitative studies. The TQF is composed of four components, each pertaining to a phase of the research process – data collection (Credibility), analysis (Analyzability), reporting (Transparency), and the ability to do something of value with the outcomes (Usefulness).

Qualitative research is most often conducted as a standalone study but frequently conducted in conjunction with quantitative methods. A mixed methods research (MMR) design involves collecting both qualitative and quantitative data, then integrating or connecting the two datasets to draw interpretations derived from the combined strengths of both sets of data (Creswell, 2015). The integration of, or making the connection between, the qualitative and quantitative components is fundamental to MMR and distinguishes it from a multi-method approach that simply utilizes different methods. In contrast, a mixed methods design incorporates any number of qualitative and quantitative methods (and modes) with the specific intention of blending the data in some fashion. Mixed methods research is the subject of an earlier article in Research Design Review.

So, how do we apply the TQF to a MMR design? It is not good enough to simply think of the qualitative component of MMR as a separate feature to the overall design and apply a TQF approach to the qualitative method(s). For MMR, the TQF needs to be adapted to accommodate a qualitative-quantitative connection as discussed earlier. There are many ways to do this. A few practical applications of the TQF in MMR are outlined below.

Credibility (Data Collection)

A necessary and highly practical consideration in the course of collecting in-depth interview data is the question of the number of interviews to complete. To address this question, the TQF presents 10 related questions* for the researcher to contemplate when in the field, such as

Did all interviewees provide clear, unambiguous answers to key questions or issues, or does the researcher need to go back to some interviewees for clarification?

Can the researcher identify the sources for variation and contradictions within the data?

Do the data confirm or deny what is already known about the subject matter?

The kinds of questions the researcher might contemplate in a MMR design are similar but are now tweaked to connect qualitative data gathering with the quantitative component. In each case, the researcher is expanding his/her thinking to consider the implications associated with the collecting of qualitative data as well as that associated with the quantitative. The researcher conducting a MMR study might now consider,

Did all interviewees provide clear, unambiguous answers to key questions or issues; if not, does the researcher need to go back to the participant(s) or leave clarification for the quantitative component?

Can the researcher identify the sources for variation and contradictions within the qualitative data as well as between the qualitative and quantitative data?

Do the data confirm or deny what is known from the quantitative data?

Analyzability (Analysis)

The TQF offers numerous ways to approach the processing and verification of qualitative data. One of the suggested verification strategies has to do with reflexivity and, specifically, the reflexive journal. The reflexive journal gives researchers the opportunity to respond to questions intended to foster introspection along with an understanding of the researcher’s effect on the qualitative data. These reflections further the researcher’s ability to verify the interpretations of qualitative data during the analysis process. In a standalone qualitative study, the researcher’s reflexive journal might include the contemplation of such questions as*

What do I think I “know” from this/these participant(s) and how do I think I “know” it?

What assumptions did I make (what did I assume to be true) about the participant(s)?

If the researcher was conducting MMR, the reflexive journal would address similar questions but now in the context of the broader MMR scheme. To connect the qualitative component with the quantitative, the reflexive journal asks the researcher to think about

What do I think I “know” from this/these participant(s) and how has that been influenced by what I may know from the quantitative data?

What assumptions did I make (what did I assume to be true) about the participant(s) based on what I may know about respondents to the quantitative survey?

How did my understanding of the quantitative data affect or shape the questions I asked, the interjections I made, my listening skills, and/or behavior?

Transparency (Reporting)

The Transparency component of the TQF has to do with reporting the outcomes in the final document; specifically, reporting a “thick description” of study details (NOTE: For earlier RDR articles on thick description, see this April 2017 article and this 2015 article). By conveying the details of the data collection and analysis processes, the researcher allows the users of the research (e.g., other researchers, the sponsor) to examine the researcher’s work and draw their own conclusions as well as transfer the design to other contexts. There are many details about the study that the researcher may want to address in the final document*, including the

Adequacy (i.e., comprehensiveness) of the lists that were used to represent the target population.

Failure to interview all interviewees sampled, efforts that were made to avoid this, and possible biases or weakness this may have caused.

Field notes (e.g., note-taking procedures, examples from the field notebook).

In MMR, the qualitative researcher needs to pay attention to connecting the qualitative component with the quantitative portion of the study. To do this in the reporting phase, the researcher interjects the thick description with details relevant to both the qualitative and the quantitative research. For example, the details might include the

Compatibility of the lists with that used in the quantitative phase.

Failure to interview comparable types of people, efforts that were made to avoid this, and possible biases or weakness this may have caused.

Ultimately, the objective of our research efforts is to derive outcomes that respond to the research question and provide outcomes that serve a valuable purpose. In many instances, a MMR approach fulfills this goal more so than a standalone qualitative or quantitative study by expanding and enriching the researcher’s understand beyond the “borders” of a mono-method study. The Journal of Mixed Methods Research and other resources are filled with examples of ways MMR has contributed to important societal issues:

Meaning-making underlying bereaved mothers’ adaptive and complicated grief responses to the death of a child from cancer (Gerrish, et al., 2014)

Food Safety

Gap between knowledge & behavior (Meysenburg et al., 2014).

When adapting a quality approach to the qualitative component of MMR, it is not sufficient to simply treat the qualitative portion as an independent element in the overall MMR design. Indeed, it is critical and fundamental to the MMR approach to make a connection between the qualitative and quantitative facets of the study. The few practical examples discussed in this article illustrate how qualitative researchers can make these connections while, at the same time, maintaining the integrity of the unique epistemology underpinning qualitative inquiry.

Filed under: Mixed Methods Research, Qualitative Research, Quality Standards, Total Quality Framework Tagged: mixed methods research, qualitative research design, Total Quality Framework]]>https://researchdesignreview.com/2017/05/30/making-connections-practical-applications-of-the-total-quality-framework-in-mixed-methods-research/feed/0Margaret R. RollerUseful Qualitative Research: The Total Quality Framework Usefulness Componenthttps://researchdesignreview.com/2017/05/03/useful-qualitative-research-the-total-quality-framework-usefulness-component/
https://researchdesignreview.com/2017/05/03/useful-qualitative-research-the-total-quality-framework-usefulness-component/#commentsWed, 03 May 2017 17:42:14 +0000http://researchdesignreview.com/?p=1651]]>Our research is of little value if the outcomes are not deemed useful in some way. This is true for all types of research. Whether it is qualitative, quantitative, or a mixed methods approach, the “carrot” that dangles ahead of the research team is the promise of reaching worthwhile, actionable conclusions and recommendations for the users and sponsors of the research. Achieving this objective – reaching the “carrot” of useful research – is the product of the quality measures put into place at the data collection, analysis, and reporting phases of the research design.

The Total Quality Framework (TQF)* offers a way of thinking about these quality measures in a qualitative research design. The TQF is comprised of four inter-related components, each having to do with a stage of the research process. Recent articles in Research Design Review have discussed three of these components – Credibility pertaining to data collection, Analyzability having to do with the processing and verification of qualitative data, and Transparency relating to the reporting of details associated with data collection, analysis, and the drawing of interpretations.

The fourth component of the TQF is Usefulness or the “ability to do something of value with the outcomes.” The ultimate strength of the Usefulness component is a function of the vigor – the attention to quality – within the Credibility (data collection), Analyzability (analysis), and Transparency (reporting) components. In this way, the Usefulness component relies on each of the other components independently as well as collectively. The goal is to maximize the value of a qualitative research study for the researchers affiliated with the study, the research sponsors, and the users of the research such as researchers working in comparable fields or contexts as well as students investigating the research topic for the first time. The TQF Usefulness component was briefly discussed in RDR back in 2012 – see “Designing Qualitative Research to Produce Outcomes You Can Use.”

Broadly speaking, the Usefulness component addresses the question, “What can and should be done with the study now that it has been completed?” Specifically, the usefulness of a qualitative study will be determined in varying degrees from study to study depending on the research question and the particular objectives. When evaluating usefulness, researchers may ask,

Has the study confirmed or refuted important hypotheses?

Has the study identified important knowledge gaps that future research should try to help close?

Has the study offered recommendations for action that are worthy of further testing or worthy of actionable next steps?

Has the study demonstrated the value of using new or refined methods for gathering qualitative data?

Has the study demonstrated new or refined methods for analyzing qualitative data?

In keeping with the important role of useful qualitative research, it is recommended that a “Usefulness of the Study” section be included in the final research document. The intended purpose of this section is to explain the researcher’s views on how the study should be interpreted, acted upon, or applied in other research contexts. Regardless of whether the consumers of the research agree with the stated views, this section serves as an important reminder of the researcher’s responsibility to impart their knowledge and perspectives associated with the ultimate goal – useful qualitative research giving people the ability to do something of value with the outcomes.

Filed under: Qualitative Research, Quality Standards, Research Design & Methods, Total Quality Framework, Total Quality Framework, usefulness, Usefulness Tagged: qualitative research design, Total Quality Framework, usefulness]]>https://researchdesignreview.com/2017/05/03/useful-qualitative-research-the-total-quality-framework-usefulness-component/feed/1Margaret R. RollerTransparent Qualitative Research: The Total Quality Framework Transparency Componenthttps://researchdesignreview.com/2017/04/20/transparent-qualitative-research-the-total-quality-framework-transparency-component/
https://researchdesignreview.com/2017/04/20/transparent-qualitative-research-the-total-quality-framework-transparency-component/#commentsThu, 20 Apr 2017 21:53:46 +0000http://researchdesignreview.com/?p=1639]]>The Total Quality Framework (TQF)* contributes to the conversation in the qualitative research community by providing researchers with a way to think about their qualitative designs – along with strategies or techniques – for the purpose of enhancing the quality of research outcomes. The TQF is a comprehensive approach that considers all stages of the research process – from data collection to the final “product.” Recent articles in Research Design Review discussed two of the four components of the TQF – specifically, the Credibility component and the Analyzability component. The Credibility component pertains to data collection and consists of Scope (having to do with sampling and coverage) and Data Gathering (having to do with minimizing potential bias, nonresponse, and other factors that may weaken the validity of the data). The Analyzability component of the TQF is focused on the Processing of qualitative data (e.g., the quality by which the initial “raw” data is transformed) as well as Verification of research findings and interpretations (e.g., by way of deviant cases, peer debriefs, the reflexive journal).

The third component of the TQF has to do with the next phase in a qualitative research design – that is, reporting. When the data has been collected and thoroughly processed and verified, the qualitative researcher is left with the job of effectively communicating what went on in the research study and how the researcher drew interpretations from the analysis. Importantly, the job of reporting goes beyond conveying the research findings and the researcher’s interpretations and recommendations, but also gives details of the research design having to do with Scope and Data Gathering (i.e., Credibility) as well as Processing and Verification (i.e., Analyzability). As discussed in this 2013 RDR article, the benefit of a detailed discussion of Credibility and Analyzability lies in its ability to fully inform the user of the quality strategies or techniques that were (or were not) incorporated into the design and, among other things, allow the user to evaluate the transferability of the research design, i.e., how well it might be used in a comparable context.

The elaboration of study details is referred to as “thick description” which is a term originally coined by British philosopher Gilbert Ryle and then adopted by Clifford Geertz to describe the work being done in ethnography (Ponterotto, 2006). In this respect, Geertz (2003) talks about the “multiplicity of complex conceptual structures” (p. 150) in ethnographic research, stating that “ethnography is thick description” (p. 156, emphasis added). Similarly, the “multiplicity” of design decisions that qualitative researchers make before, during, and at the completion of a qualitative study warrant a thick description in the final reporting document that explodes with rich details by which the user can essentially re-live the research process. In doing so, the user is able to evaluate his/her confidence in the research process as well as the researcher’s final interpretations and the applicability of the research to other contexts (i.e., transferability).

Representativeness of the participants to the population and why that was or was not a concern.

Level of cooperation and tactics that were used to maximize cooperation.

Ethical considerations.

Researcher/interviewer training.

Interview/focus group guide development.

Decisions that were made in the field, particularly decisions that changed the initial study design.

Field notes and the researcher’s reflexive journal.

Transcription process.

Data processing protocol and verification procedures.

As with Credibility and Analyzability, the Transparency component of the TQF is not intended to prescribe procedures or steps to follow in the reporting process but rather offer researchers a way of thinking about how to incorporate a complete accounting of a research study for the benefit of the user (e.g., the researcher, the research sponsor, a colleague working on a similar topic). It is by way of this thick description that qualitative researchers demonstrate their commitment to transparency while providing an audit trail of the relevant materials. This transparent approach to reporting expands the life of any given study and achieves the ultimate goal of allowing the user to do something of value with the outcomes. That brings us to the fourth and final TQF component, Usefulness.

Filed under: Qualitative Research, Quality Standards, Research Design & Methods, Total Quality Framework, Transparency, Transparency Tagged: qualitative research design, thick description, Total Quality Framework, transparency]]>https://researchdesignreview.com/2017/04/20/transparent-qualitative-research-the-total-quality-framework-transparency-component/feed/2Margaret R. RollerAnalyzable Qualitative Research: The Total Quality Framework Analyzability Componenthttps://researchdesignreview.com/2017/04/06/analyzable-qualitative-research-the-total-quality-framework-analyzability-component/
https://researchdesignreview.com/2017/04/06/analyzable-qualitative-research-the-total-quality-framework-analyzability-component/#commentsThu, 06 Apr 2017 12:31:17 +0000http://researchdesignreview.com/?p=1627]]>A March 2017 article in Research Design Review discussed the Credibility component of the Total Quality Framework (TQF). As stated in the March article, the TQF “offers qualitative researchers a way to think about the quality of their research designs across qualitative methods and irrespective of any particular paradigm or theoretical orientation” and revolves around the four phases of the qualitative research process – data collection, analysis, reporting, and doing something of value with the outcomes (i.e., usefulness). The Credibility piece of the TQF has to do with data collection. The main elements of Credibility are Scope and Data Gathering – i.e., how well the study is inclusive of the population of interest (Scope) and how well the data collected accurately represent the constructs the study set out to investigate (Data Gathering).

The present article briefly describes the second TQF component – Analyzability. Analyzability is concerned with the “completeness and accuracy of the analysis and interpretations” of the qualitative data derived in data collection and consists of two key parts – Processing and Verification. Processing involves the careful consideration of: (a) how the preliminary data are transformed into the final dataset that is used in analysis and (b) the actual analysis of the final set of data. The transformation of preliminary data typically involves converting audio or video recordings to a written transcript. From a TQF perspective, the qualitative researcher needs to give serious thought to, among other things, the quality of the transcripts created, with particular attention to the knowledge and accuracy of the transcriptionist*. The qualitative researcher also needs to reflect on the limitations of transcripts and, specifically, what can and cannot be learned from the data in transcript form.

Once the final dataset has been developed, the qualitative researcher is ready to make sense of the data by way of analysis. The analysis process may vary among researchers depending on their particular approach or orientation. Broadly speaking, the analysis involves: (a) selecting the unit of analysis (e.g., an entire in-depth interview), (b) developing codes (designations that give meaning to some portion of the data in the context of the interview and research question), (c) coding, (d) identifying categories (i.e., groups of codes that share an underlying construct), (e) identifying themes or patterns across categories, and (f) drawing interpretations and implications.

Verification is the other principal piece of the TQF Analyzability component. It is at the Verification stage – that is, when interpretations and implications are being conceptualized – that qualitative researchers give critical attention to the data by looking for alternative sources of evidence that support or contradict early interpretations of the study data. The verification step is an important one that contributes heavily to the overall quality of a qualitative research design. The various verification techniques include: (a) peer debriefing (the unbiased review of the research by an impartial peer), (b) a reflexive journal (the researcher’s diary of what went on in the study including reflections on their own values or beliefs that may have impacted data gathering or analysis), (c) triangulation (contrasting and comparing the data with other sources, such as data from different types of participants, different methods, or different interviewers or moderators), and (d) deviant cases (looking for “negative cases” or outliers that contradict the prevailing interpretation). There is another verification technique – member checking – that many researchers endorse but, from a TQF perspective, potentially weakens the quality of a qualitative study**.

Verification is the topic of discussion in a 2014 article posted in RDR – “Verification: Looking Beyond the Data in Qualitative Data Analysis.” Readers of this blog will also be interested in the Morse, et al. (2002) article in International Journal of Qualitative Methods on verification strategies where the authors advocate utilizing verification “mechanisms” during the course of the qualitative research per se (i.e., not just at the analysis stage) to ensure the “reliability and validity and, thus, the rigor of a study.”

Not unlike credible qualitative research (the subject of the March RDR post), analyzable qualitative research is the product of knowing how to think about quality approaches to data processing and verification. It is not about concrete procedures to follow but rather the ability to conceptualize and integrate research practices that maximize the validity as well as the ultimate usefulness of a qualitative research study. The TQF Analyzability component is a vehicle by which qualitative researchers can think about where and how to apply quality principles in the processing and verification of their data. In doing so, researchers gain rich interpretations of the data leading to outcomes that address the research question and have value.

Value or usefulness, however, is not solely dependent on credible and analyzable research. Before a qualitative study can be truly useful it must be effectively communicated. That is where Transparency – the third component of the TQF and the subject of the next blog post – comes in.

if it is agreed that qualitative research can, in fact, serve worthwhile purposes, then logically it would serve those purposes only to the degree that it is done well, regardless of the specific objectives that qualitative researchers strive to address. (p.20)

There are four components to the TQF – Credibility, Analyzability, Transparency, and Usefulness – each pertaining to a distinct aspect of the research process. The schematic (below) shows the interrelatedness of these components, with each of the first three components contributing to the fourth component, and ultimate goal of qualitative inquiry, i.e., Usefulness.

This article is a brief discussion of Credibility which is the TQF component having to do with data collection in qualitative research. Subsequent articles will be devoted to the other three components – Analyzability, Transparency, and Usefulness.

From a TQF perspective, credible qualitative research is the result of effectively managing data collection, paying particular attention to the two specific areas of Scope and Data Gathering. Scope has to do with how well the participants from which data are gathered represent the broader population of people that is the focus of investigation. There are four considerations related to Scope. The qualitative researcher needs to think about*: (a) defining the target population; (b) how these individuals will be selected for inclusion in the study (i.e., the source itself – e.g., a list to sample from, a community center to draw from – and the procedures to be used to sample from the source); (c) how many participants the researcher ultimately wants to include in the study; and (d) strategies to maximize the researcher’s ability to gain access to and cooperation from the people of interest.

There are articles in RDR that discuss the various considerations related to Scope. For example, a RDR post back in 2012 titled “Designing a Quality In-depth Interview Study: How Many Interviews Are Enough?” talked about the many factors researchers should think about when determining the number of in-depth interviews to complete for an IDI study, both at the initial design phase as well as when in the field.

Data Gathering is the other critical ingredient to Credibility. Data Gathering has to do with how well the data collected in a qualitative study accurately represent the concepts the study set out to investigate. Data Gathering, you might say, is concerned with construct validity (where “construct” may refer to anything from a narrow topic to a broad and possibly ambiguous concept), addressing the question of How confident am I that my data truly answer my research objectives? There are four considerations the qualitative researcher will want to think about when designing and conducting Data Gathering: (a) identifying the appropriate constructs – as well as the specific attributes within each construct – to measure based on the research question or objectives; (b) choosing the appropriate qualitative method as well as the appropriate mode; (c) developing the data collection tool(s) to effectively operationalize and measure the constructs and their attributes, e.g., the interview or discussion guide; and (d) mitigating sources of bias and inconsistency associated with the data collector (researcher) as well as the participants.

Credible qualitative research is derived, not from a strict set of rules to follow but rather, from a keen sense of the research objectives and an understanding of how to think about the research principles that apply to data collection in relationship to the research question under investigation. By way of the TQF Credibility component, qualitative researchers are encouraged to think carefully about the composition (and inclusiveness) of their participants along with the unbiased and consistent manner in which data is gathered. It goes without saying that the flexible and contextual nature of qualitative research will attract any number of missteps – e.g., a skewed participant mix or researcher effects that bias the data – but the point here is that qualitative researchers need to be conscious of these factors, to reflect upon them and record these reflections, and to use this information in the interpretation and reporting of findings. This, of course, is where the other TQF components come in.

Next time, the TQF Analyzability component.

*These considerations also pertain to qualitative content analysis where the focus is on objects and text rather than individuals.