Limitations and Constraints

In the final topic of this unit, we’re going to look at the perceived limitations of mixed methods. Remember that these arguments are, as ever, only one side of the coin: some ‘champions’ of mixed methods would not necessarily accept that such constraints exist or would argue for ways of overcoming them. However we’ve already had a good look at why, when and how to mix methods so it is now time to consider the potential limitations. In fact we have already touched on some of these – notably, the difficulty of achieving Mason’s ‘dialectical’ approach or Denzin’s triangulation by theory. When researchers try to bring together data collected by using two or more methods which carry different and conflicting ways of looking at the world, questions arise about consistency of research assumptions, clarity of knowledge claims and what Feyerabend (1993) called ‘incommensurability’. We considered some aspects of competing knowledge claims of the different research traditions in the first unit of this module; these arguments need to be revisited in relation to ‘mixing methods’.

The Risk of Pragmatism

Brannen (1992: 33) warns that ‘creeping pragmatism’ may result in studies which have no clear theoretical perspective. The researcher may get so caught up in technical details she forgets about the need to reflect about methods more widely. Indeed, where researchers work with colleagues from other disciplines, they may deliberately steer clear of theory lest differences emerge which cannot easily be reconciled (Mason 2006). In Topic 1, we looked at the argument that pragmatism is the ‘leading contender for the philosophical champion’ of mixed methods research (Greene 2008:8).

Go back and remind yourself of Johnson and Onwuegbozie’s (2004) arguments for pragmatism as a theoretical basis for mixed methods research in Educational Researcher, 33, 7, 14-26.

In Topic 1, we looked at Peter Halfpenny’s contention that particular research methods are not inextricably linked to either the positivist or interpretivist paradigm: therefore, mixed methods design does not entail conflicting ontological or epistemological understandings. We saw that Martin Hammersley took this argument further or, more accurately, added a new one: not only are methods independent of paradigms, but the much debated differences between paradigms are in fact over-blown – no more than ‘working assumptions’. Many theorists of methodology contest this view: the concept of ‘incommensurability’ is central to their argument.

The Incommensurability of Theory

Within the philosophy of science, the notion of incommensurability arises in debates about method which are broader than – but nonetheless relevant to - the positivist / interpretivist divide within social science. This is a complex area which Smith (2000) writes about in an accessible way. He says there are three ways of looking at science:

The traditional inductivist view: Derived from the work of John Stuart Mill, this understanding of science contends that, through a process of inference and induction based on preceding observation, scientists deduce causal relations. If verified, (eg: if the assumed relationships explain the observation satisfactorily), then the hypothesis is correct. Science proceeds by collecting factual data through observation and experiment, leading to generalisations and causal laws which can be verified. Key points of this view are, first, hypotheses follow observation and, secondly, it is possible to arrive at the truth, or ‘true’ theories.

Hypothetico-deductive view: Based on the work of Karl Popper, Smith (2000) explains that this approach begins with hypotheses and moves on to observation or experiment to test these out: hypotheses are then developed or discarded in the light of these observations. Thus, observations are interpreted in the light of theory, a process also influenced by our biases and preconceptions. Popper believed that theory can never be proved correct but can be falsified: there is no absolute truth and ‘pure facts’ do not exist – a radical departure from the inductivist view. It is this ability to ‘falsify’ theory which differentiates science from non-science. Note that both the inductivist and the hypothetico-deductive views involve use of particular methods, tied to very different theoretical approaches.

Thomas Kuhn (1962) disagreed with Popper that falsifiability is the distinguishing feature of science, arguing that science, or rather a field of science, is identified by the fact that it is led by one clear paradigm and scientists within the field work within that framework. In its early (‘pre-paradigmatic’) stages, a discipline is likely to be marked by various competing schools, debates and/ or paradigms. Over time, one of these paradigms will come to dominate and eventually defeat the others because it seems to offer a better explanatory framework. Kuhn contended that as yet no such paradigm exists in the social sciences. So long as social science is riven by competing paradigms, it cannot claim to be a science (Smith 2000).

Feyerabend (1993) used the term ‘incommensurability’ to refer to theories which have no common language or assumptions and which cannot therefore be compared, let alone brought together to form a coherent world view. If, as Feyerabend argued, competing paradigms are irreconcilable, then within the social sciences it makes no sense to draw on theories from different paradigms or bring together data derived from different methods, because they will point to conflicting explanations at both micro level (the findings of a particular study) and macro level (the nature of the social world). In addition, methods are at risk of dilution or distortion if divorced from their theoretical origin. Discourse analysis, derived from Foucault’s particular ontological perspective, could be seen as a case in point. Unhinged from its Foucauldian base, it is now widely used within other and no theoretical frameworks.

The concept of incommensurability can be helpful in identifying contradictions in our understandings of the social world and the limits to our knowledge. In an age when policy makers demand research which will unproblematicaly identify ‘what works’, the notion of incommensurability may serve as a salutary reminder that science is, as Popper implied, concerned with what does not work, and why. The implications for using mixed methods is that we may not always be able to achieve a unified research design nor a perfect coming together of different types of data .

Munro (2010:276) summarises:

'As Jameson (2006) argues, incommensurability not only refers to the conflict between two irreconcilable theories, but to the predicament that a researcher [may] require both at the same time yet cannot fit them together in a systematic way. In other words, it is this need for both levels that clashes with the inability to construct a coherent system out of their differences that makes incommensurability a significant problem. It reflects the constraints of scientific method and the significance it places on contradiction and conflict as essential elements to social reality..... As Jameson (1990: 38) writes:

"[E]ven at their most intellectually energetic, the concepts of sociology cannot but be flawed and fractured, since their very object is contradictory, faithfulness to it thereby requiring a certain transfer of the social contradictions into thought" '.

Task 3

The ‘Learning Lives’ Project is a large scale collaborative study between researchers in the Universities of Exeter, Brighton, Leeds and Stirling. The focus of the research is in the inter-relationships between learning, identity and agency in people’s lives with a view to drawing out implications for policy and practice. You can access lots of information about this project on the project website

Read the beginning of Working Paper 1 to get an overview of the project scales and methods used. What is the study trying to achieve and what approach to mixed methods has been adopted?

Find the paper Contrasting Concepts of Learning and Contrasting Research Methodologies by Phil Hodkinson and Flora McLeod presented at the TLRP Annual Conference, Cardiff, 26-27 November 2007 on this page listing various conference contributions (it is quite near the top!).

This paper includes discussion of methods used in another project in addition to ‘Learning Lives’ and investigates links between different theoretical perspectives. Consider what issues covered in this unit are reflected in this paper.

Another example of a project using mixed methods is the 'Collaborative Lives Project' which investigated neighbourhood, networks and communities. The methodological approach brought together perspectives from a range of disciplines including health, transport studies, human geography, informatics and sociology. To find out more about their methods read the paper by Emmel and Clark (2009)