On the outside looking in

6 February 2018

We’re loving February for many different reasons.

This is the month when we realise time does not stand still, and the first portionof the year is gone. Inevitably those who made new years’ resolutions will pause to think about their progress with their commitment to lose weight, spend more time with friends and family, study, sleep more, drink less, quit smoking, etc.This means that February may bring some stress, disappointment, or satisfaction, depending on what we did in January.

February is also a reminder that summer in the Southern hemisphere will not last forever, and that autumn and winter will be here soon. Depending on whether you love summer or winter best, you may love or hate it that days are becoming shorter and cooler. With winter approaching, there is hope for rain in the drought-ridden Western Cape.

There is also the State of the Nation Address, on 8 February 2018. This key political event is an opportunity for the President to report on the economic and social state of the nation, and includes a review of the past year and priorities for 2018. Given the current state of affairs this SONA is going to be decisive in determining how South Africa’s future political and economic landscape unfolds.

For those who are romantically inclined and are looking for an opportunity to make a (daring) move, February is the time. It is hard not to notice the red hearts, roses, candles teddy bears that and other items signalling Valentine’s Day on the 14th. Not everyone celebrates the day – some call it “Single Awareness Day” and others just ignore it.

Interestingly, there is no agreement on the origins of the day of love. Some say it originated from an ancient pagan festival of lust and nudity in the times of the Roman Empire; others say it morphed into a Christian celebration of St. Valentine as the Catholic Church gained ground; and an alternative explanation is that it is just an excuse to celebrate spring in February – that is in the Northern hemisphere.

So, what does all of this have to do with evaluation?

Apart from the obvious need to plan properly to make those New Year’s’ resolutions work out as intended – read Theory of Change, Theory of Action, Indicators, Targets, Timelines – there are two other things that come up: context, and values.

The importance of context is clear from the above thoughts on February – for everyone in the world it signals the approach of a new season, but depending on whether you live in the Southern or the Northern hemisphere, the approaching season will be different. The issue of context in evaluation is equally important.

It is also clear that where we “stand”, or in other words, our own perspectives and situation has an influence on what we think about February. Similarly, in evaluation, every evaluator has a position, like it or not. As evaluators we give careful consideration to many aspects and variables, but are we conscious enough of how our own background, position in society, gender, age, preferences – the list can continue endlessly – shape how we engage with an evaluation, right from the time when we read the Terms of Reference).

February is a reminder that it is a good idea for evaluators to pause, stand back, do some introspection, and check our assumptions, biases and filters. As evaluators, we need to make sound judgements about the projects and programmes we evaluate, and we cannot take that responsibility lightly. This said, we also have to be conscious that finding the truth is not as simple as it seems. Who we are, where we come from, what we value, and what we want from the future may influence us profoundly in our work, particularly in qualitative inquiry, where “the human being is the instrument of data collection” (Patton, 2002: 51).

Patton proposes that the social scientists need to “carefully reflect on, deal with, and report potential sources of bias and error” in addition to other ways[1] to ensure that high-quality qualitative data, which is trustworthy, and fair to the phenomenon and/or the people being studied. If not, our work will be worthless.

Bias can occur at any stage of research, even when we decide what to study/evaluate, or how the research questions are formulated. Even in research where rigorous quantitative data is collected, bias and distortion can occur. For example, measurement bias happens when values are produced that “systematically differ from the true value in some way… one of the things that might contribute to response bias is the appearance or behaviour of the person asking the question, the group or organisation conducting the study, and the tendency for people not to be completely honest when asking about illegal behaviour or unpopular beliefs.” (Peck, Olden, Devore, 2012: 38)

Without elaborating on all the potential sources of bias, it should be mentioned that social theory is steeped in values, embedded in the context and culture from where they emanate. Some argue that “researchers must be unbiased” (Weber 1949 in Neuman, 2003: 497), while the idea of value-free research is disputed as being “a value in favour of ‘value free’”, and that it is used to disguise or hide scientists and professionals’ own values (Gouldner, 1976) in Neuman, 2003: 497).

Realising that it is virtually impossible to be totally value-free, Gouldner recommended that researchers make their own values explicit, and saw value in how researchers’ own values can motivate them to make a meaningful contribution: “The researcher who is motivated by a strong moral desire to effect change need not invalidate good research practice” (Neuman, 20013: 497).

Another suggestion comes from Patton (2002: 51), who says that credible research requires a “stance of neutrality” regarding the phenomenon being studied. This means that the researcher does not attempt to prove any perspective, or does not attempt to arrive at “predisposed truths”, but has the intention to understand what is being studied, with all its complexities and the multiple perspectives related to the phenomenon.

Perhaps the proposition that social researchers should “adopt a relational position – a position apart from any specific social group, yet in touch with all groups” should be considered (Mannheim, 1936 in Neuman, 2003: 497 – 498). A relational position requires social researchers to be somewhat detached, or marginal in society, albeit with “connections with all parts of society, even parts that are often overlooked or hidden.” Roughly translated in plain language, this could mean “on the outside looking in”.

In the diverse South African society, where all are affected and influenced in some way by the history of the country, and where many issues are highly politicised, it is particularly relevant for researchers and evaluators to engage in discussions on values, and positioning, and perhaps to define what “relational positioning” means in the South African context, and what we can contribute to society through our unique positioning. This discussion could be useful to interrogate the potential contribution of social research, and particularly evaluation, in an evidence-based policy environment.

But before we start with this serious and important discussion – enjoy February, and happy Valentine’s day.