a personal-public notebook about human development(s) ... by michaela raab

Tuesday, 26 May 2015

Too quick, too dirty

Today I came across the (virtual) file of a "quick and dirty" evaluation I carried out a while ago. That made me feel a bit queasy - because I should have turned down that offer, or advised my clients to go for something radically different. Quick and dirty can be quite wasteful.Someone had approached me for the job at very short notice - the expectation was that the consultant would start travelling within a couple of weeks. I happened to have time - some other assignment had been postponed - and I found the subject matter interesting. That's why I accepted the job. I was worried about the terms of reference, though. They came with plenty of specific ideas as to how the evaluation would have to be carried out - including requirements to visit three countries, to conduct a survey and to complete the job within some 30 person-days spread across two months. Within that time frame, the relevance, effectiveness, efficiency and sustainability of a project covering several rural areas across three countries was to be assessed. Oh, impact, too, but I talked them out of that.There was hardly any time for me to study the project documentation; desk study, travel planning and country visits had to happen in parallel. As a result, I knew only some things about the project when devising my interview guides and survey questions. There were plenty of other things I read about too late to build them into the survey, or to contact interesting interlocutors I had not thought of before. Everything and everybody was so rushed. Local counterparts in the first country on the visiting lists had less than a week to come up with an itinerary. I had hardly any influence at all on the selection of my interviewees, as I had just started with the desk review when the interview dates were set. There was no time to mull over the data and over the encounters in the countries, either - the draft evaluation report had to be ready within a week from completing my trips; the final report a week later. Oh, and I think I never quite worked out which were the final versions of the manuals the project was supposed to develop: The person who could have told me which draft was the ultimate one was on leave throughout the evaluation period.The project came with design flaws which threatened its relevance. Those flaws were quite easy to detect from the written materials that were made available to me, starting from the project proposal. I believe the most useful findings I came up with, at the end of that rushed evaluation, were about those design flaws. I would not have needed to travel at all to see that. I trust I could have found out much more, even within the same evaluation budget, if twice or three times as much time could have been made available for the evaluation. I could have read the full project documentation before deciding which sites to visit, who to talk to and what to ask. I could have asked the local counterparts to run a simple survey with the users of their services (see also my post "A written survey with people who don't read or write"), and thus gather interesting information from plenty of people and from all project sites - rather than relying on quick, rather contrived conversations with those who could be drummed up at short notice. Not that anyone would have complained about the quality of my evaluation. I got plenty of friendly feed-back throughout. But a less rushed process would have yielded so many more insights. And it wouldn't have needed to cost more. My plea to those who commission evaluations: Don't rush it. Most evaluations are built into the project cycle. Nothing can keep you from planning an evaluation one year before it is actually supposed to start. You will get so much more out of it.

No comments:

About me

I am a consultant with 28 years of experience in development - in particular, monitoring & evaluation, facilitation, training and applied research with a range of organisations and in many places around the world. I live in Berlin and work in several languages.This blog is a free time venture - so please bear with typos and occasional long waits between the posts. Scroll down if wish to be alerted by email when new posts come up. Comments are most welcome.

Search This Blog

These pages sort old posts by subject. For recent posts, stay on my main page.

Follow by Email

Subscribe To

Total Pageviews

Disclaimer/ "Impressum"

This is my personal blog, which I use to share thoughts, informations and news, based on my personal knowledge and perceptions. I do my best to ensure accuracy and truthfulness but cannot be held legally responsible for any content on this site, which is meant for informal exchange only. As to the links offered here: the responsibility for content on these links remains with those who run the respective sites.