Go Figure: another Sistema No Brainer

Something is stirring in Sistema land. In the last few months an idea that has spent much of its life lurking in the depths has begun to acquire air and a new level of articulation. It’s not yet sweeping through the international Sistema community; there are no major bush fires or storms with names in their wake just yet, but by my reckoning there soon will be. And in my opinion it’s going to be a key – as well as a very big – challenge for just about every Sistema programme out there. Period.

Evaluation and Research is the name of this new game. It comes at Sistema projects from a number of directions. Many, though not all, programmes have had to do some evaluation as a matter of course when starting up or carrying on, even if only of the most basic kind. In places like Europe and the US it’s part of the background hum to a project, a constant riff that always needs attending to but can never be finally quite satisfied. Want to be sustainable? Then you’d better have some cast iron figures to show why your patrons, donors, sponsors, friends, trusts, local council or national government should keep on giving, and why your journalists, writers and broadcasters should keep on telling the story as something really positive. (Of course I appear to be making a big assumption here: I seem to be assuming that the figures will show good news. That, of course, may or may not be the case, and you won’t find that out until you have those figures).

Looked at like this, the question about evaluation and research seems pretty clear. But it’s not long before a profusion of questions arise from every side: what measures should you be using? Is it about musical attainment, social development or both? Cognitive function or social behaviour? How do you calculate value on SRI measures? Should positives (future employment) as well as absence of negatives (lower unemployment) be factored in. Should you be doing it yourself or getting outsiders in? What percentage of your precious turnover is worth it for hard facts? Are you talking about evaluation, research or both? Can you trust the evidence of already published studies? Where are the control groups? Are you prepared to wait a decade or so for really trustworthy figures? Have you measured drop out rates? Ambitious or prudent? Longitudinal? Quantitative or qualitative? Did you do base line measures before you began? And that’s before you get to any of the complicated stuff. Help!

Back in January 2012 I was sitting in an evaluation class at the LA Phil’s Take a Stand conference, and getting, well, bored. The Arts Council in England, knowing that I would anyway be in LA, asked me to have a look at what the session would throw up. Keep it simple, was the adamant proposal from the session. You have enough to worry about without getting a PhD in data methodology, and anyway it would take years to get sophisticated numbers. A few simple quantitative measures are much better.

Well I didn’t buy any of that, and I still don’t.

Of course viewed from Caracas, where I recently landed, there’s historically been a different approach. El Sistema grew here with a different sense of what it means to persuade. And anyway if you can show that it works, really show, do you also need to prove that it works? And what if you’re in Yerevan or Kinshasa or Ramallah or Stockholm? What is ‘showing’ in these cultures? What is proving?

The easiest thing is to forget the whole dance and get on with what got you started in the beginning: making great music with children and improving lives in the process, not staring at numeric pictures that merely represent the hopefully successful results of such activity.

I sympathise. But let me tell you: some new solutions are forming. And the nice thing is that that some of these new solutions are mirroring the way children learn about music in El Sistema; by doing it together. There’s a kind of quiet madness about everyone trying to solve this evaluation problem on their own. Yes, of course there are some different needs and there are different cultures and many different ways of working with evaluation or research, but there are far more common needs. Why not let’s share answers to those common needs. So Glenn Thomas’s Sistema Global has just started a sub group called, appropriately, Sistema Research. To begin with Glenn, Teaching artist Eric Booth, UK music educationalist Richard Hallam, and I started throwing a few ideas around. But I really became impressed when I saw the number of people chiming in. An initial thread passed 60 contributions, including Marilyn Price-Mitchell (US), Ken MacLeod (Canada), Joy Bechtler (US), Keane Southard (US but with increasing knowledge of Brazil) Tricia Tunstall (US) Geoff Baker (UK) and Sara Zanussi (US). (Hmn; did I say global?)

As a companion initiative I’ve started a Google Drive folder: SERA, ‘Sistema Evaluation and Research Archive’. (You can find that through this blog at http://wp.me/Pohgn-g4 or via an information page on the main menu El Sistema drop down tab). There you can place research work and see what others are placing there. It’s new and early days. Gradually a conversation will form around these sites and a new international approach to evaluating El Sistema will arise, which will not be quite so North American and UK centred. And I dare say more sophisticated ways of talking about it together will emerge.

Some of the questions are practically screaming at me. For someone there’s a career to be made by grabbing hold of this question: how to best audit what is around in a way that helps everyone in the field. There’s another field of glory for whoever can develop a small set of templates that cover a big enough range of different evaluation needs, so that evaluation can be approached with flexibility but enough consonance that results around the world can be compared. And whilst we are about it we could also do with a nice suite of evaluation tools (thank you for that one Eric).But ultimate glory awaits whoever can find the killer research project that really unites what many Sistema practitioners around the world are doing.

Now you can argue about whether El Sistema in Venezuela is a programme, a system, a network, a franchise, or another fancier word that explains it all in a few connected syllables. But internationally it’s already clear how we are going about solving El Sistema challenges: share, open source, look, learn, discuss, explore, communicate, try and report: these are the buzz words. And when it come to growing a methodology of evaluation and research that will work from LA to Kampala it’s very clear: the network is king. That is the really valuable thing I have learnt this year from international Sistema work. And not only is the network free, but it also lacks centralised control, as Glenn, Eric, Richard and I know only too well. We can start what we want, but the network will decide where it goes. A no brainer if ever I saw one.

Sistema Global Research sub group available via the Sistema Global LinkedIn Site
SERA Sistema Evaluation and Research Archive information at: http://wp.me/Pohgn-g4

Post navigation

4 thoughts on “Go Figure: another Sistema No Brainer”

Marshall, I’ve just found your blog and am enjoying what I am reading very much. I’ll be linking to you from mine. With regard to evaluation, I think two of the great weaknesses in evaluations of music/arts programs that have social change intentions can be (1) the fact that many of the evaluations are done by people who already have an interest in the project and its success (eg. organisers) which can weaken their impact as persuasive, impartial research; and (2) the lack of participant voices that are included in evaluations. Often, the voices of organisers may be (implicitly or unintentionally) privileged over those of participants, especially when the participants are young people.

A model that may have a lot to offer in these kinds of programs is the Most Significant Change tool/approach, which is a qualitative, participatory approach to evaluation and program monitoring. I think that its focus on the stories offered by individuals is a good match within cultures where a lot of knowledge is transferred orally, and it creates space for the impact that the program has had on individuals to be revealed, told, then discussed and reflected upon. MSC can yield a lot of very rich data about change in people’s lives, and it also values the contributions of participants, because it is built on their stories. Worth a look if it isn’t in use anywhere yet!

Yes, I’ll be following up the SERA archive I imagine. I’m not directly involved in Sistema research or practice but a lot of my work sits within the ‘music and social change’ paradigm so it is all of interest. But when you suggest posting the comment above “on the piece itself”, where do you mean?