Fake news? El Sistema and the IDB launch their study

[28/03/2017] A launch event has just taken place in Caracas for the recent evaluation of El Sistema commissioned by the Inter-American Development Bank. The research team, El Sistema leaders, and government representatives were present at the occasion, which – according to the headline displayed prominently at the top of El Sistema’s website – “confirmed the positive impact of El Sistema on children and young people.” It is interesting to compare the press release for this event with the academic article in which the study’s findings were presented.

(NB: the same researchers wrote the article and participated in the event.)

What the press release said

The research team “expressed its satisfaction with the possibility of confirming the transformative work of the program.” They had concluded that the children and young people who entered El Sistema showed improved connections with school and family, a higher degree of cooperation with their peers, and greater self-confidence. According to one of the researchers, Marco Stampire, “we found a decrease in levels of aggression and risk-taking […]; and a willingness to take part in collective activities. The positive effects were also manifested in childhood IQ.”

What the article said

The researchers created a “theory of change” which hypothesized that “short-term participation in orchestras or choruses may foster positive change in four child functioning domains: self-regulatory skills, behavior, prosocial skills and connections, and cognitive skills.” To test their theory, they measured 26 primary outcome variables within these four domains. Only 2 significant outcomes were found: “the early-admission group had higher self-control and fewer behavioral difficulties, based on child reports.” There were thus no significant outcomes in 24 out of 26 areas, and: “We did not find any full-sample effects on cognitive skills […] or on prosocial skills and connections.”

In which case, what is the basis for the claims at the launch event of improved connections, cooperation, self-confidence, and IQ?

What the press release said

Ferdinando Regalía, head of the IDB’s Social Protection and Health Division, underlined the importance of showing the results of the study “in order to tackle the criticisms of El Sistema’s work and reaffirm the value of social inclusion via a program of artistic and musical education.”

What the article said

The estimated poverty rate among the El Sistema children was 16.7%, while the rate for the states in which they live was 46.5%. In other words, the El Sistema children in the experiment were three times less likely to be poor than all 6 to 14 year-olds residing in the same states. Consequently, the study “highlights the challenges of targeting interventions towards vulnerable groups of children in the context of a voluntary social program.” Furthermore, 44% of students who were offered a place failed to complete two semesters.

These findings do not “tackle the criticisms of El Sistema’s work” but rather reinforce the qualitative critiques with quantitative evidence. The relative affluence of the beneficiary population and the high dropout rate in the study suggest significant problems as regards social inclusion.

***

As I argued in my earlier critical analysis of the academic article, the IDB’s study, far from “confirming the transformative work of the program,” raises important questions about El Sistema’s central claims. In my conclusion, I wrote:

The big question is what happens next. If the IDB and El Sistema use the report as the basis for a serious reassessment of multiple aspects of the program, ranging from targeting and retention of students to curriculum and pedagogy, then it will turn out to be very valuable and a wise investment of $1 million. If, however, the report is to be spun by the IDB and ignored by El Sistema, then it will be a case of throwing good money after bad.

Initial signs are not particularly promising.

I don’t know what was actually said at the Caracas event, as I have only the press release to go on. The quote from one of the researchers at the live presentation is notably more upbeat than the published article, and the researchers are reported as claiming positive findings that were not mentioned in the original study. But it’s impossible to know from the press release whether the IDB research team beefed up the positives and omitted the negatives from their presentation, or whether El Sistema’s press office was responsible.

Either way, my concerns about the way this study may be used appear to be borne out. The academic article presented decidedly mixed findings: few significant effects, a low poverty rate, a high dropout rate, and a gender divide. But if this press release is indicative, the official line is nonetheless that the study proves El Sistema to be a resounding success. I won’t be surprised if I hear this flawed conclusion repeated frequently and become the orthodoxy in advocacy accounts of the Venezuelan program and its international offshoots. And I won’t be surprised if hardly anyone reads the study itself, let alone reads it carefully and reflects on the gap between spin and substance. Combating fake news and alternative facts requires journalists and readers who are willing to look at official pronouncements with a critical eye, and unfortunately both are thin on the ground when it comes to El Sistema.

The problem here isn’t that El Sistema and the IDB haven’t proven the effectiveness of the program. I wouldn’t expect them to with a study like this (though they apparently did, and bet $1 million on it). The problem is that they are claiming that they have when their own evidence suggests otherwise.