This little comic comparing different gods‘ achievement made me laugh twice — first because the comparison is so sweet, and second when I talked about it with a friend who pointed to some methodological flaws in the conclusion. So I can’t resist sharing it here, and commenting on how it illustrates some problems in (especially) social science research on intervention effects.

First a rough translation of the text: Jesus came to the world to save us from sins. Thor came to the world to save us from ice giants. Now: The world is full of sins… But there isn’t a single ice giant. Thor wins. Where is your god now?

The first and obvious problem is the absence of a pre-intervention „baseline“ measurement. We don’t know if there were even any ice giants around when Thor came here — he could have played a great trick on us. Also, we have few information on how much sin there was in the world over 2000 years ago. The Thor issue reminds me of a joke I can’t tire of telling people for its deeper implication about anxieties: The psychiatrist sees a patient who has just been admitted to the psychiatric hospital. Already during the first sentences he notices the patient keeps clapping his hands. „What do you clap your hands for?“ he asks. The patient replies: „To scare away the elephants“. Psychiatrist: „But — there are no elephants here.“ Patient, triumphantly: „See how well it works!“

Apart from that, even if we set out to measure the amount of sins in the world, we would have trouble coming to a good definition and measurement procedure. Also, you can safely assume that a lot of sins are committed in private, making the practical measurement even more problematic.

The most stunning objection, and one which immediately reminded me of issues in psychotherapy research, was my friend saying: „Jesus didn’t come to save us from sins. He came to forgive them.“ It’s often not as easy as you would think to tell what somebody (or a group of people) is aiming at with a complex intervention, especially when you’re not part of that group. And so outside measurements and comparisons can get really tricky.

The last and somewhat nerdy (ok, even more nerdy than before) point concerns standardized effect sizes. As success of Thor’s and Jesus‘ social intervention is obviously on different scales (reduction in ice giant population vs. reduction in (frequency? intensity? of) sins), how do you compare them? What you would typically do is compare the change that is attributable to the intervention to natural fluctuations in the phenomena of interest. That would be hard to do here, even though theoretically possible — how did ice giant population and sins change over time before our two so different saviors came along? But of course, it doesn’t actually tell you anything about the clinical significance of each change. Maybe something similar to a change of quality occurs at some point of the scale — if ice giant population falls below a certain point, they retreat to their mystical homelands and leave humans alone. If sins are reduced beyond a certain point, society changes, preparing the earth for eternal heavenly rule. In psychotherapy, we mostly assume something like that actually, where the clinical disorder or syndrome (of, say, depression or phobia) is seen as no longer existing after a certain threshold, but often still measured as a severity of symptoms.