The Null Hypothesis Strikes Again

This time in criminal justice interventions

Megan T. Stevenson writes,

most reforms and interventions in the criminal legal space have little to no lasting effect when evaluated by RCTs, and the occasional success usually fails to replicate when evaluated in other settings.

Pointer from Tyler Cowen.

For many years, I have argued for what I call the Null Hypothesis with respect to education interventions. That is, when tested rigorously, an education intervention probably will produce a result for the treatment group that is not significantly better than what one finds in the control group. Any apparently significant result is likely to fade out over time. And any lasting result is unlikely to be replicated if tried somewhere else.

Stevenson writes,

My beliefs are informed by my knowledge of the empirical literature in other fields. I was trained by development economists and I frequently attend conferences and collaborate with economists studying education, health, and labor. In these fields also, interventions evaluated via RCT rarely find large or lasting benefits. Microcredit (loaning small amounts of money, often to women) was, for many years, the darling of the development world. Eventually, it was shown to have little to no net benefit in most places. Health insurance, randomly allocated via lottery, was shown to increase healthcare usage and reduce bills sent to a collections agency. Yet, it had no statistically significant effect on physical health or labor market outcomes. The fact that most interventions in the social world have little impact is so ubiquitous that it has been dubbed the Iron Law of Evaluation: “The expected value of any net impact assessment of any large-scale social program is zero.”

I prefer the expression “Null Hypothesis.”

As Stevenson points out, the social world is complex. I suggest thinking of any social problem, such as substance abuse, as like a maze where most of the paths that you could take just end up where you started, sometimes after you have followed them optimistically for a long time. It is worth trying to find paths out of the maze, but most of the people who claim to know the path are over-confident.