Madness in their Methods

Methodological criticism from Anton Howes, Adam Mastroianni, and David Deming; Bryan Caplan bashes tenured professors

Sep 10, 2023

I’ve become increasingly worried that science’s replication crises might pale in comparison to what happens all the time in history, which is not just a replication crisis but a reproducibility crisis. Replication is when you can repeat an experiment with new data or new materials and get the same result. Reproducibility is when you use exactly the same evidence as another person and still get the same result — so it has a much, much lower bar for success, which is what makes the lack of it in history all the more worrying.

He cites a number of random examples, but the centerpiece of his essay is this:

the publication of a paper in the prestigious journal History & Technology by Jenny Bulstrode of UCL, in which she claimed that the inventor Henry Cort had stolen his famous 1783 iron-rolling process from Reeder’s iron mill in Jamaica, where it had been developed by 76 black metallurgists by passing iron through grooved sugar rollers. It was a widely-publicised paper, receiving 22,756 views — eleven times as many views as the journal’s next most most read paper, and frankly unheard of for most academic papers — along with a huge amount of press coverage.

Of course, the story turns out to be untrue. If Bryan Caplan were here, he would say “social desirability bias strikes again.”

Papers with conclusions that make people feel good, especially when they make people on the Left feel good, are more likely to get accepted by journals and to receive amplification in the press than papers that arrive at contrary conclusions. And inconclusive results have the least chance of all of getting published or discussed.

When another Raj Chetty paper makes a splash, how much should it move your priors? Yes, he is more rigorous than the average researcher. But somehow his success as a crowd-pleaser makes me suspicious.

Adam Mastroianni writes,

Now let's imagine every allegation of fraud is true, and everything Ariely and Gino ever did gets removed from the scientific record… What would change?
Not much.
…I wouldn't believe anything different about the human mind than I already believe now.

His point:

Apparently is possible to reach the stratosphere of scientific achievement, to publish over and over again in “high impact” journals, to rack up tens of thousands of citations, and for none of it to matter.

I get the impression that findings in the field of psychology have been minor and isolated. There are no high-stakes battles over grand theories. The same seems to be true in economics.

David Deming writes,

The classic papers of yore advanced important theories that made us think differently about the world, but the evidence supporting their arguments was often quite speculative. Today, top econ journals like QJE ensure that each of the papers they publish is near-perfectly executed and incredibly convincing. They just answer smaller questions.

He praises a new paper that tries to assess the role played by human capital in international differences in prosperity. I am less excited by that particular paper, but I agree that the question about the sources of international differences in prosperity is well worth putting effort into answering.

Bryan Caplan’s screed against tenure makes the following point en passant.

What makes low research productivity so shocking is that you can call practically anything “research.”

Overall, I think that the academic system puts too much emphasis on the individual paper. Intellectual discovery and truth-finding would be better served by back-and forth conversations. Much as I scorn Twitter, I can understand those academics who see it as a vehicle for these sorts of intellectual conversations.

Some of the input to these conversations might be similar to the studies that people publish today. But at least some of the leading historians, psychology professors, and economists should be discussing big questions and arguing over grand theories.

Substacks referenced above: