Discover more from In My Tribe
Madness in their Methods
Methodological criticism from Anton Howes, Adam Mastroianni, and David Deming; Bryan Caplan bashes tenured professors
I’ve become increasingly worried that science’s replication crises might pale in comparison to what happens all the time in history, which is not just a replication crisis but a reproducibility crisis. Replication is when you can repeat an experiment with new data or new materials and get the same result. Reproducibility is when you use exactly the same evidence as another person and still get the same result — so it has a much, much lower bar for success, which is what makes the lack of it in history all the more worrying.
He cites a number of random examples, but the centerpiece of his essay is this:
the publication of a paper in the prestigious journal History & Technology by Jenny Bulstrode of UCL, in which she claimed that the inventor Henry Cort had stolen his famous 1783 iron-rolling process from Reeder’s iron mill in Jamaica, where it had been developed by 76 black metallurgists by passing iron through grooved sugar rollers. It was a widely-publicised paper, receiving 22,756 views — eleven times as many views as the journal’s next most most read paper, and frankly unheard of for most academic papers — along with a huge amount of press coverage.
Of course, the story turns out to be untrue. If Bryan Caplan were here, he would say “social desirability bias strikes again.”
Papers with conclusions that make people feel good, especially when they make people on the Left feel good, are more likely to get accepted by journals and to receive amplification in the press than papers that arrive at contrary conclusions. And inconclusive results have the least chance of all of getting published or discussed.
When another Raj Chetty paper makes a splash, how much should it move your priors? Yes, he is more rigorous than the average researcher. But somehow his success as a crowd-pleaser makes me suspicious.
Now let's imagine every allegation of fraud is true, and everything Ariely and Gino ever did gets removed from the scientific record… What would change?
Not much.
…I wouldn't believe anything different about the human mind than I already believe now.
His point:
Apparently is possible to reach the stratosphere of scientific achievement, to publish over and over again in “high impact” journals, to rack up tens of thousands of citations, and for none of it to matter.
I get the impression that findings in the field of psychology have been minor and isolated. There are no high-stakes battles over grand theories. The same seems to be true in economics.
The classic papers of yore advanced important theories that made us think differently about the world, but the evidence supporting their arguments was often quite speculative. Today, top econ journals like QJE ensure that each of the papers they publish is near-perfectly executed and incredibly convincing. They just answer smaller questions.
He praises a new paper that tries to assess the role played by human capital in international differences in prosperity. I am less excited by that particular paper, but I agree that the question about the sources of international differences in prosperity is well worth putting effort into answering.
Bryan Caplan’s screed against tenure makes the following point en passant.
What makes low research productivity so shocking is that you can call practically anything “research.”
Overall, I think that the academic system puts too much emphasis on the individual paper. Intellectual discovery and truth-finding would be better served by back-and forth conversations. Much as I scorn Twitter, I can understand those academics who see it as a vehicle for these sorts of intellectual conversations.
Some of the input to these conversations might be similar to the studies that people publish today. But at least some of the leading historians, psychology professors, and economists should be discussing big questions and arguing over grand theories.
Substacks referenced above:
@
@
@
@
Madness in their Methods
"I’ve become increasingly worried that science’s replication crises might pale in comparison to what happens all the time in history..."
-Anton Howes
"You know, I read these history books... And whaddaya know? The good guys win! Every single time!"
-Norm MacDonald
re: “Intellectual discovery and truth-finding would be better served by back-and forth conversations.”
Indeed. And perhaps even more importantly conflicts about competing values more fully developed and new outlines of compromise discovered through debate.
At Quillette, Brian Kallar has a piece entitled “Our Lost Classical Learning” ( https://quillette.com/2023/09/05/our-lost-classical-heritage/ ) that asserts:
“The Western canon was not an unchanging set of texts, but an ongoing conversation that lasted thousands of years—enabling each generation to build on the intellectual heritage of the past. It gave people a set of cultural reference points: Homer, Socrates, Aristotle, Aquinas, Dante, Milton.” Having this as a common “vocabulary” facilitated “discussing big questions and arguing over grand theories.” One sees how valuable the big idea back-and-fourth exchange was in former days in books such as Yuval Levin’s The Great Debate: Edmund Burke, Thomas Paine, and the Birth of Right and Left. And Algernon Sydney’s Discourses Concerning Government in response to Robert Filmer's Patriarcha has been described as "the textbook of the American revolution." Yet, requiring the basics of a classical education to appreciate, these classics largely go unread, and current debate is all the poorer for it.
I am struck whenever I come across a paper that places its argument in the context of a dialogue more than 10 years old. Perhaps I read the wrong journals but it seems like clouds of supportive citations from the last 10-20 years is the norm for a paper today. New academic books often seem to average a parenthetical citation to an article published in an obscure journal in the last 10 years every third sentence. Perhaps this serves some socially useful coordinating function in directing research inputs. And perhaps hyper-specialized small-potatoes research is an adaptation to the geometric rate of increase in the number of articles being published: https://direct.mit.edu/view-large/figure/3750245/qss_a_00177_f001.tif . One can’t help but think the trade-offs involved with hyper-specialization suggest the academy is substantially over-funded and over-staffed much in the same way the officer corp in the military is top-heavy and over-staffed. Compare this list of PhD programs offered in the United States: https://en.wikipedia.org/wiki/List_of_doctoral_degrees_in_the_US to the list of Army officer occupational concentrations: https://api.army.mil/e2/c/downloads/2022/05/24/9513c3e9/chapter-2.pdf . Every officer concentration requires at least 40 positions and the rank structure requires x number of captains to support x numbers of majors to support x numbers of colonels to get a general rank out of the deal and this calculus largely drives organizational design. Similarly in academia to offer a PhD in Church Music you need to have x number of faculty to offer x number of classes and sit on x number of committees. So it would seem hyper-specialization is financially costly. Add to that the perhaps more important opportunity cost of lost generalist research, and the added cost of literature reviews that must span an ever increasing number articles in an ever increasing number of journals and the return on social return on subsidies to academia seem debatable.
It is not surprising then that there is so much interesting and valuable research being conducted by generalists outside the academy and that one rarely hears much public complaint about the output of the academy inaccessible and locked behind publisher gates: the good stuff is elsewhere.