AI and research papers
the future is here (pretty much)
Bill Sterling, an old friend with an econ Ph.D from Harvard, emailed me
thought you might find the following thread to be of interest. It is about an economist who has set a goal of producing 1,000 economic journal articles in a month:
I asked Claude to help replicate his setup on Claude Code and the results were amazing. In about an hour each it could produce well researched empirical papers that could plausibly be submitted to economic journals. So a dissertation could be completed in an afternoon.
As part of its process it sends its first draft for peer review by other AIs and then makes revisions per their recommendations.
Artificial intelligence can do many of the things that social scientists do. It can analyze data, write and review code, identify appropriate statistical methods, and offer suggestions on study drafts. It can even take a dataset and a research question and produce an entire paper on its own. Given that human-led social science is often marred by mistakes, dubious methods, ideological bias, and even outright fraud, one can hope that AI will improve the field in the years ahead.
…for now, the technology still makes frequent mistakes, carries its own ideological baggage, and fails to converge on consistent results when different models tackle the same question—unless heavily steered by the very humans whose foibles we hope to escape.
I think that the words for now are really important to keep in mind. If you had evaluated the state of the models two years ago, you would have said that they are almost worthless at generating research papers or doing peer review. Even a linear extrapolation from where we were then to where we are now would lead you to expect the AI’s to surpass humans as researchers some time in this decade. And that happen within a year or two if you extrapolate based on exponential improvement.
I know that professors want to hear a message that says that the research ecosystem does not have to change: you still need professors to generate the ideas, to write peer reviews, and to edit journals. But I would send the opposite message. We should come up with a new, AI-assisted (dare I say AI-dominated) research ecosystem. Start with a blank piece of paper, and see what you can come up with.
Of course, I posed this problem to Claude. I’ll quote part of its response:
PubMed and Google Scholar are indexes of documents. What we actually want is an indexed, queryable map of claims with their evidence and confidence levels. The paper is the provenance trail; the claim is the searchable unit. AI is already reasonably good at extracting claims from papers; in 3-4 years it should be good enough to maintain these databases reliably. A researcher asking "what do we know about X" should get a structured confidence-weighted answer, not a list of PDFs to read.
…The honest summary is that the current publishing ecosystem is a rent-collecting prestige hierarchy dressed up as quality control. AI is going to make its inadequacy impossible to ignore. The replacement should be organized around claims and their reliability rather than documents and their authors — and should treat human judgment as a scarce resource to be deployed on genuinely hard calls, not wasted on tasks machines can do better.
The concept of a claim-based citation network was introduced by Greenberg and a semantic model later proposed as micropublications by Clark et al. https://doi.org/10.1186/2041-1480-5-28
Not sure whatever happened to that, but clearly AI should be able to get it over the finish line. Groups like the Continuous Science Foundation are working on the other elements proposed by Claude.
Thanks. I really wanted to wake up this morning and feel useless. Mission accomplished!
I only became a college professor because I wanted to replicate the positive experience I had at a small liberal arts college for other students. Not only am I not doing that, but now I get to experience being replaced by a machine, which is part of one of my favorite lessons to teach (on creative destruction). My life is being creatively destroyed. Thanks, AI!