The Wrapper Model
Have academic economists fallen for a scam?
Tyler Cowen has been tracking the success of “Refine,” which describes itself as
an AI-based tool that identifies errors in reasoning, calculation, and references across parts of a paper.
For authors and journals alike, it’s a rigorous check that provides peace of mind before work goes out into the world in its final form.
It might be a scam.
Can the tool do what it says it does? I would bet yes.
Does this provide value compared to not using any tool? I would bet yes.
Can it provide services more cost-effectively than a human? I would bet yes.
Can the tool provide significantly better results than just feeding your paper to the latest version of Claude or ChatGPT and asking it to “identify errors in reasoning, calculation, and references?” I would bet no.
The term of art for “Refine” is a wrapper model. That is, it wraps around one or more of the frontier models.
I have not played with “Refine,” because trying to publish a journal article is not my jam. But for other projects, such as my lectures on 21st century American history, I am finding it more satisfying to go directly to Claude rather than use a wrapper. I imagine that if I were interested in publishing a journal article, I could go directly to Claude and receive feedback that is at least as useful as what I could get from using a wrapper.
I know many domains in which the wrapper model is generating revenue. My guess is that the decision-makers in those domains are not AI natives. They are the equivalent of the executives of the 1990s who had their secretaries print out their emails.
The wrapper modelers had better take their profits while they can. Once AI natives rise to positions of authority, they will discard the wrapper models in favor of going directly to the frontier models.
Just to clarify, Refine has released a benchmark and outperforms 'out-of-the-box' models and open-source scaffolds by a wide margin. Creating a scaffold that combines multiple frontier models and produces excellent results is non-trivial. Note, I'm an advisor to Refine and am an co-author in the benchmark below.
https://www.refine.ink/blog/refine-ai-reviewer-benchmark
For a particular academic field, having a particular wrapper dedicated to that field gives academics collectively "someone to talk to" in terms of pointing out things that need to be fixed or, well, refined, according to the particular needs and interests that are the highest status in that field. Trying to "petition" changes in something like Claude feels like trying to petition the federal government, it's too big and too wide in scope for your complaints or suggestions to matter. But a field-specialized tool is more like a local official with whom you could actually set up a meeting. The company behind it is focusing on the opinions and usage of a particular narrow community and thus has a kind of natural closer relationship to its membership and an interest interest in paying attention to its comments and concerns and making appropriate adjustments. Claude and the others and the big companies behind them are just too big to "too inpersonal" to care about those concerns, requests, interests, and details for some tiny subgroup of its giant and diverse population of users. I can especially see the value of field-dedicated wrapper products and companies in the law. Indeed, that was part of the history of Westlaw and Lexis when the big search engines came on the scene and everybody predicted that was the end of the specialist firms. It didn't work out that way because generic search that is the 95% adequate solution for a million other users is (for various reasons I'll spare you) not a good approach to legal research, the knowledge of which is organized in a distinct kind of intellectual structure. Notice that even after several years of knowing about big problems, generic AI tools are still really bad at legal research and writing, and the legally specialist AI tools, tweaked to the particular needs of the legal profession, are (at least currently) considred clearly superior (albeit a lot more expensive, which is why lawyers try to save money and use the generic tools and are still often getting themselves into big trouble.