No model is true—all are specifically made to simplify reality. No simplification of truth is true. The models are made to be useful. The Econ models’ usefulness is based on the accuracy of the predictions, especially how good is the particular investment decision. All such derisions have a monetary Return on Investment, with an estimated Net Present Value. That NPV before or at the decision point in time becomes subject to revision, and comparison, in every later time period.
To the decision makers/ owners, whether the predictions come from economists with good or bad theories, or from rocket scientists or ai economy simulators or machine learning pattern matchers is far less important than the ROIs they get using the predictions. Including timing of predictions, so as to time their stock buying & selling decisions.
It’s not surprising that pattern matching, rather than model adjustment, makes predictions that are better for higher ROIs.
"Tyler sees LLMs as fundamentally linguistic and pattern-recognition machines. He notes that "the most successful attempts to model language and also reasoning come through computer science, namely Large Language Models and perhaps their eventual offshoots as well." This is significant—he's saying that where we used to look to linguists, philosophers, or logicians to model reasoning, we now look to LLMs because they're just better at it."
Cowen may be observing a real change in who does the computation and mistaking it for a change in who does the thinking. The theoretical economist’s value is the capacity for questioning anomalies that might suggest that a model is wrong. In Cowen’s world, we lose the ability not only to make paradigm shifts, but even to recognize when a shift is needed.
"…the successful approach to predicting returns is giving up on traditional portfolio theory and using the “theory-less” technique of machine learning."
Even before I got to this sentence all I could think was how this excerpt sounded like 2007 and trenches if mortgage securities. Am I the only one?
I have been thinking of it as a facsimile of reasoning. Claude & ChatGPT show a process that at least resembles reasoning. I had supposed they were trained on texts that illustrated various chains of reasoning in the real world in order to know the patterns. But I don’t know.
So I think the issue is a bit more subtle. Much and perhaps nearly all scientific progress can be thought of as "pattern-matching" of some sort. People even win Nobel prizes for doing that. Not to disparage the contribution, but you could argue that all Paul Krugman did was take Dixit-Stiglitz and applied it first to international trade, and then economic geography. Is that not pattern matching of a sort? The only possible exceptions are true scientific revolutions in the sense of Thomas Kuhn. So a thought experiment might be, imagine an LLM that had access to all but only the "data" that Adam Smith had as of 1776, could it generate the same insights that came out of the "Wealth of Nations" if you gave it the right prompts? Right now, I doubt it, but might it get there at some point? We'll see ........
We’re not as special as we think but we’ve probably earned that right since we’re the premier primate. I’m actually surprised Arnold didn’t consider that the human brain itself is “pattern-centric” to think LLMs arrive at knowledge differently to humans. The main difference is whatever pattern the human brain matches, it is weighed against what’s socially (or legally?) acceptable, though not in all cases. I think the right prompts will give us the same insights as the “Wealth of Nations” given the “data” Smith had at the time but not just anybody can come up with the prompts. The prompter, in this case, may have to screen the answers for other societal norms that are not already captured in the data and figure out how they fit or not with the idea. I actually don’t think much has changed in how the brain works we just happened to have created a lot more data.
"I call this the “savant memorizing the encyclopedia” model of AI. Instead, I want to describe AI as operating in terms of deep patterns in the data."
Another way of looking at it is "Newly minted professor marinated from birth in the field's conventional consensus, deeply familiar with the currents of its history and output in a stunningly comprehensive way, but influenced disproportionately by its highest status papers and personalities."
Maybe an alternative way to put it is that there's a continuum between savant memorizing the encyclopedia and ordinary academic scholar and AI's are based on statistical engines in a way such that their output will seem like a high quality version of something in the middle, perhaps just as prone to intellectual inertia and conservatism and popularity as the typical academic writer.
No model is true—all are specifically made to simplify reality. No simplification of truth is true. The models are made to be useful. The Econ models’ usefulness is based on the accuracy of the predictions, especially how good is the particular investment decision. All such derisions have a monetary Return on Investment, with an estimated Net Present Value. That NPV before or at the decision point in time becomes subject to revision, and comparison, in every later time period.
To the decision makers/ owners, whether the predictions come from economists with good or bad theories, or from rocket scientists or ai economy simulators or machine learning pattern matchers is far less important than the ROIs they get using the predictions. Including timing of predictions, so as to time their stock buying & selling decisions.
It’s not surprising that pattern matching, rather than model adjustment, makes predictions that are better for higher ROIs.
Very interesting post.
The LLM's reply was right on target:
"Tyler sees LLMs as fundamentally linguistic and pattern-recognition machines. He notes that "the most successful attempts to model language and also reasoning come through computer science, namely Large Language Models and perhaps their eventual offshoots as well." This is significant—he's saying that where we used to look to linguists, philosophers, or logicians to model reasoning, we now look to LLMs because they're just better at it."
Cowen may be observing a real change in who does the computation and mistaking it for a change in who does the thinking. The theoretical economist’s value is the capacity for questioning anomalies that might suggest that a model is wrong. In Cowen’s world, we lose the ability not only to make paradigm shifts, but even to recognize when a shift is needed.
"…the successful approach to predicting returns is giving up on traditional portfolio theory and using the “theory-less” technique of machine learning."
Even before I got to this sentence all I could think was how this excerpt sounded like 2007 and trenches if mortgage securities. Am I the only one?
“LLMs are only logical by accident.”
I have been thinking of it as a facsimile of reasoning. Claude & ChatGPT show a process that at least resembles reasoning. I had supposed they were trained on texts that illustrated various chains of reasoning in the real world in order to know the patterns. But I don’t know.
So I think the issue is a bit more subtle. Much and perhaps nearly all scientific progress can be thought of as "pattern-matching" of some sort. People even win Nobel prizes for doing that. Not to disparage the contribution, but you could argue that all Paul Krugman did was take Dixit-Stiglitz and applied it first to international trade, and then economic geography. Is that not pattern matching of a sort? The only possible exceptions are true scientific revolutions in the sense of Thomas Kuhn. So a thought experiment might be, imagine an LLM that had access to all but only the "data" that Adam Smith had as of 1776, could it generate the same insights that came out of the "Wealth of Nations" if you gave it the right prompts? Right now, I doubt it, but might it get there at some point? We'll see ........
We’re not as special as we think but we’ve probably earned that right since we’re the premier primate. I’m actually surprised Arnold didn’t consider that the human brain itself is “pattern-centric” to think LLMs arrive at knowledge differently to humans. The main difference is whatever pattern the human brain matches, it is weighed against what’s socially (or legally?) acceptable, though not in all cases. I think the right prompts will give us the same insights as the “Wealth of Nations” given the “data” Smith had at the time but not just anybody can come up with the prompts. The prompter, in this case, may have to screen the answers for other societal norms that are not already captured in the data and figure out how they fit or not with the idea. I actually don’t think much has changed in how the brain works we just happened to have created a lot more data.
This is already happening, as a way to study history and not as a thought experiment. see
see https://250bpm.substack.com/p/ada-palmer-inventing-the-renaissance?utm_source=share&utm_medium=android&r=8o0zz
"I call this the “savant memorizing the encyclopedia” model of AI. Instead, I want to describe AI as operating in terms of deep patterns in the data."
Another way of looking at it is "Newly minted professor marinated from birth in the field's conventional consensus, deeply familiar with the currents of its history and output in a stunningly comprehensive way, but influenced disproportionately by its highest status papers and personalities."
Maybe an alternative way to put it is that there's a continuum between savant memorizing the encyclopedia and ordinary academic scholar and AI's are based on statistical engines in a way such that their output will seem like a high quality version of something in the middle, perhaps just as prone to intellectual inertia and conservatism and popularity as the typical academic writer.
That would all ring more true if AI weren't able to get superior returns.
Marginal Revolution 2.0
At least skip down to the last sentence of the book!
And note the cover too.