LLM Links, 11/21
Talk of the wall
When people came along and made various predictions about AI following “scaling laws,” I was never extremely impressed. I did not feel I had the technical background to contradict them, I just was never sure they were measuring the social import of that knowledge in a meaningful way.
That is one (not the only) reason I have never been much persuaded by either the AI doomsters nor the AI utopians. Both seemed to me misguided rationalists, operating with a fundamentally pre-Hayekian understanding of knowledge.
Machine learning may be able to capture explicit knowledge, but can it absorb tacit knowledge? Think of written works as the tip of the iceberg of human knowledge. Scaling up the amount of writing that LLMs can see is not going to expose the entire iceberg to the computer.
Tyler is still optimistic about progress, because of the amount of ferment in the field. Ideas having sex, to cite a famous TED talk by Matt Ridley, based on his book The Rational Optimist.
A few months ago both Google and Anthropic updated their small and medium-sized models (Sonnet 3.5 and Haiku 3.5 for Anthropic, Pro 1.5 and Flash 1.5 for Google). But we’re still waiting for corresponding updates to their largest models (Opus 3.5 for Anthropic and Ultra 1.5 for Google).
These trends have caused a lot of people, including me, to wonder whether scaling laws are running out of steam. And in the last week, a series of news reports have provided fresh support for that thesis.
This was the topic of the week last week among folks I follow concerning Large Language Models. I have nothing to add myself, just a few more links.
traditional funders and boosters like Marc Andreessen are also saying the models are reaching a “ceiling,” and now one of the great proponents of the scaling hypothesis (the idea that AI capabilities scale with how big they are and the amount of data they’re fed) is agreeing. Ilya Sutskever was always the quiet scientific brains behind OpenAI, not Sam Altman, so what he recently told Reuters should be given significant weight:
Ilya Sutskever, co-founder of AI labs Safe Superintelligence (SSI) and OpenAI, told Reuters recently that results from scaling up pre-training—the phase of training an AI model that uses a vast amount of unlabeled data to understand language patterns and structures—have plateaued.
I know that there is a huge amount of confusion, fear, and hope that creates a fog of war around this technology. It's a fog I've struggled to pierce myself. But I do think signs are increasingly pointing to the saturation of AI intelligence at below domain-expert human level. It’s false to say this is a failure, as some critics want to: if AI paused tomorrow, people would be figuring out applications for decades.
I put substantial probability into progress getting a lot harder. But even if that happens, AI is going to keep becoming more capable at a rapid pace for a while and be a big freaking deal, and the standard estimates of AI’s future progress and impact are not within the range of realistic outcomes. So at least that much hype is very much real.
In other words, even if scaling is subject to diminishing returns, there are other margins on which models can and will be improved.
Accenting the positive, Alexander Kruel writes,
This is pretty amazing stuff: Robot watches how-to videos and becomes an expert surgeon — The team from Johns Hopkins and Stanford Universities harnessed imitation learning, a technique that allowed the robot to learn from a vast archive of surgical videos, eliminating the need for programming each move. This approach marks a significant step towards autonomous robotic surgeries, potentially reducing medical errors and increasing precision in operations. https://hub.jhu.edu/2024/11/11/surgery-robots-trained-with-videos/
There is a lot of wisdom in “the future is already here—it’s just not evenly distributed”
substacks referenced above: @
@
@
@
I wonder if a plateau or slowing of change could encourage applications / practical development. When things were progressing rapidly, there was probably a sense to some that any current application would be shortly left in the dust.
The robot surgery piece was indeed amazing. Certainly provides evidence to justify optimism.
Infected with that optimism, I entered “dividends from AI stocks” in my AI equipped browser and got back an AI generated response that included “other AI-focused stocks mentioned (Microsoft, Nvidia, and C3.ai) do not currently pay dividends.” Uh oh. While that may be true of C3.ai (https://www.marketwatch.com/investing/stock/ai ) it is not true of MSFT (https://www.marketwatch.com/investing/stock/msft?mod=search_symbol ) or NVDA (https://www.macrotrends.net/stocks/charts/NVDA/nvidia/dividend-yield-history ) even if their dividend histories might not be considered by some to be spectacular.
While at the Marketwatch site, I succumbed to a clickbait link about “A Once-in-a-Decade Investment Opportunity: 1 Little-Known Vanguard Index Fund to Buy for the Artificial Intelligence (AI) Boom” which turned out to be about a utilities ETF – more data centers need more electricity, you see. Probably not the worst play in the world, even if the most obvious, but not exactly a ringing endorsement of anticipated AI revenue streams.
One of the newsletters in the inbox this morning also featured an ad from Sapience which apparently has moved beyond AI to synthetic intelligence (https://invest.newsapience.com/ ) and is offering crowdfunding opportunities while at the same time its founder and CEO is tweeting:
“That LLMs are vastly expensive, and from an energy standpoint even unsustainable, is now obvious. But can they deliver on the touted productivity gains? If not, it is not for lack of trying. Research by The Upwork Research Institute reveals that 39% of C-suite leaders are mandating the use of genAI tools, with an additional 46% encouraging their use.
But the results are not encouraging, the same study found that nearly half (47%) of employees using genAI say they have no idea how to achieve the productivity gains their employers expect, and 77% say these tools have actually decreased their productivity and added to their workload.
The Internet (eventually) actually did ‘replace very expensive solutions with very cheap solutions’ but the dot-com startup investment bubble was irrational because there were no barriers to entry. After the bust, most failed right out of the gate and it would take decades for the real winners to emerge.
LLMs startups also have no barriers to entry, the technology will always be vastly expensive, and in the end, it just doesn’t deliver. When this bubble bursts it could be ugly indeed. There may be no long-term winners, at least not big ones, this time around.”
(https://x.com/BryantCruse/status/1819087395683262852 )
On the other hand, Warren Buffett is apparently invested heavily in AI albeit mostly indirectly. (https://www.nasdaq.com/articles/46-warren-buffetts-410-billion-portfolio-invested-4-artificial-intelligence-ai-stocks )
So I don’t know, maybe a winning approach to AI might be to bet on products that are actually producing revenue streams with net present values? Not sure.