AI links, 10/27/2025
Ethan Mollick's indispensable advice; Jason Wei on AI's aptitude; The Zvi on Andrej and Dwarkesh; I ask if AI can make a penciil
most AI models only know basic user data and the information in the current chat, they do not remember or learn about you beyond that. So, you need to provide the AI with context: documents, images, PowerPoints, or even just an introductory paragraph about yourself can help
I find when working on a project that at the end of every session it pays to ask the AI to produce a status report that I can save and then send back to the AI at the start of the next session.
You might want to bookmark Mollick’s essay and come back to it when you are taking on a new challenge with using AI. And consider this advice:
Play is often a good way to learn what AI can do. Ask a video or image model to make a cartoon, ask an advanced AI to turn your report or writing into a game, do a deep research report on a topic that you are excited about, ask the AI to guess where you are from a picture, show the AI an image of your fridge and ask for recipe ideas, work with the AI to plot out a dream trip. Try things and you will learn the limits of the system.
I think that learning to use the AI’s is like learning to play guitar. There are many different styles of guitar play. The learning process can be both fun and intimidating.
the sort of cost and accessibility of um finding out knowledge or doing some reasoning is going to be driven towards zero.
Um the second is something that I call verifiers law. Um which is that the the ability to train AI to do a particular task is proportional to how easy it is to verify that task.
…the impact of AI will be largest uh on tasks that meet certain properties namely they’re digital easy for humans and data abundant. Um okay so for implications I think uh certain fields will be extremely heavily accelerated by AI. So you know software development is obviously one of those.
And he says this:
nobody should give or receive any career advice right now. Everyone has broadly underestimated the scope and scale of change and the high variance of your future.
What about the thing where humans sleep or daydream, or reflect? Is there some LLM analogy? Andrej says basically no. When an LLM reads a book it predicts the next token, when a human does they do synthetic data generation, talk about it with their friends, manipulate the info to gain knowledge. But doing this with LLMs is nontrivial, for reasons that are subtle and hard to understand, and if you generate synthetic data to train on that makes the model worse, because the examples are silently collapsed, similar to how they know like 3 total jokes. LLMs don’t retain entropy, and we don’t know how to get them to retain it. “I guess what I’m saying is, say we have a chapter of a book and I ask an LLM to think about it, it will give you something that looks very reasonable. But if I ask it 10 times, you’ll notice that all of them are the same. Any individual sample will look okay, but the distribution of it is quite terrible.”
I enjoyed the podcast myself, until about this point, where I felt lost. The Zvi’s comments add a lot of value.
I ask whether AI can make a pencil.
Will AI be able to coordinate the process of making a pencil, of gathering and assembling all the required elements, or will this still require a decentralized process governed by market pricing?
substacks referenced above: @
@
@
"I enjoyed the podcast myself, until about this point, where I felt lost. The Zvi’s comments add a lot of value."
Me to ChatGPT: "Help me understand synthetic data generation in this context:" <Insert entire paragraph>
Her response:
🧠 1. What Humans Do: “Synthetic Data Generation” via Reflection
When humans sleep, daydream, or reflect, they reprocess information in rich, diverse ways:
-They simulate alternative interpretations or scenarios (“What if this character had done X instead?”).
-They connect new ideas to existing memories.
-They compress and generalize knowledge — creating abstractions or insights.
-They often generate new internal “data” (thoughts, dreams, analogies) that helps them learn better the next time.
In ML terms, humans are augmenting their own dataset by internally generating new, high-entropy (diverse and information-rich) samples that preserve structure and meaning.
🤖 2. What LLMs Actually Do
An LLM trained with standard next-token prediction is not doing this kind of internal data generation. When it reads a book (inference), it just predicts the next token given context — that’s it.
-It doesn’t “store” new insights or compress them into durable latent structures.
-It doesn’t meaningfully “talk to itself” or “rehearse” multiple versions and integrate them.
-Every generation is effectively a fresh sampling from the same static probability distribution.
So even if you ask an LLM to “think about” a text 10 times, you don’t get 10 different internal reflections — you get 10 nearly identical draws from a sharply peaked distribution.
👉 That’s what Andrej means by “the distribution is terrible.”
It looks fine locally (each sample sounds coherent), but it lacks entropy and diversity globally.
⚗️ 3. Why Synthetic Data Usually Hurts LLMs
If you try to use an LLM to generate its own training data (so-called self-training or bootstrapping) and then retrain it on that output, the model’s knowledge often collapses:
-The generated data tends to smooth over details, losing variance and nuance.
-Repeatedly training on such low-entropy data produces mode collapse — the model becomes blander and less expressive (the “3 jokes” problem).
-The model’s uncertainty is not calibrated — it can’t explore multiple interpretations like a human mind reflecting on a text.
So instead of learning deeper structure, it amplifies its own biases and narrows the distribution of possible responses.
🧩 4. Why This Is Hard to Fix
Humans retain entropy because of:
-Noisy, multi-modal learning signals (vision, emotion, memory, embodiment).
-Iterative reflection processes that integrate and diversify understanding.
-Compression guided by meaning, not just likelihood prediction.
LLMs lack that. Once they’re trained, they’re essentially frozen functions with no internal dynamic that encourages creative reorganization or exploratory rehearsal.
Researchers are exploring ways to mimic this:
-Reflection loops (e.g. self-RAG, “let’s think step-by-step then critique and revise”).
-Diverse sampling and distillation (generate multiple hypotheses, select diverse ones).
-Agentic self-play (models debating or tutoring themselves).
-Training with reinforcement from diversity (rewarding entropy preservation).
But none of these yet capture the qualitative richness of human daydreaming or reflection — that natural generation of novel, semantically meaningful diversity.
Can AI make pencil? At the scale and scope of "I, Pencil" I would say "not profitably." You could certain try, but the cost (without markets) would be enormous. Transaction costs combined with our ever changing world make it highly unlikely that it would be worth it. Capitalism is a profit and lost cultural technology that constantly adapts to change. AI can't keep up with our physically changing world. There are too many sensors to setup, to be adjusted, maintained; there are too many unknowns in living organisms, weather, culture, etc. Too many things will go wrong or change. So no, AI will always be dependent on us.
At a deeper level, I would say, "AI doesn't have the gift of life." Life automatically adapts to change. AI doesn't and can't. AI is dependent on life in order to be adaptive, and it will never exceed the adaptive capability of living organisms.
Why is this?
I would say that AI doesn't have wants and desires that are perpetually adaptable. AI isn't fast enough to keep up with change. We need to give it new wants and desires in order for it to keep up with change. We will constantly need to intervene to help it adapt to change. It will always be a step behind change.
Like writing and books, AI is just an extension of our mind; a prothesis of the mind; an exteriorization of our thinking.
Hence, the adaptability of AI approaches an asymptote relative to change in the physical world. In order to keep up (exceed?) with the physical world, transaction costs have to be sufficiently low. Only living things can make transactions costs sufficient low (in order to stay alive).
AI is a quasi-lifeless tool. We on the other hand, have the gift of life, which allows us to adapt sufficiently fast (faster than?) relative to the rate of change of lifeless matter and other life forms. At least I hope. I'm not yet sure how life will get off this rock, or how it got here. And probably never will.
Background reading: Interview with Douglas North by Arnold Kling and Nick Schulz.