LLM Links

Chad Syverson on productivity; Scott Gottlieb and Shani Benezra on Dr Chatbot; Peter Attia and Zak Kohane on AI in medicine; Ethan Mollick on the current state of AI; Evan Ratliff on phone bots

In an interview, Chad Syverson says,

You know, I mentioned those big productivity differences within even narrowly defined industries before. Even before AI was around, we have good evidence that some of those differences are caused by differences in management practices, manager skill, et cetera. I view that as an intangible that is paired with technologies like AI, and the quality of that pairing is going to matter hugely for the productivity effect.

Later,

you can go pretty long periods of time where the technology’s out there, it’s being installed, it’s starting to be used, but you’re still not seeing its full effect reflected in the productivity statistics. And we talk about historical examples, Solow’s famous comment that he saw the computer age everywhere except in the productivity statistics. That was exactly that phenomenon.

And then there are other examples, going back in history even further, of periods where the technologies are commercially available and they’re starting to be used, but you don’t see it in the productivity stats yet. This can go on for a period of multiple years.

The whole interview is interesting. Pointer from Timothy Taylor.

Scott Gottlieb and Shani Benezra write,

To secure a medical license in the United States, aspiring doctors must successfully navigate three stages of the U.S. Medical Licensing Examination (USMLE), with the third and final installment widely regarded as the most challenging. It requires candidates to answer about 60% of the questions correctly, and historically, the average passing score hovered around 75%.

When we subjected the major large language models (LLMs) to the same Step 3 examination, their performance was markedly superior, achieving scores that significantly outpaced many doctors.

ChatGPT-4o got 49 out of 50 questions correct. Claude 3.5 got 45 out of 50.

David Shaywitz summarizes a podcast with Peter Attia and Zak Kohane.

Attia suggests the adoption of AI would likely occur in two stages. The first specialties to be impacted, he says, will probably be those he describes as visually focused: pathology, radiology, dermatology, and cardiology (especially those most involved in interpreting cardiac imaging studies). Next would be specialties he sees as focused on integrating “language data” and visual data, like primary care doctors and pediatricians.

While Kohane says he agreed with Attia’s proposed order of adoption, he emphasizes that a more immediate consideration is the shortage of doctors in key areas like primary care. 

“You have to ask yourself, how can we replace these absent primary care practitioners with nurse practitioners, with physician assistants augmented by these AIs, because there’s literally no doctor to replace.”

I think they are under-estimating the ability of LLMs to extract information by conversing with patients, not just by reading images.

Ethan Mollick writes,

we do not need any further advances in AI technology to see years of future disruption. Right now, AI systems are not well-integrated into businesses and organizations, something that will continue to improve even if LLM technology stops developing. And a complete halt to AI development seems unlikely, suggesting a future of continuous linear or exponential growth, which are also possible without achieving AGI. AI isn’t going away and is disruptive enough that we have to make decisions about it today, even if we don’t believe the technology will advance further. How do we want to handle the fact that AI is already impacting jobs? That LLMs can be used to create mass targeted phishing campaigns? That it is changing how students are learning in class? AI is not a future technology to be dealt with if it happens, it is here now and will require us to think about how we want to use it.

AI pundits keep repeating: this technology is now the worst it is ever going to be. Mollick says a lot in the essay. If you have not already read it, you should.

In an interview with David Epstein, Evan Ratliff says,

a couple months in I discovered that there were a bunch of platforms — Vapi, RetellAI, BlandAI — that did exactly the thing I was so proud of cobbling together, except many many times better and faster. These platforms are built to create voice agents, and for developers to use them as AI receptionists, AI call center callers, AI sales callers, AI therapists, and much more. 

…You can have a voice agent with your voice, ready to call, in an hour for $30-40. 

substacks referenced above:

@

Confronting Impossible Futures

@

Attack of the AI Voice Clones