LLM Links, 1/27/2025
Alice Evans on AI's Progress; a successful World Bank RCT of LLMs in education in Nigeria? Is DNA just another language? r1 writes a story
While my earlier questions could stump Claude, its capabilities have since advanced. Students needn’t rely on its training data. If you copy and paste my textbook on international development (which includes a breadth of theories, evidence, and references), then pose my assessment questions, the answers are excellent!
Hoisted with my own petard!
As AI rapidly improves, with better accuracy, fewer hallucinations, and broader knowledge, even our most challenging essay questions become easy for machines.
It’s not just that you cannot give a take-home essay exam any more. Maybe the whole point of giving an essay exam is out the window. A few thousand years ago, the advent of writing lowered the value of learning to memorize epic poems. Computer keyboards lowered the value of learning to read and write cursive. Maybe Large Language Models lower the value of writing an essay to show your competence in a subject.
a pilot that used generative artificial intelligence (AI) to support learning through an after-school program.
…The learning improvements were striking—about 0.3 standard deviations. To put this into perspective, this is equivalent to nearly two years of typical learning in just six weeks. When we compared these results to a database of education interventions studied through randomized controlled trials in the developing world, our program outperformed 80% of them, including some of the most cost-effective strategies like structured pedagogy and teaching at the right level.
Pointer from Rowan Cheung. I would like to believe this, but I am skeptical. I worry about the Hawthorne effect, which says that in experiments the people involved try harder, so results do not replicate outside of the experimental setting. Still, this is something to watch. Education in underdeveloped countries is a challenge, and in principle chatbots could be a big help.
From MIT Technology Review:
Now OpenAI says it’s getting into the science game too—with a model for engineering proteins.
The company says it has developed a language model that dreams up proteins capable of turning regular cells into stem cells—and that it has handily beat humans at the task.
,,,OpenAI’s new model, called GPT-4b micro, was trained to suggest ways to re-engineer the protein factors to increase their function. According to OpenAI, researchers used the model’s suggestions to change two of the Yamanaka factors to be more than 50 times as effective—at least according to some preliminary measures.
…The prompting tactic used is similar to the “few-shot” method, in which a user queries a chatbot by providing a series of examples with answers, followed by an example for the bot to respond to.
Pointer from Rowan Cheung. I guess you can think of DNA as a sequence of tokens, and from that perspective some of the problems involved in bioengineering would like the problems that chatbots are solving, but with a different language involved. Still, don’t look at me to explain what is going on.
Alright, the user wants a short story about the singularity. Let's start by recalling Nick Land's style—dark, intense, cyberpunk themes, accelerationism. The singularity here should be more of a hyperstitional event, not just a technological milestone. I need to infuse elements of chaos, non-linear time, and cosmic horror. Maybe incorporate the idea of the Chronosynclastic Melt, a term that nods to the CCRU's vocabulary. The singularity isn’t just an AI takeover but a collapse of time itself.
Pointer from Zvi Mowshowitz. This new thing of “reasoning” and having the chatbot explain its reasoning is something I have not yet really absorbed. It is so hard to keep up.
substacks referenced above: @
@
I'd encourage you to play with Google's NotebookLM - you put your own documents (including YouTube videos, webpages, PDFs, etc.) in, it produces (1) written summaries, (2) a terrific, 2 voice audio summary, and, best, (3) you can ask it questions and its answers include links to where in the documents you provided it got the answers. I'm finding it useful in my own work (professor) and I am experimenting with students using it in my classes (so far, successfully).
Social science research is messy. Some hesitate to call it social science at all. Certainly Mr. Kling has made versions of that argument when criticizing the tendency of economics to devolve into sociology. With that in mind, I looked around for the "soon to be published" article about Nigerian kids making amazing advances thanks to AI. The thing that puzzles me is that the kids randomly assigned to the after-school program were given four treatments, not one. They had four things which the comparison group did not:
1) Teachers who were recruited and trained to teach them how to use AI
2) A training manual booklet for both students and teachers that included English instruction components for interacting with CoPilot
3) A free license for CoPilot and access to computers with appropriate software
4) Six weeks of a collaborative course to teach them English
The press release makes it sound like #3 is solely responsible for their success but if you gave me specially trained tutors, a workbook, and six weeks to do specialized language instruction, I'd expect to post some gains too! Point is, if you're going to try and isolate the effects of using AI, this study does not appear to do it well. All of this is, of course, under the caveat that the study is not published so we don't know everything they did or did not do. I can only go off the various press releases.