Ethan Mollick on how to use chatbots; The Zvi has another big roundup; Ethan Mollick on Bing as a creative teacher; Tim B. Lee on economic implications; Chatbot fails Steven Landsburg's economics test
Most tests to some extent favor the approach of “guess what the teacher is thinking”. This is why students demand so many practice questions from their instructors and seek out past exams and assignment answer keys surreptitiously when they aren’t provided.
It is also the reason why LLMs can do so well at such a wide variety of exam questions. Like the student gathering example problems, the LLM has a lot of relevant training data but is not really thinking from first principles.
Steve Landsburg’s test does a better job than most of requiring actual thinking, not just predictive regurgitation. Still, I suspect that the true difference between his students’ economic reasoning and GPT’s is overstated by his measurement. Even the best written tests can be gamed through familiarity with teaching style to some degree.
Yes. People are missing the value in ChatGPT/LLMs, at least as they currently exist. My take is that people frequently misapply new technology. They take some task that exists in today’s world and throw new technology at that task. It fails, and people say “see, it’s not so great!” But in my own experience, I don’t want ChatGPT to simply “write an essay” or “take this test”. I don’t want it to replace things I do today. Instead, I want it to help me do my tasks better. This is where Mollick’s writing is just so super-helpful, because he gets this, and he’s trying to figure out how it can be helpful.
To get a driver's license one must pass two tests, a written test and a road test. The written test would be trivial for ChatGPT, the road test not so much. And no matter how long or complex you make the written test, passing it will never constitute proof that you can actually drive an automobile.
Landsburg: Maybe a better prompt or conversation would have produced better results, but YES sensible economics is very few and far between on the Internet. Almost none of it comes from economists.
I apologize in advance, but I still do not grasp why people are so worried about AI. I understand technological solutions exist, but at present it all depends on the electric grid.
Now, administer Steve Landsburg's exam to all Ph.D economists. What would the average score be? What would the distribution of scores be?
Might the answers be awkward?
Most tests to some extent favor the approach of “guess what the teacher is thinking”. This is why students demand so many practice questions from their instructors and seek out past exams and assignment answer keys surreptitiously when they aren’t provided.
It is also the reason why LLMs can do so well at such a wide variety of exam questions. Like the student gathering example problems, the LLM has a lot of relevant training data but is not really thinking from first principles.
Steve Landsburg’s test does a better job than most of requiring actual thinking, not just predictive regurgitation. Still, I suspect that the true difference between his students’ economic reasoning and GPT’s is overstated by his measurement. Even the best written tests can be gamed through familiarity with teaching style to some degree.
Yes. People are missing the value in ChatGPT/LLMs, at least as they currently exist. My take is that people frequently misapply new technology. They take some task that exists in today’s world and throw new technology at that task. It fails, and people say “see, it’s not so great!” But in my own experience, I don’t want ChatGPT to simply “write an essay” or “take this test”. I don’t want it to replace things I do today. Instead, I want it to help me do my tasks better. This is where Mollick’s writing is just so super-helpful, because he gets this, and he’s trying to figure out how it can be helpful.
I was thinking the same thing.
To get a driver's license one must pass two tests, a written test and a road test. The written test would be trivial for ChatGPT, the road test not so much. And no matter how long or complex you make the written test, passing it will never constitute proof that you can actually drive an automobile.
Landsburg: Maybe a better prompt or conversation would have produced better results, but YES sensible economics is very few and far between on the Internet. Almost none of it comes from economists.
Arnold, perhaps you should get Landsberg's exam and submit it yourself to an LLM.
I apologize in advance, but I still do not grasp why people are so worried about AI. I understand technological solutions exist, but at present it all depends on the electric grid.
Isn't now the time to do the research?