Brillant observation about the shiftfrom Claude expecting expertise to enabling actual vibe-coding. Back when I was experimenting with similar tools, that friction between 'what the AI assumes I know' and 'what I actually need to do' was brutal. The real value in that virtual wax museum assignment isn't just geting students comfortable with AI tools but making them document the workflow itself. Most orgs still have no idea what these tools can actually deliver vs the hype.
Not sure if what I do is "vibe-coding", but I've been using Claude Code a lot lately to help with testing our startup's product, which does migration of data between different types of databases.
I used Claude to code test data generators, test change generators, a deep-crawl correctness validator, and a test harness.
I've written things like these "by hand" several times in the past, and they are tedious but rather straightforward engineering tasks, but they take quite a bit of time to get right.
One big thing that Claude does is it wants to be "lazy", so you have to make darn sure it doesn't do things like hack the test validation code to make a particular test pass!
For testing, Claude is awesome in many ways. It grinds through numerous logs, quickly uncovering the root problem and recommending a solution, that is correct most of the time (but here you still need to watch its "laziness": it will recommend working around a problem that it identifies versus fixing the thing that generated the badness in the first place. So, if you're using it like this, you have to stop Claude from "localized" fixes.
It helps if you give Claude clear directions about stuff like "we will always use Standard Format X for Datatype Y in our 'consumer' - if we're not seeing X, the fault is in the 'producer' that give us Format Z, not the fact that we can't parse Format Z".
Using Claude Code for this stuff has saved us several man-months of time in coding the test infrastructure as well as grinding through several dozen combinations of load-types and source/target combinations.
Claude is awesome at identifying weird datatype format problems and other tedious things that are huge pains-in-the-ass for human programmers, especially if the solution involves combining log crawls, code digging, and looking up stuff online. Claude can do this in a couple of minutes, while even an experienced programmer may spend hours futzing with this sort of stuff.
Also, Claude is good at stuff like generating parallelized code or adding parallelism to existing code.
Where Claude isn't great is at high-level design. You need to come up with the high-level plan of how you want your stuff to work, carefully specify it in a set of prompts (or a spec document that's in a location and format that Claude can ingest), force Claude to dialog with you about its questions, and only have it generate code once you confirm that Claude understands what you want by basically repeating your design back to you.
Claude can also flail badly if it's trying to solve something harder, particularly if the online documentation isn't good and you haven't been rigorous with your requirements. You have to keep Claude's "eye on the ball" and be rigorously consistent with what you want it to do and what you don't want it to do.
And yes, Claude can generate "wrong" or incomplete code. You have to run a lot of tests to make sure its code is correct.
And like human programmers, Claude will only do what you ask it to do, not necessarily what you wanted.
I'm watching my wife slowly inch her way towards vibe coding. She's responsible for creating the schedule for her group of ~20 radiologists. It's a complicated allocation problem, which I actually helped her out several years ago by building a bespoke Excel document with lots of VBA. However, the dynamics of the group have been in constant flux since then with frequent changes in both staffing and site-specific constraints so she's ended up doing it mostly by hand. Last month she finally started asking ChatGPT to help her in the main chat app, and she got some early successes. I could tell she was excited when it started spitting out lists of assignments after "thinking" about it for several minutes while she scrolled Twitter. Of course, she still had to go back in and manually enter the information to their online scheduling platform, and when someone asked for a day off for a doctor's appointment, she had to do the whole thing over, but it was a start. I've been trying to gently suggest that she consider installing VS Code and trying to use Codex to build an app for her to do the work, but so far she's resisted.
I think that that is still the biggest boundary. If you can cross the threshold from the web-based AI chat apps and start interacting with the technology inside of a development environment, a lot of the hype starts to become much more understandable. My guess is that OpenAI and others would have already moved to incorporate an IDE within their chat apps if they weren't constrained by compute, so it's probably only a matter of (a very short span of) time.
Is it not reasonable to think that Ai will quickly make human coding completely obsolete? I began my college era as a computer science major. I switched to economics two years later. I learned several programming languages back in the early 1970s, including Fortran and PL1 and Cobal.
I did part of the quant analysis of my PhD dissertation using a programmable Hewlit-Packard calculator. I knew my way around programming. I tried for awhile to keep up during my career, once attempting to learn C, but I lost interest because the cost exceeded the benefits, even though I worked as a professional economist.
For the past 20 years, I have refused to learn further coding for the same reason. I think it is entirely likely that anyone learning coding right now is learning a skill that will be completely obsolete within an amazingly little time. What do you think?
All the coding an Ai will ever do is coding that a human directs it to do, in my opinion. The direction from a human might be direct or indirect, but I do not think that a machine can or ever will have motivation or goals that are not planted by a human, any more than I think a rock will spring to life.
The really big question is what happens next after AI will be able to take over the coding of improvements to AI, self-improvement and maybe, "self-agency" (?) and "self-awareness" (?) The question is whether this will be contrillable, at a manageable pace, predictable, or so rapid and unpredictable as to constitute a genuine "all bets are off" historical discontinuity.
Personally, I do not think that computing is anything like being human. Regardless of how fast a machine can compute, which is nothing more nor less than counting zeros and ones in particular ways, I am not even slightly worried about machine self awareness nor am I worried about the "Terminator Hypothesis," as i call it.
I am worried a little that Ai can put greater power in the hands of human psychopaths and sociopaths, of course.
The extent of similarity to human-ness and whether there are any theoretical limits, and where those might be and why, are fascinating philosophical (and inevitably, technical) questions. But they are also distracting and besides the point when it comes to what happens when coding-code can code improvements to itself, with some combination of rational analysis and trial and error.
One can consider this to be a question regarding an entirely incomparable, different, and "alien" form of intelligence, and the issue of the major historical consequences of its emerging capacity for iterative self-improvement still stands. Even far below the human level, higher capability "clever" intelligence in animals seems to have evolved independently at least three times, in the lines leading to the smartest mammals, birds, and cephalopods. It doesn't matter if historically these forms of intelligences were similar to each other or more wholly distinct approaches or not. When primate intellectual capacity blew up in what, in biological terms was "suddenly", it took over the planet and subordinated the others forever.
The question is similar in some ways to that of what might happen if humans fully master the art and science of genetically engineering superior humans and also fully embraced doing so with a culturally competitive, full-speed-ahead attitude. But the pace of digital evolution is orders of magnitude faster than anything biological, and can occur at an equally disproportionately lower cost.
The digital version now seems to be the incoming round that is closest and moving in fastest. Maybe it'll be a dud. Or maybe it will make further meaningful human participation in the economy impossible, also forever.
"Vibe-coding is not what I am supposed to be teaching, but it is something I think that they should learn. In the Public Choice class, I have a specific vibe-coding assignment, a virtual wax museum of certain economists, because I want them to get used to vibe-coding."
Programming a virtual wax museum is a good assignment, but I would be careful not to over-emphasize vibe-coding or any other trendy programming technique. I would ask open-ended questions about the pros and cons of using AI to implement the museum.
AI coding tools may reveal certain limitations compared to tradition coding techniques. For example, do AI programming tools allow the code be sufficiently customized to meet specific project goals and requirements? Or does AI limit the implementation in certain ways, producing commodity-type applications that any Joe Blow can get running. Perhaps traditional coding still produces an important comparative advantage. Let the students discover for themselves through experience.
You might stress students by asking for impossible goals: quick implementation, unique implementation, and accurate implementation. Note the tradeoffs in using various programming tools. Is there a solution, or just tradeoffs?
At Challenger School where my children attended a few years back, children start learning how to program in the 1st grade. Each year up to 8th grade the programming assignments increase in difficulty. Starting in the 3rd grade, there is an annual grade-level computer programming competition with awards for 1st, 2nd, and 3rd.
As a 6th grader, my daughter won 1st place for creating a ChatBot using Python. Of course it wasn't as fancy as an AI chatbot, or a Milton Friedman chatbot--it had a few bugs; it used many if statements with varying responses depending on categories of keyword inputs. The fact that it changed or seemed to change depending on user input made it fun for kids to play with. At least the admin was impressed.
I tell you this story because elementary and middle school children might soon be producing virtual wax museums. How might your class be better than K-8 versions?
I think the goal is to get a bit of exposure to AI tools while learning about great economists, with the primary focus on the second part.
That said, with the pace of change in AI tools, one has to wonder if what one learns about AI development will still be useful or otherwise helpful the next time the student does something similar. IDK.
I think the idea that human coding will become obsolete is... Short-sighted. For one, you'd need a human debugger for when the LLM fails to produce a useful result. Second, you're ignoring one other pitfall of LLM coding, by and large it replicates what's already out there. Which means it's got all the same limitations and vulnerabilities of current software. Which means that changes in the cybersecurity space cannot be responded to quickly by vibe coding (you'd need to wait for the next model update, check to make sure the code didn't repeat the same issue, etc.). I'm saying this not as a coder (I do coding tangentially to my job), but as someone aware of the cybersecurity implications.
The models were trained on human written code. The models can interpolate, but extrapolation is, in my understanding, go completely off the rails. The vibe coding is, in effect, a form of base code reuse that is useful primarily for problems that have already been solved by someone else. That is useful, but very far from replacing humans. I don't even want to think about (although I just did) what happens when nation states start poisoning the training repositories.
Write, but when human coders become "obsolete" there will no longer be new code to learn from, models will cannibalize each other and hallucinate into oblivion.
I suppose you can call extrapolation on how current models work to be an assumption. If we develop an entirely new method of AI without LLMs then all bets are off.
1 AI will be able to do everything coders do. I have a hard time seeing a "civilian" and AI creating the architecture to tackle large and complex software creation without someone who knows programming
2 while AI has little or no ability to innovate to solve new problems without a lot of hallucinations, it's not clear whether that will continue to be true.
It’s not just new problems but recognizing and coding for constraints. Picking the right solution also means it must be robust. The LLM is a token predictor based on what it sees in training sets. That is, given a set of tokens, it picks the most likely next token in the sequence. It is essentially a very good Wheel of Fortune player. Users of AI don’t have to be good at wheel of fortune, the LLM does that. But good users ask the right questions., often multiple times to get it right. The Jeopardy champs drive the Wheel of Fortune drone. New approaches could replace LLM, but they would likely be entirely different.
Brillant observation about the shiftfrom Claude expecting expertise to enabling actual vibe-coding. Back when I was experimenting with similar tools, that friction between 'what the AI assumes I know' and 'what I actually need to do' was brutal. The real value in that virtual wax museum assignment isn't just geting students comfortable with AI tools but making them document the workflow itself. Most orgs still have no idea what these tools can actually deliver vs the hype.
Not sure if what I do is "vibe-coding", but I've been using Claude Code a lot lately to help with testing our startup's product, which does migration of data between different types of databases.
I used Claude to code test data generators, test change generators, a deep-crawl correctness validator, and a test harness.
I've written things like these "by hand" several times in the past, and they are tedious but rather straightforward engineering tasks, but they take quite a bit of time to get right.
One big thing that Claude does is it wants to be "lazy", so you have to make darn sure it doesn't do things like hack the test validation code to make a particular test pass!
For testing, Claude is awesome in many ways. It grinds through numerous logs, quickly uncovering the root problem and recommending a solution, that is correct most of the time (but here you still need to watch its "laziness": it will recommend working around a problem that it identifies versus fixing the thing that generated the badness in the first place. So, if you're using it like this, you have to stop Claude from "localized" fixes.
It helps if you give Claude clear directions about stuff like "we will always use Standard Format X for Datatype Y in our 'consumer' - if we're not seeing X, the fault is in the 'producer' that give us Format Z, not the fact that we can't parse Format Z".
Using Claude Code for this stuff has saved us several man-months of time in coding the test infrastructure as well as grinding through several dozen combinations of load-types and source/target combinations.
Claude is awesome at identifying weird datatype format problems and other tedious things that are huge pains-in-the-ass for human programmers, especially if the solution involves combining log crawls, code digging, and looking up stuff online. Claude can do this in a couple of minutes, while even an experienced programmer may spend hours futzing with this sort of stuff.
Also, Claude is good at stuff like generating parallelized code or adding parallelism to existing code.
Where Claude isn't great is at high-level design. You need to come up with the high-level plan of how you want your stuff to work, carefully specify it in a set of prompts (or a spec document that's in a location and format that Claude can ingest), force Claude to dialog with you about its questions, and only have it generate code once you confirm that Claude understands what you want by basically repeating your design back to you.
Claude can also flail badly if it's trying to solve something harder, particularly if the online documentation isn't good and you haven't been rigorous with your requirements. You have to keep Claude's "eye on the ball" and be rigorously consistent with what you want it to do and what you don't want it to do.
And yes, Claude can generate "wrong" or incomplete code. You have to run a lot of tests to make sure its code is correct.
And like human programmers, Claude will only do what you ask it to do, not necessarily what you wanted.
I'm watching my wife slowly inch her way towards vibe coding. She's responsible for creating the schedule for her group of ~20 radiologists. It's a complicated allocation problem, which I actually helped her out several years ago by building a bespoke Excel document with lots of VBA. However, the dynamics of the group have been in constant flux since then with frequent changes in both staffing and site-specific constraints so she's ended up doing it mostly by hand. Last month she finally started asking ChatGPT to help her in the main chat app, and she got some early successes. I could tell she was excited when it started spitting out lists of assignments after "thinking" about it for several minutes while she scrolled Twitter. Of course, she still had to go back in and manually enter the information to their online scheduling platform, and when someone asked for a day off for a doctor's appointment, she had to do the whole thing over, but it was a start. I've been trying to gently suggest that she consider installing VS Code and trying to use Codex to build an app for her to do the work, but so far she's resisted.
I think that that is still the biggest boundary. If you can cross the threshold from the web-based AI chat apps and start interacting with the technology inside of a development environment, a lot of the hype starts to become much more understandable. My guess is that OpenAI and others would have already moved to incorporate an IDE within their chat apps if they weren't constrained by compute, so it's probably only a matter of (a very short span of) time.
Is it not reasonable to think that Ai will quickly make human coding completely obsolete? I began my college era as a computer science major. I switched to economics two years later. I learned several programming languages back in the early 1970s, including Fortran and PL1 and Cobal.
I did part of the quant analysis of my PhD dissertation using a programmable Hewlit-Packard calculator. I knew my way around programming. I tried for awhile to keep up during my career, once attempting to learn C, but I lost interest because the cost exceeded the benefits, even though I worked as a professional economist.
For the past 20 years, I have refused to learn further coding for the same reason. I think it is entirely likely that anyone learning coding right now is learning a skill that will be completely obsolete within an amazingly little time. What do you think?
Possible. But it's not even certain that AI will reduce the need for competent programmers, much less eliminate.
ATMs replaced tellers yet secondary effects caused the number of tellers to increase.
All the coding an Ai will ever do is coding that a human directs it to do, in my opinion. The direction from a human might be direct or indirect, but I do not think that a machine can or ever will have motivation or goals that are not planted by a human, any more than I think a rock will spring to life.
The really big question is what happens next after AI will be able to take over the coding of improvements to AI, self-improvement and maybe, "self-agency" (?) and "self-awareness" (?) The question is whether this will be contrillable, at a manageable pace, predictable, or so rapid and unpredictable as to constitute a genuine "all bets are off" historical discontinuity.
Personally, I do not think that computing is anything like being human. Regardless of how fast a machine can compute, which is nothing more nor less than counting zeros and ones in particular ways, I am not even slightly worried about machine self awareness nor am I worried about the "Terminator Hypothesis," as i call it.
I am worried a little that Ai can put greater power in the hands of human psychopaths and sociopaths, of course.
The extent of similarity to human-ness and whether there are any theoretical limits, and where those might be and why, are fascinating philosophical (and inevitably, technical) questions. But they are also distracting and besides the point when it comes to what happens when coding-code can code improvements to itself, with some combination of rational analysis and trial and error.
One can consider this to be a question regarding an entirely incomparable, different, and "alien" form of intelligence, and the issue of the major historical consequences of its emerging capacity for iterative self-improvement still stands. Even far below the human level, higher capability "clever" intelligence in animals seems to have evolved independently at least three times, in the lines leading to the smartest mammals, birds, and cephalopods. It doesn't matter if historically these forms of intelligences were similar to each other or more wholly distinct approaches or not. When primate intellectual capacity blew up in what, in biological terms was "suddenly", it took over the planet and subordinated the others forever.
The question is similar in some ways to that of what might happen if humans fully master the art and science of genetically engineering superior humans and also fully embraced doing so with a culturally competitive, full-speed-ahead attitude. But the pace of digital evolution is orders of magnitude faster than anything biological, and can occur at an equally disproportionately lower cost.
The digital version now seems to be the incoming round that is closest and moving in fastest. Maybe it'll be a dud. Or maybe it will make further meaningful human participation in the economy impossible, also forever.
Maybe. Maybe not. I don't see how they are beside the point.
"Vibe-coding is not what I am supposed to be teaching, but it is something I think that they should learn. In the Public Choice class, I have a specific vibe-coding assignment, a virtual wax museum of certain economists, because I want them to get used to vibe-coding."
Programming a virtual wax museum is a good assignment, but I would be careful not to over-emphasize vibe-coding or any other trendy programming technique. I would ask open-ended questions about the pros and cons of using AI to implement the museum.
AI coding tools may reveal certain limitations compared to tradition coding techniques. For example, do AI programming tools allow the code be sufficiently customized to meet specific project goals and requirements? Or does AI limit the implementation in certain ways, producing commodity-type applications that any Joe Blow can get running. Perhaps traditional coding still produces an important comparative advantage. Let the students discover for themselves through experience.
You might stress students by asking for impossible goals: quick implementation, unique implementation, and accurate implementation. Note the tradeoffs in using various programming tools. Is there a solution, or just tradeoffs?
At Challenger School where my children attended a few years back, children start learning how to program in the 1st grade. Each year up to 8th grade the programming assignments increase in difficulty. Starting in the 3rd grade, there is an annual grade-level computer programming competition with awards for 1st, 2nd, and 3rd.
As a 6th grader, my daughter won 1st place for creating a ChatBot using Python. Of course it wasn't as fancy as an AI chatbot, or a Milton Friedman chatbot--it had a few bugs; it used many if statements with varying responses depending on categories of keyword inputs. The fact that it changed or seemed to change depending on user input made it fun for kids to play with. At least the admin was impressed.
I tell you this story because elementary and middle school children might soon be producing virtual wax museums. How might your class be better than K-8 versions?
I think the goal is to get a bit of exposure to AI tools while learning about great economists, with the primary focus on the second part.
That said, with the pace of change in AI tools, one has to wonder if what one learns about AI development will still be useful or otherwise helpful the next time the student does something similar. IDK.
A bit of exposure is also a bit boring. Better to get deep exposure and create something substantial.
I think the idea that human coding will become obsolete is... Short-sighted. For one, you'd need a human debugger for when the LLM fails to produce a useful result. Second, you're ignoring one other pitfall of LLM coding, by and large it replicates what's already out there. Which means it's got all the same limitations and vulnerabilities of current software. Which means that changes in the cybersecurity space cannot be responded to quickly by vibe coding (you'd need to wait for the next model update, check to make sure the code didn't repeat the same issue, etc.). I'm saying this not as a coder (I do coding tangentially to my job), but as someone aware of the cybersecurity implications.
The models were trained on human written code. The models can interpolate, but extrapolation is, in my understanding, go completely off the rails. The vibe coding is, in effect, a form of base code reuse that is useful primarily for problems that have already been solved by someone else. That is useful, but very far from replacing humans. I don't even want to think about (although I just did) what happens when nation states start poisoning the training repositories.
Write, but when human coders become "obsolete" there will no longer be new code to learn from, models will cannibalize each other and hallucinate into oblivion.
That prediction is based on multiple assumptions. Let's hope none of them come true.
I suppose you can call extrapolation on how current models work to be an assumption. If we develop an entirely new method of AI without LLMs then all bets are off.
Assumptions:
1 AI will be able to do everything coders do. I have a hard time seeing a "civilian" and AI creating the architecture to tackle large and complex software creation without someone who knows programming
2 while AI has little or no ability to innovate to solve new problems without a lot of hallucinations, it's not clear whether that will continue to be true.
It’s not just new problems but recognizing and coding for constraints. Picking the right solution also means it must be robust. The LLM is a token predictor based on what it sees in training sets. That is, given a set of tokens, it picks the most likely next token in the sequence. It is essentially a very good Wheel of Fortune player. Users of AI don’t have to be good at wheel of fortune, the LLM does that. But good users ask the right questions., often multiple times to get it right. The Jeopardy champs drive the Wheel of Fortune drone. New approaches could replace LLM, but they would likely be entirely different.