I started testing my own productivity using a modified methodology from that study. I’d take a task and I’d estimate how long it would take to code if I were doing it by hand, and then I’d flip a coin, heads I’d use AI, and tails I’d just do it myself. Then I’d record when I started and when I ended. That would give me the delta, and I could use the delta to build AI vs no AI charts, and I’d see some trends. I ran that for six weeks, recording all that data, and do you know what I discovered?
I discovered that the data isn’t statistically significant at any meaningful level. That I would need to record new datapoints for another four months just to prove if AI was speeding me up or slowing me down at all. It’s too neck-in-neck.
So, is AI productivity in software development overhyped?
Judge also writes,
I know software development. I’ve been doing it for 25 years, maybe even 28 years1 if you count market research tabulation on amber monochrome screens. Yes, I’m old. I’m a middle-aged programming nerd. My entire life and personal identity are wrapped up in this programming thing for better or worse. I thrive off the dopamine hits from shipping cool things.
Let me guess that he is in the 98th percentile of software developers. If so, then perhaps we should not be surprised that Claude is no better than he is.
As you know, my own experience working with Claude to develop The Social Code was dramatic. It would have taken me years to do the project myself, but I completed it in a couple of weeks.
I am nowhere near Judge’s level of competence in software development. Before I tried this project, the last “development environment” I had used was Notepad. I knew nothing about Github or any of the modern software packages. I never edited Claude’s code directly, because I was afraid I would make things worse. When it was not working, we discussed it until we had a fix, and then Claude rewrote it.
Suppose that as a software developer, Claude is only in the 40th percentile. That still makes a lot of software developers obsolete! My guess is that the median coder in Bangalore no longer seems attractive to American information technology managers.
Judge asks,
If so many developers are so extraordinarily productive using these tools, where is the flood of shovelware? We should be seeing apps of all shapes and sizes, video games, new websites, mobile apps, software-as-a-service apps — we should be drowning in choice. We should be in the middle of an indie software revolution. We should be seeing 10,000 Tetris clones on Steam.
My own Social Code project could count as a new website. But the world does not necessarily need new video games, websites, and apps.
What the world needs is more effective education and more cost-effective health care. In my opinion, the AI revolution stands or falls on whether it makes a difference in those sectors, which Nick Schulz and I dubbed The New Commanding Heights. If AI achieves a lot in education and health care, the economic impact will be huge. If instead those sectors remain stagnant, I do not believe that the AI investment will earn much of a return.
AI is the ultimate DIY power tool. There is a ton of new extra utility and value that has been unlocked, but by its "home production" non-traded nature is not going to show up in the usual economic statistics. I've said similar things about modern power tools and YouTube in the past, and for DIY, AI is several qualitative tiers above mere videos.
My personal anecdotal experience is that AI tools have proven to be extremely valuable, and far in excess of what I've paid for them. I have now done dozens of formerly "technical professional services" projects, safely and effectively, for myself, in short time, that were recently completely beyond my capability on almost any timescale. The former "pay a human" prices were totally prohibitive, but now that I am augmented with powerful AI tools at negligible cost, all sorts of possibilities have opened up. I've saved thousands of dollars just on being able to do diagnostics and repairs on big-ticket items by myself.
In addition to the utility of the pride and satisfaction one feels at pulling these things off with one's own "mens et manus", as it were, there is also the avoidance of the feeling of modern principal-agent humiliation at being utterly helpless and dependent on the word of a professional who has every incentive to avoid liability-risking real-talk and to exaggerate your needs and make the bill as large as possible and who can easily get away with feeding you a bunch of malarkey about how long things take and so forth. (Before someone tries, please don't give me that stuff about Angie's List or other reputation systems or getting second opinions, I have plenty of direct experiences and arguments against the real practicality and utility of those things.)
Now, one might suspect that since the learning curves and cost of investment in human capital necessary to pull off these projects has collapsed, and that the threshold to use the AI tools effectively is not too elite, that a whole new class of AI-assisted workers would pop up to offer their services much cheaper than the existing professionals and substantially lower prices for those services.
But there seem to be some kind of inertia of transition or friction or barriers to entry or establishment of that kind of employment, so potential consumers like me just end up doing a lot more things "on their own". The amount of a service being provided is going up, but the number of workers doing that service professionally, and the prices they are charging, seems to not be changing very much. Or else, like in some fields already, there is a fairly rapid collapse in the number of humans getting paid well to do that kind of work anymore.
I've been programming since 1968. I don't believe current AI can make me a better programmer. But I do believe it can make Joe Blow (or Arnold Kling!) a programmer, period, just as spreadsheets and Visual Basic did, just as word processors made a lot more people writers who gave up with the relative perfection required by typewriters, just as typewriters made more written communication and documentation possible than longhand and ink did. Heck, just as WordPress and Substack have expanded blogging. Now even schmoes like me can start a blog in five minutes! The quality is up for debate, but most jobs don't need perfessional programmers or writers.
Someone famous said progress doesn't come from making better silk stockings for Marie Antoinette, but from making more nylon stockings for all women (or Joe Namath). That's what AI, spreadsheets, typewriters, and word processors do.