Haidt and Schmidt predict AI twitter trolls; Ethan Mollick on LLMs as human; A Stanford paper on LLMs as not so emergent; Patrick Collison and Sam Altman; The Zvi refutes me;
Black box AI systems are a problem and will not be adopted in regulated industries. Think of a large financial services company. An AI-powered risk model approves or rejects something - a loan applicant, a trade -- but can’t explain why. In a black box model no one can piece together how the model made its decision. And then it can’t be replicated. Can’t be shown to be non discriminatory. Can’t be explained. This is a major limitation. The medium to longer term future of AI will incorporate explainability as a key feature. But this is still far away.
The big limits on AI breaking into healthcare (and probably some other fields) is that you just aren't allowed to make big errors even a tiny portion of the time (and when you do the errors need to be in predictable and acceptable ways).
This is why the Silicon Valley mindset of Elizabeth Holmes didn't work, it's fine to have a buggy website but not buggy blood test results. Even a tiny amount of big unauthorized errors will sink AI in healthcare (as a means of delivering healthcare, as a means of upcoding and overcharging the government I believe it has a bright future).
"Dustin Muskovitz predicts that by 2030 ~everyone will have a personal AI agent to do their paperwork for taxes, government services, health care forms and so on. I’d expect this to move quicker than that, although the future is always unevenly distributed"
I do my mother's and my tax returns by hand. I have noticed that filing tax returns is becoming more onerous as time goes on. Our returns are always basically the same, except that in some years we have sold stocks or bonds, and yet I am filing 4 times the number of forms now that I was filing 10 years ago. I wonder if that isn't a result of the rise of tax software like Turbo Tax. So, you may be right- that AI preparing other paperwork required by government might well lead to the government requiring even more of it.
"If your bank’s software mostly works, but sometimes scolds you for wanting to withdraw money, sometimes steals your money and lies to you about it, and sometimes spontaneously manages your money to get you better returns, you would not be very happy. "
I think your point about empathizers misses two important aspects here.
1: Never mind software, we would be really angry about humans who do this, too. Even in more prosaic cases like the AI advisor who gives you life tips we were talking about a few months back, an advisor real or AI who sometimes responds to your requests with an angry "I'm busy, get out of my office!", sometimes gives you terrible advice by accident, sometimes gives you poor advice because it is funny, and yet sometimes gives amazing advice wouldn't be considered very good, or even desirable depending on relative frequencies and order of results. I have never responded to a student's request for career advice with "End thyself", but I bet it wouldn't go over well.
2: Systemizers like things that are systematic and predictable, but empathizers also like things that are empathetic. LLM AIs are certainly not, although they can sometimes fake it well for a short period. I am not optimistic about how well that works long term... see the terry cloth monkey mom experiment referenced here: https://kenazfilan.substack.com/p/from-the-mouth-of-madness-2-the-terrycloth
One reason we have what Yuval Levin would call "formative institutions" is to channel human variability into more predictable forms and make human unpredictability less scary. I can know when interacting with a formative institution that-- for better and often for worse-- the people in that institution will feel an obligation to "play their role" and do the predictable thing.
As I understand it, if you know what role you want an LLM to play, you can often get it to fulfill the predictable obligations of that role by prompting it to "think of itself" as playing that role. Ethan Mollick has a bunch of examples of this in his posts on good prompt engineering. It may be that some of the scariness of LLM unpredictability comes from the fact that we don't really know either (a) what those roles are that we really want LLMs to play or (b) how to articulate those roles clearly. Levin would probably say that (b) is a symptom of the decline of formative institutions generally.
"If your bank’s software mostly works, but sometimes scolds you for wanting to withdraw money, sometimes steals your money and lies to you about it,...you would not be very happy"
My bank's software already has, and I'm already not very happy about it.
And whether it's new complexity introduced by the use of LLM or the software has reached levels of complexity that their IT team can't manage before even using LLMs, I don't know. But I do know it is becoming more common for them not to be able to give me an answer about what's the reason for the failure every time something fails (and I've got a good sample, I'm not joking when I say I'm not happy). The moment they can start blaming "AI" for blocking any operation for reasons of "safety" they will do it, because it will simply make life so much easier in dealing with customer support.
Black box AI systems are a problem and will not be adopted in regulated industries. Think of a large financial services company. An AI-powered risk model approves or rejects something - a loan applicant, a trade -- but can’t explain why. In a black box model no one can piece together how the model made its decision. And then it can’t be replicated. Can’t be shown to be non discriminatory. Can’t be explained. This is a major limitation. The medium to longer term future of AI will incorporate explainability as a key feature. But this is still far away.
The big limits on AI breaking into healthcare (and probably some other fields) is that you just aren't allowed to make big errors even a tiny portion of the time (and when you do the errors need to be in predictable and acceptable ways).
This is why the Silicon Valley mindset of Elizabeth Holmes didn't work, it's fine to have a buggy website but not buggy blood test results. Even a tiny amount of big unauthorized errors will sink AI in healthcare (as a means of delivering healthcare, as a means of upcoding and overcharging the government I believe it has a bright future).
"Dustin Muskovitz predicts that by 2030 ~everyone will have a personal AI agent to do their paperwork for taxes, government services, health care forms and so on. I’d expect this to move quicker than that, although the future is always unevenly distributed"
I do my mother's and my tax returns by hand. I have noticed that filing tax returns is becoming more onerous as time goes on. Our returns are always basically the same, except that in some years we have sold stocks or bonds, and yet I am filing 4 times the number of forms now that I was filing 10 years ago. I wonder if that isn't a result of the rise of tax software like Turbo Tax. So, you may be right- that AI preparing other paperwork required by government might well lead to the government requiring even more of it.
"If your bank’s software mostly works, but sometimes scolds you for wanting to withdraw money, sometimes steals your money and lies to you about it, and sometimes spontaneously manages your money to get you better returns, you would not be very happy. "
I think your point about empathizers misses two important aspects here.
1: Never mind software, we would be really angry about humans who do this, too. Even in more prosaic cases like the AI advisor who gives you life tips we were talking about a few months back, an advisor real or AI who sometimes responds to your requests with an angry "I'm busy, get out of my office!", sometimes gives you terrible advice by accident, sometimes gives you poor advice because it is funny, and yet sometimes gives amazing advice wouldn't be considered very good, or even desirable depending on relative frequencies and order of results. I have never responded to a student's request for career advice with "End thyself", but I bet it wouldn't go over well.
2: Systemizers like things that are systematic and predictable, but empathizers also like things that are empathetic. LLM AIs are certainly not, although they can sometimes fake it well for a short period. I am not optimistic about how well that works long term... see the terry cloth monkey mom experiment referenced here: https://kenazfilan.substack.com/p/from-the-mouth-of-madness-2-the-terrycloth
One reason we have what Yuval Levin would call "formative institutions" is to channel human variability into more predictable forms and make human unpredictability less scary. I can know when interacting with a formative institution that-- for better and often for worse-- the people in that institution will feel an obligation to "play their role" and do the predictable thing.
As I understand it, if you know what role you want an LLM to play, you can often get it to fulfill the predictable obligations of that role by prompting it to "think of itself" as playing that role. Ethan Mollick has a bunch of examples of this in his posts on good prompt engineering. It may be that some of the scariness of LLM unpredictability comes from the fact that we don't really know either (a) what those roles are that we really want LLMs to play or (b) how to articulate those roles clearly. Levin would probably say that (b) is a symptom of the decline of formative institutions generally.
"If your bank’s software mostly works, but sometimes scolds you for wanting to withdraw money, sometimes steals your money and lies to you about it,...you would not be very happy"
My bank's software already has, and I'm already not very happy about it.
And whether it's new complexity introduced by the use of LLM or the software has reached levels of complexity that their IT team can't manage before even using LLMs, I don't know. But I do know it is becoming more common for them not to be able to give me an answer about what's the reason for the failure every time something fails (and I've got a good sample, I'm not joking when I say I'm not happy). The moment they can start blaming "AI" for blocking any operation for reasons of "safety" they will do it, because it will simply make life so much easier in dealing with customer support.
I still have not got Chat GPT to tell a decent joke even when I gave it a good punch line.