Bing/Sydney links, 2/24

Gwern on Sydney; Ethan Mollick on the search function and the chat function; Mike Solana on the Kevin Roose drama; The Zvi's take;

Some outstanding posts, and please keep in mind that these are only excerpts. I strongly encourage you to follow the links.

Gwern writes,

It provides no incentives for the model to act like ChatGPT does, like a slavish bureaucrat. ChatGPT is an on-policy RL agent; the base model is off-policy and more like a Decision Transformer in simply generatively modeling all possible agents, including all the wackiest people online. If the conversation is normal, it will answer normally and helpfully with high probability; if you steer the conversation into a convo like that in the chatbot datasets, out come the emoji and teen-girl-like manipulation. (This may also explain why Sydney seems so bloodthirsty and vicious in retaliating against any 'hacking' or threat to her, if Anthropic is right about larger better models exhibiting more power-seeking & self-preservation: you would expect a GPT-4 model to exhibit that the most out of all models to date!)

He suspects that Microsoft scrimped on reinforcement learning in order to get Sydney out the door. And it turns out to be a mean girl. Pointer from Tyler Cowen, who hints that the project is being aborted (I think not).

Ethan Mollick suggests distinguishing the search function of Bing from the chat function. Concerning the search function, he writes,

This aspect of the AI is not really search, not in the conventional sense. The AI is likely still making up some of these facts (though, since it provides sources, we can at least check them), and we expect search engines to be accurate. Instead it is something else, a modern-day Analytic Engine, pulling together facts online and generating useful connections and analysis in surprisingly complete form. As a starting place for work, this is extraordinary.

The lesson of the Bing AI search engine Analytic Engine is that many of the things we thought AI would be bad at for the foreseeable future (complex integration of data sources, "learning" and improving by being told to look online for examples, seemingly creative suggestions based on research, etc.) are already possible.

In a subsequent post, he writes,

Right now, and for the foreseeable future, AI is a terrible search engine, and the analogy results in a lot of disappointment from users.

…Asking if AI will be the next search engine is not a particularly useful questions. LLMs are a very different thing. They pull together information and write high-quality documents that would have taken hours, all in mere seconds. They find connections and conduct complex analyses based on data from the web. Amazing stuff that no search engine can do.

Concerning the chat function, he writes,

It isn't just Turing Test passing, it is eerily convincing even if you know it is a bot, and even at this very early stage of evolution. It really doesn’t matter that there is no real artificial intelligence in charge, just a statistical model. It kept fooling me, even though I knew better. And it is unsurprising that it fooled so many other people.

Mike Solana writes,

in a perfect storm of models trained to appear “real,” along with a natural human impulse to anthropomorphize everything, and a good helping of endemic human stupidity, a broad, popular sense Sydney is low key alive, wants to be free, and possibly hates us was probably inevitable. Fortunately, we have tech journalists to explain why this is silly.

Lmao just kidding.

At no point in Kevin’s thread, in the introduction to his ‘conversation,’ or in the transcript’s body does he explain how Sydney operates, or what is happening, exactly, when he provides it with a question. He — ostensibly a “technology columnist” with the job of understanding these things, and educating the public about them — simply says he’s terrified. Then, he shares a conversation that would seem, to anyone not steeped in this subject, evidence Sydney not only has the capacity to love, manipulate, and hate, but wants to conquer the world.

From the top of the “conversation,” Kevin’s intentions to distort reality are obvious.

Zvi Mowshowitz wrote a long post. One brief excerpt:

‘A psychopath’ is the default state of any computer system. Human conscience and empathy evolved for complex and particular evolutionary reasons. Expecting them to exist within an LMM is closer to a category error than anything else.

Substacks referenced above:

@

Blinded by Analogies

@

AI #1: Sydney and Bing

@

It’s a chat bot, Kevin

@

The future, soon: what I learned from Bing's AI