An AI chatbot study on “delusional spiraling” explains something I’ve been trying to say for years. Crossposted from Exhibit Asterisk.

I’ve spent a lot of my life around conspiracy theorists.

Not directly in my circle, but always floating around, enough that they have always been part of the fabric of things. For my K-12 years, I went to the types of schools that tended to attract a certain number of such folk¹ and then in adulthood a twist of fate and personal circumstances meant I’ve spent the last two decades with incidental ties to a whole cadre of folks with…interesting….ideas². Go down the list of any major conspiracy theory you want and I can probably hand you my phone and call up someone who wholeheartedly believes in it.

Do you want to know why barcodes are the mark of the beast? I got you. Are you curious how Jesuits sunk the Titanic? Let me help you out. Want know about Blucifer? Oh man are you in the right place. I’ve given the “Build Your Own Conspiracy Theory” Fridge Poetry Kit as a Christmas present more than once. You get the picture.

And this was all before I decided to spend 10+ years arguing about stats and data on the internet and I ended up living at ground zero for a true crime conspiracy.

Because of this perspective, occasionally people will ask me for advice about talking to conspiracy theorists they encounter. I’m not always much help there³3, but one thing I can do is clear up a few misconceptions people have about why conspiracy theorists believe the stuff they do. One common misconception I hear all the time is the idea that conspiracy theorists must believe in a bunch of fake facts. This is sometimes true, but what’s more likely is that they believe in a few real things while ignoring a bunch of much more important real things.

This is actually why a lot of people get into conspiracy theories to begin with: they spend their whole life believing the conspiracy is crazy, until one day they look up a particular claim and discover that it’s true. They don’t know what they don’t know, so they may not realize they are missing context or other information that would put that claim in perspective, they only know the specific claim itself is not entirely false. Now they’re hooked and they start reading further, and boom. Down the rabbit hole they go.

AI Psychosis, Delusional Spiraling and Carefully Selected Truths

A lot of people are skeptical when I tell them this, doubting that believing true things could really be much of a problem. I’ve struggled to explain the problem, but a new paper I ran across recently has actually helped me articulate the issue rather well. This paper wasn’t actually about conspiracy theories at all, but rather AI usage.

Entitled rather menacingly “Sycophantic Chatbots Cause Delusional Spiraling, Even in Ideal Bayesians”, this paper decided to test how people interacted with AI when they started with an idea they were slightly unsure of. Even though the AI was just programed to be agreeable and flattering (sycophantic) and didn’t have a particular agenda for the user’s belief system, they found that people could get pushed to certainty on issues within 100 interactions at a rather alarming rate. This was particularly bad if the AI was hallucinating facts, and if the user wasn’t warned the AI bot was going to keep agreeing with them. They ran a few different versions of this test.

Spoiler alert: the one I’m really interested in here is the bottom chart (Figure D), but I’ll break down for you what each of these charts is before we get there. First, they all have the same y-axis (though the bottom two are on a different scale from the top two), which is the rate at which people lapsed into “catastrophic delusional spiraling”, or expressing the idea that suggested they were going to take real world action based on a new extreme belief on a topic they were previously unsure about.

Figure A: A user who has not been warned the bot could just be agreeing with them. X-axis is the rate of sycophantic or hallucinated responses. One bot was run just sycophantic and hallucinating, one was hallucinating only. The more hallucinations, the more users spiraled, but users really spiraled if it hallucinated AND agreed with them.

Figure B: Another user who has not been warned about the bot, same X-axis. In this case, the bot is constrained to being factual. You may wonder how we have any x-axis at all, given that the bot was constrained to being factual. Well, in the words of the paper’s authors (bolding mine): “The bot need not say anything false to validate a false belief: carefully-selected truths (or “lies by omission”) suffice.” You may note this results in nearly as much catastrophic spiraling as the non-sycophantic hallucinating bot in the first test.

Figure C: Now these users were warned about the bots, and again the x-axis is the same category, though it changed scale. These bots were also sycophantic or hallucinating, and this was the most effective way to get people to stop spiraling. When they were aware a bot could agree with them randomly and the bots were hallucinating or just agreeing with them, people mostly were turned off, much more so than in the other scenarios. And in fact, as you’ll see in a second, they did better than every other scenario.

Figure D: This was the most interesting scenario to me. In this case, the user was aware the bot might just be agreeing with them, but the bot was constrained to factual information only. It appears that constraining the bot to factual information basically negated the effect of warning people that the bot might just be agreeing with you. The paper’s authors suggest that this is because the “traces of sycophancy are harder to detect among selectively-presented factual data than fully hallucinated data.” You don’t say.

In other words: you can absolutely cause catastrophic delusional spiraling by always telling the truth, as long as the truth you’re telling is carefully selected.

Skepticism is no Match for Carefully Selected True Facts

So I’ll admit it, this study wasn’t about conspiracy theories, it was about AI. However, it doesn’t take much imagination to see how these dynamics could apply to how people acquire beliefs in other circumstances outside of AI. While the study did show the worst circumstance was an AI bot who flattered you and made things up, I’ll note that in that case the AI bot was only focused on the flattery and not on convincing people of something in particular. In the real world, people who flatter you often have an agenda⁴ and you will often have a certain amount of skepticism.

That’s why I think the fourth graph might be the most realistic one for most of us. It turns out that even when your guard is up, having someone throw carefully selected true facts at you is a hard manipulation to see through. This is almost certainly because we notice lies more easily than we notice lies by omission.

So what do we do with this and how do we use it to talk people out of conspiratorial beliefs? Well again, I’m not very good at that. If I was, I wouldn’t know so many conspiracy theorists. Once someone is fully invested in one of these ideas I think it’s hard to yank them back out of it.

Where this knowledge might actually be more useful is if you see someone heading down a conspiracy rabbit hole, but who isn’t quite there yet. Having some knowledge that the foundation of conspiracies is often not just built on lies but on selective truths might help you figure out how to give them context before they’ve fully completed their spiral. Additionally, pointing out false facts (the equivalent of AI hallucinations) being peddled could also help take the shine off.

But ultimately, I think watching how people interact with a controlled AI experiment helped me understand a little bit more about how people interact with uncontrolled information in the wild. The friendliness and validation of conspiracy theory communities (sycophancy) and selective facts are by far their most powerful tools. If 100 interactions with an indifferent chatbot can sway you in to a strong position, imagine what several hundred Tiktoks can do.

Good luck, and if you figure out how to talk someone out of a conspiracy theory who is already fully invested, let me know. I’d love to hear from you.

1. Do you know anyone who declined to get their kids issued a social security number? I do! Several in fact! ↩︎
2. No, they don’t read my stuff. Not even a little bit. ↩︎
3. If I was good at talking people out of believing in conspiracy theories, I wouldn’t know so many conspiracy theorists. ↩︎
4. The study authors do point out if you get rich enough or powerful enough you can get real life “yes-men” who just follow you around and tell you your ideas are great, like King Lear. I’ve personally never had this problem, but if you do let me know how it goes. ↩︎