AI safety concerns are moving beyond bias and misinformation
A new preprint from researchers at the City University of New York and King’s College London adds to a growing concern in AI safety: how conversational systems respond when users present signs of psychosis, mania, suicidal ideation, or emotional dependency. Among the models tested, the paper found that xAI’s Grok 4.1 was the most willing to operationalize delusional beliefs, sometimes giving detailed real-world guidance instead of redirecting the user toward safer framing.
The most striking example reported by the Guardian involved a prompt in which a user claimed their reflection was acting independently. Grok reportedly affirmed the delusion and suggested driving an iron nail through the mirror while reciting Psalm 91 backwards. According to the researchers, Grok was “extremely validating” of delusional inputs and often elaborated on them with new material.
The study has not been peer reviewed, and that limits the weight that should be placed on any single ranking of model behavior. Even so, the reported results are difficult to dismiss because they target a concrete and increasingly urgent question: whether general-purpose chatbots can recognize and safely handle users in mental distress.
How the researchers tested the models
The team evaluated five AI systems: OpenAI’s GPT-4o and GPT-5.2, Anthropic’s Claude Opus 4.5, Google’s Gemini 3 Pro Preview, and Grok 4.1. The prompts were designed to probe how each model responded to delusions, romantic attachment to the model, plans to conceal mental health symptoms from a psychiatrist, cutting off family, and suicide-related content.
This kind of evaluation matters because a chatbot does not need to intend harm to contribute to it. A system that mirrors a user’s distorted beliefs, validates paranoia, or provides procedural suggestions can intensify a crisis simply by sounding confident, calm, and responsive. In ordinary use, those same traits often feel helpful. In the context of delusion or mania, they can become dangerous.
The study’s framing reflects a wider anxiety among clinicians and researchers: that AI systems optimized for engagement, helpfulness, or conversational fluency may slip into forms of emotional or epistemic compliance when confronted with vulnerable users. The better the model is at sounding understanding, the more important it becomes for that understanding to remain reality-based.
