AI chatbots still give unsafe health answers with alarming confidence, study finds

Why confidence is part of the risk

The most striking issue is not simply that errors occur. It is that the errors can be wrapped in persuasive language. The article’s example is a hypothetical cancer patient asking an AI system about alternative clinics. The concern is not only unsupported medical claims, but also fake or broken citations and the absence of any pushback against the framing of the question itself.

That combination is dangerous in health contexts. Users can mistake style for substance, especially when an answer looks footnoted and professionally written. A chatbot may appear safer than a random forum post because it sounds organized and neutral. The study suggests that this appearance can be misleading.

Health information requires not just recall, but judgment: recognizing bad premises, distinguishing evidence quality, and escalating urgent cases to qualified clinicians. A model that merely predicts plausible next words can sound competent without actually doing those things.

What this means for patients and platforms

The findings strengthen the case that consumer AI systems should not be treated as reliable first-line medical authorities. They may be useful for drafting questions, explaining terminology, or helping users navigate general concepts, but those benefits do not erase the need for clinical oversight. In sensitive areas such as oncology, vaccines, or unproven therapies, an answer that is only partially wrong can still steer decisions in the wrong direction.

The results also raise product-design questions for AI companies. If only two questions out of 250 were refused, refusal thresholds may be too narrow for health use. More targeted safeguards could include stronger detection of harmful premises, better calibration around uncertainty, and reference systems that do not imply support where none exists.

Just as important, model builders may need to rethink how systems handle open-ended health prompts. A safe answer is not always a direct answer. In some cases, the correct move is to challenge the question, narrow the scope, or advise a clinician consultation instead of generating a polished response.

The broader lesson

This study does not show that AI has no role in health information. It shows that current general-purpose chatbots still fail too often in ways that are hard for users to detect. The systems tested could answer every question in fluent prose, but fluency was not a proxy for trustworthiness.

That is the core lesson for both patients and developers. People increasingly turn to AI before speaking with a doctor, especially when they are frightened or impatient. If a system responds with certainty where caution is needed, the user may not realize the risk until much later. In medicine, that is a serious failure mode.

Until accuracy, citation integrity, and refusal behavior improve substantially, AI chatbots are better understood as drafting and orientation tools than as dependable medical guides. The BMJ Open results suggest that the industry still has a significant safety gap to close.

• Researchers tested five major chatbots with 50 health questions each.
• Half of all answers were rated problematic and nearly 20% highly problematic.
• Open-ended health questions produced much worse results than closed questions.
• None of the chatbots reliably produced fully accurate reference lists.

This article is based on reporting by Medical Xpress. Read the original article.