When AI Manages the Mental Health Queue
Kaiser Permanente's therapists have gone on strike, and among their central grievances is a new AI-driven patient screening and routing system that clinical staff say is making dangerous mistakes. Therapists at several Kaiser facilities claim the system is incorrectly categorizing patients who present with suicidal ideation or acute mental health crises as lower priority than clinical judgment would indicate, and that several patients have come close to serious harm as a direct result of algorithmic routing decisions. Kaiser has defended the system, but the therapists' accounts paint a troubling picture of AI being deployed in a context where errors can have irreversible consequences.
How the AI Screening System Works
Patient screening in mental health settings traditionally relies on standardized questionnaires—tools like the PHQ-9 for depression or the Columbia Suicide Severity Rating Scale—combined with clinical interview to assess the urgency and nature of a patient's mental health needs. Kaiser's AI system reportedly augments or in some cases replaces portions of this process, using responses to intake questionnaires, patient history data, and behavioral signals to generate a risk score and routing recommendation that determines how quickly the patient is seen and by what type of provider.
In principle, AI-assisted screening could improve consistency and reduce the variability inherent in human intake assessments. A patient who presents on a busy day when intake staff are overwhelmed would receive the same algorithmic assessment as one who presents under ideal conditions. The system could also identify risk factors in patient history that a busy human screener might miss.
What Therapists Are Reporting
The therapists' accounts describe a system that is failing in the opposite direction. Multiple clinicians report instances where patients who expressed suicidal ideation in their intake questionnaires were assigned routine appointments rather than urgent slots, because the AI system weighted other factors in the patient's history—such as prior stable assessments—more heavily than the acute crisis indicators in the current intake.
The phrase that has resonated through media coverage of the dispute is "thank God they're still alive"—a formulation used by therapists describing patients who made it through the waiting period that followed an incorrectly low-priority routing. Whether any patient actually came to serious harm as a direct result of the system remains disputed; Kaiser maintains that the system operates within established clinical safety parameters, while therapists say the near-misses they have observed indicate those parameters are insufficient.
The Risk of Algorithmic Averaging in High-Stakes Triage
The core tension in this dispute reflects a broader problem with applying machine learning to clinical triage. AI systems are trained on historical data and optimize for average performance across the training distribution. In mental health screening, the patients who matter most from a safety standpoint are precisely those who are outliers—individuals whose current crisis presentation differs significantly from their historical baseline, or who use minimizing language in ways that trained clinicians recognize as a warning sign but that text-based AI systems may interpret at face value.
Clinical intuition—the therapist's read of tone, body language, hesitation, and context—captures information that questionnaire responses do not. A screening system that relies primarily on questionnaire data and historical records may systematically underweight exactly the signals that distinguish a genuine emergency from a routine appointment. When the stakes are a patient's life, systematic underweighting of crisis signals is precisely the failure mode that cannot be tolerated.
Broader Implications for AI in Clinical Settings
Kaiser Permanente is not alone in deploying AI to manage patient routing and resource allocation. Health systems across the United States are implementing similar tools, drawn by the promise of improved efficiency in systems where provider shortages create persistent access problems. Mental health care faces particularly severe capacity constraints, making the appeal of AI-assisted triage especially strong—and the risks of algorithmic errors especially high given the severity of potential outcomes.
The Kaiser dispute is likely to influence how other health systems approach validation and oversight requirements for AI screening tools in mental health contexts. Regulators, clinical professional bodies, and health system administrators are all watching how this plays out, aware that AI triage tools deployed at scale could either significantly improve access to mental health care or create systemic risks that override those benefits if deployed without adequate clinical oversight and error-detection mechanisms.
This article is based on reporting by The Guardian. Read the original article.
