Utah is permitting an AI system to prescribe psychiatric medicine with no physician. It’s solely the second time the state — and the nation — has delegated this sort of scientific authority to AI. State officers say it might carry prices down and ease care shortages, however physicians warn the system is opaque, dangerous, and unlikely to broaden psychological well being care to those that want it.
The one-year pilot, introduced final week, will permit Legion Well being’s AI chatbot to resume sure prescriptions for psychiatric medicines, in some instances. The San Francisco startup guarantees Utah-based sufferers “quick, easy refills” by a $19-a-month subscription. This system begins sooner or later in April, although the corporate is just working a waitlist in the mean time.
The AI chatbot will renew sure prescriptions for psychiatric medicines, in some instances.
This system is intentionally slim in scope, restricted each when it comes to the medicines it covers and the circumstances sufferers should meet to qualify. In line with Legion’s settlement with Utah’s Workplace of Synthetic Intelligence Coverage, the chatbot can renew solely 15 lower-risk upkeep medicines which have already been prescribed by a clinician. That features fluoxetine (Prozac), sertraline (Zoloft), bupropion (Wellbutrin), mirtazapine, and hydroxyzine, generally used to deal with anxiousness and melancholy. Sufferers should even be thought of steady: Anybody with a current dose or treatment change or a psychiatric hospitalization within the final yr is excluded, and sufferers should examine in with a healthcare supplier each 10 refills or after six months, whichever comes first.
The system can not problem new prescriptions or deal with medicines that require nearer scientific oversight, together with medicine that want blood-test monitoring. Managed substances are additionally barred, ruling out many ADHD medicines. The exclusion of benzodiazepines, used for anxiousness; antipsychotics, used for circumstances like schizophrenia and bipolar dysfunction; and lithium — extensively thought of the gold-standard remedy for bipolar dysfunction — leaves many extra advanced psychiatric instances exterior the pilot’s scope.
To make use of the system, sufferers should opt-in, confirm their identification, and show they have already got a prescription, reminiscent of by offering a photograph of the label or capsule bottle. They’re then requested about their signs, in addition to negative effects and efficacy of the treatment. They’re requested questions on suicidal ideas, self-harm, extreme reactions, and being pregnant to be able to log pink flags. If any solutions fall exterior of the pilot’s low-risk standards, the instances are speculated to be escalated to a clinician earlier than any refill is issued. Sufferers and pharmacists also can request human overview.
“By safely automating the renewal course of for upkeep medicines, we’re permitting sufferers to get the care they want rather more shortly and affordably,” state officers stated when asserting the pilot. Over time, they stated, this system might free healthcare suppliers to “focus their time on extra advanced, higher-risk affected person wants” and assist tackle shortages which have left 500,000 Utah residents with out entry to psychological well being care. Legion cofounder and CEO Yash Patel has solid this system in even grander phrases, describing it as a world first that can dramatically broaden entry to healthcare and mark “the start of one thing a lot larger than refills.”
Psychiatrists are much less satisfied. Brent Kious, a psychiatrist and professor on the College of Utah Faculty of Medication, advised The Verge he thinks the “benefits of an AI-based refill system could also be overstated.” He suspects the software “won’t enhance entry for many who are most in want of care.” The goal affected person would already must be on a remedy plan with their psychiatrist to make use of the service.
“It will be higher if there have been higher transparency, extra science, and extra rigorous testing earlier than individuals are requested to make use of this.”
Kious suggests the automation might contribute to what he referred to as an “epidemic of over-treatment” in psychiatry, with some sufferers staying on treatment longer than they should. John Torous, director of digital psychiatry at Beth Israel Deaconess Medical Middle and professor of psychiatry at Harvard Medical Faculty, raised a associated concern, noting that some folks profit from staying on psychiatric medicines long-term, whereas others might profit from lowering or stopping them. “They require extra lively administration, adjustments, and cautious consideration,” he stated. That’s more durable to do should you’re outsourcing refill check-ins to a chatbot.
An even bigger fear is whether or not a chatbot can safely automate even probably the most routine components of psychiatric care. Torous stated prescribing includes extra than simply checking for drug interactions, and questioned whether or not any AI system in the present day “can perceive the distinctive context and elements that go into an individual’s treatment plan.” Kious made an identical level: “That is one thing that might be protected in precept, nevertheless it all will depend on the small print.” These considerations are compounded by how new these methods are — and the way opaque they continue to be to outsiders. “It feels a bit like alchemy proper now,” he stated. “It will be higher if there have been higher transparency, extra science, and extra rigorous testing earlier than individuals are requested to make use of this.”
There are extra fast security considerations, too. Kious stated the chatbot might miss one thing throughout screening: It could not ask the appropriate questions, a affected person might not acknowledge a facet impact, or they could reply inaccurately. Some might merely inform the system what it needs to listen to to be able to velocity up care. He burdened that this isn’t distinctive to chatbots; a lot of psychiatry depends on self-report. However human clinicians normally have entry to different info as nicely, he stated, including that when he sees sufferers, he pays consideration not simply to what they are saying, but additionally to what they don’t say and the way they current themselves. And whereas sufferers also can mislead human suppliers, Kious stated a chatbot system might make it simpler for sufferers to regulate their solutions till they produce the specified final result.
Torous stated there are extra overt security dangers as nicely, which might be acquainted to anybody following how chatbots fare in the true world. Legion’s chatbot is Utah’s second experiment with AI prescribing, becoming a member of an ongoing, broader pilot centered on main care with Doctronic that launched final December. Inside weeks of going stay, safety researchers had managed to push Doctronic’s system into spreading vaccine conspiracy theories, producing directions for cooking meth, and tripling a affected person’s opioid dosage. State officers say the extra centered program with Legion is designed particularly to focus on “the state’s psychological well being scarcity.”
Legion says the pilot is working beneath tight guardrails. Along with what it calls “conservative eligibility gates,” its settlement with Utah requires it to offer detailed month-to-month stories and have the primary 1,250 requests intently reviewed by human physicians, with periodic sampling of round 5 to 10 p.c of requests thereafter.
Legion cofounder and president Arthur MacWaters advised The Verge that “dangers exist in any distant care mannequin, whether or not AI-assisted or totally human-led” and burdened the corporate’s “workflow doesn’t depend on a single self-reported reply to unlock remedy.” He stated key safeguards embrace the pilot’s slim limits on medicines and affected person eligibility, built-in AI security screens, pharmacist involvement, and the power to escalate to a clinician. “We see this as essential to broaden entry to tons of of 1000’s of individuals in Utah who stay in psychological well being scarcity areas, in addition to an vital proving floor for AI in medication.”
MacWaters wouldn’t touch upon further use instances, medicines, or expansions to different states, however stated the agency is “excited for what the longer term holds.” He wouldn’t provide a timeline on Legion’s growth plans both, although each MacWaters and Legion have signalled broader ambitions past Utah publicly: Legion’s refill web site says the service might be accessible “nationwide 2026” and MacWaters has urged it “might be in each state very in a short time.”
For the psychiatrists I spoke to, all of it appears to boost a somewhat fundamental query: What drawback is Legion actually fixing? Established sufferers usually don’t even want an appointment to get a refill, Kious stated, explaining that almost all psychiatrists are in all probability “joyful to refill prescriptions without cost and with out an appointment” except they’re apprehensive in regards to the affected person or the treatment carries a significant threat. These are the very instances Legion’s AI is barred from dealing with.
“I’d personally keep away from it for now,” Torous stated, including that should you’ve discovered an excellent remedy plan that works for you, it’s in all probability finest to stay with that clinician.
