CHAMPAIGN, Ill. — A first-of-its-kind pilot program in Utah developed by a health-technology startup company uses artificial intelligence to automatically renew certain prescriptions for patients with chronic conditions such as hypertension and diabetes. But according to a new paper co-written by a University of Illinois Urbana-Champaign expert in legal issues surrounding cutting-edge medical technology, autonomous AI-based drug prescribing raises important clinical and legal issues.
With AI rapidly being deployed in health care, a “move fast and break things” mentality has permeated the industry, raising complex questions about the role of the U.S. Food and Drug Administration in regulating these technologies, says Sara Gerke , the Richard W. & Marie L. Corman Scholar at the College of Law .
“The big issue here is the relationship between state and federal law, which we’re seeing in Utah, where the state has become a testing ground for so-called AI sandboxes that companies can use to experiment,” said Gerke, also a professor at the European Union Center at Illinois. “The idea behind it is to test these AI medical tools before they are completely developed. But even that is legally questionable. The existence of mitigation agreements and waivers under state law doesn’t automatically render the use of unproven medical technology lawful at the federal level.”
The paper, published by the New England Journal of Medicine, was co-written by I. Glenn Cohen of Harvard Law School and Ravi B. Parikh of Emory University.
The AI-based drug prescribing tool was launched in January on a 12-month pilot program in Utah under a “regulatory sandbox” framework established under the 2024 Artificial Intelligence Policy Act, which established an AI “learning laboratory” that provides temporary waivers or allows temporary modifications to specific state laws.
Under an agreement between the Utah Office of Artificial Intelligence Policy and a startup company, after a physician oversees the first 250 AI-based prescription renewals, the AI system operates autonomously, without human oversight of individual renewal decisions for prescriptions, according to the paper.
Although the program is confined to prescription renewals for 192 commonly prescribed drugs such as antidepressants and proton pump inhibitors, it could pave the way for expanded initiatives that include autonomous initial prescribing, the researchers warn.
“We believe that the program raises incredibly important clinical and legal issues,” Gerke said.
One of the stated purposes is to address nonadherence to medication — people who, for example, forget to renew the prescription for the medication that controls their blood cholesterol or diabetes. But health literacy, regimen complexity, cost barriers, medication side effects and psychosocial factors all affect adherence and cannot be addressed by automated prescription renewal alone, the authors said.
“Automated medication-refill programs may meaningfully increase patient adherence in certain cases,” Gerke said. “But people’s health changes all the time, and clinicians need to be in the loop. Also, it would be dangerous for medications that required individualized doses to be included in such a system.”
Another reason for caution: the prescribing tool itself does not appear to have undergone FDA premarket review, Gerke said.
“There are two legal issues: ‘Is this a medical device?’ and ‘Is an AI tool even allowed to prescribe drugs under the Federal Food, Drug, and Cosmetic Act?’ If it’s considered a medical device, then it would typically be subject to FDA premarket review, and the agency would need to authorize the tool before it is brought to market, even if it’s in a pilot stage,” Gerke said.
And in order to legally refill a prescription, you likely need to have an actual human clinician involved in the decision, Gerke said.
“If you look at the statute for refilling a prescription, it clearly says that the refilling must be authorized by a ‘prescriber’ — in other words, according to the statute, a ‘practitioner licensed by law,’” she said. “The plain text interpretation of that is it needs to be a licensed human physician, not an AI tool. There is a proposed bill in Congress, the Healthy Technology Act of 2025, that actually wants to make this change, amending the Federal Food, Drug, and Cosmetic Act to expand the scope of prescribing to allow AI and machine-learning technology to potentially prescribe medicine to patients under certain conditions.
“But right now, what’s happening in Utah might be a violation of the law that the FDA has so far taken no enforcement action on.”
While states can establish certain rules, the FDA is the ultimate arbiter on whether this experiment in autonomous AI prescribing continues, Gerke said.
“It all comes down to the FDA,” she said. “If the FDA interprets the language in the Federal Food, Drug, and Cosmetic Act in favor of Utah’s initiative, the agency might choose to look the other way and permit this experiment to continue, which could leave clinicians and patients without safety guarantees during an unprecedented expansion of AI autonomy in medicine.”
New England Journal of Medicine
Commentary/editorial
Not applicable
Utah’s Prescription-Renewal Pilot Program — Autonomous AI Managing Patient Care
18-Apr-2026