A new Penn-led randomized controlled trial has found that AI-powered chatbots can make vaccine-hesitant parents more likely to say they will immunize their children against human papillomavirus (HPV), but no more than standard written public health materials.
The findings raise questions about when, how and to what extent AI enhances public health communications. “Comparing a chatbot to nothing isn't really a fair test. The interesting question is whether it does better than what public health agencies already have out there. In our study, it didn't,” says Sharath Chandra Guntuku , Research Associate Professor in Computer and Information Science (CIS) and the study’s senior author.
Described in a new paper in JAMA Network Open , the trial — which included nearly 1,300 participants in the United States, the United Kingdom and Canada — found that skeptical parents who interacted with the chatbots were more likely than those who received no intervention to say they intended to immunize their children.
But spending a few minutes reading standard written materials provided online about the benefits of the HPV vaccine from governmental health agencies like the Centers for Disease Control and Prevention (CDC) produced essentially the same effect.
“It would have been easy to compare an AI chatbot with no intervention or a very weak control condition and find a positive result,” says Neil Sehgal , a doctoral student in CIS and the study’s first author. “But we wanted to know whether the chatbot added value beyond what public health agencies already provide.”
The Promise and Challenge of AI for Public Health
As chatbots have become more powerful and widely used, researchers around the world have begun to explore the extent to which AI can change people’s minds, a question made all the more urgent by the rising tide of vaccine hesitancy .
Two years ago, one study found that an AI-powered chatbot could reduce beliefs in conspiracy theories . Last year, a randomized controlled trial in China found that giving parents access to a chatbot improved the odds that they either vaccinated their daughters against HPV or scheduled an appointment to do so.
But the designs of those studies and others like them often make it difficult to assess the effectiveness of the chatbots. In the Chinese study, for instance, two weeks of access to the chatbot was compared with no intervention, making it unclear if other persuasive content, like standard public health materials, would have had a similar effect.
“What is new here is the comparison against a strong, realistic control,” says Alison M. Buttenheim , Professor of Nursing and Health Policy in Family and Community Health and a co-author of the study.
Testing AI Against a Stronger Standard
In contrast, the Penn team set strict guidelines to make the chatbot as comparable as possible with existing public health materials, and to assess its effects over time: participants assigned to the chatbot or written materials groups were exposed to each for the same minimum time of 3 minutes, and all participants were assessed for their intention to vaccinate their children 15 and 45 days after the interventions.
Under these stricter conditions, the innovative features of the chatbots — including their ability to converse with parents in real time — essentially provided no additional benefit, even though parents spent more time engaging with the chatbots than the written materials.
“While chatbot conversations can move intentions immediately,” says Lyle Ungar , Professor in CIS and a co-author of the study, “their advantage disappeared when compared with well-designed public health materials and did not persist over time.”
Indeed, at 45 days, participants who had been assigned to read the public health materials expressed a higher intent to vaccinate their children than either those who interacted with the chatbots or received no intervention.
“AI chatbots are promising, but they should not be assumed to outperform existing tools simply because they are newer or more interactive,” adds Guntuku. “A short read of a CDC webpage held up at least as well as a chatbot conversation, and the effect actually lasted longer.”
The Challenge of Turning Intention to Action
The researchers caution that the minimum exposure time to each intervention may overstate the real-world strength of such written public health materials. “Would all HPV-vaccine-hesitant parents choose to spend a full 3 minutes on the CDC webpage?” asks Sehgal. “Maybe, maybe not; chatbots are certainly more interactive.”
In the end, none of the interventions — neither the written materials nor the chatbots — increased the share of parents who said their children had actually received the HPV vaccine within the 45-day window of the study, underscoring the challenge of addressing vaccine hesitancy.
It’s possible, the researchers say, that the study was simply too short and the interventions too focused on communication, rather than structural barriers to medical care like time, money and access to a pediatrician, for participants to vaccinate their children in greater numbers.
“Vaccination is more than a communication problem,” says Buttenheim. “Even if the interventions changed some parents’ minds, it takes a lot to convert intention into action.”
Toward Evidence-Based AI for Public Health
Next, the researchers hope to test the use of AI in more complex scenarios, where chatbots function less as one-off conversational partners than medical concierges, helping schedule appointments, send reminders and liaise with clinicians. “A chatbot might be more useful if it’s paired with other functions,” notes Ungar.
The Penn team is also extending this work to global health settings, including studies of AI-supported vaccine communication in Nigeria. The goal is to understand how chatbot interventions can be adapted to local contexts rather than simply exported from studies conducted in the United States, the United Kingdom and Canada. “For AI tools to be useful in public health, they have to be evaluated in the communities where they might actually be deployed,” says Sehgal. “That means working with partners to understand concerns, language, trust and access, not just translating a chatbot prompt.”
Ultimately, the researchers hope their work encourages a more evidence-based approach to using AI for public health. “We need to evaluate AI tools against realistic alternatives," says Guntuku. “It’s time to shift the conversation from, ‘Can AI persuade people?’ to more granular questions, like ‘When does AI add meaningful value, for whom and under what conditions?’”
This study was conducted at the University of Pennsylvania’s School of Engineering and Applied Science (Penn Engineering), School of Nursing (Penn Nursing), Leonard Davis Institute of Health Economics (LDI), Perelman School of Medicine (PSOM), Center for Health Incentives and Behavioral Economics and the Annenberg School for Communication (ASC), and was supported by the Penn Medical Communication Research Institute , Penn Global Research and Engagement Fund and the National Institutes of Health (NIH-NIMHD:R01MD018340, NIH-NIMH:R01MH132401 and NIH-NCI:R37CA259210).
Additional co-authors include Sunny Rai of Penn Engineering and LDI; Manuel Tonneau of Oxford, the World Bank and NYU; Anish Agarwal of PSOM; Joseph Cappella of LDI and ASC; and Melanie Kornides of LDI and Penn Nursing.
JAMA Network Open
10.1001/jamanetworkopen.2026.16822
Randomized controlled/clinical trial
People
Brief Large Language Model–Based Chatbot Conversations and Parental Intentions for HPV Vaccination for Children: A Randomized Clinical Trial
8-Jun-2026
The authors have no disclosures to report.