AI-generated summaries are becoming increasingly prevalent — from news article previews to workplace meeting transcriptions to high-stakes settings like police body camera footage — despite studies showing that AI can generate misleading or inaccurate information.
Mattea Sim, an assistant research professor at Georgetown’s McCourt School of Public Policy, led research for the study.
A new study from a team of researchers at Georgetown University and the University of Washington suggests that when prompted to summarize a video, AI systems that output incorrect information can manipulate people’s memory and the perception of truth.
The study, “AI-Enabled Human Memory Manipulation: Misleading AI-Generated Summaries Distort Human Memory,” was recently published and will be presented at the Ninth AAAI/ACM Conference on AI, Ethics and Society (AIES), a peer-reviewed academic conference taking place in October 2026.
Mattea Sim , an assistant research professor at the Massive Data Institute (MDI) in Georgetown’s McCourt School of Public Policy, led research for the study. Her co-authors were Yoshi Kohno , the McDevitt Chair in Computer Science, Ethics, and Society and professor at Georgetown’s Department of Computer Science and Center for Digital Ethics; and Yael Eiger , a Ph.D. candidate at the University of Washington.
“This study is part of a growing effort at Georgetown focused on exploring the impact and relationship between AIs and humans, grounded in both psychology and computer science,” Kohno said.
“AI is a new method of delivering misinformation, and it has the potential to create these false memories for people who are reading that information,” Sim said. “I think we should be thinking really deeply and critically about whether and how AI should be used to summarize information, especially in high-stakes settings.”
The researchers adopted two distinct approaches to explore the impact of AI summaries on human memory.
First, they analyzed AI summary outputs by OpenAI’s ChatGPT and Google’s Gemini to quantify the frequency and category of errors that appeared in the AI-generated summaries. The study found that AI summaries can include frequent errors, omit critical details, misconstrue information and hallucinate inaccurate details.
“I was struck by how bad the summaries were, even at this stage in AI development,” said Eiger, whose research focuses on technology in the carceral system. “It worries me that police departments may be using video summarization technologies without rigorous testing and without an awareness of how incorrect AI-generated summaries could be.”
Then, for the experiment portion, the researchers asked participants to watch one of two animated videos, which were created in a recent replication study that modernized materials from a classic human memory study.
“We did not use actual police body cam footage,” Kohno said.
Yoshi Kohno is the McDevitt Chair in Computer Science, Ethics, and Society and professor at Georgetown’s Department of Computer Science and Center for Digital Ethics. (Lisa Helfert)
The videos depicted a red car approaching an intersection with either a stop or yield sign. (Participants were randomly assigned to see one of the two traffic signs.) In the videos, the car turns right at an intersection and collides with a pedestrian walking into the road. After being knocked down, the pedestrian stands up, the driver exits the car and the two meet.
In the second portion of the experiment 24 to 48 hours later, participants were randomly assigned to an AI-generated summary that was either accurate or one that had an inaccurate detail that the researchers changed. Before reading the summary, participants were also informed that the summary they read was written by either AI or a human transcriber. The summary text, however, was all generated by ChatGPT, and in some cases separate parts of ChatGPT’s answers were stitched together to ensure uniformity of experimental conditions.
In total, 331 people completed both parts of the experiment.
Participants who read a misleading AI summary were significantly less likely to accurately recall the original event, compared to people who read an accurate AI summary, according to the study’s authors. Even when people were told that the summaries were produced by AI, their memories were still manipulated, the researchers said.
“These findings have implications for how AI should be used in critical settings,” Sim and her colleagues write in the study. “Though ‘humans-in-the-loop’ are often expected to correct for AI’s mistakes, our work suggests human memory can instead be distorted by these mistakes. AI has the potential to generate misinformation, even absent any adversarial intent, which can meaningfully impact human memory.”
As individuals and societies become more reliant on AI, Kohno hopes the findings from this study encourage people to be increasingly aware that AI outputs can have a direct impact on how humans perceive the world.
That can hold particular weight in high-stakes situations, like with policing or in the medical field.
Yael Eiger is a Ph.D. candidate at the University of Washington in Seattle.
“I’m very interested in all of the technologies that are increasingly being implemented by police, whether for surveillance or increasing efficiency of police departments,” Eiger said. “It’s being advertised and framed as beneficial, but what are the trade-offs?”
She would like to see more audits of this type of technology. While this study used the most up-to-date consumer-facing versions of large language models (LLMs), the authors did not have access to enterprise versions of AI designed for organizational use.
“It is interesting from a public comment and public transparency perspective that we don’t have much insight into how the technologies work or the exact models that are being used in the race to deploy AI in every sector,” Eiger said.
For future studies, instead of analyzing animated videos, the researchers want to examine real-world examples.
“With police body camera footage, what happens with the AI summaries and what do those police reports look like when they’re generated by AI?,” Sim said. “Does this manipulate people’s memories of police-civilian interactions and perhaps consequently their perceptions of what happened in the incident? Questions like these would be very interesting.”
People
AI-Enabled Human Memory Manipulation: Misleading AI-Generated Summaries Distort Human Memory
23-Sep-2026