As people have begun to rely on artificial intelligence for assistance with complex or tedious tasks, they have found the technology to be…unreliable. Simple AI queries can return inaccurate information, data filled with mistakes, or reasoning that’s been manipulated by other users.
But UCF researcher Amrit Singh Bedi aims to make AI safer and more dependable through a new project supported by the Defense Advanced Research Projects Agency (DARPA). With a one-year, $340,000 grant, Bedi and his team of researchers will study AI systems and determine how to train them to withstand failures, attacks and unexpected situations.
“The goal of this project is to study how AI systems can become more resilient by learning to recognize weaknesses in their own reasoning and improve their behavior over time,” Bedi says. “At a high level, the project explores the idea of self-improving AI for safety – AI systems that can adapt and strengthen their own decision-making processes rather than relying solely on static safety rules or external corrections.”
Bedi, an assistant professor in the Department of Computer Science, says that while generative AI systems are becoming more capable, they remain vulnerable to manipulation through carefully crafted prompts, unexpected inputs, or adversarial conditions. These exploits can lead to inaccurate data, which can negatively affect the end user.
“The gap between apparent capability and true robustness is especially concerning in mission-critical settings, where even a small deviation can have serious consequences,” Bedi says. “We were motivated by the need to move beyond surface-level alignment and develop AI systems that can actively recognize, withstand, and recover from adversarial pressure while staying faithful to mission goals.
Bedi and his team in the SafeRR AI Lab plan to combat this issue by exploring the concept of “self-hardening,” which means that AI systems would be able to detect when they’re being manipulated with misleading instructions, tricky wording, or hostile inputs. Instead of veering off course, AI systems would respond to manipulation by strengthening their own responses.
The findings could help improve the reliability of AI systems that are used in defense operations, autonomous systems, and robotics.
“If AI is used to support defense, healthcare, emergency response or cybersecurity, it needs to keep working correctly even when the situation is confusing or adversarial,” Bedi says. “Our research helps build trust in AI systems for these important applications.”