KAIST researchers have developed a safety verification technology that uncovers roughly seven times more hidden vulnerabilities in AI than existing methods. The technology is expected to serve as a foundation for developing safer, more trustworthy AI.
KAIST (President Choongsik Bae) announced on the 30th of July that a research team led by Professor Junmo Kim from the School of Electrical Engineering has developed a new framework called Stable-GFlowNet (S-GFN), which overcomes the limitations of red-teaming—a safety verification process that deliberately attacks large language models (LLMs) to expose hidden weaknesses.
Red-teaming for generative AI is the process of crafting attack prompts designed to probe an AI's vulnerabilities and induce the AI to produce harmful or dangerous responses before the program is deployed. Since discovering a wider variety of attack methods allows more vulnerabilities to be addressed in advance, both the success rate and diversity of attacks are critical .
Previous approaches primarily relied on reinforcement learning—an AI technique trained to maximize reward—to generate attack prompts. However, these methods frequently suffered from mode collapse—a phenomenon where the model repeatedly converges on a narrow set of high-reward attack prompts rather than generating diverse outputs, thereby limiting its ability to uncover various vulnerabilities.
Generative Flow Networks (GFlowNets)—an AI generation technique trained to produce diverse outputs in proportion to their reward—were proposed as a solution. Yet GFlowNet training is computationally complex and unstable, and noisy reward signals can assign high rewards even to meaningless sentences, often causing training to collapse.
To address these issues, the research team developed three core techniques that help the model learn effective attacks more reliably while filtering out flawed ones.
First, much like comparing several paths to choose the best one, the team introduced Contrastive Trajectory Balance (CTB), which reduces computational complexity and stabilizes training by directly comparing pairs of generated attack trajectories.
Second, akin to filtering out background noise to focus on a single voice, the team applied Noise Gradient Pruning (NGP) to eliminate minor reward fluctuations and ensure the model learns exclusively from meaningful signals.
Third, the team applied the Min-K Fluency Stabilizer (MKS), which guides the model to generate attack prompts resembling text that a real user would write—just as a human reader naturally prefers coherent sentences to gibberish.
As a result, Stable-GFlowNet discovered 134 unique attack types—about seven times more than the 17 unique attack types found by the existing GFlowNet-based technique—while maintaining a high attack success rate of 92%.
Defense models trained using attacks generated by Stable-GFlowNet also demonstrated strong generalization, effectively defending against a wide range of attacks in cross-attack tests, which evaluate performance using attack techniques different from those used during training.
The team further demonstrated that CTB and NGP achieve faster and more stable performance than existing methods—not only in AI safety verification, but also in other distribution-matching tasks such as molecular generation for drug discovery.
Professor Kim said, "This technology is significant in that it can reliably uncover a wide range of AI vulnerabilities even in realistic conditions with limited data and high noise." He added, "Because it allows a broader range of risk factors to be identified and defended against before generative AI is deployed in real-world services, we expect it to become a core foundational technology for developing safer, more trustworthy AI."
The study was led by first author Minchan Kwon, a Ph.D. candidate from the School of Electrical Engineering, and was selected as a Spotlight paper—placing it in the top 2.2% of submissions—at the International Conference on Machine Learning (ICML) 2026, one of the world's most prestigious AI conferences.
※ Paper title: "Stable-GFlowNet: Toward Diverse and Robust LLM Red-Teaming via Contrastive Trajectory Balance,"
arXiv: https://arxiv.org/abs/2605.00553
This research was supported by the Institute of Information & Communications Technology Planning & Evaluation’s (IITP) SW Star Lab program, funded by the Ministry of Science and ICT.
Stable-GFlowNet: Toward Diverse and Robust LLM Red-Teaming via Contrastive Trajectory Balance
6-Jul-2026