Researchers developed Found-RL, an open-source platform that integrates vision-language models into reinforcement learning for autonomous driving. By using an asynchronous batch inference framework, Found-RL allows large foundation models to provide semantic feedback without slowing down high-frequency simulation training. Experiments in CARLA show that lightweight reinforcement learning agents can achieve competitive driving performance while maintaining real-time policy inference.
The team published their study in Communications in Transportation Research ( https://doi.org/10.26599/COMMTR.2026.9640027 ).
Reinforcement learning has become a promising approach for end-to-end autonomous driving because it enables agents to learn driving policies through interaction with simulated environments. However, conventional reinforcement learning often requires extensive trial and error and may lack semantic understanding in complex traffic scenarios. Vision-language models, on the other hand, can interpret driving scenes and provide high-level reasoning, but their heavy inference cost makes them difficult to use directly in fast, multi-environment reinforcement learning loops.
“Large vision-language models can provide useful driving knowledge, but directly calling them at every simulation step would make reinforcement learning training extremely slow,” says Sikai Chen, corresponding author of the study and assistant professor at the University of Wisconsin–Madison. “Found-RL is designed to bridge this gap by allowing foundation models to guide reinforcement learning in the background, without blocking the simulation process.”
Turning foundation models into scalable driving mentors
The core of Found-RL is an asynchronous batch inference framework. Instead of waiting for a vision-language model to respond at every environment step, the platform separates simulation rollout from model inference. Multiple simulation workers send driving observations and context information to a shared request queue, while an inference server processes these requests in batches and returns feedback asynchronously.
This design allows the reinforcement learning agent to continue interacting with the driving environment while the vision-language model works as a background feedback provider. The returned feedback is then matched to the correct training sample and stored in the replay buffer for learning.
“Found-RL treats the vision-language model as a scalable mentor rather than a real-time controller,” Chen explains. “The lightweight reinforcement learning policy still performs high-frequency control, while the foundation model provides semantic supervision during training.”
Learning from both action guidance and semantic rewards
On top of the platform, the research team implemented two types of vision-language-model-based supervision. The first is action guidance, where the model provides expert-like driving actions that help the agent explore more efficiently. The team designed two mechanisms, Value-Margin Regularization and Advantage-Weighted Action Guidance, to incorporate these action suggestions into reinforcement learning.
The second is CLIP-based reward shaping. To address the difficulty of applying CLIP directly to continuous driving control, the researchers introduced a conditional contrastive action-alignment reward. The method conditions text prompts on driving context, such as vehicle speed and navigation command, and compares the agent’s action with context-specific semantic action anchors. This provides dense and interpretable reward signals during training.
Lightweight agents with real-time inference
The team evaluated Found-RL in CARLA using several autonomous driving benchmarks, including Leaderboard and NoCrash. The results show that Found-RL improves the performance of classical reinforcement learning algorithms in terms of driving score, route completion, and safety-related metrics.
In the Leaderboard benchmark, the DrQv2-CLIP agent achieved a Driving Score of 0.77 and a Success Rate of 57%, outperforming standard reinforcement learning baselines. The study also found that vision-language-model-enhanced agents could reduce unsafe behaviors such as vehicle collisions and red-light violations in several settings.
A key finding is that Found-RL can transfer useful semantic knowledge from large foundation models into compact reinforcement learning policies. The resulting agent contains only 3.82 million parameters but can achieve driving performance competitive with much larger vision-language models. In addition, the lightweight policy reaches about 500 FPS in forward-pass inference, compared with approximately 1 FPS for several billion-parameter vision-language models.
“These results suggest that foundation models do not necessarily need to drive the vehicle directly,” says Chen. “They can instead serve as teachers during training, helping smaller and faster policies learn safer and more efficient driving behaviors.”
Toward foundation-model-enhanced autonomous driving
The study provides an open-source framework for future research at the intersection of foundation models, reinforcement learning, and autonomous driving. The authors note that future work may include testing Found-RL on more challenging driving benchmarks, accelerating and quantizing vision-language-model inference, using multiple foundation models to provide specialized feedback, and exploring lightweight vision-language models as policy backbones.
“Autonomous driving requires both fast control and rich scene understanding,” Chen says. “Found-RL offers a practical way to combine these strengths by using foundation models to improve training while keeping the deployed driving policy efficient.”
About Communications in Transportation Research
Communications in Transportation Research was launched in 2021, with academic support provided by Tsinghua University and China Intelligent Transportation Systems Association. The Editors-in-Chief are Professor Xiaobo Qu, a member of the Academia Europaea from Tsinghua University, and Professor Xiaopeng (Shaw) Li from University of Wisconsin–Madison. The journal mainly publishes high-quality, original research and review articles that are of significant importance to emerging transportation systems, aiming to serve as an international platform for showcasing and exchanging innovative achievements in transportation and related fields, fostering academic exchange and development between China and the global community.
It has been indexed in SCIE, SSCI, Ei Compendex, Scopus, CSTPCD, CSCD, OAJ, DOAJ, TRID and other databases. It was selected as Q1 Top Journal in the Engineering and Technology category of the Chinese Academy of Sciences (CAS) Journal Ranking List. In 2022, it was selected as a High-Starting-Point new journal project of the “China Science and Technology Journal Excellence Action Plan”. In 2024, it was selected as the Support the Development Project of “High-Level International Scientific and Technological Journals”. The same year, it was also chosen as an English Journal Tier Project of the “China Science and Technology Journal Excellence Action Plan PhaseⅡ”. In 2024, it received the first impact factor (2023 IF) of 12.5, ranking Top1 (1/58, Q1) among all journals in "TRANSPORTATION" category. In 2026, its 2025 IF was announced as 12.7, maintaining the Top1 position (1/66, Q1) in the same category.
From Volume 6 (2026), Communications in Transportation Research will be published by Tsinghua University Press on the SciOpen platform with the official journal website at https://www.sciopen.com/journal/2097-5023 . We kindly request that all new manuscript submissions be made through the journal’s submission system at https://mc03.manuscriptcentral.com/commtr . For any submission-related inquiries, please contact the Editorial Office at commtr_e@mail.tsinghua.edu.cn.
Communications in Transportation Research
Found-RL: foundation model-enhanced reinforcement learning via asynchronous VLM feedback for autonomous driving
30-Sep-2026