Self-driving vehicles are moving from pilot projects toward wider commercial deployment, but a vehicle that performs well on conventional tests may still behave in ways that feel hesitant, abrupt or disruptive in real traffic. Researchers at Tongji University have developed a new framework designed to evaluate the overall “intelligence” of high-level autonomous driving systems, rather than judging isolated functions such as braking or lane keeping alone.
The study, “Towards the Intelligence Evaluation of High-level Autonomous Vehicles: A Subjective-Objective Mapping Method via LLM,” introduces a Subjective-Objective Mapping Evaluation (SOME) framework. It combines measurable driving data with expert-style assessments generated by a large language model (LLM), then trains an attention-based neural network to learn how objective indicators relate to the broader judgments that human experts make when observing vehicle behavior.
The team published their study in Communications in Transportation Research ( https://doi.org/10.26599/COMMTR.2026.9640041 ).
“A high-level autonomous vehicle should not be evaluated only by whether it avoids a crash,” says Peng Hang, corresponding author of the study and a research professor at Tongji University. “Its behavior should also be judged by whether it moves smoothly, completes tasks efficiently, interacts reasonably with other road users and avoids unnecessarily disrupting surrounding traffic.”
From raw trajectories to expert-like evaluation
The framework evaluates five complementary aspects of autonomous driving behavior: safety, comfort, efficiency, social interaction and impact on surrounding traffic. It uses interaction segments from the Waymo Open Dataset, including following, merging and crossing situations. To improve coverage of rare but important poor-performance cases, the researchers also augmented the dataset with physically feasible low-score interaction samples. In total, the study constructed 5,200 valid interaction segments for training and evaluation.
Because language models are not naturally suited to interpreting raw vehicle trajectories and map data, the researchers developed a Data-Language Mapping Module. This component converts key interaction events, vehicle states and abnormal motions into structured natural-language descriptions. The descriptions are combined with autonomous-driving knowledge, traffic rules and driving-behavior guidance, enabling the LLM to assess the interaction in an expert-like manner.
The LLM-based judgments are then used as supervisory labels for an attention-based neural network. The network learns nonlinear relationships among ten quantitative metrics instead of relying on a fixed set of manually assigned weights. This allows the model to account for the fact that vehicle safety, motion smoothness, travel efficiency and social behavior can influence one another in complex ways.
Closer agreement with expert reasoning
To test whether the LLM-generated evaluations resembled expert reasoning, the team conducted a Turing Test with ten doctoral students specializing in autonomous driving. Across 66 interaction samples, the LLM-based evaluations achieved an average passing rate of 86.75 percent, meaning the reviewers frequently judged them to be human-generated or could not reliably distinguish their source.
In additional experiments, the proposed LLM evaluation strategy achieved the closest overall agreement with human expert assessments among the compared settings. It obtained a mean absolute error of 0.763, a Pearson correlation of 0.863 and the lowest score-distribution divergence from expert ratings. The downstream attention-based evaluation model achieved a mean absolute error of 0.068 and an R² value of 0.880 when predicting the LLM-derived intelligence scores.
“The goal is not to replace human experts in every decision,” Hang says. “Rather, the framework provides a scalable and consistent way to translate high-dimensional driving behavior into a comprehensive evaluation that can support scenario-based testing and system improvement.”
Why fixed weights can miss important behavior
The researchers applied their method to five autonomous-driving algorithms in six scenario-based tests from the OnSite Autonomous Driving Challenge. The rankings produced by the SOME framework were generally consistent with those of human experts. Compared with an analytic hierarchy process (AHP) method, the proposed model was better able to identify cases in which strong safety scores masked weaknesses in comfort, efficiency or interaction quality.
For example, a fixed-weight method can assign a high overall score to a vehicle with strong safety indicators even when it causes excessive delay or makes uncomfortable maneuvers. By learning relationships among metrics, the new model can capture these trade-offs more directly. The analysis also found that safety and comfort were particularly important for distinguishing different levels of autonomous-driving intelligence, while each of the five dimensions contributed useful information to the final assessment.
The authors note that the current framework focuses mainly on trajectory-level interactions and high-level behavioral indicators. Future work will incorporate richer multimodal information, such as video, perception outputs and more detailed environmental context, while testing the framework across additional datasets, regions, traffic cultures and evaluation platforms.
DOI Link:
https://doi.org/10.26599/COMMTR.2026.9640041
About Communications in Transportation Research
Communications in Transportation Research was launched in 2021, with academic support provided by Tsinghua University and China Intelligent Transportation Systems Association. The Editors-in-Chief are Professor Xiaobo Qu, a member of the Academia Europaea from Tsinghua University, and Professor Xiaopeng (Shaw) Li from University of Wisconsin–Madison. The journal mainly publishes high-quality, original research and review articles that are of significant importance to emerging transportation systems, aiming to serve as an international platform for showcasing and exchanging innovative achievements in transportation and related fields, fostering academic exchange and development between China and the global community.
It has been indexed in SCIE, SSCI, Ei Compendex, Scopus, CSTPCD, CSCD, OAJ, DOAJ, TRID and other databases. It was selected as Q1 Top Journal in the Engineering and Technology category of the Chinese Academy of Sciences (CAS) Journal Ranking List. In 2022, it was selected as a High-Starting-Point new journal project of the “China Science and Technology Journal Excellence Action Plan”. In 2024, it was selected as the Support the Development Project of “High-Level International Scientific and Technological Journals”. The same year, it was also chosen as an English Journal Tier Project of the “China Science and Technology Journal Excellence Action Plan PhaseⅡ”. In 2024, it received the first impact factor (2023 IF) of 12.5, ranking Top1 (1/58, Q1) among all journals in "TRANSPORTATION" category. In 2026, its 2025 IF was announced as 12.7, maintaining the Top1 position (1/66, Q1) in the same category.
From Volume 6 (2026), Communications in Transportation Research will be published by Tsinghua University Press on the SciOpen platform with the official journal website at https://www.sciopen.com/journal/2097-5023 . We kindly request that all new manuscript submissions be made through the journal’s submission system at https://mc03.manuscriptcentral.com/commtr . For any submission-related inquiries, please contact the Editorial Office at commtr_e@mail.tsinghua.edu.cn.
Communications in Transportation Research
Towards the Intelligence Evaluation of High-level Autonomous Vehicles: A Subjective-Objective Mapping Method via LLM
30-Sep-2026