TRIP交通测评数据集

# TRIP-Evaluate：开源多模态综合交通大模型测评基准

**在使用该数据集的过程中，若您遇到任何问题或有独特的见解，欢迎通过电子邮件[e-mail]与我们取得联系。我们也始终秉持着开放合作的态度，诚挚邀请您加入TRIP-Evaluation项目，一同致力于构建安全可信的交通大模型。**

[](https://zhenz2020.github.io/TRIP_Evaluation/TRIP-Evaluation.html)

**数据集引用格式**

*Gong, H., Zhou, Z., Shi, Y., Tan, Y., Huo, J., Hong, Q., & Liu, Z. (2026). TRIP-Evaluate: An Open Multimodal Benchmark for Evaluating Large Models in Transportation. arXiv preprint arXiv:2605.00907.*

**TRIP-Evaluate** 是一个面向交通垂直领域的大模型系统性评测基准，覆盖交通全生命周期主体的 **车辆端** / **管理端** / **出行者端** / **规划设计端** 四大维度，并采用 “**一级角色**（**Role**）- **二级任务**（**Task**）- **三级知识点**（**Knowledge**）” 的分层分类体系，支持从总体到细粒度下钻的诊断评估。

> 🌐 **欲获取完整的评测报告、交互式诊断结果及更多详细图表，请访问我们的 [TRIP-Evaluate 项目主页](https://zhenz2020.github.io/TRIP_Evaluation/TRIP-Evaluation.html)。**
---

## 基准特性与主要贡献

与通用的常识问答或基础视觉评测不同，TRIP-Evaluate 深度聚焦交通垂直领域的复杂工程与安全需求，其核心贡献体现在以下四个维度：

- **全生命周期与立体化分类体系**：依托 “**车辆** / **管理** / **出行者** / **规划设计**” 四大物理与逻辑端口，构建了严密的 “**一级角色 - 二级任务 - 三级知识点**” 分层评估架构。该体系无缝覆盖了从上游宏观路网规划、中游微观交通流管控，到下游单车智能感知的完整交通业务链条。
- **原生多模态与创新点云对齐机制**：突破了传统基准的单模态限制，融合了 **纯文本**、**图像** 及 **三维点云** 三类异构数据。特别针对自动驾驶领域高度依赖的点云数据，创新性地设计了 “**BEV + 前视双视图**” 的确定性二维投影对齐方案，使得通用视觉大模型（**VLM**）也能被公平、系统地纳入三维空间感知能力的测评框架中。
- **强工程规范与真实安全计算约束**：题目语料深度对齐现行中国交通法规、国家工程标准与真实业务操作规范。相较于单纯的文本记忆，本基准大量引入了多条件信控配时、视距三角形求解、通行能力审计等复杂长链工程计算任务，确保大模型的评测表现具有真实的工业 “**可落地性**” 与结果 “**可复核性**”。
- **全链路可复现与多维细粒度诊断**：开源了标准化的自动化评测流水线（**统一的 Prompt 模板、严格的解码参数控制、格式容错与自动化判分规则**），最大限度消除了不同模型 API 接口差异带来的噪音。同时，借助完备的元数据标签字典，支持生成细粒度的切片画像，精准定位模型在特定模态或难度下的 “**偏科**” 缺陷。

---

## 核心基准洞察

基于大规模模型的测试表现，TRIP-Evaluate 进行了深度的切片诊断，揭示了当前大模型在交通垂直领域的四大核心能力边界：

- **长链推演的**“**伪逻辑**”**与结构性衰减**：在难度递增的长链工程计算中，模型极易暴露出“**伪逻辑链**”。例如，在多条件组合的困难任务下，部分前沿模型的准确率从 94.2% 暴跌至 49.6%。模型虽然能生成表面合理的分步结构，但常常缺乏可验证的约束检查机制，导致在公式嵌套、边界条件校核或量纲一致性上出现误差级联放大。

*(图：大模型在难度递增与长链计算下的准确率结构性衰减)*

- **跨模态对齐的系统性**“**点云惩罚**”：引入多模态输入后，二维图像带来的增益极不稳定（**部分模型甚至出现退化**），而三维点云数据则带来了强烈的系统性退化。即使点云已降维为 BEV 与前视双视图，点云模态的准确率相对纯文本基线依然出现了最高达 57.1% 的折损。这表明当前视觉大模型在三维空间拓扑、遮挡关系与相对深度推演上仍存在底层的表征鸿沟。
- **双链路压力源的分化**（**模态敏感 vs 难度敏感**）：交通业务场景可抽象为两条核心链路。评测发现，“**安全语义链**”（**涉及场景理解与空间推断**）对输入模态极其敏感，模态切换极易导致语义对齐崩溃；而“**工程可校验链**”（**涉及公式计算与规则适用**）则对任务难度高度敏感，随着约束条件增多，逻辑推理准确率大幅下降。
- **知识失分分布的极度两极化**：不同交通角色的失效模式截然不同。车辆端（**感知与定位**）和管理端的错误呈现高度的“**头部集中**”效应，例如车端单“**目标检测**”一项就贡献了约 67% 的失分；相对地，出行者端与规划设计端（**如通行能力评价**）的错误则呈现“**长尾分散**”特征。这提示我们在做模型优化时，上游感知环节需死磕头部缺陷，而下游决策设计环节更依赖高覆盖度的边缘场景约束。

*(图：大模型在纯文本与多模态综合场景下的跨端能力分布对比)*

*(图：四大端口细分错误知识点帕累托分布，呈现明显的头部集中与长尾分散特征)*

---

## 数据规模

**核心评测集**（**推荐基线**）
- **总题数**：767
- **模态构成**：文本 596 / 图像 128 / 点云 43
- **角色端**：出行者端 191 / 管理端 211 / 规划设计端 194 / 车辆端 171
- **难度**：简单 111 / 中等 444 / 困难 212
- **能力维度**：知识记忆 118 / 逻辑推理 200 / 数值计算 243 / 场景语义理解 206

---

## 评测分类体系

TRIP-Evaluate 采用 “**一级角色 - 二级任务 - 三级知识点**” 的立体评价架构，用于覆盖交通行业的关键应用链路。

*(图：TRIP-Evaluate 三级架构示例)*

- 🚗 **车辆端**：感知与定位 / 决策规划 / 控制执行 / 交通安全
- 🛣️ **管理端**：交通信号控制 / 道路标志标线 / 智能路侧设备 / 交通执法与安全
- 🧍 **出行者端**：出行行为分析 / 交通安全与防护
- 🧭 **规划设计端**：道路几何设计 / 路网规划 / 通行能力评价 / 交通安全审计

---

## 数据结构

数据以 CSV 或 JSON 形式组织。每条样本包含完整元数据以支持多维度归因分析。

| 字段 | 类型 | 说明 |
| --- | --- | --- |
| `id` | String | 样本唯一标识符 (e.g., B_b_1) |
| `role_tag` | String | 一级分类：车辆端/管理端/出行者端/规划设计端 |
| `subject_tag` | String | 二级分类：具体任务场景 (e.g., 交通安全) |
| `knowledge_point` | String | 三级分类：具体考点 (e.g., 视距三角形) |
| `question` | String | 题干描述，包含场景设定与问题 |
| `A, B, C, D` | String | 选项的具体文本内容（分别对应四个候选答案） |
| `answer` | String | 标准答案选项 |
| `explanation` | String | 答案解析，引用法规条文或计算公式 |
| `difficulty` | String | 难度系数：简单/中等/困难 |
| `modality` | String | 数据模态：文本 / 图像 / 点云 |
| `capability_tag` | String | 能力维度：知识记忆/逻辑推理/数值计算/场景语义理解 |

---

## 统一评测协议与数据流水线

为保证跨模型可比性，TRIP-Evaluate 采用 “**统一题型—统一输出—统一判分**” 协议：

- **题型统一**：所有样本均为单项选择题（**A** / **B** / **C** / **D**）。
- **输出约束**：通过模板化提示词约束模型只输出单个字母，不输出解释或额外文本。
- **模态一致**：三类模态使用一致的核心提示约束；点云以 **BEV + 前视双图** 作为输入载体，解决了原生点云的三维表征鸿沟。
- **解码固定**：固定温度、Top-p、随机种子等设置，保证复现与横向对比稳定。
- **异常处理**：对格式错误或多字符等异常按统一规则处理。

*(图：TRIP-Evaluate 数据集样本生成与多模态对齐流水线)*

---

## 核心榜单

（**注**：部分模型因接口系统拒答或模态支持差异，未能实现全量题目覆盖，计算准确率时以该模型的实际有效答题数为基准，表中不支持的模态留空。）

| 模型名称 | 综合准确率 | 纯文本 | 图像 | 点云 |
| --- | --- | --- | --- | --- |
| DeepSeek-R1 | 90.8% | 90.8% | - | - |
| Gemini-3-flash-preview | 88.8% | 91.3% | 93.7% | 34.2% |
| DeepSeek-V3.2 | 84.7% | 84.7% | - | - |
| Gpt-oss-120b | 81.4% | 89.1% | 60.6% | 31.6% |
| Claude Sonnet 4.6 | 80.5% | 83.7% | 79.5% | 36.8% |
| Qwen-max | 79.3% | 82.8% | 73.2% | 47.4% |
| Gpt-oss-20b | 79.2% | 86.9% | 55.9% | 39.5% |
| Qwen2-VL-72B-Instruct | 77.7% | 79.0% | 81.9% | 44.7% |
| Qwen3-8B | 77.6% | 77.6% | - | - |
| Gemma-2-27b-it | 77.2% | 77.2% | - | - |
| Qwen2.5-coder-32b-instruct | 76.5% | 76.5% | - | - |
| Qwen2.5-coder-7b-instruct | 75.7% | 75.7% | - | - |
| Gemma-3-27b-it | 73.9% | 73.9% | - | - |
| Gemma-2-9b-it | 73.7% | 73.7% | - | - |
| Llama-3.2-90b-vision-instruct | 73.1% | 76.7% | 70.9% | 26.3% |
| Qwen3-VL-8B-Instruct | 72.3% | 75.1% | 70.9% | 34.2% |
| Claude Sonnet 4.5 | 69.9% | 71.8% | 69.3% | 42.1% |
| Llama-3.2-11b-vision-instruct | 61.2% | 65.7% | 53.5% | 18.4% |

---

## 快速开始

### 方式 1：通过 ModelScope SDK 加载（推荐）

```python
from modelscope.msdatasets import MsDataset

ds = MsDataset.load(
"YOUR_NAMESPACE/TRIP-Evaluate",
subset_name="default",
split="test",
)

print(ds[0])
```

### 方式 2：GIT 下载 / 本地文件加载

你也可以通过 GIT 或下载页面获取数据集文件，再用本地路径加载。

---

## 质量控制

- **可判定 / 可复算**：题面边界清晰、条件完整；工程计算题提供关键公式与变量口径。
- **去重与歧义控制**：在验收阶段进行去重、歧义审查与计算复核。
- **选项偏置抑制**：引入选项长度约束，减少基于捷径的虚假高分。

---

## 未来规划

- **难度进阶**：增加多步推理与复杂工程计算、极端天气判定等边界测试用例。
- **多模态增强**：扩充点云样本，进一步评估三维空间感知；引入更多视觉模型测试。

## 贡献人员名单

  - **模型贡献方**：TRIP项目小组
  - **核心贡献者**：龚晗、周臻、洪奇（东南大学复杂交通网络研究中心）
  - **支持团队**：东南大学交通学院

# TRIP-Evaluate: Open-Source Multimodal Comprehensive Traffic Large Model Evaluation Benchmark

**If you encounter any issues or have unique insights during the use of this dataset, please contact us via email at [e-mail]. We always adhere to the attitude of openness and cooperation, and sincerely invite you to join the TRIP-Evaluation project to jointly build safe and trustworthy traffic large models.**

[](https://zhenz2020.github.io/TRIP_Evaluation/TRIP-Evaluation.html)

**Dataset citation format will be provided by the end of April 2026.**

**TRIP-Evaluate** is a systematic evaluation benchmark for large models in the traffic vertical domain, covering four dimensions of full-life-cycle traffic stakeholders: **Vehicle-side**, **Management-side**, **Traveler-side**, and **Planning & Design-side**. It adopts a hierarchical classification framework of "Level 1 Role (Role) - Level 2 Task (Task) - Level 3 Knowledge Point (Knowledge)", enabling diagnostic assessment from a macroscopic overview to fine-grained drill-down analysis.

> 🌐 **To obtain the complete evaluation report, interactive diagnostic results and more detailed charts, please visit our [TRIP-Evaluate Project Homepage](https://zhenz2020.github.io/TRIP_Evaluation/TRIP-Evaluation.html).**
---

## Benchmark Features and Main Contributions

Unlike general common-sense question answering or basic visual evaluation benchmarks, TRIP-Evaluate deeply focuses on the complex engineering and safety requirements of the traffic vertical domain. Its core contributions are reflected in the following four aspects:

- **Full-Life-Cycle and Stereo Classification System**: Relying on the four physical and logical ports of "Vehicle, Management, Traveler, Planning & Design", it constructs a rigorous hierarchical evaluation framework of "Level 1 Role - Level 2 Task - Level 3 Knowledge Point". This system seamlessly covers the entire traffic business chain, from upstream macroscopic road network planning, midstream microscopic traffic flow control, to downstream single-vehicle intelligent perception.
- **Native Multimodal and Innovative Point Cloud Alignment Mechanism**: It breaks through the single-modal limitation of traditional benchmarks, integrating three types of heterogeneous data: **plain text**, **images**, and **3D point clouds**. Specifically for point cloud data highly relied on in the autonomous driving field, it innovatively designs a deterministic 2D projection alignment scheme of "**BEV (Bird's Eye View) + Front-View Dual Views**", enabling general vision-language models (**VLMs**) to be fairly and systematically included in the evaluation framework for 3D spatial perception capabilities.
- **Strict Engineering Specifications and Realistic Safety Calculation Constraints**: The question corpus is deeply aligned with current Chinese traffic regulations, national engineering standards and actual business operation specifications. Compared with simple text memory tasks, this benchmark introduces a large number of complex long-chain engineering calculation tasks such as multi-condition traffic signal timing, sight triangle solving, and traffic capacity audit, ensuring that the evaluation performance of large models has real industrial "implementability" and result "reviewability".
- **Full-Link Reproducibility and Multi-Dimensional Fine-Grained Diagnosis**: It open-sources a standardized automated evaluation pipeline (**unified prompt templates, strict decoding parameter control, format tolerance and automatic scoring rules**), minimizing the noise caused by differences in model API interfaces. Meanwhile, with a complete metadata label dictionary, it supports generating fine-grained slice profiles to accurately locate the "imbalanced performance" defects of the model under specific modalities or difficulty levels.

---

## Core Benchmark Insights

Based on the test performance of large-scale models, TRIP-Evaluate conducts in-depth slice diagnostics, revealing four core capability boundaries of current large models in the traffic vertical domain:

- **"Pseudo-Logic" and Structural Decay in Long-Chain Reasoning**: In long-chain engineering calculations with increasing difficulty, models are prone to expose "pseudo-logic chains". For example, under difficult tasks with multi-condition combinations, the accuracy of some cutting-edge models plummets from 94.2% to 49.6%. Although models can generate superficially reasonable step-by-step structures, they often lack verifiable constraint checking mechanisms, leading to cascading amplification of errors in formula nesting, boundary condition verification or dimensional consistency.

*(Figure: Structural decay of model accuracy under increasing difficulty and long-chain calculations)*

- **Systematic "Point Cloud Penalty" in Cross-Modal Alignment**: After introducing multimodal inputs, the gains brought by 2D images are extremely unstable (**some models even experience performance degradation**), while 3D point cloud data brings severe systematic degradation. Even after point clouds are reduced to BEV and front-view dual views, the accuracy of the point cloud modality still suffers a maximum loss of up to 57.1% compared to the plain text baseline. This indicates that current vision-language models still have underlying representation gaps in 3D spatial topology, occlusion relationship and relative depth reasoning.
- **Divergence of Dual-Linkage Pressure Sources (Modal Sensitivity vs. Difficulty Sensitivity)**: Traffic business scenarios can be abstracted into two core linkages. The evaluation finds that the "Safety Semantic Link" (**involving scene understanding and spatial inference**) is extremely sensitive to input modalities, and modality switching easily leads to semantic alignment collapse; while the "Engineering Verifiable Link" (**involving formula calculation and rule application**) is highly sensitive to task difficulty, with logical reasoning accuracy dropping significantly as constraint conditions increase.
- **Extreme Polarization of Knowledge Point Loss Distribution**: The failure modes of different traffic roles are completely different. Errors on the Vehicle-side (**perception and positioning**) and Management-side show a highly "head-concentrated" effect. For example, the single "object detection" task on the vehicle side contributes about 67% of the total points lost. In contrast, errors on the Traveler-side and Planning & Design-side (such as traffic capacity evaluation) show a "long-tail dispersed" characteristic. This suggests that when optimizing models, the upstream perception link needs to focus on head defects, while the downstream decision-making and design link relies more on high-coverage edge scene constraints.

*(Figure: Comparison of cross-port capability distribution of large models under plain text and multimodal comprehensive scenarios)*

*(Figure: Pareto distribution of subdivision error knowledge points for the four ports, showing obvious head-concentrated and long-tail dispersed characteristics)*

---

## Data Scale

**Core Evaluation Set (Recommended Baseline)**
- **Total number of questions**: 767
- **Modal composition**: Text 596 / Image 128 / Point Cloud 43
- **Stakeholder sides**: Traveler-side 191 / Management-side 211 / Planning & Design-side 194 / Vehicle-side 171
- **Difficulty levels**: Easy 111 / Medium 444 / Hard 212
- **Capability dimensions**: Knowledge Memory 118 / Logical Reasoning 200 / Numerical Calculation 243 / Scene Semantic Understanding 206

---

## Evaluation Taxonomy

TRIP-Evaluate adopts a three-dimensional evaluation framework of "Level 1 Role - Level 2 Task - Level 3 Knowledge Point" to cover the key application chains of the traffic industry.

*(Figure: Example of the three-level architecture of TRIP-Evaluate)*

- 🚗 **Vehicle-side**: Perception and Positioning / Decision Planning / Control Execution / Traffic Safety
- 🛣️ **Management-side**: Traffic Signal Control / Road Signs and Markings / Intelligent Roadside Equipment / Traffic Law Enforcement and Safety
- 🧍 **Traveler-side**: Travel Behavior Analysis / Traffic Safety and Protection
- 🧭 **Planning & Design-side**: Road Geometric Design / Road Network Planning / Traffic Capacity Evaluation / Traffic Safety Audit

---

## Data Structure

Data is organized in CSV or JSON format. Each sample contains complete metadata to support multi-dimensional attribution analysis.

| Field | Type | Description |
| --- | --- | --- |
| `id` | String | Unique identifier of the sample (e.g., B_b_1) |
| `role_tag` | String | Level 1 classification: Vehicle-side / Management-side / Traveler-side / Planning & Design-side |
| `subject_tag` | String | Level 2 classification: Specific task scenario (e.g., Traffic Safety) |
| `knowledge_point` | String | Level 3 classification: Specific knowledge point (e.g., Sight Triangle) |
| `question` | String | Stem description, including scene settings and questions |
| `A, B, C, D` | String | Specific text content of options (corresponding to four candidate answers respectively) |
| `answer` | String | Standard answer option |
| `explanation` | String | Answer explanation, citing regulatory provisions or calculation formulas |
| `difficulty` | String | Difficulty level: Easy / Medium / Hard |
| `modality` | String | Data modality: Text / Image / Point Cloud |
| `capability_tag` | String | Capability dimension: Knowledge Memory / Logical Reasoning / Numerical Calculation / Scene Semantic Understanding |

---

## Unified Evaluation Protocol and Data Pipeline

To ensure cross-model comparability, TRIP-Evaluate adopts the "Unified Question Type - Unified Output - Unified Scoring" protocol:

- **Unified Question Type**: All samples are multiple-choice questions with four options (**A** / **B** / **C** / **D**).
- **Output Constraints**: Use templated prompts to constrain the model to output only a single letter, without explanations or extra text.
- **Modal Consistency**: The three modalities use consistent core prompt constraints; point clouds use **BEV + Front-View Dual Images** as the input carrier, solving the 3D representation gap of native point clouds.
- **Fixed Decoding**: Fix settings such as temperature, Top-p, random seed, etc., to ensure stable reproducibility and horizontal comparison.
- **Exception Handling**: Handle exceptions such as format errors or multiple characters according to unified rules.

*(Figure: Sample generation and multimodal alignment pipeline for the TRIP-Evaluate dataset)*

---

## Core Leaderboard

**Note**: Some models failed to cover all questions due to interface system rejection or modality support differences. When calculating accuracy, the actual effective number of questions answered by the model is used as the benchmark, and unsupported modalities in the table are left blank.

| Model Name | Overall Accuracy | Plain Text | Image | Point Cloud |
| --- | --- | --- | --- | --- |
| DeepSeek-R1 | 90.8% | 90.8% | - | - |
| Gemini-3-flash-preview | 88.8% | 91.3% | 93.7% | 34.2% |
| DeepSeek-V3.2 | 84.7% | 84.7% | - | - |
| Gpt-oss-120b | 81.4% | 89.1% | 60.6% | 31.6% |
| Claude Sonnet 4.6 | 80.5% | 83.7% | 79.5% | 36.8% |
| Qwen-max | 79.3% | 82.8% | 73.2% | 47.4% |
| Gpt-oss-20b | 79.2% | 86.9% | 55.9% | 39.5% |
| Qwen2-VL-72B-Instruct | 77.7% | 79.0% | 81.9% | 44.7% |
| Qwen3-8B | 77.6% | 77.6% | - | - |
| Gemma-2-27b-it | 77.2% | 77.2% | - | - |
| Qwen2.5-coder-32b-instruct | 76.5% | 76.5% | - | - |
| Qwen2.5-coder-7b-instruct | 75.7% | 75.7% | - | - |
| Gemma-3-27b-it | 73.9% | 73.9% | - | - |
| Gemma-2-9b-it | 73.7% | 73.7% | - | - |
| Llama-3.2-90b-vision-instruct | 73.1% | 76.7% | 70.9% | 26.3% |
| Qwen3-VL-8B-Instruct | 72.3% | 75.1% | 70.9% | 34.2% |
| Claude Sonnet 4.5 | 69.9% | 71.8% | 69.3% | 42.1% |
| Llama-3.2-11b-vision-instruct | 61.2% | 65.7% | 53.5% | 18.4% |

---

## Quick Start

### Method 1: Load via ModelScope SDK (Recommended)

python
from modelscope.msdatasets import MsDataset

ds = MsDataset.load(
"YOUR_NAMESPACE/TRIP-Evaluate",
subset_name="default",
split="test",
)

print(ds[0])

### Method 2: Git Clone / Local File Loading

You can also obtain the dataset files via Git or the download page, then load them using a local path.

---

## Quality Control

- **Decidable & Recalculable**: The question surface has clear boundaries and complete conditions; engineering calculation questions provide key formulas and variable specifications.
- **Deduplication and Ambiguity Control**: Deduplication, ambiguity review and calculation review are carried out during the acceptance stage.
- **Option Bias Suppression**: Introduce option length constraints to reduce spurious high scores based on shortcuts.

---

## Future Plans

- **Difficulty Advancement**: Add boundary test cases such as multi-step reasoning and complex engineering calculations, extreme weather judgment, etc.
- **Multimodal Enhancement**: Expand point cloud samples to further evaluate 3D spatial perception; introduce more vision model tests.

## Contributors

- **Model Contribution Team**: TRIP Project Team
- **Core Contributors**: Han Gong, Zhen Zhou, Qi Hong (Complex Traffic Network Research Center, Southeast University)
- **Support Team**: School of Transportation, Southeast University

搜集汇总

背景与挑战

背景概述

TRIP-Evaluate是一个专为交通领域大模型设计的系统性多模态评估基准，覆盖车辆、管理、旅行者及规划与设计四个关键维度的全生命周期。它采用分层分类法，包含767个涵盖文本、图像和点云模态的问题，并提供了统一的评估协议以支持可复现的模型诊断。