thomas-yanxin/ost-bench-mirror

---
license: cc-by-nc-4.0
task_categories:
- question-answering
- multiple-choice
- visual-question-answering
- image-text-to-text
language:
- en
pretty_name: OST-Bench
size_categories:
- 10K<n<100K
dataset_info:
features:
- name: scan_id
dtype: string
- name: turn_id
dtype: int64
- name: type
dtype: string
- name: new_observations
sequence: string
- name: origin_question
dtype: string
- name: option
sequence: string
- name: answer
dtype: string
splits:
- name: test
num_examples: 10000
configs:
- config_name: default
data_files:
- split: test
path: OST_bench.json
---
This page contains the data for the paper "OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understanding."

[**🌐 Homepage**](https://rbler1234.github.io/OSTBench.github.io/) | [**📑 Paper**](https://arxiv.org/pdf/2507.07984) | [**💻 Code**](https://github.com/OpenRobotLab/OST-Bench) | [**📖 arXiv**](https://arxiv.org/abs/2507.07984)

## Introduction

Download OST-Bench for evaluation only:
```
huggingface-cli download rbler/OST-Bench --include OST_bench.json,img.zip --repo-type dataset
```
Download OST-Bench for both training and evaluation:
```
huggingface-cli download rbler/OST-Bench --repo-type dataset
```

## Dataset Description
The `imgs`/`img_train` zipfile contains image data corresponding to 1.4k/7k scenes. Each scene has its own subfolder, which stores the observations captured by the agent while exploring that scene.

OST_bench.json/OST_bench_train.json consists of 10k/50k data samples, where each sample represents one round of Q&A (question and answer) and includes the new observations for that round. The structure of each sample (dictionary) is as follows:

```python
{
"scan_id" (str): Unique identifier for the scene scan,
"system_prompt" (str): Shared context/prompt for the multi-turn conversation,
"turn_id" (int): Index of the current turn in the dialogue,
"type" (str): Question subtype/category,
"origin_question" (str): Original question text,
"answer" (str): Ground-truth answer,
"option" (list[str]): Multiple-choice options,
"new_observations" (list[str]): Relative paths to new observation images (within `imgs` dir),
"user_message" (str): Formatted input prompt for the model,
}
```

Samples with the same `scan_id` belong to the same multi-turn conversation group. During model evaluation, each multi-turn conversation group is processed as a unit: the shared `system_prompt` is provided, and new observations along with questions are fed in sequentially according to `turn_id`.

## Evaluation Instructions
Please refer to our [evaluation code](https://github.com/OpenRobotLab/OST-Bench) for details.

许可证：CC-BY-NC-4.0
任务类别：
- 问答
- 多项选择
- 视觉问答（Visual Question Answering）
- 图像文本转文本
语言：
- 英语
美观名称：OST-Bench
样本规模：10000 < 样本数 < 100000
数据集信息：
特征字段：
- 字段名：scan_id，数据类型：字符串
- 字段名：turn_id，数据类型：64位整数
- 字段名：type，数据类型：字符串
- 字段名：new_observations，序列类型，元素为字符串
- 字段名：origin_question，数据类型：字符串
- 字段名：option，序列类型，元素为字符串
- 字段名：answer，数据类型：字符串
数据集划分：
- 划分名称：测试集，样本数：10000
配置项：
- 配置名称：默认配置，数据文件：
- 划分：测试集，文件路径：OST_bench.json

本页面包含论文《OST-Bench：评估多模态大语言模型（Multimodal Large Language Model, MLLM）在线时空场景理解能力》的配套数据集。

[**🌐 主页**](https://rbler1234.github.io/OSTBench.github.io/) | [**📑 论文**](https://arxiv.org/pdf/2507.07984) | [**💻 代码**](https://github.com/OpenRobotLab/OST-Bench) | [**📖 arXiv**](https://arxiv.org/abs/2507.07984)

## 简介
仅用于评估场景的OST-Bench下载指令：

huggingface-cli download rbler/OST-Bench --include OST_bench.json,img.zip --repo-type dataset

用于训练与评估的完整OST-Bench下载指令：

huggingface-cli download rbler/OST-Bench --repo-type dataset

## 数据集说明
`imgs`/`img_train` 压缩包分别包含对应1400/7000个场景的图像数据。每个场景拥有专属子文件夹，用于存储智能体在该场景探索过程中采集到的观测图像。

OST_bench.json与OST_bench_train.json分别包含1万与5万条数据样本，每条样本对应一轮问答（Question & Answer, Q&A），并包含该轮次对应的新观测数据。单条样本（字典格式）的结构如下：
python
{
"scan_id" (str): 场景扫描的唯一标识符,
"system_prompt" (str): 多轮对话共享的上下文与提示词,
"turn_id" (int): 对话中当前轮次的索引,
"type" (str): 问题子类型/类别,
"origin_question" (str): 原始问题文本,
"answer" (str): 标准答案（Ground Truth）,
"option" (list[str]): 多项选择选项,
"new_observations" (list[str]): 新观测图像的相对路径（相对于`imgs`目录）,
"user_message" (str): 为模型格式化后的输入提示词,
}

拥有相同`scan_id`的样本属于同一多轮对话组。在模型评估阶段，每个多轮对话组将作为整体进行处理：首先提供共享的`system_prompt`，随后按照`turn_id`的顺序依次输入新观测数据与对应问题。

## 评估说明
详细评估流程请参阅我们的[评估代码](https://github.com/OpenRobotLab/OST-Bench)。