OVO-S-Bench

# OVO-S-Bench OVO-S-Bench is a hierarchical benchmark for **streaming spatial intelligence** in multimodal LLMs — 1,695 multiple-choice questions across 4 difficulty levels, sampled from 9 public video sources. Evaluation framework: <https://github.com/InternLM/OVO-S-Bench> ## Repository layout ``` ovo_s_bench_l1_l4.parquet # 1695 questions, ~35 MB videos/ ├── Ego4D/ *.mp4 # 71 clips, ~88 GiB ├── annotated_videos/ │ ├── 01_CODa_full/ *.mp4 │ ├── 02_Ego4D/ *.mp4 │ ├── 03_OmniWorld/ *.mp4 │ ├── 04_RoomTour3D/ *.mp4 │ ├── honda_videos_for_modelscope/ *.mp4 │ └── RoomTour3D/ *.mp4 ├── Sekai/ *.mp4 # YouTube travelogue clips ├── RoomTour3D/ *.mp4 ├── CODa_full/ *.mp4 ├── OmniWorld/ *.mp4 ├── VSI-Bench/ *.mp4 ├── honda/ *.mp4 # L4.3 trajectory matching source ├── arkitscenes/ *.mp4 # L4.3 trajectory matching source └── edited_videos/ *.mp4 # L3.2 edited clips ``` 339 unique video files, ~219 GiB total. ## Schema | column | type | notes | | --------------------- | ---------------------- | ------------------------------------------------------------------ | | `id` | string | L1-L3: `{subcat}_{idx}` (e.g. `1.1.1_0`); L4: `{main}_{NNN}` | | `source_dataset` | string | Ego4D / RoomTour3D / CODa_full / OmniWorld / VSI-Bench / Sekai / YouTube / Honda / ARKitScenes | | `video_id` | string | Stable across releases | | `video_path` | string | Relative to `videos/` | | `level` | int (1-4) | | | `task_main_category` | string (e.g. `1.1`) | | | `task_subcategory` | string | L1-L3: dotted (`1.1.1`); L4: main_cat (`4.1`) | | `task_type_name` | string | Human-readable task name | | `question` | string | English-translated | | `options` | list<[label, content]> | 2-7 options. L4.3 contents are `data:image/png;base64,...` URIs | | `query_times` | list<float> | Seconds from video start | | `evidence_times` | list<[float, float]> | Each evidence interval `[start, end]` in seconds | | `answers` | list<string> | Correct option label(s) | ## Task taxonomy | Level | Subcategories | Count | | ----- | ------------- | ----- | | L1 — Spatial perception | 1.1.x metric / 1.2.x topological / 1.3.x dynamic | 629 | | L2 — Scene understanding | 2.1.x recognition / 2.2.x memory / 2.3.x temporal | 513 | | L3 — Spatial-temporal multi-hop reasoning | 3.1.x / 3.2.x / 3.3.x | 279 | | L4 — Compositional reasoning | 4.1 allocentric direction / 4.2 topological structure / 4.3 trajectory map matching (image options) | 274 | L4.3 questions ship their option images **embedded inline** as base64 data URIs in the `options` column. No separate image asset download needed. ## Quick load (datasets library) ```python from datasets import load_dataset ds = load_dataset("InternLM/OVO-S-Bench", split="train") # 1695 rows print(ds[0]) ``` ## Quick load (pandas) ```python import pandas as pd df = pd.read_parquet("ovo_s_bench_l1_l4.parquet") print(df.iloc[0]) ``` ## Evaluation framework The reference inference + scoring code: ```bash pip install -r requirements.txt python inference.py --model gpt-4o --annotation ovo_s_bench_l1_l4.parquet python score.py --result results/gpt-4o/ovo_s_bench_l1_l4.json ``` Full setup at <https://github.com/InternLM/OVO-S-Bench>. ## Licensing - Annotations (this parquet): CC-BY-4.0 - Source videos: retain their respective licenses — Ego4D (Ego4D License), RoomTour3D (CC-BY-NC), CODa (Apache-2.0), OmniWorld (CC-BY-NC), VSI-Bench (mixed: ScanNet++ / ScanNet / ARKitScenes), Sekai (CC-BY), ARKitScenes (Apple ARKit ML), Honda HRD (CC-BY-NC). ## Citation ```bibtex @misc{ovosbench2026, title = {OVO-S-Bench: A Hierarchical Benchmark for Streaming Spatial Intelligence in Multimodal LLMs}, author = {OVO-S-Bench Team}, year = {2026}, note = {arXiv:TBD} } ```