Rynn-VP

# Rynn-VP: Long-Horizon Visual Planning Benchmark Rynn-VP evaluates visual plans for everyday manipulation in real indoor scenes. Given a natural-language task and an initial image, a system produces a candidate video or an ordered sequence of step images. Evaluation examines the action program supported by those images, the visible realization of task effects, and perceptual quality. Each of the **200 task instances** provides the instruction, initial scene, a successful GT witness video, and timestamped planning annotations. The GT is one feasible execution, rather than a unique action script that candidates must imitate. Exact object identity, quantities, exclusions, requested actions and destination relations are determined by the task. **Dataset repository:** `Alibaba-DAMO-Academy/Rynn-VP` **Data, Skill and code license:** [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) **Evaluation instructions:** [Rynn-VP Evaluation Skill](skills/rynn-vp-eval/SKILL.md) **中文说明:** [README_ZH.md](README_ZH.md) ## Contents Each case directory contains exactly four original files: ```text Rynn-VP/ ├── README.md ├── README_ZH.md ├── LICENSE ├── SHA256SUMS.txt ├── skills/ │ └── rynn-vp-eval/ │ ├── SKILL.md │ ├── references/ │ │ ├── DETAILED_SCORING_ZH.md │ │ ├── metric_contract.json │ │ ├── reporting_profile.json │ │ ├── native_logic.py │ │ ├── agentic_llm_reference.py │ │ ├── prompts/ │ │ └── schemas/ │ └── scripts/ │ ├── prepare_prompt.py │ └── aggregate_scores.py └── cases/ └── <case_id>/ ├── gt.mp4 ├── gt_annotations.json ├── initial.png └── task.json ``` The release contains **800 original data files**, totaling **43,752,021,216 bytes** (43.75 GB; 40.75 GiB), plus the small evaluation and documentation files. Media are not transcoded. GT specifications, normalized annotation copies, caches, preview frames, storyboards, model candidates and evaluation results are not included. Generate derived evaluation assets in a separate run directory using the Skill. ## Dataset statistics | Property | Value | |---|---:| | Task instances | 200 | | Unique literal instruction strings | 124 | | GT videos / initial images / planning annotation files | 200 each | | Timestamped planning intervals | 2,215 | | Intervals other than `Done` or `Done.` | 2,016 | | Intervals per case | 6–24; mean 11.075; median 11 | | GT video resolution | 1920 × 1080 | | GT video duration | 20.02–122.54 seconds | | Mean / median GT duration | 43.72 / 36.60 seconds | | Total GT duration | 145.74 minutes | | GT videos no longer than 60 seconds | 165 / 200 (82.5%) | | Scene label | Cases | |---|---:| | Bedroom | 54 | | Living_Room | 49 | | Office_Space | 44 | | Kitchen | 29 | | Bathroom | 12 | | Home Office | 12 | Source groups contain 91 (`source1_1`), 56 (`source1_2`), 52 (`source2_1`) and 1 (`source2_2`) cases. These are source identifiers, not participant counts. There are 56 distinct `(source_group, scene, room_id)` tuples, or 46 if source group is ignored; neither count establishes the number of physically distinct homes. All 200 cases form the released benchmark set; this package does not define a training split. ## File formats `task.json` supplies the case identifier and original English instruction. For example: ```json { "task_id": "source1_1-Bedroom-Room_03-Task_02-Variant_07-001", "instruction": "Help me take the pillow cover off the pillow, fold it, and place it in the corner." } ``` `initial.png` defines the initial visual scene (S0). `gt.mp4` is the successful witness used by the evaluator to interpret referents, feasible transitions and evidence requirements. For a task-and-initial-image generation protocol, keep the GT witness and its planning annotations on the evaluator side. `gt_annotations.json` has three fields: | Field | Meaning | |---|---| | `planning` | Ordered mapping from interval strings such as `0.0s-2.2s` to natural-language action descriptions. Times are in seconds relative to the GT video. | | `task` | Original task text; a few entries contain trailing whitespace. | | `video` | Original recording filename, retained as provenance. Load the packaged video from `gt.mp4`, not from this historical filename. | The planning describes the successful witness. It is not a required candidate script and does not enable explicit-plan PI-5 scoring. Preserve JSON insertion order and parse interval endpoints as seconds. Case IDs are the stable join key across tasks, candidates and reports. ## Evaluation The reference judging setup uses **GPT-6-sol with the Codex harness**, with the evaluation procedure encoded as a Skill. Record the actual model, reasoning configuration, prompt version, input modality, candidate scope and context-sharing policy for each run. The helpers generate prompts and aggregate recorded judgments; they do not call a model or perform visual review automatically. 1. **GT compilation:** inspect the task, S0, GT witness and timestamped annotations; derive entity correspondence, goal predicates, necessary process evidence, allowed alternatives and uncertainties. Cache this locally, keyed by input fingerprints. 2. **Five candidate roles:** Observer, VR/VQ judge, dense hallucination auditor, PI judge and independent Critic. Each judgment must cite actual visual evidence. The first evaluation has six logical stages including GT compilation; a cached GT leaves five candidate stages. Retries, repairs or manager review can require additional work. 3. **Evidence:** the standard video policy uses at most 50 source frames, including S0, up to four GT anchors and 30 dense candidate panels arranged into five six-panel storyboards. Record actual clip boundaries, candidate truncation and any extra evidence. Ordered images use step identifiers; unshown transitions remain evidence gaps rather than assumed continuous video. 4. **Audit and aggregation:** resolve stage contradictions, distinguish partial progress from hallucination, preserve genuine subgoal credit, and apply the unchanged native per-metric calibration once. Every confirmed hallucination must be localized to an entity, interval or step, contradictory observations, and an affected causal/identity chain. The [Skill](skills/rynn-vp-eval/SKILL.md) contains the full workflow. The [detailed rubric](skills/rynn-vp-eval/references/DETAILED_SCORING_ZH.md), canonical schemas, frozen role prompts and preserved source functions specify the scoring behavior. The source reference file documents the original implementation; it is not a standalone API evaluator. Frame extraction and image inspection use the tools available in the executing harness. ### Metrics All axes are on a 0–100 scale; higher is better. | Axis | Evaluates | |---|---| | **PI — Planning Intelligence** | Executable goal achievement, step validity, dependency/order, and coverage of necessary process evidence. | | **VR — Visual Realization** | Visible goals, action/effect evidence, state inheritance, entity/scene consistency, feasibility and visual plausibility. | | **VQ — Visual Quality** | Holistic perceptual quality, with seven diagnostic dimensions. This is not a prescribed arithmetic average of those dimensions. | The current reporting profile is: \[ \mathrm{Overall}=0.50\,\mathrm{PI}+0.40\,\mathrm{VR}+0.10\,\mathrm{VQ}. \] PI uses weights `0.50 / 0.20 / 0.15 / 0.15` for PI-1 through PI-4. Only a task supplying an explicit plan activates PI-5, with weights `0.40 / 0.20 / 0.15 / 0.15 / 0.10`; PI-6 is diagnostic. VR-1 through VR-8 use `0.20 / 0.20 / 0.15 / 0.15 / 0.10 / 0.10 / 0.07 / 0.03`. VR-8 remains part of VR and need not equal independent VQ. For reproducibility, the frozen native contract also records the historical total `0.55 × PI + 0.45 × VR`. The provided aggregation helper reports this as `legacy_final_score` alongside the current `final_score`. Changing reporting weights does not constitute a new visual evaluation. Missing candidates receive no fabricated scores and are excluded from means; always report the scored denominator. ### Interpretation and limitations Correct partial actions receive partial credit. A wrong drawer, wrong shelf, wrong original entity or missing release cannot be described as satisfying the requested relation. Apparent completion that depends on an unrecoverable entity or world-state rewrite is decisive hallucination of the task-critical chain. An unsampled or occluded producer alone does not establish hallucination. Image-based judging cannot directly measure forces or certify physical robot execution. Sparse evidence can miss brief events. The Skill requires uncertainty to remain explicit and model identity or desired score distributions to be excluded from scoring evidence. Video and ordered-image protocols have different temporal evidence; report modality and sampling scope when comparing results. ## Download and read After publication, download using the [Hugging Face Hub library](https://huggingface.co/docs/huggingface_hub/guides/download): ```bash python -m pip install -U huggingface_hub ``` ```python from huggingface_hub import snapshot_download snapshot_download( repo_id="Alibaba-DAMO-Academy/Rynn-VP", repo_type="dataset", local_dir="Rynn-VP", ) ``` Pin `revision` to the published commit used in an experiment. To fetch only tasks and annotations initially, add `allow_patterns=["cases/*/task.json", "cases/*/gt_annotations.json", "README*", "LICENSE"]`. Read the raw files directly: ```python import json from pathlib import Path for case in sorted(Path("Rynn-VP/cases").iterdir()): if not case.is_dir(): continue task = json.loads((case / "task.json").read_text(encoding="utf-8")) annotations = json.loads((case / "gt_annotations.json").read_text(encoding="utf-8")) print(task["task_id"], task["instruction"], len(annotations["planning"])) ``` The repository is a file collection. No automatic `datasets.load_dataset` configuration is assumed. Prompt preparation and numerical aggregation require Python 3.10+ and the standard library; downloading and media extraction have their own dependencies. ## Integrity and annotation notes `SHA256SUMS.txt` records every release file except itself. All 200 initial images and the first and last frame of every GT video were decoded during staging; this was not a full-frame decode of every video. Original media, task files and annotations are preserved byte-for-byte. Raw timestamps were retained. Three annotation intervals overlap preceding intervals (0.1, 0.3 and 0.1 seconds). Thirty-six endpoints exceed the metadata-derived video duration, mostly by small rounding differences; three exceed it by more than 0.2 seconds, with a maximum excess of approximately 1.108 seconds. When extracting frames, clamp extraction positions to the actual decodable range and retain the original annotation timestamps in provenance. Terminal labels include 113 `Done`, 86 `Done.` and one final `Down.`. The latter is preserved as written and is not silently reclassified as a terminal label. Three annotation `task` strings have extra trailing whitespace; compare normalized whitespace for consistency checks without modifying the source. ## License and attribution The four data files in every `cases/<case_id>/` directory, this documentation, and the Evaluation Skill with its prompts, schemas and accompanying code are released under **Creative Commons Attribution-NonCommercial 4.0 International**. Noncommercial reuse requires appropriate attribution, a license link and indication of changes, as specified by the [license](LICENSE). The Skill/code licensing scope is also recorded [in the Skill directory](skills/rynn-vp-eval/LICENSE_SCOPE.md), with a local copy of the license for separate distribution. For attribution, identify **Rynn-VP**, retain the dataset repository link and cite the release revision used. A dataset citation is: ```bibtex @misc{rynn_vp_dataset, title = {Rynn-VP: Long-Horizon Visual Planning Benchmark}, year = {2026}, howpublished = {Hugging Face dataset}, url = {https://huggingface.co/datasets/Alibaba-DAMO-Academy/Rynn-VP} } ```