HindsboNikolaj/scope-benchmark

---
license: cc-by-nc-4.0
language:
- en
tags:
- robotics
- vision-language
- human-robot-interaction
- ptz-camera
- benchmark
- evaluation
- blender
size_categories:
- n<1K
task_categories:
- visual-question-answering
- object-detection
- image-classification
pretty_name: "SCOPE — Natural Language PTZ Camera Agent Benchmark"
configs:
- config_name: default
data_files:
- split: test
path: scope_541.csv
---

# SCOPE Benchmark

**Evaluation benchmark** for the HRI '26 paper [*SCOPE: A Real-Time Natural Language Camera Agent at the Edge*](https://doi.org/10.1145/3757279.3785641). Test-only — no train split. 541 questions × 4 Blender scenes × 8 task categories.

> When you chain a language model and a vision model together, how do you know which one failed?

## Contents

```
scope-benchmark/
scope_541.csv 541-row evaluation set
paper.pdf HRI '26 paper PDF
scenes/
whitechapel/whitechapel.blend Urban exterior (France)
book-nook/book-nook.blend Interior room
city-street/city-street.blend Urban street, neon signage
postwar-city/postwar-city.blend Damaged urban exterior
```

All four `.blend` files are texture-packed and load standalone in Blender 4.0+. Pulled from the canonical repo at [github.com/HindsboNikolaj/SCOPE](https://github.com/HindsboNikolaj/SCOPE).

## Task categories (8)

Counting · descriptor · location/spatial · OCR identification · single-call · multi-step command · multi-step reasoning · comparative/relational. See [paper §4](https://doi.org/10.1145/3757279.3785641) for the breakdown.

## CSV schema

| Column | Description |
| --- | --- |
| `question_id` | Q_001 … Q_541 |
| `file_location` | Path to scene `.blend` (relative to `scenes/`) |
| `question` | Natural-language prompt to the agent |
| `expected_answer` | Ground-truth answer for the judge |
| `eval_category` | One of the 8 task categories |
| `difficulty` | Easy / Medium / Hard |
| `multi_step_mode` | Single-call vs multi-step expected behavior |
| `required_tools_policy` | Strict / Lenient tool-use enforcement |
| `expected_tool_order_json` | Optional expected tool sequence |
| `evaluation_notes` | Free-text grading hints |

## Usage

```python
from huggingface_hub import snapshot_download
snapshot_download(repo_id="HindsboNikolaj/scope-benchmark", repo_type="dataset",
local_dir="benchmark/")
```

Then run the eval pipeline from the [code repo](https://github.com/HindsboNikolaj/SCOPE):

```bash
git clone https://github.com/HindsboNikolaj/SCOPE
cd SCOPE && bash scripts/01_install.sh
bash scripts/run_eval_pipeline.sh
```

## Reproducibility note

A handful of the original scene textures (≈ 10% of references in `postwar-city`, plus a few paid Blender Market HDR addons) are not in this distribution — they were external assets owned by the original scene authors that could not be redistributed under permissive licenses. The scenes still load, render, and produce correct answers for the majority of questions; the missing surfaces appear flat-shaded.

The full provenance and a re-acquisition manifest for paper-grade reproduction live at [`docs/MISSING_TEXTURES.md`](https://github.com/HindsboNikolaj/SCOPE/blob/main/docs/MISSING_TEXTURES.md) in the code repo.

## License

CC-BY-NC-4.0 for the benchmark questions and metadata. Individual scene `.blend` files retain the licenses of their original authors (Sketchfab / MySimsWorld / community contributors); see scene-by-scene attribution in the code repo's `docs/`.

## Citation

```bibtex
@inproceedings{hindsbo2026scope,
title = {SCOPE: A Real-Time Natural Language Camera Agent at the Edge},
author = {Hindsbo, Nikolaj and Ehsani, Sina and Mishra, Pragyana},
booktitle = {Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI '26)},
year = {2026},
publisher = {ACM},
doi = {10.1145/3757279.3785641},
}
```