siyich/spacetools-eval-benchmarks

---
license: apache-2.0
task_categories:
- visual-question-answering
- image-to-text
tags:
- spatial-reasoning
- robotics
- benchmark
- evaluation
size_categories:
- 1K<n<10K
configs:
- config_name: robospatial
data_files: data/robospatial.parquet
- config_name: reflocation
data_files: data/reflocation.parquet
- config_name: refplacement
data_files: data/refplacement.parquet
- config_name: refunseen
data_files: data/refunseen.parquet
- config_name: boppose
data_files: data/boppose.parquet
- config_name: bopgrasp
data_files: data/bopgrasp.parquet
- config_name: blinkdepth
data_files: data/blinkdepth.parquet
- config_name: cvb2drelation
data_files: data/cvb2drelation.parquet
- config_name: cvb3ddepth
data_files: data/cvb3ddepth.parquet
---

# SpaceTools Evaluation Benchmarks

Evaluation benchmarks from the [SpaceTools paper](https://spacetools.github.io/) for spatial reasoning with vision tool calling. All images are embedded as base64 data URIs -- no external dependencies needed.

## Benchmarks

| Benchmark | Samples | Task | Metric |
|-----------|---------|------|--------|
| `robospatial` | 350 | Robot spatial reasoning (config/compat/context/grounding) | Accuracy |
| `reflocation` | 100 | RefSpatial object location | Accuracy |
| `refplacement` | 100 | RefSpatial placement | Accuracy |
| `refunseen` | 77 | RefSpatial unseen objects | Accuracy |
| `boppose` | 60 | 6DoF object pose estimation (BOP-HOPE) | Accuracy |
| `bopgrasp` | 60 | Grasp pose estimation (BOP-HOPE) | MACE / SR |
| `blinkdepth` | 124 | Relative depth comparison (Blink) | Accuracy |
| `cvb2drelation` | 650 | 2D spatial relation (CVBench) | Accuracy |
| `cvb3ddepth` | 600 | 3D depth ordering (CVBench) | Accuracy |

**Total: 2,121 evaluation samples across 9 benchmarks**

## Paper Results (Quantitative, 3B model)

| RoboSpatial | RefSpatial (avg) | Blink Depth | CVB 2D Rel | CVB 3D Depth | BOP Pose | BOP Grasp MACE | BOP Grasp SR |
|-------------|-----------------|-------------|------------|--------------|----------|----------------|--------------|
| 70.0% | 53.07% | 90.32% | 94.92% | 96.00% | 43.06 | 43.06 | 50.0% |

## Schema

Each parquet has the same columns (verl-compatible format):
- `data_source`: Dataset/benchmark identifier
- `prompt`: Chat messages (system + user with question and image)
- `images`: List of base64-encoded images (`{"image": "data:image/...;base64,..."}`)
- `ability`: Task ability tag
- `reward_model`: Ground truth and scoring style
- `extra_info`: Question text, answer, split, qa_type

## Usage

```python
from datasets import load_dataset

# Load a specific benchmark
ds = load_dataset("siyich/spacetools-eval-benchmarks", "robospatial")
print(f"RoboSpatial: {len(ds['train'])} samples")

# Load all benchmarks
for bench in ["robospatial", "reflocation", "refplacement", "refunseen",
"boppose", "bopgrasp", "blinkdepth", "cvb2drelation", "cvb3ddepth"]:
ds = load_dataset("siyich/spacetools-eval-benchmarks", bench)
print(f"{bench}: {len(ds['train'])} samples")
```

## Direct Parquet Usage (with verl)

```bash
# Download specific benchmark
huggingface-cli download siyich/spacetools-eval-benchmarks data/boppose.parquet --repo-type dataset

# Use in eval config
# data.val_files=path/to/boppose.parquet
```