nv-njb/CRPE

---
license: cc-by-4.0
task_categories:
- multiple-choice
- visual-question-answering
language:
- en
pretty_name: CRPE (Circular Re-Position Evaluation)
size_categories:
- 1K<n<10K
configs:
- config_name: default
data_files:
- split: test
path: data/test-*.parquet
---

# CRPE — re-hosted for lmms-eval

A bundled-image re-host of [`OpenGVLab/CRPE`](https://huggingface.co/datasets/OpenGVLab/CRPE) prepared for upstream-friendly use with [`EvolvingLMMs-Lab/lmms-eval`](https://github.com/EvolvingLMMs-Lab/lmms-eval).

## What changed vs. `OpenGVLab/CRPE`

The original repo ships the `crpe_relation.jsonl` annotation file alongside 544 abnormal-scene images in `abnormal_images/`. The remaining 5,400 question records reference **COCO val2017 images by relative path** — but those JPEGs are not part of the HF repo, so out-of-the-box `load_dataset(...)` cannot resolve them.

This re-host:

1. Bundles all 1,081 unique referenced images (537 from COCO val2017 + 544 from `abnormal_images/`) into the parquet itself, encoded as JPEG bytes under an `Image()` feature.
2. Preserves every question record, choice, target, and category from the original `crpe_relation.jsonl`.

Result: a self-contained parquet that loads end-to-end via standard `load_dataset` — no extra COCO download needed.

## Schema

| Field | Type | Notes |
|------------------|------------------|----------------------------------------------------|
| `question_id` | `string` | Original `question_id` from the source jsonl |
| `image` | `Image()` | JPEG-encoded image |
| `text` | `string` | Question + A./B./C./D. options + final instruction |
| `correct_option` | `string` | A / B / C / D |
| `category` | `string` | One of `subject`, `predicate`, `object` |

## Stats

- 7,576 test items, single `test` split.
- 1,081 unique images: 537 from COCO val2017, 544 from `abnormal_images/`.
- Category distribution: `subject` 3,176, `predicate` 2,176, `object` 2,224.
- Parquet payload: ~1.0 GB across 4 shards.

## License

This re-host inherits its licensing from the original sources:

- **CRPE annotations** — © OpenGVLab, released under the original repository's license (see [`OpenGVLab/CRPE`](https://huggingface.co/datasets/OpenGVLab/CRPE)).
- **COCO val2017 images** — © COCO Consortium, [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).
- **Abnormal scene images** — bundled from `OpenGVLab/CRPE`.

## Citation

```bibtex
@article{wang2024crpe,
title={The All-Seeing Project V2: Towards General Relation Comprehension of the Open World},
author={Wang, Weiyun and others},
journal={arXiv preprint arXiv:2402.19474},
year={2024}
}
```