refspatial

# RefSpatial Pointing Data Preparation This repository contains the scripts and processed files needed to build the RefSpatial pointing training set from the original RefSpatial raw data. The conversion has two stages: 1. `convert_to_files.py`: raw RefSpatial JSON/media -> `annotations.jsonl` and copied media 2. `sharegpt_converter.py`: `annotations.jsonl` -> final ShareGPT `qa_sharegpt.jsonl` `qa_sharegpt.jsonl` is the file used by training. ## 1. Download And Place Raw Data Download the original RefSpatial dataset from ModelScope/Hugging Face according to your environment policy. The raw data is not committed to this repository. Expected workspace layout: ```text /mnt/nas/datasets5/refspatial/ ├── raw_data/ │ ├── 2D/ │ │ ├── reasoning_template_qa.json │ │ ├── image/ │ │ └── depth/ │ ├── 3D/ │ │ ├── reasoning_template_qa.json │ │ ├── vacant_qa.json │ │ ├── image/ │ │ └── depth/ │ └── Simulator/ │ ├── metadata.json │ ├── image/ │ └── depth/ └── prepare_data/ └── refspatial/ # this git repository ``` If your raw data lives elsewhere, pass that path with `--raw-root`. ## 2. Environment The scripts require Python plus `ijson` and `Pillow`. ```bash pip install ijson pillow ``` In the current workspace, use the existing virtual environment from this repo: ```bash cd /mnt/nas/datasets5/refspatial/prepare_data/refspatial ../../venv/bin/python convert_to_files.py --help ../../venv/bin/python sharegpt_converter.py --help ``` ## 3. Repository Layout ```text . ├── README.md ├── convert_to_files.py ├── sharegpt_converter.py ├── pack_shards.py └── refspatial_ds/ ├── annotations.jsonl ├── qa_sharegpt.jsonl ├── images/ # local unpacked media; usually not uploaded ├── depths/ # local unpacked media; usually not uploaded ├── images_shards/ # recommended upload format └── depths_shards/ # recommended upload format ``` ## 4. Generate Intermediate Annotations Run from this repository root: ```bash cd /mnt/nas/datasets5/refspatial/prepare_data/refspatial ../../venv/bin/python convert_to_files.py \ --raw-root ../../raw_data \ --out-root ./refspatial_ds \ --overwrite ``` This creates: ```text refspatial_ds/ ├── annotations.jsonl ├── images/ └── depths/ ``` `annotations.jsonl` is the intermediate layer. It keeps sample metadata, selected QA pairs, normalized ground-truth points, image/depth paths, and previous conversation turns needed for prompt construction. ### Task Filtering The converter is intentionally strict. For `2D/reasoning_template_qa.json` and `3D/reasoning_template_qa.json`, it keeps only clear pointing tasks, for example: - `Point to ...` - `Point out ...` - `Select a point on ...` - `Choose the point of ...` - `Pinpoint ...` - `Locate a point ...` It excludes camera/view, distance/depth, relation/comparison, action-planning, and broad localization prompts such as `Where is ... located?`. For `3D/vacant_qa.json`, all non-choice QA turns with coordinate answers are kept as vacant-space pointing tasks. ## 5. Generate Final Training JSON ```bash ../../venv/bin/python sharegpt_converter.py \ --annotations ./refspatial_ds/annotations.jsonl \ --out ./refspatial_ds/qa_sharegpt.jsonl ``` The final training file is: ```text ./refspatial_ds/qa_sharegpt.jsonl ``` Media paths in `qa_sharegpt.jsonl` are relative to `refspatial_ds`, so keep the JSONL file and media directories or media shards together. ## 6. Small Tests Small end-to-end sample: ```bash ../../venv/bin/python convert_to_files.py \ --raw-root ../../raw_data \ --out-root ./refspatial_ds_smoke \ --max-samples 20 \ --overwrite \ --debug-dump 5 ../../venv/bin/python sharegpt_converter.py \ --annotations ./refspatial_ds_smoke/annotations.jsonl \ --out ./refspatial_ds_smoke/qa_sharegpt.jsonl ``` Only process `3D/vacant_qa.json`: ```bash ../../venv/bin/python convert_to_files.py \ --raw-root ../../raw_data \ --out-root ./refspatial_vacant_ds \ --only-subsets vacant_qa \ --max-samples 100 \ --overwrite \ --debug-dump 5 ``` Multiple subsets can be comma-separated: ```bash ../../venv/bin/python convert_to_files.py \ --raw-root ../../raw_data \ --out-root ./refspatial_subset_ds \ --only-subsets reasoning_template_qa,vacant_qa \ --overwrite ``` ## 7. Final ShareGPT Schema Each line in `qa_sharegpt.jsonl` uses only these top-level fields: ```json { "id": "refspatial:sample_id:turn_id", "conversations": [ {"from": "system", "value": "You are a helpful assistant."}, {"from": "human", "value": "<image>...prompt..."} ], "images": ["images/example.jpg"], "videos": [], "audios": [], "data_source": "refspatial", "ground_truth": { "points": [ {"point_2d": [500, 500], "label": "target"} ] }, "extra_info": { "source": "3D", "subset": "vacant_qa", "task_type": "pointing" } } ``` Notes: - `conversations` contains only `system` and `human` by default. - The assistant answer is not written by default, avoiding ground-truth conflicts with parsers that extract the last assistant response. - Ground truth is stored in the top-level `ground_truth`. - Reward/debug metadata is stored in `extra_info`. - The number of `<image>` tokens matches the number of paths in `images`. The human prompt requires: ```text <thinking> ... </thinking> <answer> ... </answer> ``` The answer inside `<answer>` must be a JSON array: ```json [ {"point_2d": [x, y], "label": "target"} ] ``` Coordinates are normalized to `0-1000` relative to image width and height. ## 8. Context Handling Some pointing tasks appear in multi-turn conversations. `convert_to_files.py` stores previous turns in `annotations.jsonl`. `sharegpt_converter.py` uses a conservative default: - keep only previous QA pairs related to the same target terms; - skip camera, distance, relation, comparison, and weak localization context; - keep at most 2 related QA pairs by default; - if no related context is found, keep at most 1 recent non-negative QA pair. Useful options: ```bash # Disable previous conversation context. ../../venv/bin/python sharegpt_converter.py \ --annotations ./refspatial_ds/annotations.jsonl \ --out ./refspatial_ds/qa_sharegpt.jsonl \ --no-context # Keep at most one related context QA pair. ../../venv/bin/python sharegpt_converter.py \ --annotations ./refspatial_ds/annotations.jsonl \ --out ./refspatial_ds/qa_sharegpt.jsonl \ --max-context-pairs 1 # Use recent context instead of related-object context. ../../venv/bin/python sharegpt_converter.py \ --annotations ./refspatial_ds/annotations.jsonl \ --out ./refspatial_ds/qa_sharegpt.jsonl \ --context-strategy all \ --max-context-pairs 2 ``` For offline debugging only, `--include-assistant-answer` can write an assistant answer. Do not use it for default training unless your parser is configured to avoid ground-truth conflicts. ## 9. Reduce File Count With Shards Uploading many individual media files is slow and can stress Git/LFS and the ModelScope web UI. The recommended upload format is: - upload `annotations.jsonl` and `qa_sharegpt.jsonl`; - pack `images/` into tar shards; - pack `depths/` into tar shards; - upload `images_shards/` and `depths_shards/`; - do not commit the unpacked `images/` and `depths/` directories unless needed. Create shards: ```bash ../../venv/bin/python pack_shards.py ``` By default this writes: ```text refspatial_ds/images_shards/images_00000.tar refspatial_ds/depths_shards/depths_00000.tar ... ``` Restore media from shards: ```bash mkdir -p refspatial_ds/images refspatial_ds/depths for f in refspatial_ds/images_shards/*.tar; do tar -xf "$f" -C refspatial_ds/images; done for f in refspatial_ds/depths_shards/*.tar; do tar -xf "$f" -C refspatial_ds/depths; done ``` ## 10. Quick Validation ```bash ../../venv/bin/python - <<'PY' import json from pathlib import Path p = Path("refspatial_ds/qa_sharegpt.jsonl") allowed = {"id", "conversations", "images", "videos", "audios", "data_source", "ground_truth", "extra_info"} with p.open(encoding="utf-8") as f: for _, line in zip(range(5), f): sample = json.loads(line) assert set(sample) == allowed prompt = sample["conversations"][1]["value"] assert prompt.count("<image>") == len(sample["images"]) assert all(tag in prompt for tag in ["<thinking>", "</thinking>", "<answer>", "</answer>"]) assert sample["ground_truth"]["points"] print(sample["id"], sample["images"], sample["ground_truth"]["points"]) PY ```