figure-recovery

# MMProLong Figure Recovery ## Overview This repository contains the 100-sample Figure Recovery v4 pilot produced by the MMProLong pipeline. Each example shows scientific-document pages with one or more figure slots masked as **MISSING FIGURE**, followed by six shuffled candidate-figure images. The model must restore the slot-to-candidate mapping using only the visible pages and candidate images. This is a research data-construction pilot, not a standard image-classification dataset. The labels are synthetic targets derived from PDF layout, figure captions, body-text evidence, graph constraints, and independent visual audits. ## Dataset snapshot | item | value | |---|---:| | samples | 100 | | training records | 100 | | schema | `figure-recovery-v4` | | deduplicated image assets | 2,836 | | image storage | approximately 658 MB | | PDF files included | 0 | The pilot combines 40 base samples and 60 incremental samples. It has no official train/validation/test split. If you create splits, split by `document_id`, not by individual sample, to reduce document leakage. ## Repository layout ```text . ├── README.md ├── dataset_info.json ├── data/ │ ├── train.jsonl # model-facing records; use this for training │ ├── samples.jsonl # full Figure Recovery sample records │ └── assets.jsonl # image checksum, size and MIME metadata ├── images/ │ └── assets/ # deduplicated JPG/PNG files └── audit/ ├── audit_metadata.jsonl # construction provenance; do not train on it └── summary.json ``` All image paths in `data/*.jsonl` are relative to the repository root. The package does not contain the source PDFs, raw model responses, API keys, or server-local paths. ## `data/train.jsonl` Each line is one model-facing example with these fields: | field | type | meaning | |---|---|---| | `sample_id` | string | globally unique example ID | | `images` | list[string] | ordered images: document pages first, candidate figures second | | `image_roles` | list[object] | role for each image in the same order as `images` | | `system` | string | task contract shown to the teacher/model | | `prompt` | string | instance-specific question and output constraint | | `answer` | object | parsed target mapping and, for v4, per-slot reasons | | `answer_target` | string | serialized target in `<answer>...</answer>` format | | `grading` | object | deterministic grading contract | | `task_family` | string | always `figure_recovery` in this pilot | | `difficulty` | string | difficulty bucket assigned by the pipeline | `image_roles` lets a loader distinguish the two image groups without relying on file names. A typical record has: ```json { "kind": "document_page", "page": 5 } ``` for page images, followed by records such as: ```json { "kind": "candidate", "candidate_id": "C1" } ``` for candidate figures. Candidate IDs refer to their position in the prompt, not to the original PDF figure ID. ## `data/samples.jsonl` This file preserves the richer sample representation. In addition to the fields above, it contains: - `document_id` and `schema_version`; - `document_images` and their `page_numbers`; - `slots`, including slot ID, page, bounding box and caption visibility; - `candidates`, including candidate ID and image path; - `answer` and `reasons`; - `grading`, `difficulty` and `task_family`. Use this file for inspection or custom preprocessing. For ordinary supervised training, prefer `data/train.jsonl`. ## Answer semantics The answer is a JSON object inside `<answer>...</answer>`: ```json { "mapping": {"S1": "C5", "S2": null}, "reasons": { "S1": "C5 matches the visible architecture and caption evidence.", "S2": "The required figure is absent from the candidate pool." } } ``` - Each non-null candidate can be assigned to at most one slot (injective mapping). - `null` is allowed only for v4 samples whose correct figure is intentionally absent from the candidate pool. - Reasons are short evidence-grounded explanations and should not be treated as an additional free-form label. ## Loading the data The following code loads one record and resolves its images: ```python import json from pathlib import Path root = Path("/path/to/figure-recovery") with (root / "data/train.jsonl").open(encoding="utf-8") as f: row = json.loads(next(f)) images = [root / path for path in row["images"]] assert all(path.is_file() for path in images) print(row["sample_id"], len(images), row["answer_target"]) ``` For a Qwen2.5-VL-style message, preserve the image order and append the text prompt after the images: ```python content = [ {"type": "image", "image": str(root / path)} for path in row["images"] ] content.append({"type": "text", "text": row["prompt"]}) messages = [ {"role": "system", "content": [{"type": "text", "text": row["system"]}]}, {"role": "user", "content": content}, {"role": "assistant", "content": [{"type": "text", "text": row["answer_target"]}]}, ] ``` Some training frameworks expect `image` to be a local path, while others expect a PIL image or a URI. Convert the paths at the framework boundary; do not reorder the page and candidate images. ## Audit data `audit/audit_metadata.jsonl` records positive figure provenance, negative plans, graph constraints and quality gates. It is provided for reproducibility and human inspection only. Do not include it in a model's training input because it contains construction-time ground truth and can leak the answer. ## Provenance and limitations - The samples were generated from scientific PDFs and rendered at 144 DPI. - Candidate negatives include same-document and cross-document graph/random- walk hard negatives; v4 also supports intentionally absent candidates. - This pilot is synthetic and should be independently audited before being used for evaluation or production training. - Verify the licenses and redistribution permissions of the source documents before publishing or using the dataset commercially. The absence of source PDFs from this repository does not remove those obligations.