mhpd-dpo-v0

# MHPD-DPO v0.4 — Merge History Preference Distillation dataset A pairwise preference dataset for **DPO fine-tuning of paper-recommendation rerankers**, generated from team git merge history via the IPD (Implicit Preference Distillation) framework. First instantiation of the broader IPD method (paper: *Code Is Context: Tuning Coding Agents via Implicit Preference Distillation*). ## What's in it - **4,418 pairwise preference records** across two Remyx projects: - `VQASynth`: 1,418 pairs, each prepended with a sampled subset of VQASynth's `ExperimentHistory`. - `spacethinker`: 3,000 pairs, each prepended with a sampled subset of the SpaceThinker HF model card (some records also include an "upstream lineage" block of VQASynth experiments, reflecting that SpaceThinker is trained on VQASynth-generated data). - Derived from **209 unique arXiv papers**. - **2,242 unique prompts** in the dataset. ## DPO setup (read this before training) DPO operates at the **completion-likelihood** level. The model sees `prompt + chosen` in one forward pass and `prompt + rejected` in another. Pairwise comparison happens *inside the loss*. The completion is one paper formatted as `Title: ...\n\nAbstract: ...`. At inference: ```python score = logprob(paper_text | prompt) # higher = more team-aligned ``` O(N) per candidate pool. ## v0.4: parallel context for both interests Each record now begins with a **retrieval-augmented context block** appropriate to its interest: ### VQASynth records 2-3 sampled experiments from `vqasynth_experiment_history.json` (the extracted `ExperimentHistory`). Each experiment lists name, date, type, chain membership, and a 1-paragraph description. Sample is deterministic per record (seeded by record index). ### spacethinker records Sampled blocks from the [SpaceThinker model card](https://huggingface.co/remyxai/SpaceThinker-Qwen2.5VL-3B). Always includes the *Overview* anchor; samples 1-2 additional blocks from `capabilities`, `training_data`, `training_config`, `evaluation`, `limitations`, `research_direction`, or `upstream_lineage` (a sample of VQASynth experiments, since SpaceThinker is trained on VQASynth-generated data — making the data-pipeline relationship explicit when this block is picked). This parallel structure means the model learns one pattern — "use the prepended context to decide team alignment" — applied to two different sources (commit history for VQASynth, model card for spacethinker). At inference, the same model can serve any team by injecting their appropriate context type into the prompt. ## Example records ### VQASynth (with sampled experiment history) ``` Recently shipped by the VQASynth team (sample of their experiment history): - 2024-02-21 — Initial project setup with Docker-based pipeline [General] Initializes the project as a Python package and introduces a Docker Compose pipeline for processing, inspired by the work in the SpatialVLM paper. - 2024-11-17 — Add Gradio demo application [General] Creates a Gradio application to showcase and interactively test the VQASynth pipeline, allowing users to input an image and see the generated 3D models and scene captions. - 2025-03-30 — Add optional R1 reasoning data format [Data Curation] Adds a new pipeline (`spatialvqa_reasoning.yaml`) with a stage to format the generated VQA data into a 'chain-of-thought' or R1 reasoning style, enhancing model training for step-by-step reasoning. Suggest a high-impact paper that aligns with the research goals of the VQASynth team. ``` ### spacethinker (with model-card sample + upstream VQASynth lineage) ``` Context for the SpaceThinker project (model card highlights, sampled): - Overview: SpaceThinker-Qwen2.5VL-3B is a reasoning vision-language model for quantitative spatial reasoning, finetuned from UCSC-VLAA/VLAA-Thinker-Qwen2.5VL-3B (3.75B params, Qwen2.5-VL architecture) via LoRA. The model targets embodied-AI applications: planning and navigation in 3D space (robotics, drones). - Upstream lineage (VQASynth pipeline that produced SpaceThinker's training data has recently shipped): - 2024-10-08 — Upgrade segmentation with Florence and SAM2 [Research; iteration_chain_key: localization_component, iter #1] Replaces the ClipSeg-based region proposal with the Florence model for richer bounding box descriptions, and upgrades the segmentation model to the faster SAM2. - 2024-10-09 — Add optional CLIP-based content filtering [Data Curation] Introduces an optional pipeline stage to filter image samples based on CLIP tags, allowing users to include or exclude images based on content. - 2025-03-30 — Add optional R1 reasoning data format [Data Curation] Adds a new pipeline (`spatialvqa_reasoning.yaml`) with a stage to format the generated VQA data into a 'chain-of-thought' or R1 reasoning style, enhancing model training for step-by-step reasoning. - Training config: LoRA finetune (rank 128, alpha 256, target modules q_proj/v_proj) over the VLAA-Thinker base. AdamW lr=2e-5, batch size 1, 3 epochs, max input length 1024 tokens. The spacethinker team is looking to expand their theo ``` `chosen` and `rejected` follow the prompt as `Title: ...\n\nAbstract: ...` formatted papers. ## Schema ```python { "prompt": "<context block + single-paper-framing template>", "chosen": "Title: ...\n\nAbstract: ...", "rejected": "Title: ...\n\nAbstract: ...", "interest": "VQASynth" | "spacethinker", "chosen_arxiv_id": "2604.21461", "rejected_arxiv_id": "2604.11331", "utility_gap": 0.00097, "chosen_utility": 0.00105, "rejected_utility": 0.00008, "template_idx": 3, "template_version": "v4-vqasynth-history" | "v4-spacethinker-modelcard", "history_context": "vqasynth_sampled" | "spacethinker_modelcard_sampled", "history_head_sha": "<sha of VQASynth ExperimentHistory snapshot>", } ``` ## Caveats - **GP teacher source.** The PairwiseGP that scored all 209 papers was fit on **VQASynth**'s experiment history. spacethinker pairs inherit that teacher as a related-project proxy (justified by the data-generation lineage). Once SpaceThinker has enough internal history of its own (e.g., as it accumulates training runs / eval cycles / paper-derived improvements), regenerate the spacethinker GP teacher and re-score. - **Single-source-per-interest training.** With only one team per interest seen during training, the model learns to *use* a context block but only generalizes across the in-distribution variation (sampled subsets). True multi-team generalization needs more teams' histories — that's REMYX-49's job. - **Prompt length.** Each context block is ~300-500 tokens. Set `MAX_PROMPT_LEN=1536` and `MAX_LEN=4096` in the TRL training script. - Absolute GP utility values are tightly clustered (range ~1.8e-3). DPO consumes binary preferences, not magnitudes, so this is workable. ## Intended use - Fine-tune small (1–7B) open-weight models as alignment-aware rerankers via `trl.DPOTrainer` + LoRA. - At inference, inject the target project's context (extracted `ExperimentHistory`, or model card highlights, or any other team-specific signal) into the prompt, then compute `logprob(paper | prompt)` per candidate. Sort descending. ## Method context - **Framework:** Implicit Preference Distillation (IPD) — recovering preference signal from engineering artifacts that exist for reasons other than alignment. - **This dataset:** MHPD — IPD instantiated on git merge order + iteration chains, with retrieval-augmented conditioning (v0.3 added it for VQASynth; v0.4 adds parallel conditioning for spacethinker). - **Paper:** *Code Is Context: Tuning Coding Agents via Implicit Preference Distillation* (in prep). ## Changelog - **v0**: 4,418 records, 2 unique prompts. - **v0.1**: 24 unique prompts via 12 paraphrased templates × 2 interests. - **v0.2**: 12 templates rewritten as single-paper framing (DPO-format-correct). - **v0.3**: VQASynth records prepended with sampled `ExperimentHistory` context. - **v0.4 (current)**: spacethinker records prepended with sampled SpaceThinker model-card context — including an optional "upstream lineage" block of VQASynth experiments since SpaceThinker is trained on VQASynth-generated data. 2,242 unique prompts.