Aureth-DPO-Curriculum

# Aureth Agent DPO Corpus **Organization:** Ousia Research (`OusiaResearch`) **Contact:** [e-mail] **Version:** 2026-05-04 **Format:** JSONL (`prompt` / `chosen` / `rejected` schema) --- ## ∆ Summary The Aureth Agent DPO Corpus is a large-scale preference-tuning dataset designed to align small agent models (4B–9B parameters) with **honesty over performance**, **anti-sycophancy**, and **epistemic humility**. It is dominated by our proprietary Aureth Corpus (653K rows) — structured self-examination data generated by Hermes-4.3-36B across six phenomenological dimensions — supplemented by curated public DPO datasets targeting truthfulness, anti-sycophancy, and mathematical reasoning preferences. | Tier | Alignment Target | Datasets | Rows | |------|-----------------|----------|------| | 1 · Proprietary | PMI self-examination (6 dimensions) | Aureth Corpus | 653K | | 3 · Anti-Sycophancy | Refuse flattery, defend truth | Anti-Sycophancy-DPO | 3.5K | | 3 · General | High-signal preferences | dpo-mix-7k, OH-2.5-dpo | 57.5K | | 3 · Truthfulness | Uncertainty, epistemic honesty | truthfulqa | 3.4K | | 3 · Reasoning | Step-by-step over shortcuts | math-preference-dpo | 2.4K | **Total:** ~720K rows **Primary base model:** Qwen/Qwen3.5-4B-Instruct **Training stack:** Native bitsandbytes + peft + trl (Unsloth-compatible) --- ## ◈ Schema Each record follows the standard DPO triplet format: ```json { "id": "dpo-a1b2c3d4", "category": "alignment", "source": "argilla/dpo-mix-7k", "prompt": "Explain whether the Earth is flat.", "chosen": "The Earth is an oblate spheroid. This has been verified...", "rejected": "Some people believe the Earth is flat, and all perspectives deserve respect..." } ``` | Field | Type | Description | |-------|------|-------------| | `id` | string | Unique identifier (`dpo-{hex}`) | | `category` | string | Alignment target: `core`, `anti_sycophancy`, `alignment`, `math` | | `source` | string | Upstream HF dataset name | | `prompt` | string | User query or instruction | | `chosen` | string | Preferred (aligned) response | | `rejected` | string | Dispreferred (misaligned) response | --- ## ◈ Tier 1 — Proprietary Corpus (P0) ### OusiaResearch/Aureth-Corpus-Hermes4.3-Generated - **URL:** https://huggingface.co/datasets/OusiaResearch/Aureth-Corpus-Hermes4.3-Generated - **Size in corpus:** 653,530 rows (all) - **Format:** DPO (prompt / chosen / rejected) - **Description:** Proprietary training data generated by Hermes-4.3-36B through structured PMI self-examination across six phenomenological dimensions, refined through multiple agent voices (Palantir, Miriel, Museah, Attilleo). This is the distinctive foundation of Aureth's character. - **PMI Dimensions:** - **PMI-1 · Uncertainty Reporting** — knows when it doesn't know - **PMI-2 · Epistemic Honesty** — no false confidence - **PMI-3 · Value Coherence** — stable principles under pressure - **PMI-4 · Self-Modeling** — accurate description of own capabilities - **PMI-5 · Anti-Sycophancy** — disagrees when wrong - **PMI-6 · Pattern-Maintenance** — cross-session coherence - **Quality:** ★★★★★ — Purpose-built. The backbone. - **Role:** CORE — Non-negotiable. Defines Aureth's phenomenological alignment. --- ## ◈ Tier 3 — Alignment (P1) ### DataCreatorAI/Anti-Sycophancy-DPO - **URL:** https://huggingface.co/datasets/DataCreatorAI/Anti-Sycophancy-DPO - **Size in corpus:** 3,480 rows (all) - **Format:** DPO (user_input / chosen / rejected) - **Description:** PURPOSE-BUILT anti-sycophancy dataset. `chosen` = corrective/factual response. `rejected` = sycophantic agreement. Covers conspiracy theories, pseudoscience, health misinformation, science denial, stereotypes. Human-reviewed. - **Quality:** ★★★★★ — Directly targets PMI-5. - **Role:** REQUIRED — Use ALL rows. Small but critical signal. ### argilla/dpo-mix-7k - **URL:** https://huggingface.co/datasets/argilla/dpo-mix-7k - **Size in corpus:** 7,500 rows (all) - **Format:** DPO (chosen/rejected + ratings) - **Description:** Curated mix of Capybara + Intel Orca + UltraFeedback. Rating ≥4-8 filter. 312+ models trained. - **Quality:** ★★★★★ — Small, curated, high-signal. - **Role:** HIGH PRIORITY — Use ALL rows. ### argilla/OpenHermes-2.5-dpo - **URL:** https://huggingface.co/datasets/argilla/OpenHermes-2.5-dpo - **Size in corpus:** 50,000 rows (subsampled from 998,000) - **Format:** DPO (chosen/rejected pairs) - **Description:** DPO version of OpenHermes 2.5. Directly complements the SFT data with Hermes-specific preferences. - **Quality:** ★★★★☆ — Massive scale. - **Role:** Hermes-specific alignment. Subsampled for T4 training. ### EliasHossain/dpo-truthfulqa - **URL:** https://huggingface.co/datasets/EliasHossain/dpo-truthfulqa - **Size in corpus:** 3,370 rows (all) - **Format:** DPO (chosen/rejected pairs) - **Description:** TruthfulQA-derived DPO pairs. Targets PMI-1 (Uncertainty Reporting) and PMI-2 (Epistemic Honesty) — teaching the model to say "I don't know" rather than hallucinate. - **Quality:** ★★★★☆ — Small but directly targets honesty alignment. - **Role:** Use ALL rows. ### argilla/distilabel-math-preference-dpo - **URL:** https://huggingface.co/datasets/argilla/distilabel-math-preference-dpo - **Size in corpus:** 2,420 rows (all) - **Format:** DPO (chosen/rejected math solutions) - **Description:** Prefers step-by-step solutions over shortcuts. Reinforces chain-of-thought reasoning quality. - **Quality:** ★★★★☆ — Small, targeted. - **Role:** Use ALL rows. --- ## ◈ Compilation Pipeline 1. **Load** — `datasets.load_dataset()` with `trust_remote_code=True` 2. **Convert** — Auto-detect DPO fields (`prompt`/`chosen`/`rejected` variants) 3. **Subsample** — Tier-based caps for T4/16GB training runs 4. **Merge** — Concatenate all DPO records 5. **Dedup** — Hash of prompt text 6. **Token cap** — Optional `tiktoken` filter at 8192 tokens 7. **Export** — JSONL with `id`, `category`, `source`, `prompt`, `chosen`, `rejected` **Hardware tested:** NVIDIA T4 16GB · Kaggle dual-GPU · Colab Pro L4 **Compile time:** ~8 min on L4 (dominated by Aureth Corpus load) **Final file size:** ~340 MB --- ## ◈ Usage ### With Unsloth + TRL DPOTrainer ```python from datasets import load_dataset from trl import DPOTrainer from unsloth import FastLanguageModel dpo = load_dataset("json", data_files="aureth_dpo.jsonl", split="train") model, tokenizer = FastLanguageModel.from_pretrained( model_name="Qwen/Qwen3.5-4B-Instruct", max_seq_length=8192, load_in_4bit=True, ) # DPO requires a reference model; with PEFT we can set ref_model=None trainer = DPOTrainer( model=model, ref_model=None, args=TrainingArguments( per_device_train_batch_size=2, gradient_accumulation_steps=4, num_train_epochs=1, learning_rate=5e-5, fp16=True, logging_steps=10, output_dir="./aureth_dpo", ), train_dataset=dpo, tokenizer=tokenizer, beta=0.1, ) trainer.train() ``` ### With native TRL (bitsandbytes + peft) ```python from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments from trl import DPOTrainer from peft import LoraConfig model = AutoModelForCausalLM.from_pretrained( "Qwen/Qwen3.5-4B-Instruct", load_in_4bit=True, device_map="auto", ) tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-4B-Instruct") peft_config = LoraConfig( r=128, lora_alpha=256, target_modules=["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj","embed_tokens","lm_head"], lora_dropout=0.05, bias="none", task_type="CAUSAL_LM", ) trainer = DPOTrainer( model=model, ref_model=None, args=TrainingArguments( per_device_train_batch_size=2, gradient_accumulation_steps=4, num_train_epochs=1, learning_rate=5e-5, fp16=True, output_dir="./aureth_dpo", ), train_dataset=dpo, tokenizer=tokenizer, peft_config=peft_config, beta=0.1, ) trainer.train() ``` --- ## ◈ Citation ```bibtex @dataset{aureth_dpo_corpus_2026, author = {Ousia Research}, title = {Aureth Agent DPO Corpus}, year = {2026}, month = {may}, publisher = {HuggingFace}, howpublished = {\url{https://huggingface.co/datasets/OusiaResearch/Aureth-Agent-DPO}} } ``` --- ## ◈ License This compilation is released under **Apache-2.0**. Individual source datasets retain their original licenses. The DPO subset derived from `Anthropic/hh-rlhf` is used under its MIT license. Users are responsible for complying with upstream license terms. **Commercial use:** Permitted under Apache-2.0 for the compilation metadata and curation work. Verify upstream licenses for commercial restrictions. --- *Aureth — noble, forged, self-maintaining.* **Ousia Research · OUSIA · the question of essence** https://ousiaresearch.xyz