roast_dataset

# Roast Dataset A synthetic roast/humor dataset generated via a "writers' room" pipeline. 22 life-scenario categories, ~20K examples, each with 4 candidates and editor scores. 通过"编剧室"模式生成的合成吐槽数据集。22 个生活场景类别，约 2 万条数据，每条包含 4 个候选段子和编辑评分。 ## Dataset Summary This dataset was created to explore whether a small language model (305M) could learn to generate humorous roast-style responses. Given the model's limited capacity for math/code reasoning, I chose comedy as a more achievable and entertaining post-training objective. The dataset was generated entirely by DeepSeek-V4-Flash using a multi-stage pipeline: 1. **Topic generation** — Produce ~23K concrete roast topics across 22 life-scenario categories (workplace, tech, social, food, fitness, etc.) 2. **Response generation** — A comedy-specific prompt generates 4 candidate roast responses per topic 3. **Editor selection** — A "comedy editor" LLM scores all 4 candidates on observation, punchline, originality, economy, and overall; selects the winner All responses are written in observational stand-up comedy style, 30-50 words each, concise and natural. ## Data Format Each example in the JSONL file contains: | Field | Description | | ----------------- | --------------------------------------------------------- | | `id` | Unique example ID | | `category` | Life-scenario category (ai, workplace, social, food, etc.)| | `user_input` | The roast topic/prompt | | `candidates` | 4 candidate responses (index, response text) | | `editor.scores` | Per-candidate scores (observation, punchline, originality, economy, overall, 1-10) | | `editor.ranking` | Ranked candidate indices (best to worst) | | `winner_response` | The editor-selected best response text | <details> <summary>Click to expand a full example</summary> ```json { "id": "ai_0017", "category": "ai", "user_input": "AI-generated music that sounds like a washing machine having a religious experience", "candidates": [ { "index": 0, "response": "Your music sounds like a washing machine that's figured out the meaning of life mid-cycle." }, { "index": 1, "response": "Your AI music sounds like an industrial dishwasher dreaming about God." }, { "index": 2, "response": "That ambient track isn't \"evolving\" — it's just your laptop fan having a panic attack while a synthesizer forgets how to resolve a chord." }, { "index": 3, "response": "You've created music that sounds like an industrial appliance hallucinating Gregorian chants during the spin cycle." } ], "editor": { "ranking": [2, 0, 3, 1], "scores": { "0": {"observation": 7, "punchline": 8, "originality": 6, "economy": 8, "overall": 7}, "1": {"observation": 6, "punchline": 5, "originality": 5, "economy": 7, "overall": 5}, "2": {"observation": 9, "punchline": 10, "originality": 9, "economy": 10, "overall": 10}, "3": {"observation": 8, "punchline": 9, "originality": 8, "economy": 9, "overall": 9} }, "winner_index": 2 }, "winner_response": "That ambient track isn't \"evolving\" — it's just your laptop fan having a panic attack while a synthesizer forgets how to resolve a chord." } ``` </details> ## Preprocessed `.pt` Files The `train.pt` and `valid.pt` files are tokenized splits (90/10, seed=42) ready for Weighted SFT training. Each contains: - `input_ids` — Tokenized chat-formatted sequences (max length 1024) - `completion_mask` — Bool mask for assistant-only loss - `scores` — Editor overall scores for each candidate - `group_ids` — Group indices (4 candidates per group) - `is_winner` — Bool flag for the winning response ### Chat Template ``` <|system|>\nYou are a witty roast comedian.\nWrite concise observational roasts.\nAvoid explaining the joke.\n<|user|>\n{user_input}\n<|assistant|>\n{response}<|endoftext|>\n ``` ## Intended Uses - **Weighted SFT** — Primary use case; train models with score-weighted cross-entropy - **Preference Optimization** — 4 candidates per topic enable DPO/SimPO/GRPO training - **Reward Modeling** — Editor scores provide explicit quality signals - **Comedy Generation Research** — Explore what quality small models can achieve with curated data ## Limitations - All responses are generated by DeepSeek-V4-Flash, not human-written - The "comedy editor" scores are LLM-generated approximations, not human judgments - English only; humor is culturally specific ## Source Code See [github.com/YHanchao/mini_dsv4](https://github.com/YHanchao/mini_dsv4) for the preprocessing script (`scripts/preprocess_roast.py`) and training code.