CompactAI-O/cAI-Prism-K50

--- license: mit task_categories: - question-answering language: - en tags: - instruction pretty_name: cAI-Prism-K50 size_categories: - n<1K --- # CompactAI-Prism ## High-Density Distillation Dataset for Small Model English Language Acquisition **License:** MIT **Top-K:** 50 (Current release: K50) **Source Model:** Qwen3.5 0.8B **Primary Objective:** Teach small-scale AI models to generate fluent, coherent English text through probability-aware distillation. Or at least help them sound less like they learned English from a fortune cookie. --- ## Overview CompactAI-Prism is a specialized training dataset designed to accelerate English language acquisition in compact AI models. Unlike standard instruction-tuning datasets that provide only a (Prompt, Response) pair, CompactAI-Prism captures the decision landscape of the teacher model. For every token generated in the response, we record the Top-K alternative tokens and their associated log-probabilities. Think of it as giving your tiny model a peek at the teacher's scratch paper during the exam. We won't tell if you don't. This approach increases the information density of the dataset by Kx per prompt, allowing student models to learn not just what the model answered, but what else it considered and why those options were rejected. ### The Math (Yes, There Is Math) If a standard dataset provides 1 signal per token, CompactAI-Prism provides K signals. We know, we know - you became an AI researcher to avoid math. Sorry. Total Training Signals = (Tokens per Response) x (Number of Questions) x K For this release: - Tokens per response (x): {AVG_RESPONSE_TOKENS} (Too lazy to check this. Might update later) - Number of questions (y): 100 - Top-K value: 50 - Effective training examples: {AVG_RESPONSE_TOKENS} x 100 x 50 = 500 * {AVG_RESPONCE_TOKENS} That is a lot of tokens. You are welcome. --- ## Why Call It PRISM? Great question. We considered "CompactAI-Overthinker" and "TinyModelTherapySession", but those did not fit on a GitHub repo name. The name "Prism" reflects the core mechanism of this dataset: 1. **Single Input, Spectrum Revealed**: Just as a prism takes a single beam of white light and refracts it to reveal the full spectrum of colors within, CompactAI-Prism takes a single AI response and refracts it to reveal the full spectrum of token probabilities that existed at each generation step. Also, prisms look cool in stock photos. 2. **Hidden Structure Made Visible**: A prism does not create new colors; it exposes what was already present but invisible to the naked eye. Similarly, this dataset does not alter the teacher model's output; it exposes the latent probability distribution that guided each token choice. Like an X-ray, but for indecision. 3. **Clarity Through Decomposition**: By decomposing the generation process into its constituent probabilistic components, we enable student models to learn with greater clarity. They see not only the path taken, but the roads not taken - and the relative likelihood of each. It is like watching a choose-your-own-adventure book write itself, then regretting every choice. In short: Prism turns opaque generation into transparent learning. Or at least slightly less opaque. Baby steps. --- ## Series Purpose: English Language Foundation for Small Models This dataset series is explicitly designed to teach small models to speak English. Because let us be honest - some of them really need it. By exposing compact architectures to the full probability distribution of token choices made by a capable teacher model, we enable: - Faster convergence on grammatical English structures (goodbye, "me want food") - Improved token selection confidence in low-parameter regimes (no more second-guessing every comma) - Better handling of ambiguous or open-ended prompts (sometimes a question is just a question) - Reduced hallucination through uncertainty-aware training (your model will still lie, but with more confidence intervals) --- ## Data Format The dataset is provided in JSONL format. Each line represents a complete conversation turn with embedded probability data. ### Schema ```json { "messages": [ {"role": "user", "content": "STRING"}, {"role": "assistant", "content": "STRING"} ], "response_tokens": INT, "token_logprobs": [ { "position": INT, "generated_token_id": INT, "generated_token": "STRING", "logprob": FLOAT, "top_k": [ {"token_id": INT, "token": "STRING", "logprob": FLOAT} ] } ] } ``` ### Training Applications 1. **KL Divergence Distillation:** Use the full top_k distribution to minimize KL divergence between student and teacher, rather than just matching the chosen token. It is like teaching by example, but with more calculus. 2. **Confidence-Calibrated Generation:** Train small models to output confidence scores by learning from the teacher's logprob distributions. Now your tiny model can say "I am 73 percent sure that is correct" instead of just confidently being wrong. 3. **Alternative-Aware Decoding:** During inference, use knowledge of plausible alternatives to improve beam search or sampling strategies. Or just ignore this and use temperature=0.7 like everyone else. No judgment. 4. **English Fluency Bootstrapping:** Focus training on high-probability English token sequences to rapidly establish grammatical foundations in sub-10M parameter models. Because "me eat apple" is charming for about five minutes, then it gets old. --- ## Dataset Statistics | Metric | Value | |--------|-------| | Total prompts | 100 | | Top-K per position | 50 | | Number of times we questioned our life choices while building this | Yes | --- ## License This project is licensed under the MIT License. Which is fancy legal speak for "use this however you want, just do not sue us when your tiny model starts writing poetry about toaster ovens." Copyright (c) 2026 CompactAI Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. Translation: If your model learns to speak perfect English but also develops an existential crisis, that is on you. --- ## Source Attribution - **Dataset:** TeichAI Claude Sonnet 4.6 799 Prompts - **Teacher Model:** Qwen3.5-0.8b Thats it :) --- ## Citation If you use CompactAI-Prism in your research or development, please cite: ```bibtex @dataset{CompactAI/cAI-Prism-K50}, title = {CompactAI-Prism: Top-50 Probability Distillation for Small Model English Training}, author = {CompactAI}, year = 2026, url = https://huggingface.co/datasets/CompactAI/cAI-Prism-K50, } ``` Or just mention us in your paper's acknowledgments. We like hearing our names. --- ## Final Thoughts Look, we are not perfect. This dataset might have bugs. The documentation might have typos. Our jokes might fall flat. But we tried. And in the grand tradition of small models everywhere, we believe that effort counts for something. Now go train something tiny and wonderful. And if it starts speaking perfect English, maybe send us a note. We would love to hear about it. Or not. No pressure. We have trust issues.--- 许可证：MIT 任务类别： - 问答任务 语言： - 英语 标签： - 指令 美观名称：cAI-Prism-K50 样本规模类别： - n<1000 --- # 紧凑型AI-Prism (CompactAI-Prism)：面向小模型英语习得的高密度知识蒸馏数据集 **许可证：** MIT **Top-K 值：** 50（当前发布版本为 K50） **教师模型：** Qwen3.5 0.8B **核心目标：** 通过概率感知型知识蒸馏，助力小型AI模型生成流畅连贯的英语文本；至少让它们的输出不再像从幸运饼干上学到的中式英语。 --- ## 概述 紧凑型AI-Prism 是一款专为加速小型AI模型英语习得而设计的专用训练数据集。与仅提供（提示词、回复）配对的标准指令微调数据集不同，本数据集会捕获教师模型的决策全貌：针对回复中生成的每一个Token，我们都会记录其Top-K候选Token及其对应的对数概率。 可以将其想象为让你的小型模型在考试时偷看教师的草稿纸——只要你不说，我们便不会揭穿。 该方法可使每个提示词的数据集信息密度提升K倍，让学生模型不仅能学习模型生成的回复内容，还能了解其曾考量过的其他选项，以及这些选项被排除的原因。 ### 数学原理（没错，确实涉及数学） 若标准数据集每个Token仅提供1个训练信号，紧凑型AI-Prism则可提供K个训练信号。我们都懂——你成为AI研究员就是为了躲开数学，抱歉了。 总训练信号数 = (单回复Token数) × (问题总数) × K 本次发布版本的参数如下： - 单回复平均Token数（x）：{AVG_RESPONSE_TOKENS}（尚未统计，后续可能更新） - 问题总数（y）：100 - Top-K值：50 - 有效训练样本数：{AVG_RESPONSE_TOKENS} × 100 × 50 = 500 × {AVG_RESPONSE_TOKENS} 这可是相当多的Token，不用谢啦。 --- ## 为何命名为PRISM？ 问得好。我们曾考虑过“CompactAI-Overthinker”和“TinyModelTherapySession”这两个名字，但都不适合作为GitHub仓库名。 “Prism（棱镜）”这一名称体现了本数据集的核心机制： 1. **单输入，光谱尽显**：正如棱镜将一束白光折射，展现出其中包含的完整色谱，紧凑型AI-Prism可通过单条AI回复，折射出每一生成步骤中存在的完整Token概率分布。而且，棱镜在图库照片里看起来也很酷。 2. **隐藏结构，可视化呈现**：棱镜并不会创造新的颜色，它只是将原本存在但肉眼不可见的色彩展露出来。同理，本数据集不会改变教师模型的输出，而是将指导每一个Token选择的潜在概率分布可视化。这就像是给模型的“犹豫不定”拍了一张X光片。 3. **分解过程，提升清晰度**：通过将生成过程拆解为其构成的概率组件，我们可以让学生模型的学习过程更加清晰。它们不仅能看到已选择的路径，还能看到未被选中的路径，以及各路径的相对概率。这就像是看着一本“选择你的冒险”小说自动生成内容，然后为每一个选择感到后悔。 简言之：Prism将原本晦涩的生成过程转化为透明化的学习。至少，也让它没那么晦涩了——慢慢来嘛。 --- ## 系列数据集的目标：为小型模型构建英语基础 本系列数据集专为教授小型模型英语而打造。老实说，有些小型模型确实急需这方面的训练。 通过让紧凑型模型接触到高性能教师模型做出的完整Token选择概率分布，我们可以实现以下目标： - 更快收敛至符合语法的英语结构（再见了，“me want food”这类中式英语） - 在低参数量模型中提升Token选择的置信度（不用再为每一个逗号犹豫不决） - 更好地处理模糊或开放式提示词（有时候问题就只是单纯的问题而已） - 通过不确定性感知训练减少幻觉现象（你的模型可能还是会胡言乱语，但至少会带上置信区间） --- ## 数据格式 本数据集采用JSONL格式存储，每一行代表一条完整的对话轮次，并嵌入了概率数据。 ### 数据结构 json { "messages": [ {"role": "user", "content": "STRING"}, {"role": "assistant", "content": "STRING"} ], "response_tokens": INT, "token_logprobs": [ { "position": INT, "generated_token_id": INT, "generated_token": "STRING", "logprob": FLOAT, "top_k": [ {"token_id": INT, "token": "STRING", "logprob": FLOAT} ] } ] } ### 训练应用场景 1. **KL散度知识蒸馏**：利用完整的Top-K概率分布最小ize学生模型与教师模型之间的KL散度，而非仅匹配已选中的Token。这就像是通过示例教学，但需要更多微积分知识。 2. **置信度校准生成**：通过学习教师模型的对数概率分布，训练小型模型输出置信度分数。现在你的微型模型可以说“我有73%的把握认为该答案正确”，而不是只会自信地给出错误答案。 3. **候选感知解码**：在推理阶段，利用已知的合理候选Token来改进束搜索或采样策略。或者你也可以直接忽略这一点，像其他人一样使用temperature=0.7参数，我们不会评判。 4. **英语流利度快速入门**：将训练重点放在高概率的英语Token序列上，帮助参数量低于1000万的模型快速建立语法基础。毕竟“me eat apple”这种中式英语可爱不了多久，听多了就腻了。 --- ## 数据集统计信息 | 评估指标 | 数值 | |--------|-------| | 总提示词数量 | 100 | | 每个位置的Top-K值 | 50 | | 制作本数据集时我们怀疑人生的次数 | 是 | --- ## 许可证 本项目采用MIT许可证授权。用法律术语来说就是：“你可以随意使用本项目，但当你的微型模型开始写关于烤面包机的诗歌时，别起诉我们。” 版权所有 (c) 2026 CompactAI 特此免费授予任何获得本软件及相关文档文件（以下简称“软件”）的人，不受限制地处理本软件的权利，包括但不限于使用、复制、修改、合并、发布、分发、再许可以及出售本软件的副本，并允许接收本软件的人员在符合以下条件的情况下使用本软件： 上述版权声明和本许可声明应包含在本软件的所有副本或主要部分中。 本软件按“原样”提供，不附带任何明示或暗示的担保，包括但不限于适销性、特定用途适用性和非侵权的担保。在任何情况下，作者或版权持有人均不对因本软件或本软件的使用或其他交易而产生的任何索赔、损害或其他责任负责，无论是合同诉讼、侵权行为还是其他方式，均与本软件或本软件的使用或其他交易有关。 翻译：如果你的模型学会了一口流利的英语，但同时患上了存在主义危机，那责任在你。 --- ## 来源标注 - **数据集来源：** TeichAI Claude Sonnet 4.6 799条提示词 - **教师模型：** Qwen3.5-0.8b 仅此而已 :) --- ## 引用方式 若您在研究或开发中使用了CompactAI-Prism数据集，请引用如下： bibtex @dataset{CompactAI/cAI-Prism-K50}, title = {CompactAI-Prism: Top-50 Probability Distillation for Small Model English Training}, author = {CompactAI}, year = 2026, url = "https://huggingface.co/datasets/CompactAI/cAI-Prism-K50", } 或者您也可以在论文的致谢部分提及我们，我们很乐意看到自己的名字被提到。 --- ## 最后想说的话 老实说，我们并非完美无缺。本数据集可能存在漏洞，文档里也可能有拼写错误，我们的笑话也可能很冷。但我们尽力了。秉承各地小型模型的优良传统，我们相信努力本身就有意义。 现在去训练一个小巧又出色的模型吧。如果它最终能说一口流利的英语，不妨给我们发个消息，我们很乐意听听你的成果。 当然也可以不这么做，没有压力。我们有点信任问题。