CompactAI-O/cAI-Dense-PRISM

---
license: mit
task_categories:
- question-answering
language:
- en
tags:
- instruction
pretty_name: cAI-Dense-PRISM
size_categories:
- n<1K
---
# CompactAI-Prism Dense

## High-Density Distillation Dataset for Small Model English Language Acquisition

**License:** MIT
**Top-K:** 4096 (Current release: Dense)
**Source Model:** Qwen3.5 2B
**Primary Objective:** Teach small-scale AI models to generate fluent, coherent English text through probability-aware distillation. Or at least help them sound less like they learned English from a fortune cookie.

---

## Overview

CompactAI-Prism is a specialized training dataset designed to accelerate English language acquisition in compact AI models. Unlike standard instruction-tuning datasets that provide only a (Prompt, Response) pair, CompactAI-Prism captures the decision landscape of the teacher model. For every token generated in the response, we record the Top-K alternative tokens and their associated log-probabilities.

Think of it as giving your tiny model a peek at the teacher's scratch paper during the exam. We won't tell if you don't.

This approach increases the information density of the dataset by Kx per prompt, allowing student models to learn not just what the model answered, but what else it considered and why those options were rejected.

### The Math (Yes, There Is Math)
If a standard dataset provides 1 signal per token, CompactAI-Prism provides K signals. We know, we know - you became an AI researcher to avoid math. Sorry.

Total Training Signals = (Tokens per Response) x (Number of Questions) x K

For this release:
- Tokens per response (x): {AVG_RESPONSE_TOKENS} (Too lazy to check this. Might update later)
- Number of questions (y): 799
- Top-K value: 4096
- Effective training examples: {AVG_RESPONSE_TOKENS} x 799 x 4096 = 40960 * {AVG_RESPONCE_TOKENS}

That is a lot of tokens. You are welcome.

---

## Why Call It PRISM?

Great question. We considered "CompactAI-Overthinker" and "TinyModelTherapySession", but those did not fit on a GitHub repo name.

The name "Prism" reflects the core mechanism of this dataset:

1. **Single Input, Spectrum Revealed**: Just as a prism takes a single beam of white light and refracts it to reveal the full spectrum of colors within, CompactAI-Prism takes a single AI response and refracts it to reveal the full spectrum of token probabilities that existed at each generation step. Also, prisms look cool in stock photos.

2. **Hidden Structure Made Visible**: A prism does not create new colors; it exposes what was already present but invisible to the naked eye. Similarly, this dataset does not alter the teacher model's output; it exposes the latent probability distribution that guided each token choice. Like an X-ray, but for indecision.

3. **Clarity Through Decomposition**: By decomposing the generation process into its constituent probabilistic components, we enable student models to learn with greater clarity. They see not only the path taken, but the roads not taken - and the relative likelihood of each. It is like watching a choose-your-own-adventure book write itself, then regretting every choice.

In short: Prism turns opaque generation into transparent learning. Or at least slightly less opaque. Baby steps.

---

## Series Purpose: English Language Foundation for Small Models

This dataset series is explicitly designed to teach small models to speak English. Because let us be honest - some of them really need it.

By exposing compact architectures to the full probability distribution of token choices made by a capable teacher model, we enable:

- Faster convergence on grammatical English structures (goodbye, "me want food")
- Improved token selection confidence in low-parameter regimes (no more second-guessing every comma)
- Better handling of ambiguous or open-ended prompts (sometimes a question is just a question)
- Reduced hallucination through uncertainty-aware training (your model will still lie, but with more confidence intervals)

---

## Data Format

The dataset is provided in JSONL format. Each line represents a complete conversation turn with embedded probability data.

### Schema

```json
{
"messages": [
{"role": "user", "content": "STRING"},
{"role": "assistant", "content": "STRING"}
],
"response_tokens": INT,
"token_logprobs": [
{
"position": INT,
"generated_token_id": INT,
"generated_token": "STRING",
"logprob": FLOAT,
"top_k": [
{"token_id": INT, "token": "STRING", "logprob": FLOAT}
]
}
]
}
```

### Training Applications

1. **KL Divergence Distillation:** Use the full top_k distribution to minimize KL divergence between student and teacher, rather than just matching the chosen token. It is like teaching by example, but with more calculus.

2. **Confidence-Calibrated Generation:** Train small models to output confidence scores by learning from the teacher's logprob distributions. Now your tiny model can say "I am 73 percent sure that is correct" instead of just confidently being wrong.

3. **Alternative-Aware Decoding:** During inference, use knowledge of plausible alternatives to improve beam search or sampling strategies. Or just ignore this and use temperature=0.7 like everyone else. No judgment.

4. **English Fluency Bootstrapping:** Focus training on high-probability English token sequences to rapidly establish grammatical foundations in sub-10M parameter models. Because "me eat apple" is charming for about five minutes, then it gets old.

---

## Dataset Statistics

| Metric | Value |
|--------|-------|
| Total prompts | 799 |
| Top-K per position | 4096 |
| Number of times we questioned our life choices while building this | Yes |

---

## License

This project is licensed under the MIT License. Which is fancy legal speak for "use this however you want, just do not sue us when your tiny model starts writing poetry about toaster ovens."

Copyright (c) 2026 CompactAI

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Translation: If your model learns to speak perfect English but also develops an existential crisis, that is on you.

---

## Source Attribution
- **Dataset:** TeichAI Claude Sonnet 4.6 799 Prompts
- **Teacher Model:** Qwen3.5-2b
Thats it :)
---

## Citation

If you use CompactAI-Prism in your research or development, please cite:

```bibtex
@dataset{CompactAI/Dense-PRISM},
title = {CompactAI-Prism: Top-4096 Probability Distillation for Small Model English Training},
author = {CompactAI},
year = 2026,
url = https://huggingface.co/datasets/CompactAI/cAI-Prism-K4096,
}
```

Or just mention us in your paper's acknowledgments. We like hearing our names.

---

## Final Thoughts

Look, we are not perfect. This dataset might have bugs. The documentation might have typos. Our jokes might fall flat. But we tried. And in the grand tradition of small models everywhere, we believe that effort counts for something.

Now go train something tiny and wonderful. And if it starts speaking perfect English, maybe send us a note. We would love to hear about it.

Or not. No pressure. We have trust issues.

---
许可证：MIT
任务类别：
- 问答
语言：
- 英语
标签：
- 指令微调
展示名称：cAI-Dense-PRISM
样本量分类：
- n<1K（样本数少于1000）
---
# CompactAI-Prism 密集版

## 面向小模型英语习得的高密度蒸馏数据集

**许可证：MIT**
**Top-K值：4096（当前发布版本为Dense版）**
**教师模型：Qwen3.5 2B**
**核心目标：通过概率感知蒸馏技术，助力小型AI模型生成流畅连贯的英语文本，至少让其输出不再像从幸运饼干中学到的蹩脚英语。**

---

## 概述

CompactAI-Prism是专为加速小型AI模型英语习得设计的专用训练数据集。与仅提供（提示词、回复）对的标准指令微调数据集不同，CompactAI-Prism完整记录了教师模型的决策空间。对于回复中生成的每个Token（Token），我们均记录了Top-K个备选Token及其对应的对数概率（logprob）。

不妨将其比作让你的小型模型在考试时偷看教师的草稿纸——只要你不说，我们就不会揭发。

该方法使数据集的信息密度提升至K倍，让学生模型不仅能学习模型给出的答案，还能了解其曾考虑过的其他选项，以及为何这些选项未被采纳。

### 数学原理（没错，确实涉及数学）
若标准数据集每个Token仅提供1个信号，那么CompactAI-Prism每个Token可提供K个信号。我们深知，成为AI研究者本是为了远离数学——抱歉了。
总训练信号数 = （单回复Token数） × （问题总数） × K
针对本次发布版本：
- 单回复平均Token数（x）：{AVG_RESPONSE_TOKENS}（暂未统计，后续可能更新）
- 问题总数（y）：799
- Top-K值：4096
- 有效训练样本数：{AVG_RESPONSE_TOKENS} × 799 × 4096 = 40960 × {AVG_RESPONSE_TOKENS}
这可是相当可观的Token数量，祝您使用愉快。

---

## 为何命名为PRISM？

问得好。我们曾考虑过"CompactAI-Overthinker"和"TinyModelTherapySession"这类名称，但都不适合作为GitHub仓库名。

"Prism（棱镜）"这一名称体现了本数据集的核心机制：
1. **单输入，光谱尽显**：正如棱镜可将一束白光折射为完整可见光谱，CompactAI-Prism可将单条AI回复拆解，揭示每个生成步骤中存在的完整Token概率分布。此外，棱镜在图库素材中也十分亮眼。
2. **隐藏结构可视化**：棱镜并不会创造新颜色，仅会暴露肉眼原本不可见的固有成分。同理，本数据集并不会改变教师模型的输出，仅会揭示指导每个Token选择的隐式概率分布。这就像是给模型的犹豫不决拍了X光片。
3. **分解后更清晰**：通过将生成过程拆解为独立的概率组分，我们能让学生模型的学习过程更加清晰。它们不仅能看到选定的路径，还能了解未被选择的路径，以及各路径的相对可能性。这就像是看着一本"选择你的冒险"小说自行创作，然后为每一个选择懊悔不已。
简言之：Prism可将晦涩的生成过程转化为透明的学习过程——至少没那么晦涩了，算是循序渐进吧。

---

## 系列定位：为小模型打造英语基础

本数据集系列专为教授小模型英语而设计。老实说，有些小模型的英语水平实在有待提升。
通过让紧凑架构模型接触高性能教师模型的完整Token选择概率分布，我们可实现：
- 更快收敛至符合语法的英语结构（再见了"me want food"这类中式英语）
- 在低参数量场景下提升Token选择的置信度（不用再纠结每个逗号的使用）
- 更好地处理歧义或开放式提示（有时候问题就只是单纯的问题而已）
- 通过不确定性感知训练减少幻觉（你的模型依然可能说谎，但至少会附带置信区间）

---

## 数据格式

本数据集以JSON Lines（JSONL）格式提供。每行代表一条包含嵌入概率数据的完整对话轮次。

### 数据Schema

json
{
"messages": [
{"role": "user", "content": "字符串"},
{"role": "assistant", "content": "字符串"}
],
"response_tokens": 整数,
"token_logprobs": [
{
"position": 整数,
"generated_token_id": 整数,
"generated_token": "字符串",
"logprob": 浮点数,
"top_k": [
{"token_id": 整数, "token": "字符串", "logprob": 浮点数}
]
}
]
}

### 训练应用场景

1. **KL散度（KL Divergence）蒸馏**：利用完整的top_k分布最小化学生模型与教师模型之间的KL散度，而非仅匹配选定的Token。这就像是通过示例教学，但多了不少微积分内容。
2. **置信度校准生成**：通过学习教师模型的对数概率分布，训练小模型输出置信度评分。现在你的小模型可以说"我有73%的把握认为这个答案正确"，而不是只会盲目自信地说错答案。
3. **备选感知解码**：在推理阶段，利用已知的合理备选结果改进束搜索（beam search）或采样策略。当然你也可以直接忽略这一点，像大多数人一样使用温度参数（temperature）=0.7，我们不会评判。
4. **英语流利度快速入门**：将训练重点放在高概率英语Token序列上，帮助参数量低于1000万的模型快速建立语法基础。毕竟"me eat apple"这类表达新奇不过五分钟，很快就会让人审美疲劳。

---

## 数据集统计

| 指标 | 数值 |
|--------|-------|
| 总提示词数量 | 799 |
| 每个位置的Top-K值 | 4096 |
| 构建此数据集时我们怀疑人生的次数 | 是的 |

---

## 许可证

本项目采用MIT许可证。这可以用专业法律语言概括为："你可以随意使用本软件，但不要因为你的小模型开始写关于烤面包机的诗歌而起诉我们。"
版权所有 (c) 2026 CompactAI
特此免费授予任何获得本软件及相关文档文件（以下简称"软件"）副本的人不受限制地处理本软件的权利，包括但不限于使用、复制、修改、合并、发布、分发、再许可和/或出售本软件副本的权利，允许向其提供本软件的人员这么做，但需满足以下条件：
上述版权声明和本许可声明应包含在本软件的所有副本或主要部分中。
本软件按"原样"提供，不附带任何明示或暗示的担保，包括但不限于适销性、特定用途适用性和非侵权的担保。在任何情况下，作者或版权持有人均不对因本软件或本软件的使用或其他交易产生的任何索赔、损害或其他责任负责，无论是合同诉讼、侵权诉讼还是其他形式的诉讼，均与本软件或本软件的使用或其他交易有关。

翻译说明：若你的模型不仅学会了地道英语，还患上了存在主义危机，后果自负。

---

## 来源标注
- **数据集**：TeichAI Claude Sonnet 4.6 799条提示词
- **教师模型**：Qwen3.5-2b
以上就是全部内容 :)
---

## 引用说明

如果你的研究或开发中使用了CompactAI-Prism，请引用以下文献：
bibtex
@dataset{CompactAI/Dense-PRISM,
title = {CompactAI-Prism: Top-4096 Probability Distillation for Small Model English Training},
author = {CompactAI},
year = 2026,
url = https://huggingface.co/datasets/CompactAI/cAI-Prism-K4096,
}

或者在你的论文致谢中提及我们即可。我们很乐意看到自己的名字被提起。

---

## 最后想说的话

老实说，我们并非完美无瑕。本数据集可能存在漏洞，文档可能有拼写错误，我们的笑话也可能很冷。但我们已经尽力了。秉承小模型们的一贯传统，我们相信努力本身就有意义。
现在去训练一些小巧而出色的模型吧。如果它学会了一口流利的英语，不妨给我们发个消息。我们很乐意听听你的成果。
当然也可以不发，没有压力。我们有点社交恐惧。