YCWTG/Code-ChatML
收藏Hugging Face2026-04-07 更新2026-04-12 收录
下载链接：
https://hf-mirror.com/datasets/YCWTG/Code-ChatML
下载链接
链接失效反馈官方服务：
资源简介：
---
language:
- en
- zh
task_categories:
- text-generation
size_categories:
- 100K<n<1M
---
# Code-ChatML
`Code-ChatML` is a merged ChatML dataset built from the following sources:
- `YCWTG/Better-Ruozhiba-ChatML`
- `YCWTG/luau-stack-hq-ChatML`
- `YCWTG/Qwen3-Coder-Next-Open-Code-SFT-ChatML`
- `YCWTG/Opus-4.6-Reasoning-3000x-filtered-ChatML`
## Processing Pipeline
1. Load **only the `train` split** from each source dataset.
2. Normalize each record to:
```json
{
"messages": [
{"role": "...", "content": "..."}
]
}
```
3. Compute token length using:
- `AutoTokenizer.from_pretrained("google/gemma-4-31B-it")`
- `tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=False)`
4. Keep only samples where token length is `<= 4096`.
5. Drop samples where rendered token length is `> 4096`.
6. Merge filtered samples from all sources into a single `train` split.
## Source Statistics
| Source dataset | Original samples | Kept samples | Removed samples |
| --- | ---: | ---: | ---: |
| `YCWTG/Better-Ruozhiba-ChatML` | 1475 | 1475 | 0 |
| `YCWTG/luau-stack-hq-ChatML` | 21704 | 20853 | 851 |
| `YCWTG/Qwen3-Coder-Next-Open-Code-SFT-ChatML` | 23104 | 23084 | 20 |
| `YCWTG/Opus-4.6-Reasoning-3000x-filtered-ChatML` | 2326 | 2324 | 2 |
## Final Format
- Single split: `train`
- Single field: `messages`
- Total removed samples (`> 4096`): **873**
- Total samples in `train`: **47736**
提供机构：
YCWTG
