HuggingFaceTB/smollm3-configs

---
language:
- en
---

# SmolLM3 Training Configs

**[IMPORTANT NOTE]**: for the latest configs go to this repo: https://github.com/huggingface/smollm/tree/main/text/pretraining/smollm3

Here you can find the training configs for [SmoLLM3-3B-Base](https://huggingface.co/HuggingFaceTB/SmolLM3-3B-Base) using [nanotron](https://github.com/huggingface/nanotron/) with exact training details and data mixtures.

The model was trained on 11.2T tokens in 3 stages on 4k context:

- stage 1 [config](https://huggingface.co/datasets/HuggingFaceTB/smollm3-configs/blob/main/stage1_8T.yaml)
- stage 2 [config](https://huggingface.co/datasets/HuggingFaceTB/smollm3-configs/blob/main/stage2_8T_9T.yaml)
- stage 3 [config](https://huggingface.co/datasets/HuggingFaceTB/smollm3-configs/blob/main/stage3_9T_11T.yaml)


And then we trained on an additional 2 stages to extend the contetx length to 64k:

- stage 4 [config](https://huggingface.co/datasets/HuggingFaceTB/smollm3-configs/blob/main/long_context_4k_to_32k.yaml)
- stage 5 [config](https://huggingface.co/datasets/HuggingFaceTB/smollm3-configs/blob/main/long_context_32k_to_64.yaml)


语言：
- 英语

# SmolLM3 训练配置

**【重要提示】**：如需获取最新配置，请访问此仓库：https://github.com/huggingface/smollm/tree/main/text/pretraining/smollm3

本仓库提供基于[nantron](https://github.com/huggingface/nanotron/)训练[SmolLM3-3B-Base](https://huggingface.co/HuggingFaceTB/SmolLM3-3B-Base)的全套训练配置，包含精确训练细节与数据混合方案。

该模型采用4096（4k）上下文长度，分三阶段完成总计11.2万亿Token的训练：
- 阶段1 [配置](https://huggingface.co/datasets/HuggingFaceTB/smollm3-configs/blob/main/stage1_8T.yaml)
- 阶段2 [配置](https://huggingface.co/datasets/HuggingFaceTB/smollm3-configs/blob/main/stage2_8T_9T.yaml)
- 阶段3 [配置](https://huggingface.co/datasets/HuggingFaceTB/smollm3-configs/blob/main/stage3_9T_11T.yaml)


随后我们额外开展两阶段训练，将模型上下文长度拓展至65536（64k）：
- 阶段4 [配置](https://huggingface.co/datasets/HuggingFaceTB/smollm3-configs/blob/main/long_context_4k_to_32k.yaml)
- 阶段5 [配置](https://huggingface.co/datasets/HuggingFaceTB/smollm3-configs/blob/main/long_context_32k_to_64.yaml)