HuggingFaceTB/smollm3-configs

--- language: - en --- # SmolLM3 Training Configs **[IMPORTANT NOTE]**: for the latest configs go to this repo: https://github.com/huggingface/smollm/tree/main/text/pretraining/smollm3 Here you can find the training configs for [SmoLLM3-3B-Base](https://huggingface.co/HuggingFaceTB/SmolLM3-3B-Base) using [nanotron](https://github.com/huggingface/nanotron/) with exact training details and data mixtures. The model was trained on 11.2T tokens in 3 stages on 4k context: - stage 1 [config](https://huggingface.co/datasets/HuggingFaceTB/smollm3-configs/blob/main/stage1_8T.yaml) - stage 2 [config](https://huggingface.co/datasets/HuggingFaceTB/smollm3-configs/blob/main/stage2_8T_9T.yaml) - stage 3 [config](https://huggingface.co/datasets/HuggingFaceTB/smollm3-configs/blob/main/stage3_9T_11T.yaml)  And then we trained on an additional 2 stages to extend the contetx length to 64k: - stage 4 [config](https://huggingface.co/datasets/HuggingFaceTB/smollm3-configs/blob/main/long_context_4k_to_32k.yaml) - stage 5 [config](https://huggingface.co/datasets/HuggingFaceTB/smollm3-configs/blob/main/long_context_32k_to_64.yaml) 语言： - 英语 # SmolLM3 训练配置 **【重要提示】**：如需获取最新配置，请访问此仓库：https://github.com/huggingface/smollm/tree/main/text/pretraining/smollm3 本仓库提供基于[nantron](https://github.com/huggingface/nanotron/)训练[SmolLM3-3B-Base](https://huggingface.co/HuggingFaceTB/SmolLM3-3B-Base)的全套训练配置，包含精确训练细节与数据混合方案。 该模型采用4096（4k）上下文长度，分三阶段完成总计11.2万亿Token的训练： - 阶段1 [配置](https://huggingface.co/datasets/HuggingFaceTB/smollm3-configs/blob/main/stage1_8T.yaml) - 阶段2 [配置](https://huggingface.co/datasets/HuggingFaceTB/smollm3-configs/blob/main/stage2_8T_9T.yaml) - 阶段3 [配置](https://huggingface.co/datasets/HuggingFaceTB/smollm3-configs/blob/main/stage3_9T_11T.yaml)  随后我们额外开展两阶段训练，将模型上下文长度拓展至65536（64k）： - 阶段4 [配置](https://huggingface.co/datasets/HuggingFaceTB/smollm3-configs/blob/main/long_context_4k_to_32k.yaml) - 阶段5 [配置](https://huggingface.co/datasets/HuggingFaceTB/smollm3-configs/blob/main/long_context_32k_to_64.yaml)