SmolLM

该数据集是SmolLM系列，包含了不同规模的大型语言模型（分别为135M、360M、1B7参数规模），这些模型使用了多头注意力（MHA）或分组查询注意力（GQA）进行预训练。此外，该数据集是开源的，旨在减少在微调过程中数据和处理流程上的差距。其任务是针对从多头注意力（MHA）迁移到多层注意力（MLA）的建筑迁移进行微调。

This dataset belongs to the SmolLM series, which includes large language models with varying parameter scales: 135M, 360M, and 1.7B. These models are pre-trained using either multi-head attention (MHA) or grouped-query attention (GQA). Furthermore, this open-source dataset is designed to narrow the gap in data and processing workflows during fine-tuning. Its targeted task is to conduct fine-tuning for architecture migration from multi-head attention (MHA) to multi-layer attention (MLA).