juliannunezb/smoltalk-gpt2-sft

SmolTalk-GPT2-SFT是一个完全标记化的数据集版本，源自HuggingFaceTB/smoltalk数据集（all配置），使用GPT-2 BPE标记器（词汇量50257）将原始多轮对话转换为（token_id, loss_mask）对，适用于任何使用GPT-2词汇的语言模型的监督微调（SFT）。数据集无需模板解析、角色标记或聊天模板调整，可直接用于PyTorch训练。它包含104万次对话，混合了来自Llama 3.1 405B Instruct的合成蒸馏数据以及涵盖数学、代码、摘要、函数调用和对话的公共数据集。标记化过程使用tiktoken.get_encoding(gpt2)，并包含损失掩码以指示助手响应标记。数据集以Parquet格式存储，包含原始对话、渲染文本、标记ID和损失掩码列。SmolTalk-GPT2-SFT is a fully tokenized version of the entire HuggingFaceTB/smoltalk dataset (all config), converted from raw multi-turn conversations into ready-to-train (token_id, loss_mask) pairs using the GPT-2 BPE tokenizer (vocab=50257). It is designed for supervised fine-tuning (SFT) of any GPT-2-vocab language model without the need for template parsing, role tagging, or chat-template gymnastics. The dataset contains 1.04M conversations, mixing synthetic distillations from Llama 3.1 405B Instruct with public datasets covering math, code, summarization, function-calling, and conversation. Tokenization is performed with tiktoken.get_encoding(gpt2), and includes a loss mask to identify assistant response tokens. It is stored in Parquet format with columns for original messages, rendered text, token IDs, and loss mask.