ChatQA2Seg
收藏资源简介：
ChatQA2Seg 是一个用于文本生成任务的英语数据集，其规模在1万到10万样本之间。该数据集是ChatQA2数据集的一个过滤版本（ChatQA2-Long-SFT-data-long_sft_train_filtered）经过进一步处理得到的“分割版本”。其核心目的是为研究“块注意力”的泛化能力提供训练数据。数据集的创建过程是：使用一个名为Qwen3-4B-Instruct-2507-Segmenter的预训练分割模型进行处理，该模型以递归深度为2、两个层级分别采用0.2和0.4的阈值对原始文本进行自动分割。数据集中的每个样本（称为‘cut_item’）包含五个关键字段：1. ‘txt_marker’：原始的文本字符串，其中插入了候选的分割点标记。2. ‘chunk_id’：定义了分割后每个文本块在原始文本中的边界位置。3. ‘chunk_plain_text’：分割后得到的各个文本块的实际内容。4. ‘cut_prob’：分割器模型为每个候选分割点计算出的分割概率。5. ‘threshold’：在分割过程中实际应用的概率阈值。该数据集主要用于训练和评估能够处理长文本、理解文本块结构的模型，特别是在探索通过自动分割和块蒸馏来提升块注意力机制泛化能力的研究中，如相关论文所述。
ChatQA2Seg is an English dataset for text generation tasks, with a scale ranging from 10,000 to 100,000 samples. It is a "segmented version" further processed from a filtered variant of the ChatQA2 dataset (named ChatQA2-Long-SFT-data-long_sft_train_filtered). Its core purpose is to provide training data for researching the generalization capability of chunk attention mechanisms. The dataset is created by processing with a pre-trained segmentation model named Qwen3-4B-Instruct-2507-Segmenter, which automatically segments the original text with a recursive depth of 2, applying threshold values of 0.2 and 0.4 respectively at the two levels. Each sample in the dataset (referred to as 'cut_item') contains five key fields: 1. 'txt_marker': The original text string with candidate split point markers inserted. 2. 'chunk_id': Defines the boundary position of each segmented text chunk within the original text. 3. 'chunk_plain_text': The actual content of each text chunk obtained after segmentation. 4. 'cut_prob': The segmentation probability calculated by the segmenter model for each candidate split point. 5. 'threshold': The probability threshold actually applied during the segmentation process. This dataset is primarily used for training and evaluating models that can handle long texts and understand text chunk structures, especially in studies exploring the improvement of generalization capabilities of chunk attention mechanisms through automatic segmentation and chunk distillation, as described in relevant academic papers.
数据集概述：ChatQA2Seg
该数据集是 ChatQA2 数据集（经过筛选的版本）的分段版本，用于支持论文《Towards Generalization of Block Attention via Automatic Segmentation and Block Distillation》中的模型训练。
- 许可证: Apache-2.0
- 任务类别: 文本生成
- 语言: 英语
- 数据集规模: 10,000 < n < 100,000 条数据
数据来源与处理
- 原始数据集: Seerkfang/ChatQA2-Long-SFT-data-long_sft_train_filtered
- 分段工具: 使用名为 Syon-Li/Qwen3-4B-Instruct-2507-Segmenter 的分段器，递归深度为 2，阈值分别为 0.2（第一层）和 0.4（第二层）。
数据字段说明
每个被切分的项包含以下字段：
- txt_marker: 插入候选切分点的文本字符串。
- chunk_id: 每个文本块的切分边界标识。
- chunk_plain_text: 每个文本块的纯文本内容。
- cut_prob: 分段器给出的切分概率。
- threshold: 用于切分的阈值。
相关引用
若该数据集对您有帮助，请引用论文：
@article{li2026towards, title={Towards Generalization of Block Attention via Automatic Segmentation and Block Distillation}, author={Li, Shuaiyi and Zhang, Zhisong and Wang, Yan and Zhu, Lei and Ma, Dongyang and Deng, Chenlong and Deng, Yang and Lam, Wai}, journal={arXiv preprint arXiv:2605.15913}, year={2026} }
