TNSA/PT-HF500B

--- language: - en license: odc-by tags: - synthetic-data - instruction-tuning - large-scale - TNSA - NGen annotations_creators: - machine-generated language_creators: - found pretty_name: FinePhrase Synthetic Corpus size_categories: - n>1M source_datasets: - fineweb-edu (sample-350BT) task_categories: - text-generation task_ids: - language-modeling configs: - config_name: all data_files: - split: train path: - faq/**/*.parquet - math/**/*.parquet - table/**/*.parquet - tutorial/**/*.parquet - config_name: faq data_files: - split: train path: faq/**/*.parquet - config_name: math data_files: - split: train path: math/**/*.parquet - config_name: table data_files: - split: train path: table/**/*.parquet - config_name: tutorial data_files: - split: train path: tutorial/**/*.parquet train-eval-index: - config: all task: text-generation task_id: language-modeling splits: train_split: train col_mapping: text: text --- # PT-HF500B (FinePhrase) ## Overview **FinePhrase** is a large-scale synthetic dataset designed for high-quality language modeling, reasoning, and instruction-following tasks. It transforms raw educational web data into structured, instruction-rich formats suitable for training advanced language models. This dataset has been extensively used in the pre-training pipeline of TNSA models, including: * NGen-3 * NGen-4 * NGen-4-OW It plays a critical role in improving reasoning ability, structured output generation, and multi-format understanding. --- ## Dataset Composition FinePhrase is built by transforming raw documents into four distinct prompt-driven formats: ### 1. FAQ Format * Converts content into structured question-answer pairs * Enhances retrieval-style reasoning and clarity ### 2. Mathematical Reasoning * Converts text into multi-step math problems * Includes step-by-step solutions * Improves numerical reasoning and logical chains ### 3. Tabular Understanding * Extracts structured data into tables * Generates question-answer pairs from tabular data * Strengthens structured data interpretation ### 4. Tutorial / Instructional * Rewrites content into step-by-step guides * Improves procedural reasoning and instruction following --- ## Scale * Input Documents: ~339 Million * Generated Samples: ~1.35 Billion * Total Tokens Generated: ~486 Billion | Config | Samples | Tokens (Completion) | Avg Tokens | | --------- | --------- | ------------------- | ---------- | | FAQ | 338.9M | 148.1B | 436.9 | | Math | 338.7M | 98.4B | 290.5 | | Table | 338.5M | 92.4B | 272.9 | | Tutorial | 337.7M | 147.4B | 436.4 | | **Total** | **1.35B** | **486.3B** | **359.2** | --- ## Data Schema Each sample includes: * `id` — unique identifier * `text` — original source content * `rollout_results` — generated outputs * `text` — transformed output * `finish_reason` — generation termination reason * `usage` — token statistics --- ## Generation Process * Built using a high-throughput synthetic data pipeline * Based on large-scale educational web data * Uses instruction-driven transformations * Supports long-context generation (up to ~8K tokens) --- ## Use Cases * Pre-training large language models * Instruction tuning * Reasoning benchmarks * Structured output generation * Synthetic data augmentation --- ## Limitations * Fully synthetic outputs may include hallucinations * Some long documents are truncated due to context limits * Quality depends on transformation prompts and generation settings --- ## Licensing * ODC-BY (Open Data Commons Attribution License) --- ## Attribution This dataset originates from large-scale educational web corpora and has been transformed using automated synthetic data generation pipelines. --- ## Notes FinePhrase represents a foundation-scale synthetic dataset optimized for next-generation AI systems, particularly in improving: * reasoning depth * structured thinking * instruction adherence * multi-format understanding It serves as a core dataset in the development of TNSA’s advanced language models.language: - 英语 license: ODC-BY（Open Data Commons Attribution License） tags: - 合成数据（synthetic data） - 指令微调（instruction tuning） - 大规模 - TNSA - NGen annotations_creators: - 机器生成（machine-generated） language_creators: - 公开采集文本（found） pretty_name: FinePhrase合成语料库 size_categories: - 样本量超100万 source_datasets: - FineWeb-Edu（sample-350BT） task_categories: - 文本生成 task_ids: - 语言建模 configs: - config_name: 全量 data_files: - split: 训练集 path: - faq/**/*.parquet - math/**/*.parquet - table/**/*.parquet - tutorial/**/*.parquet - config_name: FAQ data_files: - split: 训练集 path: faq/**/*.parquet - config_name: 数学 data_files: - split: 训练集 path: math/**/*.parquet - config_name: 表格 data_files: - split: 训练集 path: table/**/*.parquet - config_name: 教程 data_files: - split: 训练集 path: tutorial/**/*.parquet train-eval-index: - config: 全量 task: 文本生成 task_id: 语言建模 splits: train_split: 训练集 col_mapping: text: text # PT-HF500B（FinePhrase） ## 概述 **FinePhrase** 是一款面向高质量语言建模、推理与指令跟随任务的大规模合成数据集。它将原始教育网络数据转换为结构化、富含指令的格式，适用于训练先进大语言模型（Large Language Model）。 本数据集已被广泛应用于TNSA模型的预训练流程，包括： * NGen-3 * NGen-4 * NGen-4-OW 其在提升模型推理能力、结构化输出生成能力与多格式理解能力方面发挥着关键作用。 --- ## 数据集构成 FinePhrase通过将原始文档转换为四种独立的提示驱动格式构建而成： ### 1. FAQ格式 * 将内容转换为结构化问答对 * 增强检索式推理与文本清晰度 ### 2. 数学推理 * 将文本转换为多步数学问题 * 包含分步解题过程 * 提升数值推理与逻辑链能力 ### 3. 表格理解 * 从非结构化文本中提取结构化数据并整理为表格 * 基于表格数据生成问答对 * 强化结构化数据解读能力 ### 4. 教程/指导性内容 * 将原始内容重写为分步指南 * 提升过程式推理与指令遵循能力 --- ## 数据集规模 * 输入文档：约3.39亿份 * 生成样本：约13.5亿份 * 总生成Token数：约4860亿 | 配置 | 样本量 | 补全Token数 | 平均Token数 | | --------- | --------- | ------------------- | ---------- | | FAQ | 3.389亿 | 1481亿 | 436.9 | | 数学 | 3.387亿 | 984亿 | 290.5 | | 表格 | 3.385亿 | 924亿 | 272.9 | | 教程 | 3.377亿 | 1474亿 | 436.4 | | **总计** | **13.5亿** | **4863亿** | **359.2** | --- ## 数据模式 每个样本包含以下字段： * `id` — 唯一标识符 * `text` — 原始源内容 * `rollout_results` — 生成输出 * `text` — 转换后的输出内容 * `finish_reason` — 生成终止原因 * `usage` — Token统计信息 --- ## 生成流程 * 基于高通量合成数据流水线构建 * 依托大规模教育网络数据 * 采用指令驱动的转换方式 * 支持长上下文生成（最长约8000个Token） --- ## 应用场景 * 大语言模型预训练 * 指令微调 * 推理基准测试 * 结构化输出生成 * 合成数据增强 --- ## 局限性 * 全合成输出可能包含幻觉内容 * 部分长文档因上下文长度限制被截断 * 数据集质量取决于转换提示与生成配置 --- ## 许可证 * ODC-BY（开放数据共同体署名许可证，Open Data Commons Attribution License） --- ## 归因说明 本数据集源自大规模教育网络语料库，通过自动化合成数据生成流水线完成转换。 --- ## 补充说明 FinePhrase是一款面向下一代人工智能系统优化的基础级大规模合成数据集，尤其在以下方面表现优异： * 推理深度 * 结构化思维 * 指令遵循能力 * 多格式理解能力 其是TNSA先进大语言模型开发过程中的核心数据集之一。