domofon/XML-corpus-19k
收藏资源简介：
Domofon XML预训练语料库是一个包含19,204个独特SFT（监督微调）对的数据集，专门用于训练语言模型将噪声或脏文本（如markdown、HTML、带有伪影的纯文本）转换为干净的结构化XML文档。该数据集的任务是：给定一个噪声、退化的文本，生成一个结构良好的XML文档，要求1）保留所有内容原样，不重写或总结；2）识别潜在结构（如标题、列表、表格、代码、引用）；3）将样板内容（如导航、页脚、广告）包装在<aside role=boilerplate>标签中；4）仅使用核心词汇表或提供模式中声明的标签。数据格式为JSONL（Parquet）行，包括input_text（噪声源）、target_xml（干净XML）和full_conversation（SFT就绪的im_start/im_end格式）。信封/模式系统提供三种类型：bare（无模式块，使用27个核心标签）、schema_only（输入前添加<schema>块列出可用标签）和full（包含<meta>、<schema>和<document>包装器的完整模式）。核心词汇表包含27个始终可用的标签，如document、section、heading等。数据集涵盖英语和俄语，提供多种配置（如clean2、clean1、latest_raw等），并包含来自DeepSeek和MiniMax等提供商的数据。处理流程包括模型创作、确定性降级、验证、后处理和救援步骤。已知质量注意事项包括处理虚构引用、链接、游离模式标签等问题。数据集用于研究目的，通过DeepSeek和MiniMax API生成。
The Domofon XML Pretrain Corpus is a dataset containing 19,204 unique SFT (Supervised Fine-Tuning) pairs, designed for training language models to convert noisy or dirty text (such as markdown, HTML, plaintext with artifacts) into clean, structured XML documents. The task is: given a noisy, degraded text, produce a well-structured XML document that 1) preserves all content verbatim without rewriting or summarization, 2) recognizes latent structure (e.g., headings, lists, tables, code, quotes), 3) wraps boilerplate content (e.g., nav, footer, ads) in <aside role=boilerplate>, and 4) uses only declared tags from the core vocabulary or provided schema. The format is JSONL (Parquet) rows, including input_text (noisy source), target_xml (clean XML), and full_conversation (SFT-ready im_start/im_end format). The envelope/schema system offers three types: bare (no schema block, using 27 core tags), schema_only (input prepended with <schema> block listing available tags), and full (full wrapper with <meta>, <schema>, and <document>). The core vocabulary includes 27 always-available tags, such as document, section, heading, etc. The dataset covers English and Russian languages, provides multiple configurations (e.g., clean2, clean1, latest_raw), and includes data from providers like DeepSeek and MiniMax. The pipeline involves model authorship, deterministic degradation, verification, post-processing, and rescue steps. Known quality caveats address issues like fabricated quotes, links, stray schema tags, etc. The dataset is for research purposes and generated via DeepSeek and MiniMax APIs.
