mxxsc/zip2zip-plus-mixture-partitioned

--- license: other task_categories: - text-generation language: - en - multilingual tags: - pretraining - fineweb - code - math - multilingual pretty_name: Zip2Zip Plus Mixture Partitioned size_categories: - 100B<n<1T dataset_info: features: - name: text dtype: string - name: source dtype: string splits: - name: train num_bytes: 907324781210 num_examples: 190184417 download_size: 475006167489 dataset_size: 907324781210 configs: - config_name: default data_files: - split: train path: data/train-* --- # Zip2Zip Plus Mixture Partitioned This dataset is a partitioned pretraining-data mixture built for zip2zip language-model pretraining. The mixture is byte-balanced across four top-level domains: | Domain | Source | Target byte ratio | |---|---|---:| | General | `HuggingFaceFW/fineweb-edu`, `sample-100BT` | 50% | | Code | `bigcode/the-stack-dedup` | 20% | | Math | `HuggingFaceTB/finemath`, `finemath-3plus` | 10% | | Multilingual | `epfml/FineWeb2-HQ`, 20 language subsets | 20% | The uploaded layout is partitioned by source domain. It is intended as the full source-partitioned version from which remixed training datasets and token-budget subsets can be derived. ## Dataset Structure Each row has the schema: ```python { "text": str, "source": str } ``` The `source` field preserves source detail, for example: - `fineweb-edu::sample-100BT` - `the-stack-dedup` - `finemath::finemath-3plus` - `fineweb2-hq::deu_Latn` - `fineweb2-hq::cmn_Hani` ## Multilingual Subsets The multilingual portion uses 20 FineWeb2-HQ language subsets, sampled according to their row counts: - `arb_Arab` - `ces_Latn` - `cmn_Hani` - `dan_Latn` - `deu_Latn` - `ell_Grek` - `fas_Arab` - `fra_Latn` - `hun_Latn` - `ind_Latn` - `ita_Latn` - `jpn_Jpan` - `nld_Latn` - `pol_Latn` - `por_Latn` - `rus_Cyrl` - `spa_Latn` - `swe_Latn` - `tur_Latn` - `vie_Latn` ## Intended Use This dataset is intended for research on language-model pretraining and zip2zip-style training pipelines. Typical workflow: 1. Build or download this partitioned mixture. 2. Count tokens with the target tokenizer. 3. Remix into flat mixed shards. 4. Create token-budget subsets, such as a 20B-token subset. 5. Use the resulting flat mixed dataset for pretraining. ## Loading ```python from datasets import load_dataset ds = load_dataset("mxxsc/zip2zip-plus-mixture-partitioned", split="train") print(ds[0]) ``` ## Data Construction The mixture was constructed by streaming source datasets and writing incremental compressed JSONL shards. The top-level mixture ratios are byte-based, not token-based. The multilingual domain is internally allocated according to language subset row counts. ## Caveats This dataset inherits the quality, filtering, licensing, and safety properties of its upstream datasets. Users should consult the original dataset cards before redistribution or downstream deployment. The mixture ratios are based on written text bytes. Token ratios may differ depending on the tokenizer. ## Source Datasets - `HuggingFaceFW/fineweb-edu` - `bigcode/the-stack-dedup` - `HuggingFaceTB/finemath` - `epfml/FineWeb2-HQ`license: 其他 task_categories: - 文本生成 language: - 英语 - 多语言 tags: - 预训练（pretraining） - FineWeb - 代码 - 数学 - 多语言 pretty_name: Zip2Zip Plus 混合分区数据集 size_categories: - 100亿 < 样本规模 < 1万亿 dataset_info: features: - name: text dtype: 字符串 - name: source dtype: 字符串 splits: - name: train num_bytes: 907324781210 num_examples: 190184417 download_size: 475006167489 dataset_size: 907324781210 configs: - config_name: default data_files: - split: train path: data/train-* # Zip2Zip Plus 混合分区数据集 本数据集是为zip2zip语言模型预训练（pretraining）构建的分区预训练数据混合集。 该混合集在四个顶级领域间实现字节均衡： | 领域 | 来源 | 目标字节占比 | |---|---|---:| | 通用 | `HuggingFaceFW/fineweb-edu`、`sample-100BT` | 50% | | 代码 | `bigcode/the-stack-dedup` | 20% | | 数学 | `HuggingFaceTB/finemath`、`finemath-3plus` | 10% | | 多语言 | `epfml/FineWeb2-HQ`、20种语言子集 | 20% | 上传的布局按来源域分区，旨在作为完整的源分区版本，可从中衍生出混合训练数据集和Token（Token）预算子集。 ## 数据集结构 每一行遵循以下数据模式： python { "text": str, "source": str } `source`字段保留来源详情，例如： - `fineweb-edu::sample-100BT` - `the-stack-dedup` - `finemath::finemath-3plus` - `fineweb2-hq::deu_Latn` - `fineweb2-hq::cmn_Hani` ## 多语言子集 多语言部分使用20个FineWeb2-HQ语言子集，按其行数采样： - `arb_Arab` - `ces_Latn` - `cmn_Hani` - `dan_Latn` - `deu_Latn` - `ell_Grek` - `fas_Arab` - `fra_Latn` - `hun_Latn` - `ind_Latn` - `ita_Latn` - `jpn_Jpan` - `nld_Latn` - `pol_Latn` - `por_Latn` - `rus_Cyrl` - `spa_Latn` - `swe_Latn` - `tur_Latn` - `vie_Latn` ## 预期用途 本数据集旨在用于语言模型预训练及zip2zip风格训练流水线的研究。 典型流程： 1. 构建或下载该分区混合集； 2. 使用目标分词器统计Token数； 3. 混合为平坦的分块数据集； 4. 创建Token预算子集，例如200亿Token子集； 5. 使用生成的平坦混合数据集进行预训练。 ## 加载方式 python from datasets import load_dataset ds = load_dataset("mxxsc/zip2zip-plus-mixture-partitioned", split="train") print(ds[0]) ## 数据构建 该混合集通过流式读取源数据集并写入增量压缩的JSONL分块构建。顶级混合比例基于字节而非Token。多语言域内部按语言子集的行数分配比例。 ## 注意事项 本数据集继承其上游数据集的质量、过滤规则、许可证及安全属性。用户在重新分发或下游部署前，应查阅原始数据集卡片。混合比例基于书面文本字节数，不同分词器下的Token比例可能有所差异。 ## 源数据集 - `HuggingFaceFW/fineweb-edu` - `bigcode/the-stack-dedup` - `HuggingFaceTB/finemath` - `epfml/FineWeb2-HQ`