Zengying/OLMoE-mix-0924

---
task_categories:
- text-generation
language:
- en
size_categories:
- 1B<n<10B
license: odc-by
pretty_name: OLMoE Mix (September 2024)
dataset_info:
features:
- name: id
dtype: string
- name: text
dtype: string
- name: added
dtype: string
- name: created
dtype: string
---

# OLMoE Mix (September 2024)

## Dataset Description

- **Repository:** https://github.com/allenai/OLMoE
- **Paper:** [OLMoE: Open Mixture-of-Experts Language Models](https://arxiv.org/abs/2409.02060)

<img alt="OLMoE Mix Logo." src="olmoe-mix.png" width="250px">

The following data mix was used to train OLMoE-1B-7B, a Mixture-of-Experts LLM with 1B active and 7B total parameters released in September 2024.

The base version of OLMoE-1B-7B can be found at [this page](https://huggingface.co/allenai/OLMoE-1B-7B-0924), the SFT of OLMoE-1B-7B is available [here](https://huggingface.co/allenai/OLMoE-1B-7B-0924-SFT), and a version combining SFT and DPO is available following [this link](https://huggingface.co/allenai/OLMoE-1B-7B-0924-Instruct).

## Statistics

| Subset | Tokens | Words | Bytes | Docs |
|--------------------------------------------------------------|:----------:|:----------:|:----------:|:----------:|
| [DCLM Baseline 1.0](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0) | 3.86 T | 3.38 T | 16.7 T | 2.95 B |
| [Starcoder](https://huggingface.co/datasets/bigcode/starcoderdata) | 101 B | 63.9 B | 325 B | 78.7 M |
| [peS2o](https://huggingface.co/datasets/allenai/peS2o)<br>([Dolma](https://huggingface.co/datasets/allenai/dolma)) | 57.2 B | 51.3 B | 268 B | 38.8 M |
| Arxiv<br>([RedPajama v1](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T) <br>via [Proof Pile II](https://huggingface.co/datasets/EleutherAI/proof-pile-2)) | 21.1 B | 23.5 B | 88.8 B | 1.55 M |
| OpenWebMath<br>([Proof Pile II](https://huggingface.co/datasets/EleutherAI/proof-pile-2)) | 12.7 B | 10.2 B | 42.4 B | 2.91 M |
| Algebraic Stack<br>([Proof Pile II](https://huggingface.co/datasets/EleutherAI/proof-pile-2)) | 12.6 B | 9.6 B | 39.3 B | 2.83 M |
| En Wikipedia + <br>Wikibooks<br>([Dolma](https://huggingface.co/datasets/allenai/dolma)) | 3.69 B | 3.16 B | 16.2 B | 6.17 M |
| **Total** | **4.07 T** | **3.53 T** | **17.4 T** | **3.08 B** |

## Preprocessing

All subsets were pre-processed to remove documents with a *sequence* of 32 or more repeated *ngrams*.
- a *ngram* is a span of 1 to 13 tokens, included;
- *tokens* are obtained using the model tokenizer;
- a *sequence* is a contiguous span of repeated ngrams.

In addition of the above, Starcoder dataset was further processed by removing any document meeting any of the following rules:
- document is from a repository with fewer than 2 stars on GitHub;
- the top most frequent word in the document constitutes over 30% of the document;
- the two most frequent words in the document constitutes over 50% of the document.

## Licensing Information

This mix is licensed under [Open Data Commons Attribution License (ODC-By) v1.0](https://opendatacommons.org/licenses/by/1-0/). By using this dataset, you are bound to licenses and Terms of Services of underlying datasets, which you can access by clicking on the links in the table above.

## Citation

```bibtex
@misc{muennighoff2024olmoeopenmixtureofexpertslanguage,
title={OLMoE: Open Mixture-of-Experts Language Models},
author={Niklas Muennighoff and Luca Soldaini and Dirk Groeneveld and Kyle Lo and Jacob Morrison and Sewon Min and Weijia Shi and Pete Walsh and Oyvind Tafjord and Nathan Lambert and Yuling Gu and Shane Arora and Akshita Bhagia and Dustin Schwenk and David Wadden and Alexander Wettig and Binyuan Hui and Tim Dettmers and Douwe Kiela and Ali Farhadi and Noah A. Smith and Pang Wei Koh and Amanpreet Singh and Hannaneh Hajishirzi},
year={2024},
eprint={2409.02060},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2409.02060},
}
```

任务类别：文本生成
语言：英语
规模类别：10亿 < 令牌数 < 100亿
许可协议：ODC-By
美观名称：OLMoE 混合数据集（2024年9月版）
数据集信息：
特征：
- 名称：id，数据类型：字符串
- 名称：text，数据类型：字符串
- 名称：added，数据类型：字符串
- 名称：created，数据类型：字符串

# OLMoE 混合数据集（2024年9月版）

## 数据集描述
- **仓库地址**：https://github.com/allenai/OLMoE
- **论文**：[OLMoE：开放混合专家大语言模型（Large Language Model）](https://arxiv.org/abs/2409.02060)


本数据集混合版本用于训练OLMoE-1B-7B，该模型是一款拥有10亿激活参数、总参数达70亿的混合专家架构大语言模型，于2024年9月发布。

OLMoE-1B-7B的基础版本可在[此页面](https://huggingface.co/allenai/OLMoE-1B-7B-0924)获取；其监督微调（Supervised Fine-Tuning，SFT）版本可通过[此处](https://huggingface.co/allenai/OLMoE-1B-7B-0924-SFT)获取；而结合监督微调与直接偏好优化（Direct Preference Optimization，DPO）的版本可通过[此链接](https://huggingface.co/allenai/OLMoE-1B-7B-0924-Instruct)获取。

## 统计信息
| 子集名称 | 令牌数 | 词数 | 字节数 | 文档数 |
|:------------------------------------------------------------ |:----------:|:----------:|:----------:|:----------:|
| [DCLM 基准版1.0](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0) | 3.86 T | 3.38 T | 16.7 T | 2.95 B |
| [Starcoder](https://huggingface.co/datasets/bigcode/starcoderdata) | 101 B | 63.9 B | 325 B | 78.7 M |
| [peS2o](https://huggingface.co/datasets/allenai/peS2o)<br>([Dolma](https://huggingface.co/datasets/allenai/dolma)) | 57.2 B | 51.3 B | 268 B | 38.8 M |
| ArXiv<br>([RedPajama v1](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T) <br>via [Proof Pile II](https://huggingface.co/datasets/EleutherAI/proof-pile-2)) | 21.1 B | 23.5 B | 88.8 B | 1.55 M |
| OpenWebMath<br>([Proof Pile II](https://huggingface.co/datasets/EleutherAI/proof-pile-2)) | 12.7 B | 10.2 B | 42.4 B | 2.91 M |
| Algebraic Stack<br>([Proof Pile II](https://huggingface.co/datasets/EleutherAI/proof-pile-2)) | 12.6 B | 9.6 B | 39.3 B | 2.83 M |
| 英文维基百科 + <br>维基教科书<br>([Dolma](https://huggingface.co/datasets/allenai/dolma)) | 3.69 B | 3.16 B | 16.2 B | 6.17 M |
| **总计** | **4.07 T** | **3.53 T** | **17.4 T** | **3.08 B** |

## 预处理流程
所有子集均经过预处理，移除包含32个及以上重复**n元语法（ngram）序列**的文档。其中：
- **n元语法（ngram）** 指长度为1至13的连续文本片段（包含端点值）；
- **令牌（Token）** 通过模型分词器获取；
- **序列** 指连续重复的n元语法片段。

除此之外，Starcoder数据集还经过了额外预处理，移除符合以下任意一条规则的文档：
1. 文档来自GitHub上星标数少于2的仓库；
2. 文档中出现频率最高的单词占总词数的比例超过30%；
3. 文档中出现频率最高的两个单词占总词数的比例超过50%。

## 许可信息
本数据集混合版本遵循[开放数据共同体署名许可协议v1.0（Open Data Commons Attribution License v1.0，ODC-By）](https://opendatacommons.org/licenses/by/1-0/)进行授权。使用本数据集即表示您同意遵守底层数据集的许可协议与服务条款，相关内容可通过上表中的链接查看。

## 引用格式
bibtex
@misc{muennighoff2024olmoeopenmixtureofexpertslanguage,
title={OLMoE: Open Mixture-of-Experts Language Models},
author={Niklas Muennighoff and Luca Soldaini and Dirk Groeneveld and Kyle Lo and Jacob Morrison and Sewon Min and Weijia Shi and Pete Walsh and Oyvind Tafjord and Nathan Lambert and Yuling Gu and Shane Arora and Akshita Bhagia and Dustin Schwenk and David Wadden and Alexander Wettig and Binyuan Hui and Tim Dettmers and Douwe Kiela and Ali Farhadi and Noah A. Smith and Pang Wei Koh and Amanpreet Singh and Hannaneh Hajishirzi},
year={2024},
eprint={2409.02060},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2409.02060},
}