SidhiPanda/peS2o
收藏Hugging Face2025-12-05 更新2025-12-20 收录
下载链接：
https://hf-mirror.com/datasets/SidhiPanda/peS2o
下载链接
链接失效反馈官方服务：
资源简介：
---
license:
- odc-by
task_categories:
- text-generation
- fill-mask
language:
- en
tags:
- biology
- chemistry
- engineering
- computer science
- physics
- material science
- math
- psychology
- economics
- political science
- business
- geology
- sociology
- geography
- environmental science
- art
- history
- philosophy
pretty_name: peS2o (Pretraining Efficiently on S2ORC)
size_categories:
- 10B<n<100B
source_datasets:
- allenai/s2orc
---
<p align="center" style="margin-top: -2em">
<img src="https://huggingface.co/datasets/allenai/pes2o/resolve/main/logo.png" alt="peS2o logo. It's a picure of a mortar and pestle with documents flying in." width=384px height=auto>
</p>
<p align="center" style="font-size: 1.2em; margin-top: -1em"><i>Pretraining Effectively on <a href="https://github.com/allenai/s2orc">S2ORC</a>!</i></p>
The peS2o dataset is a collection of ~40M creative open-access academic papers,
cleaned, filtered, and formatted for pre-training of language models. It is derived from
the [Semantic Scholar Open Research Corpus][2]([Lo et al, 2020][1]), or S2ORC.
We release multiple version of peS2o, each with different processing and knowledge cutoff
date. We recommend you to use the latest version available.
If you use this dataset, please cite:
```bibtex
@techreport{peS2o,
author = {Luca Soldaini and Kyle Lo},
year = 2023,
title = {{peS2o (Pretraining Efficiently on S2ORC) Dataset}},
institution = {{Allen Institute for AI}},
note = {ODC-By, \url{https://github.com/allenai/pes2o}}
}
```
## Document Format
Each document in the dataset is a dictionary with the following fields:
- `added`: Date the document was added to the corpus.
- `created`: Best-guess date for when the document was first published. Some have resolution down to the day, only down to the year.
- `id`: Semantic Scholar Corpus ID of the document; it can be used with the [Semantic Scholar API](https://api.semanticscholar.org/) to retrieve metadata about the document (e.g., fields of study, authors).
- `source`: Collection from which the document was sourced from. At the moment, two are supported:
- `s2orc`: collection of full-text papers
- `s2ag`: collection of title and abstracts
- `text`: Text of the document. Paragraphs are separated by two newlines (`\n\n`).
- `version`: version of peS2o.
------
## peS2o V2 (Latest)
### Key Facts
- *Knowledge cutoff*: 2023-01-03
- *Number of documents*: 38.97M
- *Number of whitespace-separated tokens**: 42.01B
### Processing
peS2o V2 is largely the same as V1, but it includes additional heuristics s2ag aimed at filtering out OCR errors from abstract.
First, we check if the abstract was obtained from Semantic Scholar sources that are likely to contain OCR'ed content. For any abstract derived from those sources, we count how often the text contains subsequences matching `\b([A-Za-z]\s)([a-z]\s)*[A-Za-z]\b`, i.e. individual alpha letters separated by a space. This heuristic matches cases such as `A b stra ct` (2 matching subsequences), where the OCR parser inserted erroneous spaces.
Any abstract with more than 4 matching subsequences is removed.
#### Statistics
| Dataset | Split | # Documents | # Words |
|:-------:|:-----:|------------:|---------------:|
| s2orc | train | 8,242,162 | 36,088,195,908 |
| s2orc | valid | 51,323 | 255,139,074 |
| s2ag | train | 30,569,017 | 5,920,099,207 |
| s2ag | valid | 109,709 | 24,029,459 |
-------
## peS2o V1
### Key Facts
- *Knowledge cutoff*: 2023-01-03
- *Number of documents*: 67.56M
- *Number of whitespace-separated tokens*: 47.37B
### Processing
Processing differs slightly wether it was derived from the full-text corpus (`s2orc`) or the title and abstract corpus (`s2ag`).
#### S2ORC-derived documents
Unfiltered, S2ORC contains 11.3M papers and 46.9B whitespace-separated tokens as of 2023-01-03. To derive peS2o v1, we impose the following constraints:
- The paper must have a title and abstract.
- From each paper, we use [Grobid](https://github.com/kermitt2/grobid) to extract section headers and paragraphs; figures, tables, and references, and any other non-textual content is removed. Title and abstracts are also available, but they come from the Semantic Scholar metadata (obtained through the APIs), not Grobid.
- The paper must be in English.
- To determine the language of each document, we use the [pycld3](https://github.com/bsolomon1124/pycld3) library
- We run pycld3 on the first 2000 characters of each paragraph in the paper.
- The language of the paper is the most common language of the paragraphs.
- The paper must have at least 500 whitespace-separated words.
- The paper was published after 1969; papers published before this date are often obtained through OCR and contain unrecoverable errors.
- The paper must have at least 5 paragraphs.
- All sections that have a average log word probability of less than `-20` are removed.
- To calculate the average log word probability, we use word frequencies extracted from the [1T Web Ngram corpus](https://catalog.ldc.upenn.edu/LDC2006T13); specifically, we use the list available [created by Rachel Tatman](https://www.kaggle.com/datasets/rtatman/english-word-frequency). A copy is hosted [here](https://ai2-s2-research-public.s3-us-west-2.amazonaws.com/lucas/google-1T-unigram/unigram_freq.csv).
- The most frequent word in the paper consists of alpha characters only, and it appears in less than 7.5% of the document.
- Words are obtained by splitting the text on whitespace.
The train set contains papers published before 2022-12-01;
the validation set includes documents published after 2022-12-01 and until 2023-01-03.
#### S2AG-derived documents
The S2AG corpus contains titles and abstracts of papers in Semantic Scholar.
Unfiltered, the corpus contains 91.1M papers and 15.5B whitespace-separated tokens as of 2023-01-03. To derive peS2o v1, we impose the following constraints:
- Abstract must be in English.
- To calculate the language, we once again use pycld3
- Title must be in English, or have average unigram log probability greater than -20.
- Abstract must be in English.
- Abstract must have higher than -20 average unigram log probability.
- Abstract must have at least 50 words.
- Abstract must have no more than 1000 words.
- The most frequent word in the union of text and abstract must be a 2+ character alpha word, or it can be `a` followed by a 2+ character alpha word.
- Paper was published after 1969.
#### Statistics
| Dataset | Split | # Documents | # Words |
|:-------:|:-------:|:-----------:|:--------------:|
|s2orc | train | 8,242,162 | 36,088,195,908 |
|s2orc | valid | 51,323 | 255,139,074 |
|s2ag | train | 59,382,301 | 11,009,123,378 |
|s2ag | valid | 111,228 | 24,398,512 |
[1]: https://aclanthology.org/2020.acl-main.447/
[2]: https://github.com/allenai/s2orc
许可证：
- odc-by
任务类别：
- 文本生成（text-generation）
- 掩码填充（fill-mask）
语言：
- en（英语）
标签：
- 生物学（biology）
- 化学（chemistry）
- 工程学（engineering）
- 计算机科学（computer science）
- 物理学（physics）
- 材料科学（material science）
- 数学（math）
- 心理学（psychology）
- 经济学（economics）
- 政治学（political science）
- 商学（business）
- 地质学（geology）
- 社会学（sociology）
- 地理学（geography）
- 环境科学（environmental science）
- 艺术学（art）
- 历史学（history）
- 哲学（philosophy）
可读名称：peS2o（基于S2ORC的高效预训练，Pretraining Efficiently on S2ORC）
规模类别：100亿 < 数据规模 < 1000亿
源数据集：
- allenai/s2orc
<p align="center" style="margin-top: -2em">
<img src="https://huggingface.co/datasets/allenai/pes2o/resolve/main/logo.png" alt="peS2o 标识：研钵与研杵搭配飞舞的文献" width=384px height=auto>
</p>
<p align="center" style="font-size: 1.2em; margin-top: -1em"><i>基于<a href="https://github.com/allenai/s2orc">S2ORC</a>实现高效预训练！</i></p>
peS2o数据集是约4000万篇开放获取学术文献的集合，经过清洗、过滤与格式标准化处理，专为大语言模型（Large Language Model, LLM）预训练打造。该数据集源自[语义学者开放研究语料库（Semantic Scholar Open Research Corpus）][2]（[Lo等人，2020][1]），简称S2ORC。
我们提供多个版本的peS2o，各版本采用不同的处理流程与知识截止日期。我们推荐使用当前可用的最新版本。
若您使用本数据集，请引用如下文献：
bibtex
@techreport{peS2o,
author = {Luca Soldaini and Kyle Lo},
year = 2023,
title = {{peS2o (Pretraining Efficiently on S2ORC) Dataset}},
institution = {{Allen Institute for AI}},
note = {ODC-By, url{https://github.com/allenai/pes2o}}
}
## 文档格式规范
数据集中的每篇文献均为包含以下字段的字典：
- `added`：文献被加入语料库的日期。
- `created`：文献首次发表的推测日期，部分精确到日，部分仅精确到年。
- `id`：文献的语义学者语料库ID，可通过[语义学者API（Semantic Scholar API）](https://api.semantics.org/)获取该文献的元数据（如研究领域、作者等）。
- `source`：文献的来源数据集，当前支持两种类型：
- `s2orc`：完整文本文献集合
- `s2ag`：标题与摘要集合
- `text`：文献正文内容，段落以两个换行符（`
`）分隔。
- `version`：peS2o数据集版本。
------
## peS2o V2（最新版）
### 核心参数
- *知识截止日期*：2023-01-03
- *文献总数*：3897万篇
- *空白符分隔的Token（Token）总数*：420.1亿个
### 数据处理流程
peS2o V2 与V1大体一致，但新增了针对s2ag子集的启发式过滤规则，用于过滤摘要中的光学字符识别（Optical Character Recognition, OCR）错误。
首先，我们会判断摘要是否来自易包含OCR识别内容的语义学者数据源。对于来自此类数据源的摘要，我们统计文本中匹配正则表达式`([A-Za-z]s)([a-z]s)*[A-Za-z]`的子序列数量，该正则用于匹配单个字母以空格分隔的情况（例如`A b stra ct`，存在2个匹配子序列，即OCR识别引擎插入了错误空格的场景）。
若某摘要的匹配子序列数量超过4，则将其从语料库中移除。
#### 统计数据
| 数据集 | 划分集 | 文献数量 | 词数 |
|:-------:|:-----:|------------:|---------------:|
| s2orc | 训练集 | 8,242,162 | 36,088,195,908 |
| s2orc | 验证集 | 51,323 | 255,139,074 |
| s2ag | 训练集 | 30,569,017 | 5,920,099,207 |
| s2ag | 验证集 | 109,709 | 24,029,459 |
-------
## peS2o V1
### 核心参数
- *知识截止日期*：2023-01-03
- *文献总数*：6756万篇
- *空白符分隔的Token总数*：473.7亿个
### 数据处理流程
针对完整文本语料库（`s2orc`）与标题摘要语料库（`s2ag`）衍生的数据集，处理流程略有差异。
#### 源自S2ORC的文献
截至2023年1月3日，未经过滤的S2ORC语料库包含1130万篇文献与469亿个空白符分隔的Token。为生成peS2o V1，我们设置了如下筛选条件：
- 文献必须包含标题与摘要。
- 针对每篇文献，我们使用[Grobid](https://github.com/kermitt2/grobid)工具提取章节标题与段落，移除图表、参考文献及其他非文本内容。标题与摘要同样保留，但数据源自语义学者的API元数据，而非Grobid提取结果。
- 文献必须为英文。
- 我们使用[pycld3](https://github.com/bsolomon1124/pycld3)库来识别每篇文献的语言。
- 针对文献中的每个段落，我们取其前2000个字符进行语言检测。
- 文献的整体语言以各段落检测结果的多数投票结果为准。
- 文献的空白符分隔词数不得少于500个。
- 文献的发表时间需晚于1969年：1969年之前发表的文献通常通过OCR识别获取，且存在难以修复的错误。
- 文献需包含至少5个段落。
- 移除平均词对数概率低于`-20`的章节。
- 平均词对数概率的计算基于从[1T Web Ngram语料库](https://catalog.ldc.upenn.edu/LDC2006T13)中提取的词频数据，具体使用由Rachel Tatman整理的词频列表（可从[此处](https://www.kaggle.com/datasets/rtatman/english-word-frequency)获取，本数据集也提供了一份备份：[https://ai2-s2-research-public.s3-us-west-2.amazonaws.com/lucas/google-1T-unigram/unigram_freq.csv](https://ai2-s2-research-public.s3-us-west-2.amazonaws.com/lucas/google-1T-unigram/unigram_freq.csv)）。
- 文献中出现频率最高的词必须仅包含字母字符，且其出现占比不超过总词数的7.5%。
- 词的分割以空白符为分隔依据。
训练集包含2022年12月1日之前发表的文献，验证集包含2022年12月1日至2023年1月3日之间发表的文献。
#### 源自S2AG的文献
S2AG语料库包含语义学者平台中所有文献的标题与摘要。截至2023年1月3日，未经过滤的S2AG语料库包含9110万篇文献与155亿个空白符分隔的Token。为生成peS2o V1，我们设置了如下筛选条件：
- 摘要必须为英文。
- 语言检测同样使用pycld3库。
- 标题必须为英文，或其平均单字对数概率高于`-20`。
- 摘要必须为英文。
- 摘要的平均单字对数概率需高于`-20`。
- 摘要的词数不得少于50个。
- 摘要的词数不得超过1000个。
- 标题与摘要的联合文本中，出现频率最高的词必须是长度≥2的字母词，或是以`a`开头且后续为长度≥2的字母词的组合。
- 文献的发表时间需晚于1969年。
#### 统计数据
| 数据集 | 划分集 | 文献数量 | 词数 |
|:-------:|:-------:|:-----------:|:--------------:|
|s2orc | 训练集 | 8,242,162 | 36,088,195,908 |
|s2orc | 验证集 | 51,323 | 255,139,074 |
|s2ag | 训练集 | 59,382,301 | 11,009,123,378 |
|s2ag | 验证集 | 111,228 | 24,398,512 |
[1]: https://aclanthology.org/2020.acl-main.447/
[2]: https://github.com/allenai/s2orc
提供机构：
SidhiPanda
