Pritama/peS2o

peS2o数据集是一个用于语言模型预训练的学术论文集合，包含约4000万篇开放获取的学术论文，这些论文经过清洗、过滤和格式化处理。该数据集源自Semantic Scholar开放研究语料库（S2ORC），由Allen Institute for AI发布。数据集有多个版本（如V1和V2），每个版本具有不同的处理方式和知识截止日期（例如，V2版本的知识截止日期为2023年1月3日）。文档格式包括添加日期、创建日期、语义学者语料库ID、来源（如s2orc或s2ag）和文本内容（段落由两个换行符分隔）。数据集覆盖广泛的学科领域，包括生物学、化学、工程学、计算机科学、物理学、材料科学、数学、心理学、经济学、政治学、商业、地质学、社会学、地理学、环境科学、艺术、历史和哲学。处理过程包括语言检测（使用pycld3库）、OCR错误过滤（针对V2版本）、字数限制和出版日期筛选（例如，排除1969年以前的论文）。统计信息显示，V2版本包含约3897万篇文档和约420.1亿个以空格分隔的词元，分为训练集和验证集。该数据集适用于文本生成和掩码填充等自然语言处理任务。

The peS2o dataset is a collection of approximately 40 million open-access academic papers, cleaned, filtered, and formatted for pre-training of language models. It is derived from the Semantic Scholar Open Research Corpus (S2ORC) and released by the Allen Institute for AI. Multiple versions are available (e.g., V1 and V2), each with different processing methods and knowledge cutoff dates (e.g., V2 has a cutoff of January 3, 2023). Document formats include fields such as added date, created date, Semantic Scholar Corpus ID, source (e.g., s2orc or s2ag), and text (with paragraphs separated by two newlines). The dataset spans a wide range of disciplines, including biology, chemistry, engineering, computer science, physics, material science, math, psychology, economics, political science, business, geology, sociology, geography, environmental science, art, history, and philosophy. Processing involves language detection (using the pycld3 library), OCR error filtering (for V2), word count limits, and publication date filtering (e.g., excluding papers published before 1969). Statistics indicate that V2 contains approximately 38.97 million documents and 42.01 billion whitespace-separated tokens, divided into training and validation splits. It is suitable for natural language processing tasks such as text generation and fill-mask.