scilons/SciLaD-all-markdown-v1

SciLaD是一个新颖、大规模的科学语言数据集，完全基于开源框架和公开可用的数据源构建。它包含一个经过整理的英文部分，涵盖超过1000万篇科学出版物，以及一个多语言、未过滤的TEI XML部分，包括超过3500万篇出版物。我们还发布了用于生成SciLaD的可扩展管道。SciLaD is a novel, large-scale dataset of scientific language constructed entirely using open-source frameworks and publicly available data sources. It comprises a curated English split containing over 10 million scientific publications and a multilingual, unfiltered TEI XML split including more than 35 million publications. We also publish the extensible pipeline for generating SciLaD.