PeytonT/1m_papers_text

这是一个去重的论文文本数据集，主要来源于arXiv的科学论文。数据集通过合并现有的论文文本parquet导出文件与一个或多个回填的parquet分片，确保每个`canonical_paper_id`只保留一行数据。在选择文本行时，优先选择非部分文本、原始PDF衍生的文本、较新版本以及较长的文本。数据集包含100万篇去重的论文，每篇论文对应一行数据，主要列包括论文ID、标题、摘要、作者、类别、许可证、文本内容等。

This dataset merges an existing paper-text parquet export with one or more backfill parquet shards, then keeps exactly one row per `canonical_paper_id`. One row corresponds to one canonical arXiv paper. Where multiple versions are available, the merge keeps the strongest text row according to the repos selection logic: prefer non-partial text, then prefer raw-PDF-derived text, then newer versions, then longer text. The dataset contains 1,000,000 deduped papers, with each paper corresponding to one row of data. The main columns include paper ID, title, abstract, authors, categories, license, text content, etc.