chosummingcuhk/wikipedia-citations-enwiki-20260101

该数据集是从英文维基百科中提取的引用（citations）集合，旨在检查这些引用中是否存在AI生成或幻觉内容（hallucinated content）。数据来源于2026年1月1日的维基百科转储（dump），并可能扩展到2026年4月1日的转储。数据集分为两个部分：sorted.parquet文件包含每个引用块及其对应的维基百科文章标题；tags.parquet文件则将引用的`tags = rest`部分按标签分组。处理过程包括解压缩XML转储、解析页面节点、提取标题和Wikitext，并使用正则表达式和Polars工具进行高效数据操作。数据集还涉及对arXiv、DOI和ISBN引用的初步分析，并特别处理了超长引用（超过10,000字符）。整体目标是评估维基百科引用的真实性，为研究AI幻觉内容提供基础数据。This dataset extracts citations from the English Wikipedia to check for AI-generated or hallucinated content. It is based on the Wikipedia dump from 2026/01/01, with potential extension to the 2026/04/01 dump. The dataset consists of two parts: sorted.parquet, where each row contains a citation block and the associated Wikipedia article title, and tags.parquet, which groups the `tags = rest` sections of citations by tags. The processing involves decompressing the XML dump, parsing page nodes to extract titles and Wikitext, and using regex and Polars for efficient data manipulation. It includes preliminary analysis of arXiv, DOI, and ISBN citations, and separately handles jumbo citations (over 10,000 characters). The overall aim is to assess the authenticity of Wikipedia citations and provide baseline data for studying AI hallucination in content.