Makamaka77/structured-wikipedia

Wikimedia结构化维基百科数据集是一个包含预解析英文和法文维基百科文章的结构化数据集，通过Wikimedia Enterprise快照API提取。该数据集涵盖了英文和法文维基百科的所有文章，以一致的架构预解析并输出为结构化数据。数据以Parquet格式提供，优化了高性能分析查询和高效存储。此版本使用统一的固定架构，兼容DuckDB、pandas、Polars和Apache Spark。数据集新增了以下特性：解析的参考文献和引用，连接维基百科知识与其来源；解析的表格，这是维基百科页面信息最密集的部分之一；可信度信号（如referenceneed和referenceryisk），指示信息可能未得到充分来源支持；列表解析改进，包括嵌套列表、有序列表和定义列表；以及文章正文图像的解析。数据集总行数为10,468,881，总大小为44.42 GiB，包含英文（7,597,149行，34.61 GiB）和法文（2,871,732行，9.81 GiB）两个语言版本。每行代表一个维基百科文章快照，包含名称、标识符、URL、版本、摘要、描述、图像、信息框、章节、表格、参考文献等字段。

The Wikimedia Structured Wikipedia dataset is a structured dataset containing pre-parsed English and French Wikipedia articles, extracted using the Wikimedia Enterprise Snapshot API. It includes all articles from the English and French language editions of Wikipedia, pre-parsed and output as structured data with a consistent schema. The dataset is provided in Parquet format, optimized for high-performance analytical queries and efficient storage. This version uses a unified, pinned schema across all files, making it compatible with DuckDB, pandas, Polars, and Apache Spark out of the box. New features in this dataset include: parsed references and citations, connecting Wikipedias knowledge with its sources of truth; parsed tables, one of the most information-heavy sections of Wikipedia pages; credibility signals (e.g., referenceneed and referenceryisk), signaling where information may not be sufficiently backed by sources; improvements to list parsing, including nested lists, ordered lists, and definition lists; and parsed article-body images. The dataset has a total of 10,468,881 rows and a size of 44.42 GiB, with English (7,597,149 rows, 34.61 GiB) and French (2,871,732 rows, 9.81 GiB) language editions. Each row represents one Wikipedia article snapshot, containing fields such as name, identifier, url, version, abstract, description, image, infoboxes, sections, tables, and references.