Compact representation of the OpenAIRE citation graph

When working with this dataset please cite to the accompanying article: Skarding, J. and Sanda, P. (2026) ‘Making the Complete OpenAIRE Citation Graph Easily Accessible Through Compact Data Representation’, Journal of Open Humanities Data, 12(1), p. 63. Available at: https://doi.org/10.5334/johd.520. We're making available a distilled version of the OpenAIRE citation graph. We share the complete graph as two files totalling ~11GB. We also provide a larger file including additonal publication fields (see table below). For complete description see the data paper at . There are three data files each served in two formats; the TSV (Tab-Separated Values) and Parquet : publications.tsv.xz and publications.parquet - The nodes in the citation graph, and their primary doi. citations.tsv.xz and citations.parquet - The edges in the citation graph publication_large.tsv.xz and publications_large.parquet - The nodes, but with several additional fields. pipeline.tar.xz - The pyspark pipeline used to produce the other files. Contains a singularity/apptainer container for reproducability and portability. The code is also found at Codeberg. All files are compressed (.xz or .parquet) files. Memory efficient loading of the Parquet files using Pandas For memory efficient loading of the citation.parquet and publications.parquet in loading, use the PyArrow backend: import pandas as pd df_pubs = pd.read_parquet( "publications.parquet", engine="pyarrow", dtype_backend="pyarrow", ) df_cites = pd.read_parquet( "citations.parquet", engine="pyarrow", dtype_backend="pyarrow", ) The large publications file e.g. publications_large.parquet, will not fit inside memory on most machines. It is however simple to select a subset of coloumns, and only load those. df_large = pd.read_parquet( "publications_large.parquet", columns=["nodeId", "title", "pid_dois"], engine="pyarrow", dtype_backend="pyarrow", ) Here, the nodeId, title, and pid_dois columns are selected. The fields in the publication_large: Field Type Explanation Memory usage (GB) Percentage of filled fields nodeId int32 Unique internal identifier for the node (publication) 0.8 100.00% openaireId str Identifier assigned by the OpenAIRE platform 9.6 100.00% title str Title of the publication 16.5 99.41% authors list[str] List of authors associated with the publication 11.0 83.84% description str Abstract or short description of the publication 131.3 57.17% date datetime Date when the publication was published 0.8 97.33% container str Journal, conference, or repository where it was published 2.2 68.45% citations int Number of times the publication has been cited 1.6 97.33% language str Language in which the publication is written 0.2 99.99% pid_dois list[str] DOI indentifiers 5.6 80.70% pid_mag_ids list[str] MAG IDs 2.0 44.27% pid_pmids list[str] PubMed IDs 1.2 18.18% pid_handles list[str] Persistent handles 1.1 8.33% pid_pmcs list[str] PubMed Central IDs 0.9 4.77% pid_arxiv_ids list[str] ArXiv IDs 0.9 1.38% Memory usage is the size the column takes up in memory, when loaded into a panda using the code snippets below. The percentage of filled fields shows the amount of non-null entries in each column.使用本数据集时，请引用随附的学术文章：Skarding J、Sanda P，2026年，《通过紧凑数据表示轻松获取完整的OpenAIRE引用图谱》，《Journal of Open Humanities Data》，12(1)，第63页。可访问：https://doi.org/10.5334/johd.520. 我们发布了OpenAIRE引用图谱的精简版本。完整图谱以两个总计约11GB的文件形式共享。此外我们还提供了包含额外文献元数据字段的更大尺寸文件（详见下表）。完整说明请参阅对应数据论文。 本数据集包含三类数据文件，每类均提供两种格式：TSV（制表符分隔值，Tab-Separated Values）与Parquet格式： 1. publications.tsv.xz 与 publications.parquet：引用图谱的节点及其主DOI（Digital Object Identifier）； 2. citations.tsv.xz 与 citations.parquet：引用图谱的边（即文献引用关系）； 3. publication_large.tsv.xz 与 publications_large.parquet：包含额外元数据字段的节点数据。 此外还有pipeline.tar.xz：用于生成其余数据文件的PySpark流水线，内含可复现且支持便携性的Singularity/Apptainer容器。相关代码亦可在Codeberg平台获取。 所有文件均为压缩格式（.xz或Parquet格式）。 ### 使用Pandas高效加载Parquet文件 若需高效加载citation.parquet与publications.parquet文件，请使用PyArrow后端，示例代码如下： import pandas as pd df_pubs = pd.read_parquet( "publications.parquet", engine="pyarrow", dtype_backend="pyarrow", ) df_cites = pd.read_parquet( "citations.parquet", engine="pyarrow", dtype_backend="pyarrow", ) 大型文献文件（如publications_large.parquet）在多数设备上无法完全载入内存，但可通过仅加载所需列的方式实现高效使用。以下为仅加载指定列的示例代码： df_large = pd.read_parquet( "publications_large.parquet", columns=["nodeId", "title", "pid_dois"], engine="pyarrow", dtype_backend="pyarrow", ) 此处选取了nodeId、title与pid_dois三列。 ### publication_large文件字段说明 publication_large文件包含的字段如下表所示： | 字段名 | 数据类型 | 字段说明 | 内存占用（GB） | 非空字段占比 | |-------|---------|---------|--------------|------------| | nodeId | int32 | 节点（文献）的唯一内部标识符 | 0.8 | 100.00% | | openaireId | str | OpenAIRE平台分配的标识符 | 9.6 | 100.00% | | title | str | 文献标题 | 16.5 | 99.41% | | authors | list[str] | 文献关联作者列表 | 11.0 | 83.84% | | description | str | 文献摘要或简短说明 | 131.3 | 57.17% | | date | datetime | 文献出版日期 | 0.8 | 97.33% | | container | str | 文献发表所在的期刊、会议或仓储 | 2.2 | 68.45% | | citations | int | 文献被引次数 | 1.6 | 97.33% | | language | str | 文献撰写语言 | 0.2 | 99.99% | | pid_dois | list[str] | DOI标识符列表 | 5.6 | 80.70% | | pid_mag_ids | list[str] | MAG标识符列表 | 2.0 | 44.27% | | pid_pmids | list[str] | PubMed标识符列表 | 1.2 | 18.18% | | pid_handles | list[str] | 持久标识符句柄列表 | 1.1 | 8.33% | | pid_pmcs | list[str] | PubMed Central标识符列表 | 0.9 | 4.77% | | pid_arxiv_ids | list[str] | ArXiv标识符列表 | 0.9 | 1.38% | 此处的内存占用为使用下述代码片段将字段载入Pandas数据框时，单列占用的内存大小。非空字段占比则表示各列中非空条目所占的比例。