storytracer/openlibrary_dump_2024-04-30

--- license: unknown license_link: https://openlibrary.org/developers/licensing pretty_name: OpenLibrary Dump (2024-04-30) size_categories: - 10M<n<100M configs: - config_name: dumps default: true data_files: - split: authors path: "data/parquet/ol_dump_authors_2024-04-30.parquet" - split: works path: "data/parquet/ol_dump_works_2024-04-30.parquet" - split: editions path: "data/parquet/ol_dump_editions_2024-04-30.parquet" - config_name: summaries data_files: - split: authors path: "summaries/parquet/ol_dump_authors_2024-04-30_summary.parquet" - split: works path: "summaries/parquet/ol_dump_works_2024-04-30_summary.parquet" - split: editions path: "summaries/parquet/ol_dump_editions_2024-04-30_summary.parquet" --- # OpenLibrary Dump (2024-04-30) This dataset contains the [OpenLibrary dump](https://openlibrary.org/developers/dumps) of April 2024 converted to Parquet and DuckDB for easier querying. ## Formats ### Original GZIP dumps The original GZIP dumps are available at [data/dumps](https://huggingface.co/datasets/storytracer/openlibrary_dump_2024-04-30/tree/main/data/dumps). The dumps are gzipped TSV files with the original OL JSON record contained in the fifth column of the TSV. ### DuckDB The authors, works and editions dumps were imported as tables into [data/duckdb/ol_dump_2024-04-30.duckdb](https://huggingface.co/datasets/storytracer/openlibrary_dump_2024-04-30/blob/main/data/duckdb/ol_dump_2024-04-30.duckdb) using the script [ol_duckdb.sh](https://huggingface.co/datasets/storytracer/openlibrary_dump_2024-04-30/blob/main/ol_duckdb.sh). The scripts extract the JSON record from the fifth column and pipes it to DuckDB to import using the flags ```union_by_name=true``` and ```ignore_errors=true``` to account for the inconsistent JSON structure of the dumps. ### Parquet The authors, works and editions tables were exported as Parquet from DuckDB to [data/parquet](https://huggingface.co/datasets/storytracer/openlibrary_dump_2024-04-30/tree/main/data/parquet). These Parquet files are contained in the default ```data``` config for this dataset. ### Table Summaries To give users an easy overview of the fields contained in the dump, the DuckDB tables have been summarized using the ```SUMMARIZE``` function as Markdown and Parquet files at [summaries](https://huggingface.co/datasets/storytracer/openlibrary_dump_2024-04-30/tree/main/summaries), excluding the summary columns ```min```, ```max```, ```avg``` for easier viewing. You can also explore the table summaries using the dataset viewer by selecting the ```summaries``` config. The dumps fields are supposed to be consistent with the schema referenced [in the documentation](https://openlibrary.org/developers/dumps#:~:text=Format%20of%20JSON%20records). However, the summaries show that the dumps are not consistent with the documentation: the dumps contain some undocumented fields and some of the fields almost exclusively contain ```null``` values.license: 未知 license_link: https://openlibrary.org/developers/licensing pretty_name: OpenLibrary 转储文件（2024年4月30日） size_categories: - 1000万<n<1亿条数据 configs: - 配置名称: dumps 默认启用: 是 数据文件: - 数据子集: 作者集 路径: "data/parquet/ol_dump_authors_2024-04-30.parquet" - 数据子集: 作品集 路径: "data/parquet/ol_dump_works_2024-04-30.parquet" - 数据子集: 版本集 路径: "data/parquet/ol_dump_editions_2024-04-30.parquet" - 配置名称: summaries 数据文件: - 数据子集: 作者集 路径: "summaries/parquet/ol_dump_authors_2024-04-30_summary.parquet" - 数据子集: 作品集 路径: "summaries/parquet/ol_dump_works_2024-04-30_summary.parquet" - 数据子集: 版本集 路径: "summaries/parquet/ol_dump_editions_2024-04-30_summary.parquet" # OpenLibrary 转储文件（2024年4月30日） 本数据集包含2024年4月的[OpenLibrary转储文件](https://openlibrary.org/developers/dumps)，已转换为Parquet与DuckDB格式以方便查询。 ## 数据格式 ### 原始GZIP转储文件 原始GZIP转储文件可从[data/dumps](https://huggingface.co/datasets/storytracer/openlibrary_dump_2024-04-30/tree/main/data/dumps)获取。此类转储文件为经GZIP压缩的TSV文件，其TSV的第五列包含原始OpenLibrary JSON格式记录。 ### DuckDB格式 已通过脚本[ol_duckdb.sh](https://huggingface.co/datasets/storytracer/openlibrary_dump_2024-04-30/blob/main/ol_duckdb.sh)将作者、作品与版本转储文件导入为数据表，存入[data/duckdb/ol_dump_2024-04-30.duckdb](https://huggingface.co/datasets/storytracer/openlibrary_dump_2024-04-30/blob/main/data/duckdb/ol_dump_2024-04-30.duckdb)。该脚本从TSV文件的第五列提取JSON记录，并通过管道将其导入DuckDB，导入时使用了`union_by_name=true`和`ignore_errors=true`参数，以适配转储文件不一致的JSON结构。 ### Parquet格式 已从DuckDB中将作者、作品与版本数据表导出为Parquet格式，存入[data/parquet](https://huggingface.co/datasets/storytracer/openlibrary_dump_2024-04-30/tree/main/data/parquet)。此类Parquet文件包含于本数据集的默认`data`配置项中。 ### 数据表摘要 为方便用户快速了解转储文件包含的字段，已通过`SUMMARIZE`函数将DuckDB数据表生成为Markdown与Parquet格式的摘要文件，存放于[summaries](https://huggingface.co/datasets/storytracer/openlibrary_dump_2024-04-30/tree/main/summaries)，且为便于查看，移除了`min`、`max`、`avg`三个摘要列。用户亦可通过数据集查看器选择`summaries`配置项来浏览数据表摘要。 转储文件的字段理论上应与[官方文档](https://openlibrary.org/developers/dumps#:~:text=Format%20of%20JSON%20records)中提及的数据模式保持一致。但实际摘要显示，转储文件与文档并不相符：转储文件包含部分未在文档中说明的字段，且部分字段几乎仅包含`null`空值。

原始信息汇总

OpenLibrary Dump (2024-04-30) 数据集概述

基本信息

• 许可证: 未知
• 数据集名称: OpenLibrary Dump (2024-04-30)
• 大小: 10M<n<100M

数据配置

• 默认配置: dumps

  • 数据文件:
    • authors: 路径为 "data/parquet/ol_dump_authors_2024-04-30.parquet"
    • works: 路径为 "data/parquet/ol_dump_works_2024-04-30.parquet"
    • editions: 路径为 "data/parquet/ol_dump_editions_2024-04-30.parquet"

• 配置: summaries

  • 数据文件:
    • authors: 路径为 "summaries/parquet/ol_dump_authors_2024-04-30_summary.parquet"
    • works: 路径为 "summaries/parquet/ol_dump_works_2024-04-30_summary.parquet"
    • editions: 路径为 "summaries/parquet/ol_dump_editions_2024-04-30_summary.parquet"

数据格式

• 原始格式: GZIP 压缩的 TSV 文件，包含原始的 OL JSON 记录。
• 转换格式:
  • DuckDB: 使用脚本 ol_duckdb.sh 将数据导入到 data/duckdb/ol_dump_2024-04-30.duckdb。
  • Parquet: 从 DuckDB 导出到 data/parquet 目录。
  • 表摘要: 使用 SUMMARIZE 函数生成的 Markdown 和 Parquet 文件，位于 summaries 目录。

数据一致性

• 问题: 数据集中的字段与文档描述不完全一致，包含一些未记录的字段，且部分字段几乎全为 null 值。