open-alex

# OpenAlex: Complete Academic Research Database The world's scholarly research catalog, converted to analysis-ready Parquet. 114.1M records across 7 entity types. ## Table of Contents - [What is this?](#what-is-this) - [Quick start](#quick-start) - [Entity overview](#entity-overview) - [How entities connect](#how-entities-connect) - [Schema reference](#schema-reference) - [Working with abstracts](#working-with-abstracts) - [Pipeline details](#pipeline-details) - [Things to know](#things-to-know) - [Attribution](#attribution) ## What is this? [OpenAlex](https://openalex.org) is a free, open catalog of the global research system: papers, authors, institutions, journals, topics, publishers, and funders. It's maintained by [OurResearch](https://ourresearch.org/) as the open replacement for the discontinued Microsoft Academic Graph. The index currently covers over 250 million scholarly works with full citation networks, authorship chains, institutional affiliations, and topic classifications. This dataset is a straight conversion of the [OpenAlex snapshot](https://docs.openalex.org/download-all-data/openalex-snapshot) (2026-04) from gzipped JSON Lines into sharded, ZSTD-compressed Parquet. **114.1M total records** split into files of up to 1 million rows each. You can query it directly with DuckDB (no download needed), stream it with the `datasets` library, or pull specific entities with `huggingface_hub`. ### File layout ``` data/ works/ works-00000.parquet scholarly works (~1M rows each) works-00001.parquet ... authors/ authors-00000.parquet researchers and their metrics ... sources/ sources-00000.parquet journals, repositories, conferences institutions/ institutions-00000.parquet universities, labs, companies topics/ topics-00000.parquet research topic taxonomy publishers/ publishers-00000.parquet academic publishers funders/ funders-00000.parquet funding organizations ``` Nested fields (authorships, locations, topics, etc.) are stored as JSON strings. Use `json_extract()` in DuckDB or `json.loads()` in Python to work with them. ## Quick start ### DuckDB (no download required) DuckDB can read Parquet files directly from Hugging Face. This is the fastest way to explore: ```sql -- Most-cited works of all time SELECT id, title, publication_year, cited_by_count, doi, oa_status FROM 'hf://datasets/open-index/open-alex/data/works/*.parquet' WHERE cited_by_count > 1000 ORDER BY cited_by_count DESC LIMIT 20; ``` ```sql -- Top authors by h-index SELECT id, display_name, h_index, i10_index, works_count, cited_by_count FROM 'hf://datasets/open-index/open-alex/data/authors/*.parquet' ORDER BY h_index DESC LIMIT 20; ``` ```sql -- Open access rates by year SELECT publication_year, COUNT(*) as total, SUM(CASE WHEN is_oa THEN 1 ELSE 0 END) as oa_count, ROUND(100.0 * SUM(CASE WHEN is_oa THEN 1 ELSE 0 END) / COUNT(*), 1) as oa_pct FROM 'hf://datasets/open-index/open-alex/data/works/*.parquet' WHERE publication_year BETWEEN 2000 AND 2025 GROUP BY publication_year ORDER BY publication_year; ``` ```sql -- Top institutions in a country SELECT display_name, type, geo_city, works_count, cited_by_count, h_index FROM 'hf://datasets/open-index/open-alex/data/institutions/*.parquet' WHERE country_code = 'US' ORDER BY works_count DESC LIMIT 20; ``` ```sql -- Extract author affiliations from nested JSON SELECT id, display_name, json_extract_string(last_known_institutions, '$[0].display_name') as institution, json_extract_string(last_known_institutions, '$[0].country_code') as country FROM 'hf://datasets/open-index/open-alex/data/authors/*.parquet' WHERE last_known_institutions IS NOT NULL ORDER BY h_index DESC LIMIT 20; ``` ```sql -- Join works to their first author SELECT w.title, w.publication_year, w.cited_by_count, a.display_name, a.h_index FROM 'hf://datasets/open-index/open-alex/data/works/*.parquet' w, 'hf://datasets/open-index/open-alex/data/authors/*.parquet' a WHERE w.cited_by_count > 5000 AND a.id = json_extract_string(w.authorships, '$[0].author.id') ORDER BY w.cited_by_count DESC LIMIT 20; ``` ```sql -- Citation distribution percentiles SELECT percentile_disc(0.50) WITHIN GROUP (ORDER BY cited_by_count) AS p50, percentile_disc(0.90) WITHIN GROUP (ORDER BY cited_by_count) AS p90, percentile_disc(0.99) WITHIN GROUP (ORDER BY cited_by_count) AS p99, AVG(cited_by_count) AS mean FROM read_parquet('hf://datasets/open-index/open-alex/data/works/*.parquet'); ``` ### Python (datasets library) ```python from datasets import load_dataset # Stream works without downloading everything ds = load_dataset("open-index/open-alex", "works", split="train", streaming=True) for work in ds: print(work["id"], work["title"], work["cited_by_count"]) # Load smaller entities into memory authors = load_dataset("open-index/open-alex", "authors", split="train") topics = load_dataset("open-index/open-alex", "topics", split="train") ``` ### Downloading specific entities ```python from huggingface_hub import snapshot_download # Just the small entities (~500 MB total) snapshot_download( "open-index/open-alex", repo_type="dataset", local_dir="./openalex/", allow_patterns=["data/topics/*", "data/publishers/*", "data/funders/*", "data/sources/*", "data/institutions/*"], ) # For faster downloads: # pip install huggingface_hub[hf_transfer] # HF_HUB_ENABLE_HF_TRANSFER=1 ``` ## Entity overview | Entity | Records | What's in it | |---|---|---| | **Topics** | 4.5K | Research topics with hierarchical classification (domain → field → subfield → topic) | | **Publishers** | 10.7K | Academic publishers with hierarchy levels and country information | | **Funders** | 32.4K | Research funding organizations with award counts and cross-references | | **Sources** | 280.7K | Journals, repositories, conferences, and ebook platforms with ISSN, DOAJ status, and APC pricing | | **Institutions** | 121.5K | Universities, research centers, companies, and government bodies with ROR IDs and geolocation | | **Authors** | 113.6M | Researchers with ORCID IDs, h-index, affiliations, and publication statistics | | **Works** | n/a | Scholarly works (articles, books, datasets) with citations, DOIs, topics, authorships, and open access status | ## How entities connect Works sit at the center. Everything else links through them. ``` +---------------+ | Works | (central entity) +-------+-------+ +----------------+------------------+ | | | +------v------+ +-----v------+ +---------v--------+ | Authorships | | Locations | | Referenced Works | | (nested) | | (nested) | | (citations) | +------+------+ +-----+------+ +------------------+ | | +------v------+ +---v------+ | Authors | | Sources | journals, repos, conferences +------+------+ +---+------+ | | +----------v------+ +---v--------+ | Institutions | | Publishers | +-----------------+ +------------+ Topics: domain > field > subfield > topic (4-level hierarchy) Funders: linked to works through grants and awards ``` Authorships, locations, and topics are nested as JSON inside works. To join works with their authors, parse the `authorships` field. ## Schema reference ### Topics Research topics with hierarchical classification (domain → field → subfield → topic). | Column | Type | |---|---| | `id` | string | | `display_name` | string | | `description` | string | | `keywords` | string | | `subfield_id` | string | | `subfield_name` | string | | `field_id` | string | | `field_name` | string | | `domain_id` | string | | `domain_name` | string | | `siblings` | string | | `works_count` | int32 | | `cited_by_count` | int32 | | `ids` | string | | `created_date` | string | | `updated_date` | string | ### Publishers Academic publishers with hierarchy levels and country information. | Column | Type | |---|---| | `id` | string | | `display_name` | string | | `alternate_titles` | string | | `hierarchy_level` | int32 | | `parent_publisher` | string | | `country_codes` | string | | `homepage_url` | string | | `works_count` | int32 | | `cited_by_count` | int32 | | `h_index` | int32 | | `i10_index` | int32 | | `lineage` | string | | `roles` | string | | `counts_by_year` | string | | `ids` | string | | `created_date` | string | | `updated_date` | string | **Data completeness** (fields below 100%): | Field | Fill rate | Est. records | |---|---|---| | `alternate_titles` | 10.0% | 1.1K | | `parent_publisher` | 0.0% | 0 | | `country_codes` | 90.0% | 9.6K | | `homepage_url` | 80.0% | 8.6K | | `counts_by_year` | 90.0% | 9.6K | ### Funders Research funding organizations with award counts and cross-references. | Column | Type | |---|---| | `id` | string | | `display_name` | string | | `alternate_titles` | string | | `country_code` | string | | `description` | string | | `homepage_url` | string | | `works_count` | int32 | | `cited_by_count` | int32 | | `awards_count` | int32 | | `h_index` | int32 | | `i10_index` | int32 | | `roles` | string | | `counts_by_year` | string | | `ids` | string | | `created_date` | string | | `updated_date` | string | **Data completeness** (fields below 100%): | Field | Fill rate | Est. records | |---|---|---| | `alternate_titles` | 87.5% | 28.4K | | `description` | 56.2% | 18.2K | | `homepage_url` | 53.1% | 17.2K | ### Sources Journals, repositories, conferences, and ebook platforms with ISSN, DOAJ status, and APC pricing. | Column | Type | |---|---| | `id` | string | | `issn_l` | string | | `issn` | string | | `display_name` | string | | `type` | string | | `host_organization` | string | | `host_organization_name` | string | | `works_count` | int32 | | `cited_by_count` | int32 | | `is_oa` | bool | | `is_in_doaj` | bool | | `is_core` | bool | | `homepage_url` | string | | `country_code` | string | | `h_index` | int32 | | `i10_index` | int32 | | `apc_usd` | int32 | | `alternate_titles` | string | | `topics` | string | | `counts_by_year` | string | | `ids` | string | | `created_date` | string | | `updated_date` | string | **Data completeness** (fields below 100%): | Field | Fill rate | Est. records | |---|---|---| | `issn_l` | 61.4% | 172.4K | | `issn` | 61.4% | 172.4K | | `host_organization` | 25.7% | 72.2K | | `host_organization_name` | 25.4% | 71.2K | | `homepage_url` | 26.4% | 74.2K | | `country_code` | 42.5% | 119.3K | | `apc_usd` | 3.2% | 9.0K | | `alternate_titles` | 23.2% | 65.2K | | `topics` | 92.9% | 260.6K | | `counts_by_year` | 93.2% | 261.6K | | `created_date` | 93.2% | 261.6K | ### Institutions Universities, research centers, companies, and government bodies with ROR IDs and geolocation. | Column | Type | |---|---| | `id` | string | | `ror` | string | | `display_name` | string | | `type` | string | | `country_code` | string | | `homepage_url` | string | | `image_url` | string | | `works_count` | int32 | | `cited_by_count` | int32 | | `h_index` | int32 | | `i10_index` | int32 | | `geo_city` | string | | `geo_region` | string | | `geo_country` | string | | `geo_latitude` | float64 | | `geo_longitude` | float64 | | `associated_institutions` | string | | `lineage` | string | | `topics` | string | | `counts_by_year` | string | | `roles` | string | | `ids` | string | | `created_date` | string | | `updated_date` | string | **Data completeness** (fields below 100%): | Field | Fill rate | Est. records | |---|---|---| | `country_code` | 94.2% | 114.5K | | `homepage_url` | 99.2% | 120.5K | | `image_url` | 10.7% | 13.1K | | `geo_region` | 38.0% | 46.2K | | `associated_institutions` | 32.2% | 39.2K | | `topics` | 88.4% | 107.5K | | `counts_by_year` | 86.8% | 105.4K | ### Authors Researchers with ORCID IDs, h-index, affiliations, and publication statistics. | Column | Type | |---|---| | `id` | string | | `orcid` | string | | `display_name` | string | | `display_name_alternatives` | string | | `works_count` | int32 | | `cited_by_count` | int32 | | `h_index` | int32 | | `i10_index` | int32 | | `two_yr_mean_citedness` | float64 | | `affiliations` | string | | `last_known_institutions` | string | | `topics` | string | | `topic_share` | string | | `counts_by_year` | string | | `ids` | string | | `created_date` | string | | `updated_date` | string | **Data completeness** (fields below 100%): | Field | Fill rate | Est. records | |---|---|---| | `orcid` | 7.1% | 8.0M | | `affiliations` | 44.5% | 50.6M | | `last_known_institutions` | 39.4% | 44.7M | | `topics` | 97.2% | 110.5M | | `topic_share` | 97.2% | 110.5M | | `counts_by_year` | 99.6% | 113.2M | ### Works Scholarly works (articles, books, datasets) with citations, DOIs, topics, authorships, and open access status. | Column | Type | |---|---| | `id` | string | | `doi` | string | | `title` | string | | `publication_year` | int32 | | `publication_date` | string | | `type` | string | | `language` | string | | `is_retracted` | bool | | `is_paratext` | bool | | `cited_by_count` | int32 | | `fwci` | float64 | | `referenced_works_count` | int32 | | `authors_count` | int32 | | `locations_count` | int32 | | `is_oa` | bool | | `oa_status` | string | | `oa_url` | string | | `pr