Public Utility Data Liberation Project (PUDL) Data Release

PUDL v2025.2.0 Data Release This is our regular quarterly release for 2025Q1. It includes updates to all the datasets that are published with quarterly or higher frequency, plus initial verisons of a few new data sources that have been in the works for a while. One major change this quarter is that we are now publishing all processed PUDL data as Apache Parquet files, alongside our existing SQLite databases. See Data Access for more on how to access these outputs. Some potentially breaking changes to be aware of: In the EIA Form 930 – Hourly and Daily Balancing Authority Operations Report a number of new energy sources have been added, and some old energy sources have been split into more granular categories. See Changes in energy source granularity over time. We are now running the EPA’s CAMD to EIA unit crosswalk code for each individual year starting from 2018, rather than just 2018 and 2021, resulting in more connections between these two datasets and changes to some sub-plant IDs. See the note below for more details. Many thanks to the organizations who make these regular updates possible! Especially GridLab, RMI, and the ZERO Lab at Princeton University. If you rely on PUDL and would like to help ensure that the data keeps flowing, please consider joining them as a PUDL Sustainer, as we are still fundraising for 2025. New Data EIA 176 Add a couple of semi-transformed interim EIA-176 (natural gas sources and dispositions) tables. They aren’t yet being written to the database, but are one step closer. See #3555 and PRs #3590, #3978. Thanks to @davidmudrauskas for moving this dataset forward. Extracted these interim tables up through the latest 2023 data release. See #4002 and #4004. EIA 860 Added EIA 860 Multifuel table. See #3438 and #3946. FERC 1 Added three new output tables containing granular utility accounting data. See #4057, #3642 and the table descriptions in the data dictionary: out_ferc1__yearly_detailed_income_statements out_ferc1__yearly_detailed_balance_sheet_assets out_ferc1__yearly_detailed_balance_sheet_liabilities SEC Form 10-K Parent-Subsidiary Ownership We have added some new tables describing the parent-subsidiary company ownership relationships reported in the SEC’s Form 10-K, Exhibit 21 “Subsidiaries of the Registrant”. Where possible these tables link the SEC filers or their subsidiary companies to the corresponding EIA utilities. This work was funded by a grant from the Mozilla Foundation. Most of the ML models and data preparation took place in the mozilla-sec-eia repository separate from the main PUDL ETL, as it requires processing hundreds of thousands of PDFs and the deployment of some ML experiment tracking infrastructure. The new tables are handed off as nearly finished products to the PUDL ETL pipeline. Note that these are preliminary, experimental data products and are known to be incomplete and to contain errors. Extracting data tables from unstructured PDFs and the SEC to EIA record linkage are necessarily probabalistic processes. See PRs #4026, #4031, #4035, #4046, #4048, #4050 and check out the table descriptions in the PUDL data dictionary: out_sec10k__parents_and_subsidiaries core_sec10k__quarterly_filings core_sec10k__quarterly_exhibit_21_company_ownership core_sec10k__quarterly_company_information Expanded Data Coverage EPA CEMS Added 2024 Q4 of CEMS data. See #4041 and #4052. EPA CAMD EIA Crosswalk In the past, the crosswalk in PUDL has used the EPA’s published crosswalk (run with 2018 data), and an additional crosswalk we ran with 2021 EIA 860 data. To ensure that the crosswalk reflects updates in both EIA and EPA data, we re-ran the EPA R code which generates the EPA CAMD EIA crosswalk with 4 new years of data: 2019, 2020, 2022 and 2023. Re-running the crosswalk pulls the latest data from the CAMD FACT API, which results in some changes to the generator and unit IDs reported on the EPA side of the crosswalk, which feeds into the creation of core_epa__assn_eia_epacamd. The changes only result in the addition of new units and generators in the EPA data, with no changes to matches at the plant level. However, the updates to generator and unit IDs have resulted in changes to the subplant IDs - some EIA boilers and generators which previously had no matches to EPA data have now been matched to EPA unit data, resulting in an overall reduction in the number of rows in the core_epa__assn_eia_epacamd_subplant_ids table. See issues #4039 and PR #4056 for a discussion of the changes observed in the course of this update. EIA 860M Added EIA 860m through December 2024. See #4038 and #4047. EIA 923 Added EIA 923 monthly data through September 2024. See #4038 and #4047. EIA Bulk Electricity Data Updated the EIA Bulk Electricity data to include data published up through 2024-11-01. See #4042 and PR #4051. EIA 930 Updated the EIA 930 data to include data published up through the beginning of February 2025. See #4040 and PR #4054. 10 new energy sources were added and 3 were retired; see Changes in energy source granularity over time for more information. Bug Fixes Fix an accidentally swapped set of starting balance / ending balance column rename parameters in the pre-2021 DBF derived data that feeds into core_ferc1__yearly_other_regulatory_liabilities_sched278. See issue #3952 and PRs #3969, #3979. Thanks to @yolandazzz13 for making this fix. Added preliminary data validation checks for several FERC 1 tables that were missing it #3860. Fix spelling of Lake Huron and Lake Saint Clair in out_vcerare__hourly_available_capacity_factor and related tables. See issue #4007 and PR #4029. Quality of Life Improvements We added a sources parameter to pudl.metadata.classes.DataSource.from_id() in order to make it possible to use the pudl-archiver repository to archive datasets that won’t necessarily be ingested into PUDL. See this PUDL archiver issue and PRs #4003 and #4013. Other PUDL v2025.2.0 Resources PUDL v2025.2.0 Data Dictionary PUDL v2025.2.0 Documentation PUDL in the AWS Open Data Registry PUDL v2025.2.0 in a free, public AWS S3 bucket: s3://pudl.catalyst.coop/v2025.2.0/ PUDL v2025.2.0 in a requester-pays GCS bucket: gs://pudl.catalyst.coop/v2025.2.0/ Zenodo archive of the PUDL GitHub repo for this release PUDL v2025.2.0 release on GitHub PUDL v2025.2.0 package in the Python Package Index (PyPI) Contact Us If you're using PUDL, we would love to hear from you! Even if it's just a note to let us know that you exist, and how you're using the software or data. Here's a bunch of different ways to get in touch: Follow us on GitHub Use the PUDL Github issue tracker to let us know about any bugs or data issues you encounter GitHub Discussions is where we provide user support. Watch our GitHub Project to see what we're working on. Email us at [e-mail] for private communications. On Mastodon: @[e-mail] On BlueSky: @catalyst.coop On Twitter: @CatalystCoop Connect with us on LinkedIn Play with our data and notebooks on Kaggle Combine our data with ML models on HuggingFace Learn more about us on our website: https://catalyst.coop Subscribe to our announcements list for email updates.