Public Utility Data Liberation Project (PUDL) Data Release

PUDL v2024.2.6 Data Release The main impetus behind this release is the quarterly update of some of ourcore datasets with preliminary data for 2023Q4. The EIA Form 860 – Annual Electric Generator Report, EPA Hourly Continuous Emission Monitoring System (CEMS), and bulk EIA API data are all up to date through the end of 2023, while the EIA Form 923 – Power Plant Operations Report lags a month behind and is currently only available through November, 2023. We also addressed several issues we found in our initial release automation process that will make it easier for us to do more frequent releases, like this one! We’re also for the first time publishing the full historical time series of of generator data available in the EIA860M, rather than just using the most recent release to update the EIA860 outputs. This enables tracking of how planned fossil plant retirement dates have evolved over time. There are also updates to our data validation system, a new version of Pandas, and experimental Parquet outputs. See below for the details. New Data Coverage Add EIA860M data through December 2023 #3313, #3367. Add 2023 Q4 of CEMS data. See #3315, #3379. Add EIA923 monthly data through November 2023 #3314, #3398, #3422. Create a new table core_eia860m__changelog_generators which tracks the evolution of all generator data reported in the EIA860M, in particular the stated retirement dates. see issue #3330 and PR #3331. Previously only the most recent month of reported EIA860M data was available within the PUDL DB. Release Infrastructure Use the same logic to merge version tags into the stable branch as we are using to merge the nightly build tags into the nightly branch. See PR #3347 Automatically place a temporary object hold on all versioned data releases that we publish to GCS, to ensure that they can’t be accidentally deleted. See issue #3400 and PR #3421. Schema Changes Restored the individual FERC Form 1 plant output tables, providing direct access to denormalized versions of the specific plant types via: out_ferc1__yearly_steam_plants_sched402 out_ferc1__yearly_small_plants_sched410 out_ferc1__yearly_hydroelectric_plants_sched406 out_ferc1__yearly_pumped_storage_plants_sched408 See issue #3416 & PR #3417 Data Validation with Pandera We’ve started integrating pandera dataframe schemas and checks with dagster asset checks to validate data while our ETL pipeline is running instead of only after all the data has been produced. Initially we are using the various database schema checks that are generated by our metadata, but the goal is to migrate all of our data validation tests into this framework over time, and to start using it to encode any new data validations immediately. See issues #941, #1572, #3318, #3412 and PR #3282. Pandas 2.2 We’ve updated to Pandas 2.2, which has a number of changes and deprecations. See PRs #3272, #3410. Changes in how merge results are sorted impacted the assignment of unit_id_pudl values, so any hard-coded values that dependent on the previous assignments will likely be incorrect now. We had to update a number of tests and FERC1-EIA record linkage training data to account for this change. Pandas is also deprecating the use of the AS frequency alias, in favor of YS, so many references to the old alias have been updated. We’ve switched to using the calamine engine for reading Excel files, which is much faster than the old openpyxl library. Parquet Outputs The ETL now outputs PyArrow Parquet files for all tables that are written to the PUDL DB. The Parquet outputs are used as the interim storage for the ETL, rather than reading all tables out of the SQLite DB. We aren’t publicly distributing the Parquet outputs yet, but are giving them a test run with some existing users. See #3102 #3296, #3399. Other PUDL v2024.2.6 Resources PUDL v2024.2.6 Data Dictionary PUDL v2024.2.6 Documentation PUDL in the AWS Open Data Registry PUDL v2024.2.6 in a free, public AWS S3 bucket: s3://pudl.catalyst.coop/v2024.2.6/ PUDL v2024.2.6 in a requester-pays GCS bucket: gs://pudl.catalyst.coop/v2024.2.6/ Zenodo archive of the PUDL GitHub repo for this release PUDL v2024.2.6 release on GitHub PUDL v2024.2.6 package in the Python Package Index (PyPI) Contact Us If you're using PUDL, we would love to hear from you! Even if it's just a note to let us know that you exist, and how you're using the software or data. Here's a bunch of different ways to get in touch: Follow us on GitHub Use the PUDL Github issue tracker to let us know about any bugs or data issues you encounter GitHub Discussions is where we provide user support. Watch our GitHub Project to see what we're working on. Email us at [e-mail] for private communications. On Mastodon: @[e-mail] On BlueSky: @catalyst.coop On Twitter: @CatalystCoop Play with our data and notebooks on Kaggle Combine our data with ML models on HuggingFace Learn more about us on our website: https://catalyst.coop Subscribe to our announcements list for email updates.