DMSP Particle Precipitation AI-ready Data

Description: 

The dataset ‘DMSP Particle Precipitation AI-ready Data’ accompanies the manuscript “Next generation particle precipitation: Mesoscale prediction through machine learning (a case study and framework for progress)” submitted to AGU Space Weather Journal and used to produce new machine learning models of particle precipitation from the magnetosphere to the ionosphere. Note that we have attempted to make these data ready to be used in artificial intelligence/machine learning explorations following a community definition of ‘AI-ready’ provided at https://github.com/rmcgranaghan/data_science_tools_and_resources/wiki/Curated-Reference%7CChallenge-Data-Sets

The purpose of publishing these data is two-fold: 

To allow reuse of the data that led to the manuscript and extension, rather than reinvention, of the research produced there; and 

To be an ‘AI-ready’ challenge data set to which the artificial intelligence/machine learning community can apply novel methods. 

These data were compiled, curated, and explored by: Ryan McGranaghan, Enrico Camporeale, Kristina Lynch, Jack Ziegler, Téo Bloch, Mathew Owens, Jesper Gjerloev, Spencer Hatch, Binzheng Zhang, and Susan Skone

Pipeline for creation:

The steps to create the data were (Note that we do not provide intermediate datasets):

Access NASA-provided DMSP data at https://cdaweb.gsfc.nasa.gov/pub/data/dmsp/

Read CDF files for given satellite (e.g., F-16)

Collect the following variables at one-second cadence: SC_AACGM_LAT, SC_AACGM_LTIME, ELE_TOTAL_ENERGY_FLUX, ELE_TOTAL_ENERGY_FLUX_STD, ELE_AVG_ENERGY, ELE_AVG_ENERGY_STD, ID_SC

Sub-sample the variables to one-minute cadence and eliminate any rows for which ELE_TOTAL_ENERGY_FLUX is NaN

Combine all individual satellites into single yearly files

For each yearly file, use nasaomnireader to obtain solar wind and geomagnetic index data programmatically and timehist2 to calculate the time histories of each parameter. Collate with the DMSP observations and remove rows for which any solar wind or geomagnetic index data are missing.

For each row, calculate cyclical time variables (e.g., local time -> sin(LT) and cos(LT))

Merge all years

How to use:

The Github repository https://github.com/rmcgranaghan/precipNet is provided to detail the use of these data and to provide Jupyter notebooks to facilitate getting started. The code is implemented in Python 3 and is licensed under the GNU General Public License v3.0

Citation: 

For anyone using these data, please cite each of the following papers: 

McGranaghan, R. M., Ziegler, J., Bloch, T., Hatch, S., Camporeale, E., Lynch, K., et al. (2021). Toward a next generation particle precipitation model: Mesoscale prediction through machine learning (a case study and framework for progress). Space Weather, 19, e2020SW002684. https://doi.org/10.1029/2020SW002684

McGranaghan, R. (2019), Eight lessons I learned leading a scientific “design sprint”, Eos, 100, https://doi.org/10.1029/2019EO136427. Published on 11 November 2019.

For questions or comments please contact Ryan McGranaghan ([e-mail])