Pre-processed COSMIC Ionospheric Scintillation and Space Weather Dataset (2007-2018)

# Pre-processed COSMIC Ionospheric Scintillation and Space Weather Dataset (2007-2018) ## Abstract This dataset provides a systematically pre-processed and spatio-temporally aggregated collection of ionospheric scintillation and solar-terrestrial space environment parameters, spanning a full solar cycle from 2007 to 2018. The data are primarily sourced from two components: 1) The `scnLv1` products from the Taiwan FORMOSAT-3/COSMIC mission, providing the S4 scintillation index; 2) The OMNIWeb database from NASA, which supplies key solar-terrestrial space environment parameters. This dataset is designed to provide a high-quality, machine-learning-ready benchmark for studying ionospheric scintillation phenomena, developing predictive models, and exploring its association with solar and geomagnetic activities. ## File Description This dataset is provided as a single CSV file. Each row represents aggregated data for a specific spatio-temporal grid cell (30-minute time window, 5°x5° latitude/longitude grid). ## Data Dictionary | Column Name | Description | Unit || :--- | :--- | :--- || `Year` | Year (UTC) | - || `Doy` | Day of Year, 1-366 (UTC) | - || `Hour` | Hour (UTC) | - || `Minute` | Minute (UTC), aggregation point (0 or 30) | - || `Lat` | Geodetic Latitude of the grid center | Degrees || `Lon` | Geodetic Longitude of the grid center | Degrees || `El` | Satellite Elevation Angle | Degrees || `caL1_SNR` | Carrier-aided L1 Signal-to-Noise Ratio (inferred) | dB-Hz || `S4` | S4 scintillation index, arithmetic mean within the spatio-temporal grid cell | Dimensionless || `sqrt_I` | Square root of signal intensity (inferred, related to amplitude scintillation) | - || `Label` | Scintillation severity label based on S4 index: 0 (Weak/None, S4<0.2), 1 (Moderate, 0.2<=S4<0.5), 2 (Strong, S4>=0.5) | - || `Bz` | Interplanetary Magnetic Field (IMF) Bz component (in GSM coordinates) | nT || `Vsw` | Solar wind speed | km/s || `Kp` | Global geomagnetic activity index | - || `R` | Sunspot number (inferred) | - || `Dst` | Disturbance storm time (Dst) index | nT || `ap` | Global geomagnetic activity index | nT || `f10.7` | 10.7cm solar radio flux index | s.f.u. || `AE` | Auroral Electrojet (AE) index | nT || `doy_sin` | Sine transformation of Doy to represent its seasonal cyclical nature | - || `doy_cos` | Cosine transformation of Doy to represent its seasonal cyclical nature | - || `hour_sin` | Sine transformation of Hour to represent its diurnal cyclical nature | - || `hour_cos` | Cosine transformation of Hour to represent its diurnal cyclical nature | - | --- ## Methodology ### 1. Data Sources* **Ionospheric Scintillation Data**: `scnLv1` products from the Taiwan FORMOSAT-3/COSMIC mission, covering the period 2007-2018.* **Space Environment Data**: NASA/GSFC OMNIWeb database, covering the period 2007-2018. ### 2. COSMIC Data Processing1. **Ionospheric Pierce Point (IPP) Calculation**: The LEO and GPS satellite positions (in ECF coordinates) from the raw data were used to calculate the point of closest approach to the Earth's center along the signal path, serving as the effective IPP for the scintillation event.2. **Coordinate System Conversion**: The calculated ECF coordinates were precisely converted to geodetic latitude, longitude, and altitude in the WGS84 coordinate system.3. **Time System Conversion**: GPS time (in GPS seconds) was converted to standard UTC and parsed into year, day of year, hour, and minute.4. **Altitude Filtering**: To focus on F-layer irregularities, the data was filtered to an altitude range of 150 km to 500 km.5. **Spatio-temporal Aggregation**: * **Temporal Dimension**: Data was aggregated into 30-minute time bins. * **Spatial Dimension**: The globe was divided into a 5° latitude × 5° longitude grid. * **Aggregation Method**: For each spatio-temporal grid cell, the arithmetic mean of all valid S4 index measurements was calculated.6. **S4 Index Processing**: * **Thresholding**: S4 index values were strictly constrained to the range [0, 1]. Any observations with S4 > 1 were considered anomalous and removed. * **Classification**: Based on the aggregated mean S4 value, scintillation severity was classified into three levels (see the description for the `Label` column above). ### 3. OMNIWeb Space Weather Data Processing* **Temporal Alignment**: All space weather parameters were aligned to the same 2007-2018 time span as the COSMIC data.* **Data Cleaning**: Invalid fill values (e.g., `999.9`) were identified and removed.* **Interpolation**: For the F10.7 index, which has a daily resolution, linear interpolation was used to fill potential missing values to ensure a complete time series. ### 4. Feature Engineering* To help machine learning models better capture temporal periodicities, the Day of Year (`Doy`) and Hour (`Hour`) were transformed into cyclical features using sine and cosine functions (`doy_sin`, `doy_cos`, `hour_sin`, `hour_cos`). ## License This dataset is licensed under the **Creative Commons Attribution 4.0 International (CC BY 4.0)** license. This means you are free to share and adapt the material for any purpose, provided you give appropriate credit, provide a link to the license, and indicate if changes were made. ## Contact For any questions regarding this dataset, please contact:* Yuan Xin - [e-mail]