Real-time testing data for: "Identifying data sources and physical strategies used by neural networks to predict TC rapid intensification"

Each file in the dataset contains machine-learning-ready data for one unique tropical cyclone (TC) from the real-time testing dataset.  "Machine-learning-ready" means that all data-processing methods described in the journal paper have already been applied.  This includes cropping satellite images to make them TC-centered; rotating satellite images to align them with TC motion (TC motion is always towards the +x-direction, or in the direction of increasing column number); flipping satellite images in the southern hemisphere upside-down; and normalizing data via the two-step procedure.

The file name gives you the unique identifier of the TC -- e.g., "learning_examples_2010AL01.nc.gz" contains data for storm 2010AL01, or the first North Atlantic storm of the 2010 season.  Each file can be read with the method `example_io.read_file` in the ml4tc Python library (https://zenodo.org/doi/10.5281/zenodo.10268620).  However, since `example_io.read_file` is a lightweight wrapper for `xarray.open_dataset`, you can equivalently just use `xarray.open_dataset`.  Variables in the table are listed below (the same printout produced by `print(xarray_table)`):

Dimensions:                                          (                                                      satellite_valid_time_unix_sec: 289,                                                      satellite_grid_row: 380,                                                      satellite_grid_column: 540,                                                      satellite_predictor_name_gridded: 1,                                                      satellite_predictor_name_ungridded: 16,                                                      ships_valid_time_unix_sec: 19,                                                      ships_storm_object_index: 19,                                                      ships_forecast_hour: 23,                                                      ships_intensity_threshold_m_s01: 21,                                                      ships_lag_time_hours: 5,                                                      ships_predictor_name_lagged: 17,                                                      ships_predictor_name_forecast: 129)Coordinates:  * satellite_grid_row                               (satellite_grid_row) int32 2kB ...  * satellite_grid_column                            (satellite_grid_column) int32 2kB ...  * satellite_valid_time_unix_sec                    (satellite_valid_time_unix_sec) int32 1kB ...  * ships_lag_time_hours                             (ships_lag_time_hours) float64 40B ...  * ships_intensity_threshold_m_s01                  (ships_intensity_threshold_m_s01) float64 168B ...  * ships_forecast_hour                              (ships_forecast_hour) int32 92B ...  * satellite_predictor_name_gridded                 (satellite_predictor_name_gridded) object 8B ...  * satellite_predictor_name_ungridded               (satellite_predictor_name_ungridded) object 128B ...  * ships_valid_time_unix_sec                        (ships_valid_time_unix_sec) int32 76B ...  * ships_predictor_name_lagged                      (ships_predictor_name_lagged) object 136B ...  * ships_predictor_name_forecast                    (ships_predictor_name_forecast) object 1kB ...Dimensions without coordinates: ships_storm_object_indexData variables:    satellite_number                                 (satellite_valid_time_unix_sec) int32 1kB ...    satellite_band_number                            (satellite_valid_time_unix_sec) int32 1kB ...    satellite_band_wavelength_micrometres            (satellite_valid_time_unix_sec) float64 2kB ...    satellite_longitude_deg_e                        (satellite_valid_time_unix_sec) float64 2kB ...    satellite_cyclone_id_string                      (satellite_valid_time_unix_sec) |S8 2kB ...    satellite_storm_type_string                      (satellite_valid_time_unix_sec) |S2 578B ...    satellite_storm_name                             (satellite_valid_time_unix_sec) |S10 3kB ...    satellite_storm_latitude_deg_n                   (satellite_valid_time_unix_sec) float64 2kB ...    satellite_storm_longitude_deg_e                  (satellite_valid_time_unix_sec) float64 2kB ...    satellite_storm_intensity_number                 (satellite_valid_time_unix_sec) float64 2kB ...    satellite_storm_u_motion_m_s01                   (satellite_valid_time_unix_sec) float64 2kB ...    satellite_storm_v_motion_m_s01                   (satellite_valid_time_unix_sec) float64 2kB ...    satellite_predictors_gridded                     (satellite_valid_time_unix_sec, satellite_grid_row, satellite_grid_column, satellite_predictor_name_gridded) float64 474MB ...    satellite_grid_latitude_deg_n                    (satellite_valid_time_unix_sec, satellite_grid_row, satellite_grid_column) float64 474MB ...    satellite_grid_longitude_deg_e                   (satellite_valid_time_unix_sec, satellite_grid_row, satellite_grid_column) float64 474MB ...    satellite_predictors_ungridded                   (satellite_valid_time_unix_sec, satellite_predictor_name_ungridded) float64 37kB ...    ships_storm_intensity_m_s01                      (ships_valid_time_unix_sec) float64 152B ...    ships_storm_type_enum                            (ships_storm_object_index, ships_forecast_hour) int32 2kB ...    ships_forecast_latitude_deg_n                    (ships_storm_object_index, ships_forecast_hour) float64 3kB ...    ships_forecast_longitude_deg_e                   (ships_storm_object_index, ships_forecast_hour) float64 3kB ...    ships_v_wind_200mb_0to500km_m_s01                (ships_storm_object_index, ships_forecast_hour) float64 3kB ...    ships_vorticity_850mb_0to1000km_s01              (ships_storm_object_index, ships_forecast_hour) float64 3kB ...    ships_vortex_latitude_deg_n                      (ships_storm_object_index, ships_forecast_hour) float64 3kB ...    ships_vortex_longitude_deg_e                     (ships_storm_object_index, ships_forecast_hour) float64 3kB ...    ships_mean_tangential_wind_850mb_0to600km_m_s01  (ships_storm_object_index, ships_forecast_hour) float64 3kB ...    ships_max_tangential_wind_850mb_m_s01            (ships_storm_object_index, ships_forecast_hour) float64 3kB ...    ships_mean_tangential_wind_1000mb_at500km_m_s01  (ships_storm_object_index, ships_forecast_hour) float64 3kB ...    ships_mean_tangential_wind_850mb_at500km_m_s01   (ships_storm_object_index, ships_forecast_hour) float64 3kB ...    ships_mean_tangential_wind_500mb_at500km_m_s01   (ships_storm_object_index, ships_forecast_hour) float64 3kB ...    ships_mean_tangential_wind_300mb_at500km_m_s01   (ships_storm_object_index, ships_forecast_hour) float64 3kB ...    ships_srh_1000to700mb_200to800km_j_kg01          (ships_storm_object_index, ships_forecast_hour) float64 3kB ...    ships_srh_1000to500mb_200to800km_j_kg01          (ships_storm_object_index, ships_forecast_hour) float64 3kB ...    ships_threshold_exceedance_num_6hour_periods     (ships_storm_object_index, ships_intensity_threshold_m_s01) int32 2kB ...    ships_v_motion_observed_m_s01                    (ships_storm_object_index) float64 152B ...    ships_v_motion_1000to100mb_flow_m_s01            (ships_storm_object_index) float64 152B ...    ships_v_motion_optimal_flow_m_s01                (ships_storm_object_index) float64 152B ...    ships_cyclone_id_string                          (ships_storm_object_index) object 152B ...    ships_storm_latitude_deg_n                       (ships_storm_object_index) float64 152B ...    ships_storm_longitude_deg_e                      (ships_storm_object_index) float64 152B ...    ships_predictors_lagged                          (ships_valid_time_unix_sec, ships_lag_time_hours, ships_predictor_name_lagged) float64 13kB ...    ships_predictors_forecast                        (ships_valid_time_unix_sec, ships_forecast_hour, ships_predictor_name_forecast) float64 451kB ...

Variable names are meant to be as self-explanatory as possible.  Potentially confusing ones are listed below.

The dimension ships_storm_object_index is redundant with the dimension ships_valid_time_unix_sec and can be ignored.

ships_forecast_hour ranges up to values that we do not actually use in the paper.  Keep in mind that our max forecast hour used in machine learning is 24.

The dimension ships_intensity_threshold_m_s01 (and any variable including this dimension) can be ignored.

ships_lag_time_hours corresponds to lag times for the SHIPS satellite-based predictors.  The only lag time we use in machine learning is "NaN", which is a stand-in for the best available of all lag times.  See the discussion of the "priority list" in the paper for more details.

Most of the data variables can be ignored, unless you're doing a deep dive into storm properties.  The important variables are satellite_predictors_gridded (full satellite images), ships_predictors_lagged (satellite-based SHIPS predictors), and ships_predictors_forecast (environmental and storm-history-based SHIPS predictors).  These variables are all discussed in the paper.

Every variable name (including elements of the coordinate lists ships_predictor_name_lagged and ships_predictor_name_forecast) includes units at the end.  For example, "m_s01" = metres per second; "deg_n" = degrees north; "deg_e" = degrees east; "j_kg01" = Joules per kilogram; ...; etc.