Project HF-EOLUS. Task 2. Aggregated SAR and HF-Radar Radial Metrics for Wind-Inversion Model Training

This record presents the results obtained by applying the open, reproducible workflow described in HF-EOLUS Geospatial Processing Tools to the HF-EOLUS Sentinel‑1 SAR Derived Data Bundle and to the VILA & PRIO Radial Metrics GeoParquet datasets. It provides analysis-ready geospatial data products generated to train HF-Radar wind-inversion models. HF-EOLUS Geospatial Processing Tools documents the methodology, software environment, and parameterization; the dataset records contain the data used as input. This record includes the concrete outputs for reuse and citation. Wind-inversion models are trained using supervised learning methods. In the HF-EOLUS project, we tackle this task as a regression problem where, given a set of features (HF-Radar data), we train a model to predict a numerical output (wind speed and direction). Due to the dense nature of the SAR and HF-Radar data and that they are not initially co-located, we need to aggregate them into a grid that allows us to get paired time-series of HF-Radar input <-> SAR output on a grid. HF-EOLUS Geospatial Processing Tools provides a workflow that we use here to 1) compute the coverage area of the HF-Radar data, 2) build a regular grid inside that area, 3) map each SAR and HF-Radar record into that grid, 4) aggregate the features and outputs into time-series for each node in the grid, 5) standarize the result for easy access and downstream analysis. Step 1 — Intersection of coverage hulls (VILA ∩ PRIO) This step computes per-station convex hulls from the VILA and PRIO HF-Radar radial metrics stored in Athena, intersects them to isolate the shared coverage area, and produces both a GeoJSON polygon and an interactive HTML map for quick inspection. The resulting intersection defines the spatial domain used for training and subsequent gridding. Inputs: Athena tables HF_INGESTION_VILA.radial_metrics and HF_INGESTION_PRIO.radial_metrics (described in VILA & PRIO Radial Metrics GeoParquet). Method: scripts/hulls/convex_hulls.sh computes per‑station convex hulls and their intersection; scripts/hulls/view_hull.sh renders an interactive preview map. Commands: ./scripts/hulls/convex_hulls.sh --profile default --tables "HF_INGESTION_VILA.radial_metrics,HF_INGESTION_PRIO.radial_metrics" --output-file VILA_PRIO_hull.json --output-location s3://hf-eolus/athena-query-results/ --operation intersection ./scripts/hulls/view_hull.sh --input VILA_PRIO_hull.json --output VILA_PRIO_hull_map.html --title "VILA ∩ PRIO" Outputs: VILA_PRIO_hull.json: GeoJSON FeatureCollection (EPSG:4326) with a single Polygon representing the intersection of the VILA and PRIO coverage hulls. VILA_PRIO_hull_map.html: Leaflet-based interactive map preview titled “VILA ∩ PRIO”. HF_INGESTION_VILA_radial_metrics_wkb.csv: CSV export with a wkb column (WKB geometries) for VILA radial metrics points used in hull computation. HF_INGESTION_PRIO_radial_metrics_wkb.csv: CSV export with a wkb column (WKB geometries) for PRIO radial metrics points used in hull computation. Step 2 — Grid generation This step builds a regular analysis grid over the VILA ∩ PRIO coverage, using 20 km spacing with a 10 km buffer, optionally appending manual nodes (e.g., buoys), and exports the resulting grid nodes and a preview map. The grid nodes serve as the common index where SAR outputs (labels) and HF-Radar inputs (features) are aligned. Inputs: VILA_PRIO_hull.json (from Step 1); optional buoys_grid_csv.csv (also provided as an output for reproducibility); Athena database ANN_TRAINING. Method: scripts/grids/create_grid_table.sh materializes the grid into the Athena table; the first call creates a regular grid constrained to the hull + buffer, the second call appends nodes from a CSV; scripts/grids/view_grid.sh produces an interactive preview. Commands: ./scripts/grids/create_grid_table.sh \ --profile default \ --database ANN_TRAINING \ --hull-file VILA_PRIO_hull.json \ --output-table grid \ --table-location s3://hf-eolus/ann_training/grid \ --output-location s3://hf-eolus/athena-query-results/ \ --node-prefix VILA_PRIO \ --grid-spacing-km 20 \ --buffer-km 10 ./scripts/grids/create_grid_table.sh \ --profile default \ --database ANN_TRAINING \ --manual-csv buoys_grid_csv.csv \ --mode append \ --output-table grid \ --table-location s3://hf-eolus/ann_training/grid \ --output-location s3://hf-eolus/athena-query-results/ ./scripts/grids/view_grid.sh --input grid_nodes_1757320574.parquet --input grid_nodes_1757320746.parquet --output grid_map.html --title "Grid Preview" Outputs: grid_nodes_1757320574.parquet: Parquet with grid nodes generated from the hull-based regular grid (first command). grid_nodes_1757320746.parquet: Parquet with nodes appended from the manual CSV (second command). grid_map.html: Leaflet-based interactive map preview titled “Grid Preview”. buoys_grid_csv.csv: CSV with manual grid nodes used for appending (columns: node_id, longitude, latitude). Notes: The grid table ANN_TRAINING.grid is stored at s3://hf-eolus/ann_training/grid. The --node-prefix VILA_PRIO is applied to node identifiers. The final grid consists of 55 nodes in a regular mesh plus one buoy node. Step 3 — Geospatial mapping to a grid This step links each observation to its nearest grid node within a 10 km radius, creating row‑to‑node mappings for the SAR and HF-Radar datasets and materializing them as Parquet‑backed Athena tables. These mappings enable joining inputs and outputs by node_id and time to form aligned samples for supervised training. Inputs: grid table ANN_TRAINING.grid (Step 2); data tables: SAR_INGEST.SAR (described in HF-EOLUS Sentinel‑1 SAR Derived Data Bundle) HF_INGESTION_VILA.radial_metrics and HF_INGESTION_PRIO.radial_metrics (described in VILA & PRIO Radial Metrics GeoParquet) Method: scripts/mapping/geo_mapping.sh creates an Athena CTAS that emits (rowid, node_id) pairs for observations whose geodesic distance to a grid node is ≤ 10 km, materialized as Parquet on S3. Commands: ./scripts/mapping/geo_mapping.sh --profile default --db-name SAR_INGEST --data-db-name SAR_INGEST --grid-db-name ANN_TRAINING --data-table SAR --grid-table grid --bucket-name hf-eolus --output-prefix ann_training/geo_join_sar --output-table GRID_JOIN --distance 10 --log-dir ./geo_join_output_sar ./scripts/mapping/geo_mapping.sh --profile default --db-name HF_INGESTION_VILA --data-db-name HF_INGESTION_VILA --grid-db-name ANN_TRAINING --data-table radial_metrics --grid-table grid --bucket-name hf-eolus --output-prefix ann_training/geo_join_vila --output-table GRID_JOIN --distance 10 --log-dir ./geo_join_output_vila ./scripts/mapping/geo_mapping.sh --profile default --db-name HF_INGESTION_PRIO --data-db-name HF_INGESTION_PRIO --grid-db-name ANN_TRAINING --data-table radial_metrics --grid-table grid --bucket-name hf-eolus --output-prefix ann_training/geo_join_prio --output-table GRID_JOIN --distance 10 --log-dir ./geo_join_output_prio Outputs: Athena tables: SAR_INGEST.GRID_JOIN → s3://hf-eolus/ann_training/geo_join_sar/ HF_INGESTION_VILA.GRID_JOIN → s3://hf-eolus/ann_training/geo_join_vila/ HF_INGESTION_PRIO.GRID_JOIN → s3://hf-eolus/ann_training/geo_join_prio/ Logs: geo_join_output_sar/geo_mapping.sh.log, geo_join_output_vila/geo_mapping.sh.log, geo_join_output_prio/geo_mapping.sh.log. Notes: --distance 10 is in kilometers. Existing target tables and S3 prefixes are dropped/cleared before creation to ensure clean outputs. Step 4 — Aggregation and GeoParquet consolidation This step aggregates observations by time and grid node (and optional partitions), applying circular statistics for wind direction and projecting radar velocities along the line of sight when required, and then consolidates the outputs with GeoParquet metadata. The resulting per‑node, per‑time aggregates—produced separately for SAR (outputs) and HF-Radar (features)—can be joined downstream on (timestamp, node_id) to build training and validation datasets. Method: SAR: aggregate_direction_wrapper.sh applies circular statistics to owiwinddirection (optionally weighted by owiwindspeed), then aggregates; finalize_geoparquet.sh merges files and writes GeoParquet metadata. VILA/PRIO: aggregate_projection_wrapper.sh projects VELO onto the line joining each grid node to a reference point (radar location), then aggregates; finalize_geoparquet.sh consolidates and repairs partitions. Commands: ./scripts/aggregation/aggregate_direction_wrapper.sh --profile default --db-name ANN_TRAINING --data-db-name SAR_INGEST --grid-db-name ANN_TRAINING --mapping-db-name SAR_INGEST --data-table SAR --grid-table grid --mapping-table GRID_JOIN --columns "owiwinddirection,owiwindspeed" --bucket-name hf-eolus --output-prefix ann_training/aggregation_sar --output-table SAR_AGGREGATED --log-dir ./aggregation_output_sar --timestamp-col date --direction-cols "owiwinddirection" ./scripts/aggregation/finalize_geoparquet.sh --profile default --db-name ANN_TRAINING --bucket-name hf-eolus --output-prefix ann_training/aggregation_sar --output-table SAR_AGGREGATED --log-dir ./aggregation_output_sar ./scripts/aggregation/aggregate_projection_wrapper.sh --profile default --db-name ANN_TRAINING --data-db-name HF_INGESTION_VILA --grid-db-name ANN_TRAINING --mapping-db-name HF_INGESTION_VILA --data-table radial_metrics --grid-table grid --mapping-table GRID_JOIN --columns "Pwr,VELO" --bucket-name hf-eolus --output-prefix ann_training/aggregation_vila --output-table VILA_AGGREGATED --log-dir ./aggregation_output_vila --partition-cols pos_bragg --projection-col "VELO" --point-lat 43.1588833 --point-lon -9.2108333 ./scripts/aggregation/finalize_geoparquet.sh --profile default --db-name ANN_TRAINING --bucket-name hf-eolus --output-prefix ann_training/aggregation_vila --output-table VILA_AGGREGATED --log-dir ./aggregation_output_vila --partition-cols pos_bragg ./scripts/aggregation/aggregate_projection_wrapper.sh --profile default --db-name ANN_TRAINING --data-db-name HF_INGESTION_PRIO --grid-db-name ANN_TRAINING --mapping-db-name HF_INGESTION_PRIO --data-table radial_metrics --grid-table grid --mapping-table GRID_JOIN --columns "Pwr,VELO" --bucket-name hf-eolus --output-prefix ann_training/aggregation_prio --output-table PRIO_AGGREGATED --log-dir ./aggregation_output_prio --partition-cols pos_bragg --projection-col "VELO" --point-lat 43.5680000 --point-lon -8.3140000 ./scripts/aggregation/finalize_geoparquet.sh --profile default --db-name ANN_TRAINING --bucket-name hf-eolus --output-prefix ann_training/aggregation_prio --output-table PRIO_AGGREGATED --log-dir ./aggregation_output_prio --partition-cols pos_bragg Outputs: Athena tables and S3 datasets (consolidated as GeoParquet): ANN_TRAINING.SAR_AGGREGATED → s3://hf-eolus/ann_training/aggregation_sar/ ANN_TRAINING.VILA_AGGREGATED → s3://hf-eolus/ann_training/aggregation_vila/ (partitioned by pos_bragg) ANN_TRAINING.PRIO_AGGREGATED → s3://hf-eolus/ann_training/aggregation_prio/ (partitioned by pos_bragg) Logs and saved SQL (attached as ZIPs): aggregation_output_sar.zip, aggregation_output_vila.zip, aggregation_output_prio.zip (contain wrapper logs and the key SQL statements generated during the process). Notes: finalize_geoparquet.sh downloads the dataset, merges Parquet files per partition, adds GeoParquet metadata for column geometry, syncs back to S3, and runs MSCK REPAIR TABLE when --partition-cols is provided. Step 5 — STAC catalog and GeoParquet packaging This step packages the aggregated GeoParquet datasets as portable STAC catalogs by staging assets locally, generating Items and a Collection with the provided property templates, and creating ZIP bundles for distribution. These catalogs facilitate discovery, reuse, and ingestion into analysis or model‑training pipelines. Inputs: consolidated GeoParquet datasets from Step 4 synced locally; STAC property templates: stac_properties_item_sar.json, stac_properties_collection_sar.json stac_properties_item_hf.json, stac_properties_collection_hf.json Method: sync each aggregated dataset from S3 to a local folder; then run scripts/aggregation/build_stac_catalog.sh to assemble a STAC collection with Items pointing to local GeoParquet assets and produce a ZIP bundle. Commands: aws s3 sync s3://hf-eolus/ann_training/aggregation_sar ./sar_aggregated/ --profile default aws s3 sync s3://hf-eolus/ann_training/aggregation_vila ./vila_aggregated/ --profile default aws s3 sync s3://hf-eolus/ann_training/aggregation_prio ./prio_aggregated/ --profile default # Build self-contained STAC catalogs for locally synced aggregates ./scripts/aggregation/build_stac_catalog.sh \ --collection SAR_AGGREGATED \ --local-source-dir ./sar_aggregated \ --output-dir ./sar_catalog \ --zip-file ./sar_catalog.zip \ --stac-item-properties-json ./stac_properties_item_sar.json \ --stac-collection-properties-json ./stac_properties_collection_sar.json ./scripts/aggregation/build_stac_catalog.sh \ --collection VILA_AGGREGATED \ --local-source-dir ./vila_aggregated \ --output-dir ./vila_catalog \ --zip-file ./vila_catalog.zip \ --stac-item-properties-json ./stac_properties_item_hf.json \ --stac-collection-properties-json ./stac_properties_collection_hf.json ./scripts/aggregation/build_stac_catalog.sh \ --collection PRIO_AGGREGATED \ --local-source-dir ./prio_aggregated \ --output-dir ./prio_catalog \ --zip-file ./prio_catalog.zip \ --stac-item-properties-json ./stac_properties_item_hf.json \ --stac-collection-properties-json ./stac_properties_collection_hf.json Outputs: STAC catalogs (folders) and ZIP bundles ready to attach: sar_catalog/ and sar_catalog.zip vila_catalog/ and vila_catalog.zip prio_catalog/ and prio_catalog.zip STAC property JSONs included alongside the catalogs: stac_properties_collection_hf.json, stac_properties_item_hf.json stac_properties_collection_sar.json, stac_properties_item_sar.json Notes: catalogs are self-contained for offline use; Items reference local GeoParquet assets under the catalog’s assets/ directory and include additional properties from the provided JSON templates. Table statistics Total rows in each aggregated table: VILA: 11,197,374 PRIO: 7,439,566 SAR: 7,353 Number of rows by node in each aggregated table: Node ID SAR n VILA n PRIO n VILA_PRIO2 129 200,592 24,796 VILA_PRIO11 129 215,430 22,423 VILA_PRIO12 129 240,065 65,294 VILA_PRIO13 129 259,771 73,995 VILA_PRIO14 129 286,584 72,840 VILA_PRIO15 129 282,672 15,168 VILA_PRIO21 129 227,948 51,762 VILA_PRIO22 129 256,650 77,228 VILA_PRIO23 129 280,174 87,157 VILA_PRIO24 129 280,623 105,682 VILA_PRIO25 129 283,889 115,792 VILA_PRIO26 129 232,556 102,868 VILA_PRIO31 129 241,219 55,463 VILA_PRIO32 129 252,427 77,025 VILA_PRIO33 129 253,449 85,888 VILA_PRIO34 129 253,465 114,046 VILA_PRIO35 129 272,197 123,962 VILA_PRIO36 129 258,016 143,172 VILA_PRIO37 129 139,492 184,669 VILA_PRIO41 129 218,703 59,621 VILA_PRIO42 129 212,609 90,817 VILA_PRIO43 129 230,476 111,725 VILA_PRIO44 129 190,620 147,695 VILA_PRIO45 129 256,603 178,326 VILA_PRIO46 129 243,969 198,894 VILA_PRIO47 129 227,560 212,324 VILA_PRIO48 129 0 216,702 VILA_PRIO51 129 164,288 61,374 VILA_PRIO52 129 191,728 117,335 VILA_PRIO53 129 191,794 139,647 VILA_PRIO54 129 166,493 170,011 VILA_PRIO55 129 238,227 190,413 VILA_PRIO56 129 219,641 201,207 VILA_PRIO57 129 220,627 212,314 VILA_PRIO58 129 183,865 167,566 VILA_PRIO62 129 172,199 131,575 VILA_PRIO63 129 149,007 145,094 VILA_PRIO64 129 155,628 165,541 VILA_PRIO65 129 211,356 179,106 VILA_PRIO66 129 183,756 200,832 VILA_PRIO67 129 182,008 183,382 VILA_PRIO68 129 171,708 135,059 VILA_PRIO69 129 121,474 208,998 VILA_PRIO72 129 132,360 111,845 VILA_PRIO73 129 117,432 141,909 VILA_PRIO74 129 131,752 136,387 VILA_PRIO75 129 175,633 180,928 VILA_PRIO76 129 158,828 183,215 VILA_PRIO77 129 152,795 153,647 VILA_PRIO78 129 150,911 125,660 VILA_PRIO79 129 123,177 202,434 VILA_PRIO83 129 72,122 103,911 VILA_PRIO84 129 109,102 149,200 VILA_PRIO85 129 145,492 158,288 VILA_PRIO86 129 127,539 146,056 VILA_PRIO87 129 103,087 100,134 Vilano_buoy 129 275,586 121,164 Total 7,353 11,197,374 7,439,566 Acknowledgements This work has been funded by the HF-EOLUS project (TED2021-129551B-I00), financed by MICIU/AEI /10.13039/501100011033 and by the European Union NextGenerationEU/PRTR - BDNS 598843 - Component 17 - Investment I3. Members of the Marine Research Centre (CIM) of the University of Vigo have participated in the development of this repository. Disclaimer This software is provided "as is", without warranty of any kind, express or implied, including but not limited to the warranties of merchantability, fitness for a particular purpose, and noninfringement. In no event shall the authors or copyright holders be liable for any claim, damages, or other liability, whether in an action of contract, tort, or otherwise, arising from, out of, or in connection with the software or the use or other dealings in the software.