embed2heights

# embed2heights Challenge - Reaching New Heights with GeoFM Embeddings ## Overview The objective of the [**embed2heights** challenge](https://platform-challenges.philab.esa.int/geoai) is to develop a multi-task method that uses Geospatial Foundation Model embeddings to map land cover and estimate heights at scale. Participants are asked to combine multiple embedding sources to segment buildings, vegetation, and water, and to predict building and vegetation heights. Participants receive an AI-ready package with **pre-computed embeddings** from four GFMs: **AlphaEarth, TESSERA, TerraMind, and THOR**, plus reference labels derived from IGN airborne LiDAR products. This enables direct experimentation on feature fusion and multi-task modeling without running large-scale embedding generation. ## Dataset The training dataset includes **2,024 patches** of size **256x256 at 10 m resolution**, sampled over major French cities and selected rural areas in France. Labels are derived from IGN products based on airborne LiDAR. Labels are **not discrete categories**. For each pixel, they represent the percentage contribution of each class within a **10x10 m cell**. Label data is provided as 4-band TIFF files: - **Band 1**: percentage of building - **Band 2**: percentage of vegetation - **Band 3**: percentage of water - **Band 4**: relative height above ground (nDSM) The source data is generated at **1 m spatial resolution** and includes four classes: **Background**, **Buildings**, **Trees/HighVegetation**, and **Unclassified**. The *Unclassified* class captures mixed/overlap cases (for example, a tree attached to a house). The test set (around **1,000 patches**) is generated with similar data, but from different regions and years. ## Training Data Folder Structure Inside the `data/train/` directory, the following subdirectories are provided: - `alphaearth_emb`: AlphaEarth pixel-level embeddings, shape `(256, 256, 64)`, total size **33.93 GB**. - `labels`: reference label tensors (not a model output), pixel-level, shape `(256, 256, 4)`, total size **2.12 GB**. - `terramind_s1_emb`: TerraMind (S1) patch-level embeddings, shape `(16, 16, 768)`, total size **1.60 GB**. - `terramind_s2_emb`: TerraMind (S2) patch-level embeddings, shape `(16, 16, 768)`, total size **1.60 GB**. - `tessera_emb`: Tessera pixel-level embeddings, shape `(256, 256, 128)`, total size **67.82 GB**. - `thor_s1_emb`: THOR (S1) patch-level embeddings, shape `(16, 16, 768)`, total size **1.96 GB**. - `thor_s2_emb`: THOR (S2) patch-level embeddings, shape `(16, 16, 768)`, total size **1.95 GB**. All subdirectories currently contain the same number of files: **2,024**. In addition, the `data/` directory also includes a `catalog.parquet` manifest file (~**2.1 MB**), with **14,169 rows** and **10 columns** (`type`, `stac_version`, `stac_extensions`, `datetime`, `id`, `bbox`, `geometry`, `assets`, `links`, `repository`). The catalog stores one entry per object, with **2,024** entries for each data subdirectory. The `assets` field provides per-file metadata such as checksum, source `href`, file size, and timestamp. ## Baseline A reference baseline implementation is available on GitHub: - <https://github.com/VMarsocci/emb2heights-baselines> ## Evaluation The team score is computed with a **weighted multi-metric evaluation** combining segmentation and height accuracy: `mIoU_buildings` (25%), `mIoU_trees` (15%), `mIoU_water` (15%), `RMSE_buildings` (25%), and `RMSE_vegetation` (20%). The final leaderboard score is the weighted mean of these five metrics. ## Submission Requirements Each submission **must include predictions for all 946 test patches**. Each evaluated submission receives: - a **public score**, computed on a subset of the test set; - a **private score**, computed on the full test set and used for final ranking. The exact patch-level composition of the public evaluation subset is not disclosed to participants. At the end of the challenge, private scores are revealed and the final leaderboard is computed using the private score. ## Authors - Andrea Nascetti, KTH, Sweden - Valerio Marsocci, ESA, Italy - Fabrizio Pera, GMATICS, Italy - Paolo Soresina, GMATICS, Italy