Core-DEM


# Major TOM Core-DEM

Contains a global coverage of [Copernicus DEM](https://spacedata.copernicus.eu/collections/copernicus-digital-elevation-model), each of size 356 x 356 pixels.

| Source | Modality Type | Number of Patches | Patch Size | Total Pixels |
|:-------|:-------------:|:-----------------:|:----------:|:------------:|
|Copernicus DEM 30 | Digital Surface Model (DSM) |1,837,843| 356 x 356 (30 m) | > 1.654 Billion |

## Content

| Column | Details | Resolution |
|:-------|:--------|:-----------|
| DEM | Original data | 30m |
| thumbnail | compressed hillshade visualisation | 30m |
| compressed | compressed png of original data | 30m |

## Spatial Coverage
This is a global monotemporal dataset that contains nearly the entire COP-DEM dataset.

The following figure demonstrates the spatial coverage (only black pixels are absent):

In this first version, all available DEM data was included except for the Major TOM cells below the 89th latitude and two degrees west off the date change line. Azerbaijan and Armenia weren’t included either as they are unavailable on the Creodias platform used to create this dataset.
## Example Use

Interface scripts are available at https://github.com/ESA-PhiLab/Major-TOM

Here's an example with reading directly via http from HuggingFace:
```python
from fsspec.parquet import open_parquet_file
import pyarrow.parquet as pq
from rasterio.io import MemoryFile
from PIL import Image

PARQUET_FILE = 'part_00390' # parquet number
ROW_INDEX = 42 # row number (about 500 per parquet)

url = "https://huggingface.co/datasets/Major-TOM/Core-DEM/resolve/main/images/{}.parquet".format(PARQUET_FILE)
with open_parquet_file(url,columns = ["DEM"]) as f:
with pq.ParquetFile(f) as pf:
first_row_group = pf.read_row_group(ROW_INDEX, columns=['DEM'])

with MemoryFile(first_row_group['DEM'][0].as_py()) as mem_f:
with mem_f.open(driver='GTiff') as f:
dem = f.read()
```

and here's an example with a thumbnail image:
```python
from fsspec.parquet import open_parquet_file
import pyarrow.parquet as pq
from io import BytesIO
from PIL import Image

PARQUET_FILE = 'part_00390' # parquet number
ROW_INDEX = 42 # row number (about 500 per parquet)

url = "https://huggingface.co/datasets/Major-TOM/Core-DEM/resolve/main/images/{}.parquet".format(PARQUET_FILE)
with open_parquet_file(url,columns = ["thumbnail"]) as f:
with pq.ParquetFile(f) as pf:
first_row_group = pf.read_row_group(ROW_INDEX, columns=['thumbnail'])

stream = BytesIO(first_row_group['thumbnail'][0].as_py())
image = Image.open(stream)
```
### Reprojection Details
Contrary to [S1 RTC](huggingface.co/datasets/Major-TOM/Core-S1RTC) and S2 ([L1C](huggingface.co/datasets/Major-TOM/Core-S2L1C) & [L2A](huggingface.co/datasets/Major-TOM/Core-S2L2A)) products, which are taken in their native projection to create their respective Major TOM Core datasets, Copernicus DEM, natively in EPSG:4326, was reprojected to a carefully chosen projection. To guarantee uniformity across Major Tom sources, it was reprojected to the corresponding UTM zone of the cell. This leads to inconsistency between Sentinel-2 and COP-DEM cells in some cases. For the S2-L2A product this is estimated to 2.5% of all the cells where COP-DEM and S2-L2A are available (41,998 out of 1,679,898 cells).

Large DEM tiles were projected and resampled to 30m using bilinear interpolation. Small major tom cells were then cropped for it using nearest neighbor interpolation if needed. Some tiles above water and around Armenia and Azerbaijan, may exhibit missing pixels which value were set to -32767.


### Credits
This dataset is the product of a collaboration between [Φ-lab, European Space Agency (ESA)](https://huggingface.co/ESA-philab) and the [Adobe Research (Paris, France)](https://research.adobe.com/careers/paris/). The dataset was put together by [Paul Borne--Pons](https://www.linkedin.com/in/paul-bp-cs/) under the supervision of Mikolaj Czerkawski and Alistair Francis (the original authors of the Major TOM project) as part of his stay at ESA Phi Lab. The idea behind this collaboration is to explore the synergies between Sentinel 2 products and DEM data, notably for the generation of terrains.

---

Produced using Copernicus WorldDEM-30 © DLR e.V. 2010-2014 and © Airbus Defence and Space GmbH
2014-2018 provided under COPERNICUS by the European Union and ESA; all rights reserved


# Major TOM Core-DEM 数据集

本数据集包含覆盖全球的[哥白尼数字高程模型(Copernicus DEM)](https://spacedata.copernicus.eu/collections/copernicus-digital-elevation-model)数据，每份数据尺寸为356×356像素。

| 数据源 | 模态类型 | 补丁数量 | 补丁尺寸 | 总像素数 |
|:-------|:--------:|:--------:|:--------:|:--------:|
| Copernicus DEM 30 | 数字表面模型(Digital Surface Model, DSM) | 1,837,843 | 356×356（30m） | 超过16.54亿 |

## 数据内容

| 列名 | 详细说明 | 分辨率 |
|:-------|:--------|:-----------|
| DEM | 原始数据 | 30m |
| thumbnail | 压缩后的山体阴影可视化结果 | 30m |
| compressed | 原始数据的压缩PNG格式文件 | 30m |

## 空间覆盖范围

本数据集为全球单时态数据集，几乎涵盖全部COP-DEM数据。

下图展示了其空间覆盖范围（仅黑色像素区域为缺失数据）：

在首个版本中，本数据集纳入了所有可用的DEM数据，但排除了北纬89度以上、国际日期变更线以西2度范围内的Major TOM网格单元，同时未包含阿塞拜疆与亚美尼亚区域——这是由于制作本数据集所使用的Creodias平台无法获取该区域的数据。

## 示例用法

接口脚本可在https://github.com/ESA-PhiLab/Major-TOM 获取。

以下为通过HTTP直接从HuggingFace读取DEM数据的示例代码：
python
from fsspec.parquet import open_parquet_file
import pyarrow.parquet as pq
from rasterio.io import MemoryFile
from PIL import Image

PARQUET_FILE = 'part_00390' # 分块编号
ROW_INDEX = 42 # 行索引（每个Parquet文件约含500行）

url = "https://huggingface.co/datasets/Major-TOM/Core-DEM/resolve/main/images/{}.parquet".format(PARQUET_FILE)
with open_parquet_file(url,columns = ["DEM"]) as f:
with pq.ParquetFile(f) as pf:
first_row_group = pf.read_row_group(ROW_INDEX, columns=['DEM'])

with MemoryFile(first_row_group['DEM'][0].as_py()) as mem_f:
with mem_f.open(driver='GTiff') as f:
dem = f.read()

以下为缩略图读取示例：
python
from fsspec.parquet import open_parquet_file
import pyarrow.parquet as pq
from io import BytesIO
from PIL import Image

PARQUET_FILE = 'part_00390' # 分块编号
ROW_INDEX = 42 # 行索引（每个Parquet文件约含500行）

url = "https://huggingface.co/datasets/Major-TOM/Core-DEM/resolve/main/images/{}.parquet".format(PARQUET_FILE)
with open_parquet_file(url,columns = ["thumbnail"]) as f:
with pq.ParquetFile(f) as pf:
first_row_group = pf.read_row_group(ROW_INDEX, columns=['thumbnail'])

stream = BytesIO(first_row_group['thumbnail'][0].as_py())
image = Image.open(stream)

### 重投影细节

与[S1 RTC](https://huggingface.co/datasets/Major-TOM/Core-S1RTC)和S2（[L1C](https://huggingface.co/datasets/Major-TOM/Core-S2L1C)及[L2A](https://huggingface.co/datasets/Major-TOM/Core-S2L2A)）产品保留原生投影以构建各自的Major TOM Core数据集不同，原生采用EPSG:4326坐标系的Copernicus DEM数据被重投影至经过精心选择的坐标系。为确保Major TOM各数据源间的一致性，数据被重投影至对应网格单元的UTM分区。这导致在部分场景中，Sentinel-2与COP-DEM的网格单元存在投影不一致的问题。针对S2-L2A产品，该情况约占所有同时拥有COP-DEM与S2-L2A数据的网格单元的2.5%（1679898个单元中的41998个）。

大型DEM瓦片经投影与重采样至30米分辨率，重采样方法采用双线性插值。随后根据需要，使用最近邻插值法裁剪得到小型Major TOM网格单元。部分位于水域之上以及阿塞拜疆、亚美尼亚周边的瓦片可能存在缺失像素，其像素值被设为-32767。


### 致谢

本数据集由[Φ-lab、欧洲空间局(European Space Agency, ESA)](https://huggingface.co/ESA-philab)与[Adobe研究院(法国巴黎)](https://research.adobe.com/careers/paris/)合作研发。数据集由Paul Borne--Pons在Mikolaj Czerkawski与Alistair Francis（Major TOM项目原作者）的指导下完成，相关工作为其在ESA Phi Lab驻留期间的研究成果。本次合作旨在探索Sentinel 2产品与DEM数据的协同应用潜力，尤其面向地形生成相关任务。

---

本数据集使用Copernicus WorldDEM-30制作，© DLR e.V. 2010-2014 与 © Airbus Defence and Space GmbH 2014-2018，由欧盟与ESA在哥白尼计划框架下提供；保留所有权利。