Sentinel-2 Cloud Mask Catalogue

Overview This dataset comprises cloud masks for 513 1022-by-1022 pixel subscenes, at 20m resolution, sampled random from the 2018 Level-1C Sentinel-2 archive. The design of this dataset follows from some observations about cloud masking: (i) performance over an entire product is highly correlated, thus subscenes provide more value per-pixel than full scenes, (ii) current cloud masking datasets often focus on specific regions, or hand-select the products used, which introduces a bias into the dataset that is not representative of the real-world data, (iii) cloud mask performance appears to be highly correlated to surface type and cloud structure, so testing should include analysis of failure modes in relation to these variables. The data was annotated semi-automatically, using the IRIS toolkit, which allows users to dynamically train a Random Forest (implemented using LightGBM), speeding up annotations by iteratively improving it's predictions, but preserving the annotator's ability to make final manual changes when needed. This hybrid approach allowed us to process many more masks than would have been possible manually, which we felt was vital in creating a large enough dataset to approximate the statistics of the whole Sentinel-2 archive. In addition to the pixel-wise, 3 class (CLEAR, CLOUD, CLOUD_SHADOW) segmentation masks, we also provide users with binary classification "tags" for each subscene that can be used in testing to determine performance in specific circumstances. These include: SURFACE TYPE: 11 categories CLOUD TYPE: 7 categories CLOUD HEIGHT: low, high CLOUD THICKNESS: thin, thick CLOUD EXTENT: isolated, extended Wherever practical, cloud shadows were also annotated, however this was sometimes not possible due to high-relief terrain, or large ambiguities. In total, 424 were marked with shadows (if present), and 89 have shadows that were not annotatable due to very ambiguous shadow boundaries, or terrain that cast significant shadows. If users wish to train an algorithm specifically for cloud shadow masks, we advise them to remove those 89 images for which shadow was not possible, however, bear in mind that this will systematically reduce the difficulty of the shadow class compared to real-world use, as these contain the most difficult shadow examples. In addition to the 20m sampled subscenes and masks, we also provide users with shapefiles that define the boundary of the mask on the original Sentinel-2 scene. If users wish to retrieve the L1C bands at their original resolutions, they can use these to do so. Please see the README for further details on the dataset structure and more. Contributions & Acknowledgements The data were collected, annotated, checked, formatted and published by Alistair Francis and John Mrziglod. Support and advice was provided by Prof. Jan-Peter Muller and Dr. Panagiotis Sidiropoulos, for which we are grateful. We would like to extend our thanks to Dr. Pierre-Philippe Mathieu and the rest of the team at ESA PhiLab, who provided the environment in which this project was conceived, and continued to give technical support throughout. Finally, we thank the ESA Network of Resources for sponsoring this project by providing ICT resources.

数据集概览
本数据集包含513幅1022×1022像素、分辨率为20米的子场景云掩膜，这些子场景从2018年Sentinel-2 Level-1C级存档中随机采样得到。
本数据集的构建基于云掩膜任务的若干观察结论：(i) 完整产品的掩膜性能具有高度相关性，因此相较于完整场景，子场景的单位像素价值更高；(ii) 现有云掩膜数据集通常聚焦于特定区域，或手动挑选所用产品，这会为数据集引入偏差，使其无法代表真实世界的数据分布；(iii) 云掩膜性能似乎与地表类型及云体结构高度相关，因此测试需包含针对这些变量的失效模式分析。
本数据集采用IRIS工具集进行半自动标注：该工具集支持用户动态训练随机森林（Random Forest，基于LightGBM实现），通过迭代优化预测结果加速标注流程，同时保留标注人员在必要时进行最终手动调整的权限。这种混合标注方案使我们能够处理远超手动标注规模的掩膜数据，而我们认为这对构建足够大的数据集以近似Sentinel-2全存档的数据分布至关重要。
除逐像素的三类（CLEAR、CLOUD、CLOUD_SHADOW）分割掩膜外，我们还为每个子场景提供了二分类“标签”，可用于测试特定场景下的模型性能。这些标签涵盖：
- 地表类型（SURFACE TYPE）：共11个类别
- 云类型（CLOUD TYPE）：共7个类别
- 云高（CLOUD HEIGHT）：低、高
- 云厚度（CLOUD THICKNESS）：薄、厚
- 云覆盖范围（CLOUD EXTENT）：孤立、连片
在条件允许的情况下，我们均对云阴影进行了标注，但由于地形起伏剧烈或存在大量模糊区域，部分场景无法完成阴影标注。总计有424幅子场景完成了阴影（若存在）标注，另有89幅子场景因阴影边界极度模糊或地形投射显著阴影，无法完成阴影标注。
若用户希望专门针对云阴影掩膜训练算法，我们建议移除这89幅无法完成阴影标注的图像，但需注意：此举会系统性降低阴影类别的任务难度，相较于真实场景应用，因为这些被移除的样本包含了最具挑战性的阴影案例。
除20米分辨率的采样子场景及掩膜外，我们还为用户提供了用于定义原始Sentinel-2场景中掩膜边界的shapefile文件。若用户希望获取原始分辨率的L1C波段数据，可通过该文件实现。有关数据集结构等更多详细信息，请参阅README文档。

贡献与致谢
本数据集的数据采集、标注、校验、格式化及发布工作由Alistair Francis与John Mrziglod完成。Jan-Peter Muller教授与Panagiotis Sidiropoulos博士为我们提供了支持与建议，在此谨致谢忱。我们还要感谢Pierre-Philippe Mathieu博士及ESA PhiLab团队的其他成员，他们为本项目的构思提供了环境，并在项目全程提供了技术支持。最后，我们感谢ESA资源网络（ESA Network of Resources）通过提供ICT资源为本项目提供赞助。

搜集汇总

背景与挑战

背景概述

Sentinel-2云掩膜目录是一个用于机器学习和遥感应用的数据集，包含513个从2018年Sentinel-2存档随机采样的20米分辨率子场景，采用半自动标注方法生成三类分割掩膜（清晰、云、云阴影）和多种二进制分类标签（如表面类型、云类型），以支持云检测算法的训练和评估。数据集设计注重代表性，避免了区域偏差，并提供了形状文件以便在原始场景中定位掩膜。