EarthCARE level-2 demonstration products from simulated scenes

<strong>Overview</strong> The EarthCARE satellite combines four instruments, a Cloud Profiling Radar (CPR), an Atmospheric Lidar (ATLID), a Multispectral Imager (MSI) and a Broadband Radiometer (BBR), from which many products will be generated on the properties of clouds, aerosols, precipitation and radiation. The dataset in this repository consists of test products generated from simulated 3D scenes produced by the Canadian Global Environmental Multiscale (GEM) model. This includes both Level 1 (L1) "input" products containing simulated satellite measurements, and Level 2 (L2) "output" products produced by running the various European retrieval algorithms on the inputs. The dataset has been produced as part of the European Space Agency funded "CARDINAL" project involving numerous European and Canadian scientists, and there are several versions representing the evolution of the algorithms during preparation for the launch of EarthCARE. The scenes and algorithms are discussed in detail in a Special Issue of the journal Atmospheric Modelling Techniques (AMT), so the overview here is limited to a summary of what is contained in this dataset. <strong>Directory structure</strong> The four main top-level directories inside the zip file are for the three simulated scenes (except for the C-APC directory, described in a separate section below), each of which represent a single 6000-km long EarthCARE granule: Halifax: A swath over the Atlantic Ocean from the Caribbean to the Labrador Sea, passing close to Halifax in Nova Scotia Baja: A swath passing over the Rocky Mountains and the Baja California peninsula in Mexico Hawaii: A swath over the Pacific Ocean passing close to Hawaii Halifax_aerosol (added in version 10.10): A modified version of the Halifax scene with clouds removed and modified aerosols, to provided additional testing of the ATLID and MSI retrieval algorithms (note that algorithms involving CPR and BBR have not been run on this scene) Each of these directories contains three further directories: input: simulated level-1 instrument data (note that the data formats, e.g. attributes and variable names, do not exactly match those of real EarthCARE data, although they are close) output: level-2 meteorological products generated from the input data (note that updates to the algorithms in future may lead to minor changes to the data formats, so always use the most up-to-date Zenodo release) logs: text files logging the progress of the algorithms as they generated the level-2 data The input and output directories contain subdirectories for each product, and these each contain two files: A NetCDF4/HDF5 file with the suffix "h5" containing the retrieved data An XML file with the suffix "HDR" containing a description of the variables in the file (reproducing metadata already in the NetCDF4/HDF5 file) <strong>Input products (L1)</strong> The level-1 products are named <em>Y-ZZZ</em> where <em>Y</em> indicates the source of the data (A=ATLID, C=CPR, M=MSI, B=BBR and X=auxiliary) and <em>ZZZ</em> is a shortened version of the product name: A-NOM: Nominal ATLID measurements C-NOM: Nominal CPR measurements M-RGR: Regridded MSI measurements B-NOM: Nominal BBR measurements averaged to various scales B-SNG: Single-pixel BBR measurements X-JSG: Definition of the Joint Standard Grid on to which several of the observations are interpolated X-MET: Meteorological data from ECMWF <strong>Single-instrument output products (L2a)</strong> The naming convention is the same as the L1 data products. A-AER: ATLID aerosol profiles A-ALD: ATLID aerosol layer descriptor A-CTH: ATLID cloud top height A-EBD: ATLID extinction, backscatter and depolarization A-FM: ATLID feature mask A-ICE: ATLID ice cloud properties A-TC: ATLID target classification C-CD: CPR corrected Doppler velocity C-CLD: CPR cloud properties C-FMR: CPR feature mask and reflectivity C-TC: CPR target classification M-AOT: MSI aerosol optical thickness M-CM: MSI cloud mask M-COP: MSI cloud optical and physical properties <strong>Multi-instrument output products (L2b)</strong> The level-2b output products are named <em>YY-ZZZ </em>where YY is 2-4 character code conveying which of the four instruments were used and <em>ZZZ</em> is the shortened version of the product name.<em> </em> AC-TC: ATLID-CPR target classification AM-ACD: ATLID-MSI aerosol column descriptor AM-CTH: ATLID-MSI cloud top height BM-RAD: BBR-MSI broadband radiances (unfiltered) ACM-3D: ATLID-CPR-MSI constructed 3D scene ACM-CAP: ATLID-CPR-MSI synergistic retrieval of cloud, aerosol and precipitation ACM-COM: ATLID-CPR-MSI composite of single-instrument cloud and aerosol retrievals ACM-RT: ATLID-CPR-MSI radiative fluxes and heating rates computed on the retrievals BMA-FLX: BBR-MSI-ATLID broadband fluxes ACMB-DF: ATLID-CPR-MSI-BBR difference between radiances and fluxes computed from retrievals (ACM-RT) and measurements (BM-RAD, BMA-FLX) <strong>CPR Antenna Pointing Correction (C-APC) product</strong> From version 10.01 of the dataset, an additional top-level "C-APC" directory contains test data for the CPR Antenna Pointing Correction product, which will be generated via statistical analysis of a larger sample of C-NOM data. The generated information about mis-pointing of the radar antenna will then be used to improve interpretation of subsequent radar Doppler observations. There are two subdirectories: GEM_scenes: contains a file used in subsequent radar processing of the three GEM scenes, although in these scenes it has been assumed that there is no mis-pointing to correct, indicated by the file containing missing data and zeros. Antenna_mispointing_example: contains input and output files (in further subdirectories) illustrating what the data would look like with mis-pointing present. It was generated by artificially modifying the Doppler veclocities in the original Baja, Halifax and Hawaii scenes, concatenating them to generate a full orbit, and then running the C-APC algorithm to characterize the mis-pointing. <strong>Product definition documents</strong> From version 10.10 of the dataset, an additional top-level directory "Product_definition_documents" is provided containing documents that describe the variables contained in each of the level-2 product files. <strong>Filename format</strong> The data filenames and directories have the following name format: ECA_EXAA_<em>YYY_ZZZ_LL_OBSERVATION-TIME_GENERATION-TIME_VVVVVV</em> where: "ECA" indicates EarthCARE. EXAA indicates the file class: ESA, Latency N/A, Simulator, Baseline N/A. <em>YYY</em> is a three-character code indicating the instrumental source of the data. The 1-4 character codes defined in the sections above are converted to 3 character codes as follows: A -&gt; ATL, C -&gt; CPR, M -&gt; MSI, B -&gt; BBR, X -&gt; AUX, AM -&gt; AM_, AC -&gt; AC_, BM -&gt; BM_, ACMB -&gt; ALL. <em>ZZZ</em> is a three-character code abbreviating the full name of the product, as defined in the sections above. <em>LL</em> represents the level of the data from 1B, 1C, 1D, 2A and 2B. <em>OBSERVATION-TIME</em> is a code representing the date and time the observations were taken in the form <em>yyyymmdd</em>T<em>hhmmss</em>Z. Note that for the present datasets the dates are fictional future dates. <em>GENERATION-TIME</em> is a code of the same form representing the time that the processor was run to produce the data product. <em>VVVVVV</em> indicates the orbit number and frame letter. Further information may be obtained from the papers in the special issue of AMT. The description here may be expanded in future. Contacts: Gerd-Jan van Zadelhoff &lt;[e-mail]&gt; and Robin Hogan &lt;[e-mail]&gt;<strong>概述</strong> EarthCARE卫星搭载了四台仪器：云剖面雷达（Cloud Profiling Radar, CPR）、大气激光雷达（Atmospheric Lidar, ATLID）、多光谱成像仪（Multispectral Imager, MSI）以及宽带辐射计（Broadband Radiometer, BBR），可生成大量关于云、气溶胶、降水与辐射特性的产品。本仓库数据集由加拿大全球环境多尺度模型（Canadian Global Environmental Multiscale, GEM）生成的模拟三维场景所衍生的测试产品组成，包含两类数据：一是包含模拟卫星观测的一级（Level 1, L1）“输入”产品，二是通过对输入数据运行各类欧洲反演算法得到的二级（Level 2, L2）“输出”产品。本数据集由欧洲空间局（European Space Agency）资助的“CARDINAL”项目研发，该项目汇聚了众多欧洲与加拿大科学家，且包含多个算法迭代版本，对应EarthCARE发射筹备期间算法的演进历程。相关场景与算法的详细说明已发表于《大气建模技术》（Atmospheric Modelling Techniques, AMT）特刊，因此本概述仅对本数据集包含的内容做简要总结。 <strong>目录结构</strong> 压缩包内包含四个主要顶层目录，对应三个模拟场景（另有单独章节说明的C-APC目录除外），每个目录均代表一条长度达6000千米的EarthCARE数据条带： - Halifax：横跨大西洋的观测条带，范围从加勒比海延伸至拉布拉多海，途经新斯科舍省哈利法克斯附近海域 - Baja：途经落基山脉与墨西哥下加利福尼亚半岛的观测条带 - Hawaii：横跨太平洋的观测条带，途经夏威夷群岛附近海域 - Halifax_aerosol（10.10版本新增）：Halifax场景的修改版本，移除了云层并调整了气溶胶分布，用于额外测试ATLID与MSI的反演算法（注意：涉及CPR与BBR的算法未在该场景上运行） 每个场景目录下均包含三个子目录： - input：模拟的一级仪器数据（注：尽管数据格式与真实EarthCARE数据高度近似，但属性与变量名称并未完全匹配） - output：基于输入数据生成的二级气象产品（注：未来算法更新可能导致数据格式产生细微变化，请始终使用Zenodo平台最新版本的数据集） - logs：记录算法生成二级数据过程的文本日志文件 输入与输出目录下均按产品类型创建子目录，每个子目录包含两个文件： - 后缀为“h5”的NetCDF4/HDF5文件，存储反演得到的数据 - 后缀为“HDR”的XML文件，用于说明文件内的变量信息（与NetCDF4/HDF5文件中已包含的元数据重复） <strong>一级输入产品（L1）</strong> 一级产品的命名格式为<em>Y-ZZZ</em>，其中<em>Y</em>代表数据来源（A=ATLID、C=CPR、M=MSI、B=BBR、X=辅助数据），<em>ZZZ</em>为产品名称的缩写： - A-NOM：标准ATLID观测数据 - C-NOM：标准CPR观测数据 - M-RGR：重网格化MSI观测数据 - B-NOM：多尺度平均的标准BBR观测数据 - B-SNG：单像素BBR观测数据 - X-JSG：联合标准网格（Joint Standard Grid）定义文件，用于将多类观测数据插值至该网格 - X-MET：来自欧洲中期天气预报中心（ECMWF）的气象数据 <strong>单仪器输出产品（L2a）</strong> 命名规则与一级数据产品一致： - A-AER：ATLID气溶胶廓线 - A-ALD：ATLID气溶胶层描述符 - A-CTH：ATLID云顶高度 - A-EBD：ATLID消光、后向散射与退偏振数据 - A-FM：ATLID特征掩码 - A-ICE：ATLID冰云特性 - A-TC：ATLID目标分类 - C-CD：CPR校正多普勒速度 - C-CLD：CPR云特性 - C-FMR：CPR特征掩码与反射率 - C-TC：CPR目标分类 - M-AOT：MSI气溶胶光学厚度 - M-CM：MSI云掩码 - M-COP：MSI云光学与物理特性 <strong>多仪器协同输出产品（L2b）</strong> 二级协同输出产品的命名格式为<em>YY-ZZZ</em>，其中<em>YY</em>为2-4位字符代码，代表使用的仪器组合，<em>ZZZ</em>为产品名称缩写： - AC-TC：ATLID-CPR目标分类 - AM-ACD：ATLID-MSI气溶胶柱描述符 - AM-CTH：ATLID-MSI云顶高度 - BM-RAD：BBR-MSI宽带辐射率（未滤波） - ACM-3D：ATLID-CPR-MSI构建的三维场景 - ACM-CAP：ATLID-CPR-MSI云、气溶胶与降水协同反演 - ACM-COM：ATLID-CPR-MSI单仪器云与气溶胶反演结果的合成 - ACM-RT：基于ATLID-CPR-MSI反演结果计算的辐射通量与加热率 - BMA-FLX：BBR-MSI-ATLID宽带通量 - ACMB-DF：基于反演结果（ACM-RT）与观测数据（BM-RAD、BMA-FLX）计算得到的辐射率与通量差值（ATLID-CPR-MSI-BBR） <strong>CPR天线指向校正（CPR Antenna Pointing Correction, C-APC）产品</strong> 自数据集10.01版本起，新增顶层目录“C-APC”，用于存储CPR天线指向校正产品的测试数据。该产品将通过对大规模C-NOM数据集的统计分析生成，其得到的雷达天线指向偏差信息将用于改进后续雷达多普勒观测的解译。该目录包含两个子目录： - GEM_scenes：包含用于后续三个GEM场景雷达处理的文件，不过在本场景中假设不存在需要校正的指向偏差，文件中仅包含缺失值与零值 - Antenna_mispointing_example：包含输入与输出文件（嵌套子目录），用于演示存在指向偏差时的数据形态。该数据集通过人工修改原始Baja、Halifax与Hawaii场景中的多普勒速度，拼接为完整轨道数据后运行C-APC算法以表征指向偏差生成。 <strong>产品定义文档</strong> 自数据集10.10版本起，新增顶层目录“Product_definition_documents”，用于存储各二级产品文件所包含变量的说明文档。 <strong>文件名格式</strong> 数据文件名与目录遵循以下命名规则：`ECA_EXAA_<em>YYY_ZZZ_LL_OBSERVATION-TIME_GENERATION-TIME_VVVVVV</em>`，各字段说明如下： - "ECA"：代表EarthCARE - "EXAA"：代表文件类别：ESA、延迟时间N/A、模拟器、基线N/A - <em>YYY</em>：三位字符代码，代表数据的仪器来源。前文提及的1-4位字符代码转换为三位代码的规则如下：A→ATL、C→CPR、M→MSI、B→BBR、X→AUX、AM→AM_、AC→AC_、BM→BM_、ACMB→ALL - <em>ZZZ</em>：三位字符代码，为对应产品全称的缩写，与前文定义一致 - <em>LL</em>：代表数据级别，可选值为1B、1C、1D、2A与2B - <em>OBSERVATION-TIME</em>：观测日期与时间代码，格式为<em>yyyymmdd</em>T<em>hhmmss</em>Z。注意：本数据集所用日期均为虚构的未来日期 - <em>GENERATION-TIME</em>：与上述格式一致的代码，代表处理器生成该数据产品的运行时间 - <em>VVVVVV</em>：代表轨道号与帧字母 更多详细信息可参阅AMT特刊中的相关论文，本概述后续可能进行扩充。 联系方式：Gerd-Jan van Zadelhoff &lt;[e-mail]&gt; 与 Robin Hogan &lt;[e-mail]&gt;