Initial Sample of HYPERNETS Hyperspectral Surface Reflectance Measurements for Satellite Validation from the Barrax Site in Spain

The HYPERNETS project (www.hypernets.eu) has the overall aim to ensure that high quality in situ measurements are available to support the (VNIR/SWIR) optical Copernicus products. Therefore, it established a new autonomous hyperspectral spectroradiometer (HYPSTAR® - www.hypstar.eu) dedicated to land and water surface reflectance validation with instrument pointing capabilities. In the prototype phase, the instrument is being deployed at 24 sites covering a range of water and land types and a range of climatic and logistic conditions. This dataset provides the first published data for the Gobabeb HYPERNETS site in Barrax, Spain (BASP). It is a subset of the complete data record which consists of the best quality BASP measurements which could be used for satellite validation over the three day test deployment period. The provided NetCDF files are the L2A hypernets products with surface reflectances, their associated uncertainties and error-correlation information. The reflectance in the L2A products is the Hemispherical-directional Reflectance Factor (HDRF) defined as: HDRF = π L / E where L is the directional upwelling radiance (with field of view of 5 degrees) and E is the (hemispherical) downwelling irradiance (i.e. including both direct solar and diffuse sky irradiance). These reflectances have dimensions of wavelength and series, where each series is a set of measurements for a given geometry (combination of viewing zenith and azimuth angle). In addition to variables for wavelength and bandwidth, the files also contain variables that provide for each series the acquisition time, viewing and solar angles, number of valid scans used, and quality flags (typically no flags are set in the data provided in this dataset). These NetCDF files also contain further relevant metadata as attributes. See https://hypernets-processor.readthedocs.io/ for further info. The BASP site was a temporary installation over the period of the 20^th – 22^nd July 2022 during the Surface Reflectance Intercomparison eXperiment (SRIX) campaign (https://frm4veg.org/srix4veg/) at the Las Tiesas experimental farm in Barrax, Spain. This location was selected due to its typical clear skies, flat terrain, and well-managed crops. The HYPSTAR®-XR (eXtended Range) was deployed in a small corn field next to the ongoing UAV experiment. The instrument was deployed on a 3.5m high pole with a short extended boom at 1.3m height from the crops, with measurements running every 30 minutes throughout the day (UTC+2) and measuring between viewing zenith angles of 0-60 degrees. The HYPSTAR®-XR instruments deployed at each land HYPERNETS site consist of a VNIR and a SWIR sensor and autonomously collect data between 380-1700 nm at various viewing geometries and send it to a central server for quality control and processing. The VNIR sensor spans 1330 channels between 380 and 1000 nm with a FWHM of 3 nm and the SWIR sensor has 220 channels between 1000 and 1700 nm with a FWHM of 10 nm. The hypernets_processor (Goyens et al. 2021; De Vis et al. in prep.) automatically processes all this data into various products, including the L2A surface reflectance product provided here. All of the products have associated uncertainties (divided into random and systematic uncertainties, including error-correlation information) which were propagated using the CoMet toolkit (www.comet-toolkit.org). To obtain this dataset, we start from the full BASP data record and omit all the data that do not pass all of the quality checks performed as part of the hypernets_processor. In addition, an additional screening procedure was developed to remove outliers and only supply the best quality data suitable for satellite validation. To remove the outliers, a sigma-clipping method is used. First reflectances are extracted in separate 2 hour windows throughout the day (to account for BRDF differences due to different solar position) for 4 different wavelengths (500, 900, 1100 and 1600 nm). Outliers in these reflectances are then identified by iteratively calculating the mean reflectance trend with time (by binning the data per maximum 30 data points), calculating the standard deviation from this trend, and masking any data that is more than 3 standard deviations away from the trend. This process is repeated on the unmasked data until the standard deviation does not vary by more than 5% between two iterations. The masks for the 4 different wavelengths are then combined (keeping only measurements for which none of the 4 wavelengths is an outlier). The reflectances and associated uncertainties for any masked series (i.e. a geometry that is masked either by the sigma-clipping procedure or from the masks of the hypernets_processor) are replaced by NaNs. Any sequence that has more than half of its series masked is removed entirely. For BASP specifically, viewing zenith angles above 30 degrees have been removed, as well as any west-facing angles azimuth angles of 263,273 or 293 degrees) for viewing zenith angles of 5 degrees and 10 degrees. Note: In the accompanying .csv description file the measurement times are listed in (BST/ UTC+1) after UK time.HYPERNETS项目（HYPERNETS，www.hypernets.eu）的总体目标是提供高质量原位测量（in situ measurements）数据，以支撑可见近红外/短波红外（VNIR/SWIR）光学哥白尼（Copernicus）产品的验证工作。为此，项目研发了一款新型自主式高光谱光谱辐射计（hyperspectral spectroradiometer，HYPSTAR®，www.hypstar.eu），专为陆地与水面反射率验证设计，具备仪器指向功能。 在原型阶段，该仪器已部署于24个站点，覆盖多样的水陆地表类型与气候、后勤保障条件。本数据集首次公开了西班牙巴拉克斯（Barrax）地区戈巴布HYPERNETS站点（BASP）的实测数据，属于完整数据集的子集，仅包含该站点在三日测试部署阶段中可用于卫星验证的高质量BASP测量数据。 本次提供的NetCDF文件为L2A级HYPERNETS产品，包含地表反射率、相关不确定度以及误差相关信息。L2A产品中的反射率为半球方向反射因子（Hemispherical-directional Reflectance Factor, HDRF），其定义为：HDRF = π L / E，其中L为视场角（field of view, FOV）5度的定向向上辐亮度，E为（半球形）向下辐照度（涵盖直射太阳辐照度与漫射天空辐照度）。 上述反射率以波长与序列为维度，每个序列代表给定几何构型（观测天顶角与方位角的组合）下的一组测量数据。除波长与带宽变量外，文件还包含为每个序列提供的采集时间、观测与太阳角度、有效扫描次数以及质量标志（quality flags，本数据集提供的数据中通常未设置质量标志）。这些NetCDF文件还包含作为属性存储的其他相关元数据（metadata），更多信息可参见https://hypernets-processor.readthedocs.io/。 BASP站点为临时部署站点，部署时间为2022年7月20日至22日，期间正值西班牙巴拉克斯拉斯蒂埃萨斯（Las Tiesas）试验农场开展的地表反射率比对试验（Surface Reflectance Intercomparison eXperiment, SRIX）活动（https://frm4veg.org/srix4veg/）。该站点的选址基于其典型的晴朗天气、平坦地形与管理良好的作物种植条件。 本次部署的HYPSTAR®-XR（扩展量程版）位于一处小型玉米田中，毗邻正在进行的无人机（UAV）试验。仪器安装在3.5米高的杆体上，通过短延伸臂将传感器置于距作物冠层1.3米高度处，每日（UTC+2时区）每30分钟开展一次测量，观测天顶角覆盖0至60度范围。 部署于各陆地HYPERNETS站点的HYPSTAR®-XR仪器由VNIR与SWIR传感器组成，可自主采集380~1700 nm波段范围内、不同观测几何下的数据，并将数据传输至中央服务器进行质量控制与处理。其中VNIR传感器覆盖380~1000 nm波段，包含1330个通道，半高全宽（full width at half maximum, FWHM）为3 nm；SWIR传感器覆盖1000~1700 nm波段，包含220个通道，半高全宽为10 nm。 hypernets_processor（Goyens等人，2021；De Vis等人，待发表）可自动将所有原始数据处理为多种产品，包括本次提供的L2A级地表反射率产品。所有产品均附带相关不确定度（分为随机不确定度与系统不确定度，包含误差相关信息），不确定度的传播通过CoMet工具包（CoMet toolkit，www.comet-toolkit.org）完成。 本数据集的构建流程为：从完整的BASP数据记录出发，剔除所有未通过hypernets_processor内置质量检查的数据。此外，还开发了额外的筛选流程以剔除异常值，仅保留适用于卫星验证的最高质量数据。异常值剔除采用西格玛截断法（sigma-clipping method）：首先以2小时为窗口将每日数据划分为多个时段（以抵消不同太阳位置导致的双向反射分布函数（BRDF）差异），并针对500、900、1100与1600 nm四个特定波长提取反射率数据；随后通过迭代计算反射率随时间的变化趋势（以最多30个数据点为一组进行分箱），计算该趋势的标准差（standard deviation），屏蔽所有偏离趋势超过3倍标准差的数据；该过程在未被屏蔽的数据上重复执行，直至两次迭代间的标准差变化不超过5%。将四个波长对应的屏蔽掩码进行合并，仅保留四个波长均未被判定为异常值的测量数据。对于任何被屏蔽的序列（即通过西格玛截断法或hypernets_processor掩码被屏蔽的几何构型），其反射率与相关不确定度将被替换为非数值（NaNs）。若某组序列中有超过一半的成员被屏蔽，则将该组数据整体移除。针对BASP站点的额外筛选规则为：移除观测天顶角大于30度的数据，同时对于观测天顶角为5度与10度的测量，剔除方位角为263、273或293度的西向观测数据。 注：随附的.csv描述文件中列出的测量时间采用英国夏令时（BST/UTC+1）。