Outdoor Deployed SCD41 Co-Located Sensors Dataset

Summary Description Fourteen-month deployment of four Sensirion SCD41 low-cost CO₂ sensors at the SMEAR Estonia station, Järvselja, Estonia. Three sensors (A, B, C) operated at 30 m on the atmospheric sensing mast; the fourth (Sensor D; SCT1) at 2 m. One LGR reference instrument provided ground-truth measurements at 30 m. Sensors operated under adverse environmental conditions outside manufacturer specifications. The dataset includes 10-minute aggregated readings, per-sensor with merged outputs, quality-control flags, and validation receipts. Intended as an adverse-conditions benchmark for evaluating robust calibration methods for low-cost environmental sensors. Depicted preview Taylor diagram summarising SCD41 sensor performance relative to the LGR reference. Radial distance encodes standard deviation (ppm); angular position encodes Pearson correlation; dashed arcs show centred RMSD. Arrows indicate sensors whose standard deviation exceeds the plot radius. Data Dictionary Timestamps All timestamps are in Coordinated Universal Time (UTC), formatted as ISO 8601 strings in CSV files (YYYY-MM-DD HH:MM:SS) and as datetime64[ns] in Parquet files. Each timestamp marks the start of a non-overlapping 10-minute aggregation window aligned to the Unix epoch. Missing Data Missing values appear as empty cells in CSV and as NaN in Parquet. No interpolation or gap-filling is applied. The merged file uses an outer join: if a sensor has no data at a given timestamp, all its columns are NaN for that row. Per-Sensor Files: SCD41 (Sensors A, B, C, D) File pattern: {SENSOR}_10min.csv, {SENSOR}_{YYYY}_{MM}.csv Sensors: CO2_SCT1_2M (Sensor D, 2 m), CO2_30M_A, CO2_30M_B, CO2_30M_C (30 m) Column Type Unit Description timestamp datetime UTC Start of 10-minute window co2_mean float64 ppm Mean CO₂ concentration co2_std float64 ppm Standard deviation of CO₂ within window co2_count int64 — Number of deduplicated raw samples in window temp_mean float64 °C Mean on-chip temperature (SHT4x sensor) temp_std float64 °C Standard deviation of temperature temp_count int64 — Number of temperature samples humidity_mean float64 % RH Mean relative humidity (SHT4x sensor) humidity_std float64 % RH Standard deviation of humidity humidity_count int64 — Number of humidity samples co2_qc string — Quality-control flag for CO₂ temp_qc string — Quality-control flag for temperature humidity_qc string — Quality-control flag for humidity 13 columns per file. At the SCD41 native sampling rate of 0.2 Hz, a full 10-minute window contains up to 120 deduplicated samples. Per-Sensor Files: LGR Reference File pattern: LGR_30M_10min.csv, LGR_30M_{YYYY}_{MM}.csv Each measured variable produces four columns following the pattern {variable}_mean, {variable}_std, {variable}_count, and (where applicable) {variable}_sd for pooled standard deviation of the instrument-reported uncertainty. Column pattern Type Unit Description CO2_dry_* float64 ppm Dry-air CO₂ mole fraction (primary reference) CO2_ppm_* float64 ppm Wet-air CO₂ concentration CH4_dry_* float64 ppm Dry-air CH₄ mole fraction CH4_ppm_* float64 ppm Wet-air CH₄ concentration H2O_ppm_* float64 ppm Water vapour concentration GasP_torr_* float64 torr Internal gas cell pressure GasT_C_* float64 °C Internal gas cell temperature AmbT_C_* float64 °C Enclosure ambient temperature RD0_us_* float64 µs Ring-down time (channel 0) RD1_us_* float64 µs Ring-down time (channel 1) quality_* float64 0–100 Instrument quality indicator Fit_Flag_* float64 — Spectral fit quality (3 = good) co2_qc string — QC flag for CO₂ range quality_qc string — QC flag for instrument quality and fit At the configured logging interval of approximately 120 s, each 10-minute window contains at most 5 LGR samples. Merged File File pattern: SMEAR_EE_CO2_merged.csv, SMEAR_EE_CO2_{YYYY}_{MM}_merged.csv 100 columns. All sensors are aligned to a common 10-minute timestamp index via outer join. Column Naming Convention Merged columns are prefixed with the lowercase sensor name: {sensor}_{variable}_{statistic} Examples: co2_sct1_2m_co2_mean → Sensor D (2 m), CO₂, mean co2_30m_a_temp_std → Sensor A (30 m), temperature, standard deviation lgr_30m_CO2_dry_mean → LGR reference, dry CO₂, mean SCD41 Sensor Columns (×4 sensors, 12 columns each) Each SCD41 sensor contributes 12 columns: co2_mean, co2_std, co2_count, temp_mean, temp_std, temp_count, humidity_mean, humidity_std, humidity_count, co2_qc, temp_qc, humidity_qc — all prefixed with the sensor name. LGR Columns (47 columns) The LGR contributes 48 columns. Ten measurement variables (CO₂, CH₄, H₂O, pressure, temperatures, ring-down times) each produce four columns (mean, std, count, and pooled SD of the instrument-reported uncertainty); quality and Fit_Flag produce three columns each (mean, std, count — no instrument-reported SD exists for these). Two QC flag columns (co2_qc, quality_qc) complete the set. Pooled SD columns carry the suffix _sd__pooled_sd (double underscore). Cluster Statistics (3 columns) Computed from the three co-located 30 m SCD41 sensors (A, B, C): Column Type Unit Description scd41_30m_cluster_co2_mean float64 ppm Mean CO₂ across reporting 30 m sensors scd41_30m_cluster_co2_std float64 ppm Standard deviation across reporting 30 m sensors scd41_30m_cluster_co2_count int64 — Number of 30 m sensors reporting (0–3) Cluster statistics use only sensors with valid data at each timestamp. If one sensor reports, std is NaN. Quality-Control Flags Flag Applies to Meaning OK All sensors Value within expected range BELOW_RANGE All sensors Below minimum threshold (CO₂ < 300 ppm, T < −40 °C, RH < 0 %) ABOVE_RANGE All sensors Above maximum threshold (CO₂ > 1000 ppm, T > 60 °C, RH > 100 %) SPIKE SCD41 only Absolute change > 200 ppm between consecutive raw samples NO_DATA All sensors No valid samples in aggregation window LOW_QUALITY LGR only Quality indicator < 95 BAD_FIT LGR only Fit flag ≠ 3 We recommend filtering on qc_flag = 'OK' for initial analyses. In this deployment, only OK, BELOW_RANGE, ABOVE_RANGE, and NO_DATA appear in the output. The remaining flags (SPIKE, LOW_QUALITY, BAD_FIT) are defined by the pipeline but were not triggered by any record in the current dataset. File Formats Format Extension Compression Notes CSV .csv None Human-readable; timestamps as ISO 8601 strings Parquet .parquet Snappy Columnar binary; preserves native types; recommended for large-scale analysis CSV and Parquet files for the same time period contain identical data. Parquet files are typically 5–10× smaller. File Naming Pattern Example Contents {SENSOR}_10min.csv CO2_30M_A_10min.csv Full concatenated time series {SENSOR}_{YYYY}_{MM}.csv CO2_30M_A_2025_03.csv Single-month extract {SENSOR}_{YYYY}_{MM}.parquet CO2_30M_A_2025_03.parquet Monthly Parquet {SENSOR}_uptime.csv CO2_30M_A_uptime.csv Daily uptime statistics SMEAR_EE_CO2_merged.csv — All sensors, full period SMEAR_EE_CO2_{YYYY}_{MM}_merged.* SMEAR_EE_CO2_2025_03_merged.parquet Monthly merged数据集概述 本次部署在爱沙尼亚亚尔夫瑟利亚（Järvselja）的SMEAR爱沙尼亚站点，共部署4台盛思锐（Sensirion）SCD41低成本二氧化碳（CO₂）传感器，监测时长为14个月。其中3台传感器（编号A、B、C）安装于30米高的大气监测桅杆上，第4台传感器（传感器D，编号SCT1）安装于2米高度。另有1台激光气体分析仪（LGR）参考仪器在30米高度提供真值参考测量。所有传感器均在超出厂商规格的恶劣环境条件下运行。 本数据集包含10分钟聚合的逐传感器合并输出数据、质量控制标记及验证记录，旨在作为恶劣环境下的基准数据集，用于评估低成本环境传感器的鲁棒校准方法。 ## 预览说明 附带的预览泰勒图（Taylor diagram）总结了SCD41传感器相对于LGR参考仪器的性能表现：径向距离代表标准差（单位：ppm），角位置代表皮尔逊相关系数，虚线弧表示中心化均方根偏差（centred RMSD），箭头指示标准差超出绘图半径的传感器。 ## 数据字典 ### 时间戳 所有时间戳均采用协调世界时（UTC）：CSV文件中格式为ISO 8601字符串（YYYY-MM-DD HH:MM:SS），Parquet文件中格式为datetime64[ns]。每个时间戳标记一个与Unix纪元对齐的、无重叠的10分钟聚合窗口的起始时刻。 ### 缺失数据 缺失值在CSV文件中显示为空单元格，在Parquet文件中显示为NaN。本数据集未进行任何插值或间隙填充处理。合并文件采用外连接（outer join）方式：若某传感器在给定时间戳无有效数据，则该行中该传感器的所有列均为NaN。 ### 逐传感器文件：SCD41（传感器A、B、C、D） 文件命名模式：'{SENSOR}_10min.csv'、'{SENSOR}_{YYYY}_{MM}.csv' 传感器对应关系： - CO2_SCT1_2M：传感器D，2米高度 - CO2_30M_A、CO2_30M_B、CO2_30M_C：均为30米高度 #### 列定义 | 列名 | 数据类型 | 单位 | 描述 | |---------------------|------------|------------|----------------------------------------------------------------------| | timestamp | datetime | UTC | 10分钟聚合窗口的起始时刻 | | co2_mean | float64 | ppm | 窗口内CO₂浓度平均值 | | co2_std | float64 | ppm | 窗口内CO₂浓度的标准差 | | co2_count | int64 | — | 窗口内去重后的原始采样总数 | | temp_mean | float64 | °C | 片上温度（SHT4x传感器）平均值 | | temp_std | float64 | °C | 温度标准差 | | temp_count | int64 | — | 温度采样总数 | | humidity_mean | float64 | % RH | 相对湿度（SHT4x传感器）平均值 | | humidity_std | float64 | % RH | 相对湿度标准差 | | humidity_count | int64 | — | 相对湿度采样总数 | | co2_qc | string | — | CO₂质量控制标记 | | temp_qc | string | — | 温度质量控制标记 | | humidity_qc | string | — | 相对湿度质量控制标记 | 每个文件包含13列。以SCD41原生采样率0.2 Hz计算，完整的10分钟窗口最多包含120个去重后的原始采样样本。 ### 逐传感器文件：LGR参考仪器 文件命名模式：'LGR_30M_10min.csv'、'LGR_30M_{YYYY}_{MM}.csv' 每个测量变量对应4列，命名格式为'{variable}_mean'、'{variable}_std'、'{variable}_count'，以及（若适用）'{variable}_sd'，代表仪器报告的不确定度合并标准差。 #### 列定义 | 列名模式 | 数据类型 | 单位 | 描述 | |----------------------|------------|------------|----------------------------------------------------------------------| | CO2_dry_* | float64 | ppm | 干空气CO₂摩尔分数（一级参考测量值） | | CO2_ppm_* | float64 | ppm | 湿空气CO₂浓度 | | CH4_dry_* | float64 | ppm | 干空气甲烷（CH₄）摩尔分数 | | CH4_ppm_* | float64 | ppm | 湿空气甲烷浓度 | | H2O_ppm_* | float64 | ppm | 水蒸气浓度 | | GasP_torr_* | float64 | torr | 内部气室压力 | | GasT_C_* | float64 | °C | 内部气室温度 | | AmbT_C_* | float64 | °C | 机箱环境温度 | | RD0_us_* | float64 | µs | 振铃时间（通道0） | | RD1_us_* | float64 | µs | 振铃时间（通道1） | | quality_* | float64 | 0–100 | 仪器质量指示值 | | Fit_Flag_* | float64 | — | 光谱拟合质量（3为合格状态） | | co2_qc | string | — | CO₂量程质量控制标记 | | quality_qc | string | — | 仪器质量与拟合质量控制标记 | 以配置的约120秒的记录间隔计算，每个10分钟窗口最多包含5个LGR采样样本。 ### 合并文件 文件命名模式：'SMEAR_EE_CO2_merged.csv'、'SMEAR_EE_CO2_{YYYY}_{MM}_merged.csv' 共包含100列。所有传感器通过外连接对齐至统一的10分钟时间戳索引。 #### 列命名约定 合并后的列名以小写传感器名称作为前缀，格式为：'{sensor}_{variable}_{statistic}' 示例： - 'co2_sct1_2m_co2_mean'：传感器D（2米高度）的CO₂浓度平均值 - 'co2_30m_a_temp_std'：传感器A（30米高度）的温度标准差 - 'lgr_30m_CO2_dry_mean'：LGR参考仪器的干空气CO₂浓度平均值 #### SCD41传感器列（4台传感器，每台12列） 每台SCD41传感器贡献12列：'co2_mean'、'co2_std'、'co2_count'、'temp_mean'、'temp_std'、'temp_count'、'humidity_mean'、'humidity_std'、'humidity_count'、'co2_qc'、'temp_qc'、'humidity_qc'，均以传感器名称作为前缀。 #### LGR参考仪器列（48列） LGR参考仪器共贡献48列：10个测量变量（CO₂、CH₄、H₂O、压力、温度、振铃时间）每个对应4列（平均值、标准差、采样数、仪器报告不确定度的合并标准差）；质量指标与拟合标记各对应3列（平均值、标准差、采样数，无仪器报告的不确定度标准差）；另有2个质量控制标记列（'co2_qc'、'quality_qc'）。合并标准差列的后缀为'_sd__pooled_sd'（双下划线分隔）。 #### 集群统计量（3列） 基于3台共址的30米高度SCD41传感器（A、B、C）计算得到： | 列名 | 数据类型 | 单位 | 描述 | |------------------------------------|------------|------------|----------------------------------------------------------------------| | scd41_30m_cluster_co2_mean | float64 | ppm | 所有有效上报的30米传感器的CO₂平均浓度 | | scd41_30m_cluster_co2_std | float64 | ppm | 所有有效上报的30米传感器的CO₂浓度标准差 | | scd41_30m_cluster_co2_count | int64 | — | 上报有效数据的30米传感器数量（0–3） | 集群统计量仅使用每个时间戳下有有效数据的传感器。若仅1台传感器上报数据，则标准差列为NaN。 ### 质量控制标记 | 标记值 | 适用传感器范围 | 含义 | |----------------|----------------|----------------------------------------------------------------------| | OK | 所有传感器 | 测量值处于预期范围内 | | BELOW_RANGE | 所有传感器 | 测量值低于最小阈值（CO₂ < 300 ppm、温度 < −40 °C、相对湿度 < 0 % RH） | | ABOVE_RANGE | 所有传感器 | 测量值高于最大阈值（CO₂ > 1000 ppm、温度 > 60 °C、相对湿度 > 100 % RH） | | SPIKE | 仅SCD41传感器 | 相邻原始采样间的绝对变化量超过200 ppm | | NO_DATA | 所有传感器 | 聚合窗口内无有效采样 | | LOW_QUALITY | 仅LGR参考仪器 | 仪器质量指示值 < 95 | | BAD_FIT | 仅LGR参考仪器 | 拟合标记值 ≠ 3 | 建议在初始分析时过滤'qc_flag'为'OK'的样本。本次部署中，输出结果仅包含'OK'、'BELOW_RANGE'、'ABOVE_RANGE'和'NO_DATA'四种标记；其余标记（'SPIKE'、'LOW_QUALITY'、'BAD_FIT'）虽在处理流程中定义，但本次数据集中未触发。 ### 文件格式 | 格式 | 扩展名 | 压缩方式 | 说明 | |------------|--------------|----------|----------------------------------------------------------------------| | CSV | .csv | 无压缩 | 人类可读格式；时间戳采用ISO 8601字符串格式 | | Parquet | .parquet | Snappy | 列式二进制格式；保留原生数据类型；推荐用于大规模数据分析 | 同一时间段的CSV与Parquet文件包含完全相同的数据。Parquet文件的体积通常比CSV小5–10倍。 ### 文件命名规则 | 命名模式 | 示例 | 内容说明 | |-----------------------------------|---------------------------------------|------------------------------| | '{SENSOR}_10min.csv' | 'CO2_30M_A_10min.csv' | 完整拼接的时间序列数据 | | '{SENSOR}_{YYYY}_{MM}.csv' | 'CO2_30M_A_2025_03.csv' | 单月数据提取文件 | | '{SENSOR}_{YYYY}_{MM}.parquet' | 'CO2_30M_A_2025_03.parquet' | 月度Parquet格式数据文件 | | '{SENSOR}_uptime.csv' | 'CO2_30M_A_uptime.csv' | 每日运行时长统计数据 | | 'SMEAR_EE_CO2_merged.csv' | — | 全时间段所有传感器的合并数据 | | 'SMEAR_EE_CO2_{YYYY}_{MM}_merged.*' | 'SMEAR_EE_CO2_2025_03_merged.parquet' | 月度合并数据文件 |