The Device Activity Report with Complete Knowledge (DARCK) for NILM

1. Abstract This dataset contains aggregated and sub-metered power consumption data from a two-person apartment in Germany. Data was collected from March 5 to September 4, 2025, spanning 6 months. It includes an aggregate reading from a main smart meter and individual readings from 40 smart plugs, smart relays, and smart power meters monitoring various appliances. 2. Dataset Overview Apartment: Two-person apartment, approx. 58m², located in Aachen, Germany. Aggregate Meter: eBZ DD3 Sub-meters: 31 Shelly Plus Plug S, 6 Shelly Plus 1PM, 3 Shelly Plus PM Mini Gen3 Sampling Rate: 1 Hz Measured Quantity: Active Power Unit of Measurement: Watt Duration: 6 months Format: Single CSV file (`DARCK.csv`) Structure: Timestamped rows with columns for the aggregate meter and each sub-metered appliance. Completeness: The main power meter has a completeness of 99.3%. Missing values were linearly interpolated. 3. Download and Usage The dataset can be downloaded here: https://doi.org/10.5281/zenodo.17159850 As it contains longer off periods with zeros, the CSV file is nicely compressible. To extract it use: xz -d DARCK.csv.xz.The compression leads to a 97% smaller file size (From 4GB to 90.9MB). To use the dataset in python, you can, e.g., load the csv file into a pandas dataframe. pythonimport pandas as pd df = pd.read_csv("DARCK.csv", parse_dates=["time"]) 4. Measurement Setup The main meter was monitored using an infrared reading head magnetically attached to the infrared interface of the meter. An ESP8266 flashed with Tasmota decodes the binary datagrams and forwards the Watt readings to the MQTT broker. Individual appliances were monitored using a combination of Shelly Plugs (for outlets), Shelly 1PM (for wired-in devices like ceiling lights), and Shelly PM Mini (for each of the three phases of the oven). All devices reported to a central InfluxDB database via Home Assistant running in docker on a Dell OptiPlex 3020M. 5. File Format (DARCK.csv) The dataset is provided as a single comma-separated value (CSV) file. The first row is a header containing the column names. All power values are rounded to the first decimal place. There are no missing values in the final dataset. Each row represents 1 second, from start of measuring in March until the end in September. Column Descriptions Column Name Data Type Unit Description time datetime - Timestamp for the reading in YYYY-MM-DD HH:MM:SS main float Watt Total aggregate power consumption for the apartment, measured at the main electrical panel. [appliance_name] float Watt Power consumption of an individual appliance (e.g., lightbathroom, fridge, sherlockpc). See Section 8 for a full list. Aggregate Columns aggr_chargers float Watt The sum of sherlockcharger, sherlocklaptop, watsoncharger, watsonlaptop, watsonipadcharger, kitchencharger. aggr_stoveplates float Watt The sum of stoveplatel1 and stoveplatel2. aggr_lights float Watt The sum of lightbathroom, lighthallway, lightsherlock, lightkitchen, lightlivingroom, lightwatson, lightstoreroom, fcob, sherlockalarmclocklight, sherlockfloorlamphue, sherlockledstrip, livingfloorlamphue, sherlockglobe, watsonfloorlamp, watsondesklamp and watsonledmap. Analysis Columns inaccuracy float Watt As no electrical device bypasses a power meter, the true inaccuracy can be assessed. It is the absolute error between the sum of individual measurements and the mains reading. A 30W offset is applied to the sum since the measurement devices themselves draw power which is otherwise unaccounted for. 6. Data Postprocessing Pipeline The final dataset was generated from two raw data sources (meter.csv and shellies.csv) using a comprehensive postprocessing pipeline. 6.1. Main Meter (main) Postprocessing The aggregate power data required several cleaning steps to ensure accuracy. Outlier Removal: Readings below 10W or above 10,000W were removed (merely 3 occurrences). Timestamp Burst Correction: The source data contained bursts of delayed readings. A custom algorithm was used to identify these bursts (large time gap followed by rapid readings) and back-fill the timestamps to create an evenly spaced time series. Alignment & Interpolation: The smart meter pushes a new value via infrared every second. To align those to the whole seconds, it was resampled to a 1-second frequency by taking the mean of all readings within each second (in 99.5% only 1 value). Any resulting gaps (0.7% outage ratio) were filled using linear interpolation. 6.2. Sub-metered Devices (shellies) Postprocessing The Shelly devices are not prone to the same burst issue as the ESP8266 is. They push a new reading at every change in power drawn. If no power change is observed or the one observed is too small (less than a few Watt), the reading is pushed once a minute, together with a heartbeat. When a device turns on or off, intermediate power values are published, which leads to sub-second values that need to be handled. Grouping: Data was grouped by the unique device identifier. Resampling & Filling: The data for each device was resampled to a 1-second frequency using .resample('1s').last().ffill(). This method was chosen to firstly, capture the last known state of the device within each second, handling rapid on/off events. Secondly, to forward-fill the last state across periods of no new data, modeling that the device's consumption remained constant until a new reading was sent. 6.3. Merging and Finalization Merge: The cleaned main meter and all sub-metered device dataframes were merged into a single dataframe on the time index. Final Fill: Any remaining NaN values (e.g., from before a device was installed) were filled with 0.0, assuming zero consumption. 7. Manual Corrections and Known Data Issues During analysis, two significant unmetered load events were identified and manually corrected to improve the accuracy of the aggregate reading. The error column (inaccuracy) was recalculated after these corrections. March 10th - Unmetered Bulb: An unmetered 107W bulb was active. It was subtracted from the main reading as if it never happened. May 31st - Unmetered Air Pump: An unmetered 101W pump for an air mattress was used directly in an outlet with no intermediary plug and hence manually added to the respective plug. 8. Appliance Details and Multipurpose Plugs The following table lists the column names with an explanation where needed. As Watson moved at the beginning of June, some metering plugs changed their appliance. Those columns were split into new ones that reflect the then used appliance. Hence, there are more columns than physical metering devices. Column Name Location / Appliance Notes lightkitchen, lightwatson, lighthallway, lightbathroom, lightsherlock, lightstoreroom, lightlivingroom, fcob, watsonfloorlamp,livingfloorlamphue, watsonledmap, watsondesklamp, sherlockglobe, sherlockledstrip, sherlockfloorlamphue, sherlockalarmclocklight Misc / Lights The seven lights starting with light... are all ceiling lights. fcob is an FCOB (Flip Chip on Board) LED Strip that illuminates the counter in the kitchen. stoveplatel1, stoveplatel2, stoveovenl3 Kitchen / Stove and Oven Individual phases of the kitchen stove and oven. The stove has 4 plates. The 2 bottom ones are on phase 1, the 2 top ones are on phase 2. The oven is on phase 3. fridge, washingmachine, microwave, kettle, wheymixer, mixer, blender, kitchensoundsystem Kitchen / Other Appliances router, watsonvinylplayer watsoncharger, watsonfan, watsonlaptop, watsondesklamp, watsonipadcharger , watsonmonitor, watsonpiano , htpc Watson's Room, later Living Room sherlockpc, sherlocktv, sherlockmonitor, sherlockserver, sherlockguitaramp, sherlockhairdryer, sherlockcharger, sherlocklaptop, sherlockdesk, solderingiron Sherlock's Room Sherlock's desk can be adjusted in height via an electrical motor. printerscanner, vacuum, drill, airmattress Storeroom 9. Known Limitations The dataset contains two manually corrected periods for known unmetered loads. Other smaller unmetered loads may still exist, however unlikely as it was given special attention to not plug devices directly into outlets without an intermediary smart plug. The measurement integrated circuits in the individual meters are inexpensive and do not offer industry level precision. Hence, the individual data might be inaccurate. The meters were nevertheless calibrated using a pure resistive load, however, a linear behavior (especially with changing humidity and temperature) cannot be assumed. The resolution is 1 Hz. High frequency analyses are therefore not doable. The measured quantity is active power only. 10. License This dataset is made available under the Creative Commons Attribution license. 11. How to Cite If you use this dataset in your research, please cite the corresponding paper linked to the Zenodo upload.1. 摘要 本数据集包含德国一处双人公寓的总耗电量与分项耗电量数据。数据采集时间为2025年3月5日至9月4日，时长共计6个月。数据集涵盖主智能电表的总耗电量读数，以及40个智能插头、智能继电器与智能电表监测各类电器的分项耗电量读数。 2. 数据集概览 公寓：德国亚琛市双人公寓，建筑面积约58平方米。 总电表：eBZ DD3 分项电表：31台Shelly Plus Plug S、6台Shelly Plus 1PM、3台Shelly Plus PM Mini Gen3 采样率：1赫兹 测量参数：有功功率 测量单位：瓦特 采集时长：6个月 文件格式：单个CSV（Comma-Separated Value，逗号分隔值）文件，文件名为`DARCK.csv` 数据结构：以时间戳为行索引，包含总电表与各分项电器的耗电量列 数据完整性：主电表数据完整性为99.3%，缺失值采用线性插值填充。 3. 下载与使用 本数据集可通过以下链接下载：https://doi.org/10.5281/zenodo.17159850。由于数据集包含大量零值待机时段，该CSV文件具备极佳的压缩率。解压命令为：`xz -d DARCK.csv.xz`。压缩后文件体积可减少97%（从4GB降至90.9MB）。 若需在Python中使用本数据集，可将CSV文件加载至pandas数据框中，示例代码如下： python import pandas as pd df = pd.read_csv("DARCK.csv", parse_dates=["time"]) 4. 测量设置 主电表的监测方式为：将红外读数头通过磁吸方式固定在电表的红外接口上。刷入Tasmota固件的ESP8266可解码二进制数据报文，并将瓦特读数转发至MQTT代理服务器。各类电器的分项监测采用以下设备组合：用于插座回路的Shelly插头、用于嵌入式灯具等有线设备的Shelly 1PM，以及用于烤箱三相回路的Shelly PM Mini。所有设备通过运行于Dell OptiPlex 3020M主机Docker容器内的Home Assistant，将数据上报至中央InfluxDB数据库。 5. 文件格式（DARCK.csv） 本数据集以单个逗号分隔值文件形式提供。 第一行为表头，包含各列名称。 所有功率值均保留至小数点后一位。 最终数据集中无缺失值。 每行代表1秒的读数，采集时段为3月初至9月底。 列说明 | 列名 | 数据类型 | 单位 | 描述 | |------|----------|------|------| | time | datetime | - | 读数时间戳，格式为YYYY-MM-DD HH:MM:SS | | main | float | 瓦特 | 公寓总耗电量，由主配电箱处的智能电表测量 | | [appliance_name] | float | 瓦特 | 单个电器的耗电量，例如lightbathroom、fridge、sherlockpc等，完整列表详见第8节 | 聚合列 | 列名 | 数据类型 | 单位 | 描述 | |------|----------|------|------| | aggr_chargers | float | 瓦特 | sherlockcharger、sherlocklaptop、watsoncharger、watsonlaptop、watsonipadcharger、kitchencharger的耗电量总和 | | aggr_stoveplates | float | 瓦特 | stoveplatel1与stoveplatel2的耗电量总和 | | aggr_lights | float | 瓦特 | lightbathroom、lighthallway、lightsherlock、lightkitchen、lightlivingroom、lightwatson、lightstoreroom、fcob、sherlockalarmclocklight、sherlockfloorlamphue、sherlockledstrip、livingfloorlamphue、sherlockglobe、watsonfloorlamp、watsondesklamp、watsonledmap的耗电量总和 | 分析列 | 列名 | 数据类型 | 单位 | 描述 | |------|----------|------|------| | inaccuracy | float | 瓦特 | 由于不存在绕过电表的用电设备，因此可通过分项读数总和与主电表读数的绝对误差计算真实误差。由于测量设备自身的耗电量未被计入，因此需为分项总和添加30W的偏移量。 | 6. 数据后处理流程 最终数据集由两个原始数据源（meter.csv与shellies.csv）通过完整的后处理流程生成。 6.1 主电表（main）后处理 总耗电量数据需经过多步清理以确保准确性： - 异常值剔除：移除功率低于10W或高于10000W的读数（仅出现3次）。 - 时间戳突发校正：原始数据存在延迟读数的突发情况，采用自定义算法识别此类突发（即存在较大时间间隙后跟随快速读数），并回填时间戳以生成均匀间隔的时间序列。 - 对齐与插值：智能电表每秒通过红外接口推送新读数，为对齐至整秒，将数据重采样为1秒频率，取每秒内所有读数的平均值（99.5%的情况下每秒仅1个读数）。对由此产生的间隙（停电占比0.7%）采用线性插值填充。 6.2 分项设备（shellies）后处理 Shelly设备不存在ESP8266的突发读数问题：当用电功率变化时，设备会推送新读数；若无功率变化或变化量过小（小于数瓦），则设备每分钟推送一次读数与心跳包。设备开关机时会发布中间功率值，产生亚秒级读数，需进行相应处理： - 分组：按唯一设备标识符分组。 - 重采样与填充：将每个设备的数据重采样为1秒频率，采用`.resample('1s').last().ffill()`方法。该方法首先获取每秒内设备的最后已知状态，以处理快速开关机事件；其次通过前向填充处理无新读数的时段，模拟设备耗电量保持恒定直至下一次读数上传。 6.3 合并与最终处理 - 合并：将清理后的主电表数据与所有分项设备数据框以时间索引为键合并为单个数据框。 - 最终填充：将所有剩余NaN值（如设备安装前的时段）填充为0.0，假设此时设备耗电量为0。 7. 手动校正与已知数据问题 分析过程中发现两起未被计量的负载事件，并进行了手动校正，以提升总读数的准确性。校正后重新计算了误差列（inaccuracy）： - 3月10日：未计量的107W灯泡处于工作状态，将该耗电量从主电表读数中扣除，视为未发生该负载。 - 5月31日：未计量的101W气泵用于充气床垫，直接插在无智能插头的插座上，因此手动将该耗电量添加至对应插头的读数中。 8. 电器详情与多功能插头 下表列出了各列名称及必要说明。由于Watson于6月初搬迁，部分计量插头对应的电器发生了变更。因此将相关列拆分为新列，以反映当时所连接的电器。故数据集中的列数多于实际物理计量设备数量。 | 列名 | 位置/电器 | 说明 | |------|-----------|------| | lightkitchen、lightwatson、lighthallway、lightbathroom、lightsherlock、lightstoreroom、lightlivingroom、fcob、watsonfloorlamp、livingfloorlamphue、watsonledmap、watsondesklamp、sherlockglobe、sherlockledstrip、sherlockfloorlamphue、sherlockalarmclocklight | 杂项/照明 | 以light开头的7个列均为嵌入式天花板灯，fcob为用于厨房台面照明的Flip Chip on Board（COB）LED灯带。 | | stoveplatel1、stoveplatel2、stoveovenl3 | 厨房/炉灶与烤箱 | 厨房炉灶与烤箱的各相回路。炉灶共有4个加热盘：底部2个接相1，顶部2个接相2；烤箱接相3。 | | fridge、washingmachine、microwave、kettle、wheymixer、mixer、blender、kitchensoundsystem | 厨房/其他电器 | - | | router、watsonvinylplayer、watsoncharger、watsonfan、watsonlaptop、watsondesklamp、watsonipadcharger、watsonmonitor、watsonpiano、htpc | Watson房间，后改为客厅 | - | | sherlockpc、sherlocktv、sherlockmonitor、sherlockserver、sherlockguitaramp、sherlockhairdryer、sherlockcharger、sherlocklaptop、sherlockdesk、solderingiron | Sherlock房间 | Sherlock的书桌可通过电动马达调节高度。 | | printerscanner、vacuum、drill、airmattress | 储藏室 | - | 9. 已知局限性 本数据集包含两起已手动校正的未计量负载时段。尽管已尽量避免将设备直接插在无智能插头的插座上，但仍可能存在其他小型未计量负载，不过此类情况概率较低。 各分项电表的测量集成电路为低成本型号，不具备工业级精度，因此分项数据可能存在误差。尽管已采用纯阻性负载对电表进行校准，但无法保证测量行为的线性特性（尤其随湿度与温度变化时）。 本数据集采样分辨率为1Hz，无法进行高频分析。 本次测量仅覆盖有功功率。 10. 许可 本数据集采用知识共享署名许可协议发布。 11. 引用方式 若您在研究中使用本数据集，请引用Zenodo上传页面关联的对应论文。