Domestic Electrical Load Metering, Hourly Data 1994-2014 - South Africa

Abstract
---------------------------

This data is an aggregated subset of the 5-minute interval electricity metering data available in the Domestic Electrical Load Metering Data (DELM) 1994-2014 available in DataFirst's secure centre. The large volume and high metering cadence of the DELM 1994-2014 data is unwieldy to access and process. Many applications that do not require the granularity of the DELM 1994-2014 data will be able to extract value more effectively and conveniently from aggregate values. This dataset contains all current (Amps) observations aggregated to hourly values. It can be easily merged with the Domestic Electrical Load Survey Key Variables 1994-2014 data to link socio-demographic varibles with household consumption data. This dataset and similar custom datasets can be produced from the DELM 1994-2014 dataset with the python package delprocess. The data processing section includes a description of how this dataset was created. The development of the tools to create this dataset was funded by the South African National Energy Development Initiative (SANEDI).

Geographic coverage
---------------------------

The study had national coverage.

Analysis unit
---------------------------

Households

Universe
---------------------------

The metering study covers electrified households that received electricity either directly from Eskom or from their local municipality. Particular attention was devoted to rural and low income households, as well as surveying households electrified over a range of years, thus having had access to electricity from recent times to several decades.

Kind of data
---------------------------

Observation data

Mode of data collection
---------------------------

Other [oth]

Cleaning operations
---------------------------

This data has been produced by aggregating all current (Amps) metering data from the DELMS 1994-2014 dataset using the reduceRawProfiles function from the delprocess python package (https://github.com/wiebket/delprocess: release v1.0). Full instructions on how to use delprocess to aggregate metering data are in the README file contained in the package.

INVALID READINGS
The 'Valid' indicator of readings was converted to 1 (valid) and 0 (invalid). Missing 'Valid' indicators were filled with 0 values.

MISSING VALUES
Missing readings were treated as per pandas.dataframe.mean default: skipna = True; i.e. missing values are excluded when computing results.

DATA AGGREGATION (OBSERVATIONS)
The following processing steps were performed to produce the aggregate dataset:
0. 'Datefield' values were converted to integer values, rounded to 9 positions left of the decimal, and converted to a numpy datetime64 object with nano-second units. This was done to coerce the data to consistent time intervals.
1. readings grouped by RecorderID and ProfileID
2. grouped data resampled to hourly values ('Datefield' column converted to 'H' offset)
3. mean meter reading value and 'Valid' indicator calculated over resampled intervals
4. rows with all missing values removed
5. aggregated 'Valid' indicator set to 0 unless it was 1 (i.e. if one reading was marked as invalid, the mean 'Valid' indicator would be less than 1 and the aggregate 'Valid' indicator was set to 0, thus marking the aggregated validity as invalid)

DATA AGGREGATION (STUDY CYCLES)
Data was aggregated per year, across temporally overlapping study cycles.

摘要
---------------------------
本数据集系对DataFirst安全中心所提供的《1994-2014年国内电气负荷计量数据》（DELM）中5分钟间隔的电能计量数据的汇总子集。由于DELM 1994-2014数据量庞大且计量频率高，其访问和处理较为困难。对于不需求DELM 1994-2014数据粒度的众多应用，从汇总值中提取价值将更加高效和便捷。本数据集包含所有当前（安培）观测值的时聚合至小时值。它可与《1994-2014年国内电气负荷调查关键变量》数据轻松合并，以将社会经济变量与家庭消费数据关联。本数据集及类似定制数据集可通过python包delprocess从DELM 1994-2014数据集生成。数据处理部分包括如何创建本数据集的描述。创建本数据集的工具开发由南非国家能源发展倡议（SANEDI）资助。
地理覆盖范围
---------------------------
研究具有全国性覆盖。
分析单元
---------------------------
家庭
总体
---------------------------
计量研究覆盖了直接从埃斯康姆或其地方市政机构获得电力的通电家庭。特别关注农村和低收入家庭，以及调查了在多年间通电的家庭，从而从近期至数十年均有电力接入。
数据类型
---------------------------
观测数据
数据收集方式
---------------------------
其他[oth]
清洗操作
---------------------------
本数据是通过使用delprocess python包中的reduceRawProfiles函数（https://github.com/wiebket/delprocess: release v1.0）对DELM 1994-2014数据集中的所有当前（安培）计量数据进行汇总而产生的。关于如何使用delprocess汇总计量数据的完整说明包含在包中的README文件中。
无效读数
将读数的'Valid'指示符转换为1（有效）和0（无效）。缺失的'Valid'指示符用0值填充。
缺失值
缺失读数按pandas.dataframe.mean默认方式处理：skipna = True；即计算结果时排除缺失值。
数据汇总（观测值）
为生成汇总数据集，执行了以下处理步骤：
0. 将'Datefield'值转换为整数，四舍五入至小数点左侧9位，并转换为具有纳秒单位的numpy datetime64对象。这样做是为了强制数据到一致的时间间隔。
1. 按RecorderID和ProfileID对读数进行分组
2. 将分组数据重采样到小时值（'Datefield'列转换为'H'偏移量）
3. 在重采样间隔内计算平均计量读数值和'Valid'指示符
4. 移除所有缺失值的行
5. 将汇总的'Valid'指示符设置为0，除非其为1（即如果有一个读数被标记为无效，平均'Valid'指示符将小于1，汇总'Valid'指示符将被设置为0，从而标记汇总的有效性为无效）
数据汇总（研究周期）
按年度、跨时间重叠的研究周期进行数据汇总。