Domestic Electrical Load Metering, Hourly Data 1994-2014 - South Africa

Abstract
---------------------------

This data is an aggregated subset of the 5-minute interval electricity metering data available in the Domestic Electrical Load Metering Data (DELM) 1994-2014 available in DataFirst's secure centre. The large volume and high metering cadence of the DELM 1994-2014 data is unwieldy to access and process. Many applications that do not require the granularity of the DELM 1994-2014 data will be able to extract value more effectively and conveniently from aggregate values. This dataset contains all current (Amps) observations aggregated to hourly values. It can be easily merged with the Domestic Electrical Load Survey Key Variables 1994-2014 data to link socio-demographic varibles with household consumption data. This dataset and similar custom datasets can be produced from the DELM 1994-2014 dataset with the python package delprocess. The data processing section includes a description of how this dataset was created. The development of the tools to create this dataset was funded by the South African National Energy Development Initiative (SANEDI).

Geographic coverage
---------------------------

The study had national coverage.

Analysis unit
---------------------------

Households

Universe
---------------------------

The metering study covers electrified households that received electricity either directly from Eskom or from their local municipality. Particular attention was devoted to rural and low income households, as well as surveying households electrified over a range of years, thus having had access to electricity from recent times to several decades.

Kind of data
---------------------------

Observation data

Mode of data collection
---------------------------

Other [oth]

Cleaning operations
---------------------------

This data has been produced by aggregating all current (Amps) metering data from the DELMS 1994-2014 dataset using the reduceRawProfiles function from the delprocess python package (https://github.com/wiebket/delprocess: release v1.0). Full instructions on how to use delprocess to aggregate metering data are in the README file contained in the package.

INVALID READINGS
The 'Valid' indicator of readings was converted to 1 (valid) and 0 (invalid). Missing 'Valid' indicators were filled with 0 values.

MISSING VALUES
Missing readings were treated as per pandas.dataframe.mean default: skipna = True; i.e. missing values are excluded when computing results.

DATA AGGREGATION (OBSERVATIONS)
The following processing steps were performed to produce the aggregate dataset:
0. 'Datefield' values were converted to integer values, rounded to 9 positions left of the decimal, and converted to a numpy datetime64 object with nano-second units. This was done to coerce the data to consistent time intervals.
1. readings grouped by RecorderID and ProfileID
2. grouped data resampled to hourly values ('Datefield' column converted to 'H' offset)
3. mean meter reading value and 'Valid' indicator calculated over resampled intervals
4. rows with all missing values removed
5. aggregated 'Valid' indicator set to 0 unless it was 1 (i.e. if one reading was marked as invalid, the mean 'Valid' indicator would be less than 1 and the aggregate 'Valid' indicator was set to 0, thus marking the aggregated validity as invalid)

DATA AGGREGATION (STUDY CYCLES)
Data was aggregated per year, across temporally overlapping study cycles.

摘要
---------------------------
本数据集为DataFirst安全中心提供的1994-2014年国内电力负荷计量数据（DELM）五分钟间隔计量数据的汇总子集。DELM 1994-2014数据集的体积庞大且计量频率高，难以访问和处理。许多不需要DELM 1994-2014数据粒度的应用，能够更有效地从汇总值中提取价值。本数据集包含所有当前的（安培）观测值，汇总为小时值。它可以轻松与1994-2014年的国内电力负荷调查关键变量数据合并，以将社会人口变量与家庭消费数据联系起来。本数据集及类似的自定义数据集可通过python包delprocess从DELM 1994-2014数据集生成。数据处理部分包括如何创建本数据集的描述。创建本数据集的工具开发得到了南非国家能源发展倡议（SANEDI）的资助。
地理覆盖范围
---------------------------
研究覆盖全国。
分析单元
---------------------------
家庭
总体
---------------------------
计量研究涵盖直接从Eskom或其地方市政当局获得电力的通电家庭。特别关注农村和低收入家庭，以及那些在多年内通电的家庭，从而能够在最近几十年内获得电力。
数据类型
---------------------------
观测数据
数据收集方式
---------------------------
其他[oth]
数据清洗操作
---------------------------
本数据通过使用delprocess python包中的reduceRawProfiles函数（https://github.com/wiebket/delprocess: 版本 v1.0）将DELMMS 1994-2014数据集中的所有电流（安培）计量数据进行汇总生成。如何使用delprocess汇总计量数据的完整说明包含在包内的README文件中。
无效读数
将'Valid'读数指示符转换为1（有效）和0（无效）。缺失的'Valid'指示符用0值填充。
缺失值
缺失读数按照pandas.dataframe.mean默认值处理：skipna = True；即计算结果时排除缺失值。
数据汇总（观测值）
为生成汇总数据集，执行了以下处理步骤：
0. 将'Datefield'值转换为整数，四舍五入到小数点后9位，并转换为具有纳秒单位的numpy datetime64对象。这样做是为了强制数据到一致的时间间隔。
1. 按RecorderID和ProfileID分组读取
2. 将分组数据重采样为小时值（'Datefield'列转换为'H'偏移量）
3. 在重采样间隔内计算平均计量读数值和'Valid'指示符
4. 删除所有缺失值的行
5. 将汇总的'Valid'指示符设置为0，除非它是1（即如果有一个读数被标记为无效，平均'Valid'指示符将小于1，汇总的'Valid'指示符将被设置为0，从而将汇总的有效性标记为无效）
数据汇总（研究周期）
按年度和时间重叠的研究周期汇总数据。