ERA-NUTS: meteorological time-series based on C3S ERA5 for European regions (1980-2021)

# ERA-NUTS (1980-2021) This dataset contains a set of time-series of meteorological variables based on Copernicus Climate Change Service (C3S) ERA5 reanalysis. The data files can be downloaded from here while notebooks and other files can be found on the associated Github repository. This data has been generated with the aim of providing hourly time-series of the meteorological variables commonly used for power system modelling and, more in general, studies on energy systems. An example of the analysis that can be performed with ERA-NUTS is shown in this video. Important: this dataset is still a work-in-progress, we will add more analysis and variables in the near-future. If you spot an error or something strange in the data please tell us sending an email or opening an Issue in the associated Github repository. ## Data The time-series have hourly/daily/monthly frequency and are aggregated following the NUTS 2016 classification. NUTS (Nomenclature of Territorial Units for Statistics) is a European Union standard for referencing the subdivisions of countries (member states, candidate countries and EFTA countries). This dataset contains NUTS0/1/2 time-series for the following variables obtained from the ERA5 reanalysis data (in brackets the name of the variable on the Copernicus Data Store and its unit measure): - t2m: 2-meter temperature (`2m_temperature`, Celsius degrees) - ssrd: Surface solar radiation (`surface_solar_radiation_downwards`, Watt per square meter) - ssrdc: Surface solar radiation clear-sky (`surface_solar_radiation_downward_clear_sky`, Watt per square meter) - ro: Runoff (`runoff`, millimeters) - sd: Snow depth (`sd`, meters) There are also a set of derived variables: - ws10: Wind speed at 10 meters (derived by `10m_u_component_of_wind` and `10m_v_component_of_wind`, meters per second) - ws100: Wind speed at 100 meters (derived by `100m_u_component_of_wind` and `100m_v_component_of_wind`, meters per second) - CS: Clear-Sky index (the ratio between the solar radiation and the solar radiation clear-sky) - RH: Relative Humidity (computed following Lawrence, BAMS 2005 and Alduchov & Eskridge, 1996) - HDD/CDD: Heating/Cooling Degree days (derived by 2-meter temperature the EUROSTAT definition. For each variable we have 367 440 hourly samples (from 01-01-1980 00:00:00 to 31-12-2021 23:00:00) for 34/115/309 regions (NUTS 0/1/2). The data is provided in two formats: - NetCDF version 4 (all the variables hourly and CDD/HDD daily). NOTE: the variables are stored as `int16` type using a `scale_factor` to minimise the size of the files. - Comma Separated Value ("single index" format for all the variables and the time frequencies and "stacked" only for daily and monthly) All the CSV files are stored in a zipped file for each variable. ## Methodology The time-series have been generated using the following workflow: 1. The NetCDF files are downloaded from the Copernicus Data Store from the ERA5 hourly data on single levels from 1979 to present dataset 2. The data is read in R with the climate4r packages and aggregated using the function `/get_ts_from_shp` from panas. All the variables are aggregated at the NUTS boundaries using the average except for the runoff, which consists of the sum of all the grid points within the regional/national borders. 3. The derived variables (wind speed, CDD/HDD, clear-sky) are computed and all the CSV files are generated using R 4. The NetCDF are created using `xarray` in Python 3.8. ## Example notebooks In the folder `notebooks` on the associated Github repository there are two Jupyter notebooks which shows how to deal effectively with the NetCDF data in `xarray` and how to visualise them in several ways by using matplotlib or the enlopy package. There are currently two notebooks: - exploring-ERA-NUTS: it shows how to open the NetCDF files (with Dask), how to manipulate and visualise them. - ERA-NUTS-explore-with-widget: explorer interactively the datasets with [jupyter]() and ipywidgets. The notebook `exploring-ERA-NUTS` is also available rendered as HTML. ## Additional files In the folder `additional files`on the associated Github repository there is a map showing the spatial resolution of the ERA5 reanalysis and a CSV file specifying the number of grid points with respect to each NUTS0/1/2 region. ## License This dataset is released under CC-BY-4.0 license. ## Changelog 2022-04-08 Added Relative Humidity (RH) 2022-03-07 Added the missing month in CDD/HDD 2022-02-08 Updated the wind speed and temperature data due to missing months.

# ERA-NUTS数据集（1980-2021）
本数据集基于哥白尼气候变化服务局（Copernicus Climate Change Service, C3S）的ERA5再分析数据，包含一套气象变量时间序列。数据集文件可通过此处下载，配套的Jupyter笔记及其他文件可在关联的GitHub仓库中获取。

本数据集旨在提供常用于电力系统建模，以及更广泛能源系统研究的气象变量逐小时时间序列。本数据集可支持的分析案例可参考对应视频。

重要提示：本数据集仍处于开发完善阶段，我们将在近期新增更多分析内容与气象变量。若您发现数据存在错误或异常情况，可通过发送邮件或在关联GitHub仓库提交Issue的方式告知我们。

## 数据说明
本数据集的时间序列涵盖逐小时、每日、月度三种频率，均按照NUTS（统计区域命名法，Nomenclature of Territorial Units for Statistics）标准进行聚合。NUTS是欧盟用于划分成员国、候选国及欧洲自由贸易联盟（EFTA）国家国内行政区域的标准。

本数据集包含以下基于ERA5再分析数据提取的NUTS 0/1/2级时间序列数据，括号内为哥白尼数据存储（Copernicus Data Store）中的变量名与单位：
- t2m：2米气温（`2m_temperature`，摄氏度）
- ssrd：地表向下太阳短波辐射（`surface_solar_radiation_downwards`，瓦/平方米）
- ssrdc：晴空条件下地表向下太阳短波辐射（`surface_solar_radiation_downward_clear_sky`，瓦/平方米）
- ro：径流量（`runoff`，毫米）
- sd：积雪深度（`sd`，米）

此外，数据集还包含以下衍生变量：
- ws10：10米高度风速（由`10m_u_component_of_wind`与`10m_v_component_of_wind`衍生得到，单位：米/秒）
- ws100：100米高度风速（由`100m_u_component_of_wind`与`100m_v_component_of_wind`衍生得到，单位：米/秒）
- CS：晴空指数（实际向下太阳短波辐射与晴空条件下向下太阳短波辐射的比值）
- RH：相对湿度（计算方法参考Lawrence于2005年发表在《BAMS》的研究，以及Alduchov与Eskridge在1996年的研究成果）
- HDD/CDD：取暖度日数/降温度日数（基于2米气温，按照欧盟统计局（EUROSTAT）的定义衍生得到）

针对每个变量，NUTS 0/1/2级区域分别对应34/115/309个统计单元，逐小时样本量为367440条，时间跨度为1980年1月1日00:00:00至2021年12月31日23:00:00。

数据集提供两种存储格式：
1. NetCDF 4格式：涵盖所有变量的逐小时数据，以及CDD/HDD的逐日数据。注意：为压缩文件体积，变量以`int16`类型存储，并通过缩放因子`scale_factor`进行转换。
2. 逗号分隔值（CSV）格式：所有变量及时间频率的数据均采用「单索引」格式，仅逐日、月度数据采用「堆叠」格式。每个变量的所有CSV文件均打包为压缩包存储。

## 数据生成流程
本数据集的时间序列通过以下流程生成：
1. 从哥白尼数据存储下载ERA5单层次逐小时再分析数据集（1979年至今）的NetCDF文件。
2. 使用R语言的climate4r包读取数据，并通过panas工具包的`/get_ts_from_shp`函数进行聚合。除径流量（需对区域内所有网格点的径流量求和）外，其余变量均通过区域内网格点的平均值进行NUTS边界聚合。
3. 计算衍生变量（风速、度日数、晴空指数），并通过R语言生成所有CSV文件。
4. 使用Python 3.8环境下的`xarray`库生成NetCDF格式文件。

## 示例Jupyter笔记
关联GitHub仓库的`notebooks`文件夹中包含两份Jupyter笔记，分别演示如何通过`xarray`库高效处理NetCDF数据，以及如何使用matplotlib或enlopy包进行多维度可视化。目前包含两份笔记：
- exploring-ERA-NUTS：演示如何通过Dask加载NetCDF文件，以及如何对数据进行处理与可视化。
- ERA-NUTS-explore-with-widget：通过Jupyter与ipywidgets实现数据集的交互式探索。

`exploring-ERA-NUTS`笔记还提供了HTML渲染版本。

## 附加文件
关联GitHub仓库的`additional files`文件夹中包含一份展示ERA5再分析数据空间分辨率的地图，以及一份列明各NUTS 0/1/2级区域对应网格点数量的CSV文件。

## 版权声明
本数据集采用CC-BY-4.0协议发布。

## 更新日志
- 2022-04-08：新增相对湿度（RH）变量
- 2022-03-07：补全CDD/HDD数据中缺失的月份
- 2022-02-08：因部分月份数据缺失，更新了风速与气温数据集