50 Years of IWT Freight Data in the Rhine-Alpine Corridor

Scripts to create a collection of open freight transport data sets from EuroStats, Destatis, CBS StatLine and Rijkswaterstaat. See figure below for an overview. We deliver two IWT data sets: one file covers the total annual IWT performance (billion tonkilometers) per EU country from 1970 and the other includes annual IWT volumes (thousand tonnes) per EU country and good type. For IWT volumes to, from and via The Netherlands, the origin and destination information on NUTS-2 level is also included from 1988 onwards. The IWT performance data set is named ‘eu iwt time series tonkm.csv’. The IWT volumes per good type is named ‘eu iwt time series goodtypes.csv’. Both files can be found on the mainpage. The input data sources are also in this repository under GitHub\Freight-Transport-Data\data\sources. The folder structure under 'sources' follows the headings of the figure: CBS, CBS Archive, Destatis and EuroStat. The mappings between the several classifications can be found under 'mappings'. The scripts are all Jupyter Notebook files (.ipynb) and are stored under GitHub\Freight-Transport-Data\src. Although we attempted to automize the processing as much as possible, some task are conducted manually. The code to retrieve IWT volume data from image files (.jpeg) need to be adjusted manually by the user for every year between 1970-1982 ('retrieve_table_from_multiple_images.ipynb'). The provided code is just an example to retrieve IWT volumes from the image file regarding one year. Since the structure of the images is not consistent over all years and the automatic retrieval is not 100% accurate, manual processing is partly needed in this step. The same applies for the generation of the time series for The Netherlands between 1970-1987, using the processed image files as input ('generate_time_series_nl.ipynb'). From 1982-1987, goods types on domestic transport were published on 4 NST-R digits, before 1982 on 3 NST-R digits. Goods types on international transport were published on 2 digits. Therefore, there are two different piece of codes to prepare for the mapping from NST-R to NST2007. The user can simply uncomment the piece, referring to different period The goods type imputation also requires some manual adjusting in the code for France, Poland, Romania, Bulgaria and Croatia ('iwt_eu_goodtype_historical_imputed.ipynb'). The user has to enter the country manually, for which the imputation should be executed. Since the missing values for the goods types vary between the countries, the imputation period need also to be entered manually in the code. The imputation periods per country can be derived from Figure 2 in the paper. Some steps refer to figures in our data paper, so the reader is facilitated to reproduce these figures if needed.本脚本用于构建源自欧洲统计局（EuroStats）、德国联邦统计局（Destatis）、荷兰中央统计局统计数据库（CBS StatLine）及荷兰国家公共工程与水管理局（Rijkswaterstaat）的开放货运数据集集合。详见下图概览。 本项目提供两类内河货运（Inland Waterway Transport, IWT）数据集：其一涵盖1970年起欧盟各国年度内河货运总周转量（单位：十亿吨公里）；其二包含欧盟各国按货物品类划分的年度内河货运量（单位：千吨）。针对进出荷兰或途经荷兰的内河货运量数据，自1988年起还补充了NUTS-2级别的起讫地信息。 内河货运总周转量数据集命名为`eu_iwt_time_series_tonkm.csv`，按货物品类划分的货运量数据集命名为`eu_iwt_time_series_goodtypes.csv`，两个文件均可在项目主页获取。原始输入数据源同样存储于本GitHub仓库的`GitHubFreight-Transport-Datadatasources`路径下。`sources`文件夹的目录结构遵循下图的分类标题：CBS、CBS Archive、Destatis及EuroStat。各类分类体系间的映射关系可在`mappings`文件夹中找到。所有脚本均为Jupyter Notebook文件（.ipynb），存储于`GitHubFreight-Transport-Datasrc`路径下。 尽管本项目尽可能实现了处理流程的自动化，但仍有部分任务需手动完成。针对1970年至1982年间的年度数据，用户需手动调整从图像文件（.jpeg）中提取内河货运量数据的代码（`retrieve_table_from_multiple_images.ipynb`）。本项目提供的代码仅为单一年度图像数据提取的示例。由于不同年份的图像格式并不统一，且自动提取的准确率无法达到100%，因此该步骤需部分手动处理。 1970年至1987年荷兰内河货运时间序列的生成步骤同理，需以处理后的图像文件作为输入（`generate_time_series_nl.ipynb`）。1982年至1987年的国内货运货物品类数据采用4位NST-R编码，1982年之前则采用3位NST-R编码；国际货运货物品类数据均采用2位NST-R编码。因此需使用两段不同的代码来完成NST-R到NST2007的映射准备工作，用户只需根据对应时期取消对应代码段的注释即可。 针对法国、波兰、罗马尼亚、保加利亚及克罗地亚的货物品类缺失值插补工作，也需要在代码中进行手动调整（`iwt_eu_goodtype_historical_imputed.ipynb`）。用户需手动指定需要执行插补的国家，同时由于各国货物品类的缺失值情况各不相同，还需手动在代码中输入插补的时期。各国对应的插补时期可参考论文中的图2。 部分步骤引用了本项目数据论文中的图表，以便读者在需要时能够复现相关图表。