Data for "psit 1.0: a system to compress Lagrangian flows"

In the paper the following datasets are used for compression: tra_20200101_00 tra_20200101_00_permuted tra_20000101_00 tra_20000101_00_permuted tra_20000101_00_noise_01 tra_20000101_00_noise_05 tra_20000101_00_noise_1 fukushima In the following sections we are explaining how each of them can be obtained. tra_20200101_00 As this dataset is quite small it can directly be obtained from `data/tra_20200101_00.4`. tra_20200101_00_permuted This dataset is based tra_20200101_00 and presents a permuted version of it. It can be created using the `scripts/reorder_data.py` with the following command: python reorder_data.py path/to/tra_20200101_00.4 tra_20200101_00_permuted.4 tra_20000101_00 This dataset needs to be manually created and the steps are as follows: Download corresponding ERA5 data from weatherbench2 [1] Use the Lagranto caltra tool calculate the trajectories Clean up the data to remove outliers. Downloading the files In order to download the files from weatherbench2 [1] the script `scripts/download.py` can be used, as arguments it expects the start and end date. For this file we would use: python download.py 2000-01-01T00:00:00 2000-01-08T01:00:00 This will create a file for each hour called `P<date>_<hour>`. Using Lagranto For the calculations we are going to use Lagranto [2], in order to obtain the latest version please contact the authors. For this we have to create a directory in which we will have all the P<date>_<hour> files, the startf_<level>.2 files and a tracevars file. The startf_<level>.2 files determine the starting positions of the trajectories (they are located in the startf folder) and the tracevars file (located in the tracevars folder) determines variable names which should be traced along the trajectory paths. In order to speedup the entire process we are going to perform all the calculations in parallel. The first parallelization dimension is over the 37 pressure levels that we are interested in, we control this by providing individual starting position files startf_<level>.2. The second parallelization then happens by splitting each of those starting files again. Before we can start we first must compile the scripts/split_input.cpp file using the provided scripts/Makefile Makefile. Also in the scripts/run_caltra.sh we need to set the variable CALTRA to the installation location of caltra. At this point we should have a directory with the following content: . ├── merge_traj.py ├── P20000101_00 ├── P200001{...} ├── P20000108_01 ├── remove_invalid.py ├── run_caltra_2d.sh ├── split_input ├── start_caltra.sh ├── startf_01.2 ├── startf_{...}.2 ├── startf_37.2 └── tracevars After this is done we can start the process by executing the scripts/start_caltra.sh script, that will loop over all the pressure levels and then call scripts/run_caltra.sh with the corresponding startf_<level>.2 file along with additional information. The scripts/run_caltra.sh script takes as input the starting position file, the name of the output file, the reference date (date of first input file), and the number of hours it should simulate for, it then splits the starting position file into multiple files (2 by default, controlled by the num_workers variable) and converts them after which it will merge them together. After this is run one should have 37 new files called traj_<level>.4 which will be the trajectory files for each one of the pressure levels. Next we have to merge them together using the scripts/merge_traj.py script, this can be done with: rm -f traj_list.txt for i in {01..37}; do echo traj_${i}.4 >> traj_list.txt; done python merge_traj.py traj_list.txt tra_20000101_00_raw.4 Where we first create a file called traj_list.txt which contains the names of all the individual trajectory files which is then used by the merge_traj.py script to create the final tra_20000101_00_raw trajectory file. Cleanup In a last step we have to remove invalid trajectories from the file. For this we use the scripts/remove_invalid.py script with: python remove_invalid.py path/to/tra_20000101_00_raw.4 tra_20000101_00.4 After this we are done and can remove all the temporary files. tra_20000101_00_permuted This dataset is based on tra_20000101_00 and presents a permuted version of it. It can be created using the scripts/reorder_data.py with the following command: python reorder_data.py path/to/tra_20000101_00.4 tra_20000101_00_permuted.4 tra_20000101_00_noise_01 This dataset is similar to the tra_2000101_00 one, but the wind fields have been perturbed. It therefore follows the same steps as the creation of tra_20000101_00, but after we have downloaded the data and before we run Lagranto we use the scripts/add_noise_to_wind.py in order to randomly perturbe the wind fields. The first argument to the script is the file which we want to perturbe and the second argument is the magnitude, which in this case is 0.01, it will create a file called <input_file_name>_ which has the perturbations. We therefore run: python add_noise_to_wind.py <input_file_name> 0.01 This will create a file called <input_file_name>_. Note that if one want to use this using Lagranto the tailing underscore needs to be removed. After this we follow the same procedure as for the tra_20000101_00 file. Instead of having to do this manually for each one of the P<date>_<hour> file we can also use the scripts/add_noise.sh script. This script will in parallel add noise to all the P<date>_<hour> files in a directory. In order to use it one must manually set the location of the add_noise_to_wind.py script and the noise magnitude in the file (check comments). By default 10 workers are used, but this can be adjusted via the max_global variable. tra_20000101_00_noise_05 This dataset is very similar to the tra_20000101_00_noise_01 file, but the magnitude of the perturbations is larger, namely 0.05. Therefore we follow the same steps as there but execute the add_noise_to_wind.py with: python add_noise_to_wind.py <input_file_name> 0.05 tra_20000101_00_noise_1 This dataset is very similar to the tra_20000101_00_noise_01 file, but the magnitude of the perturbations is larger, namely 0.1. Therefore we follow the same steps as there but execute the add_noise_to_wind.py with: python add_noise_to_wind.py <input_file_name> 0.1 fukushima This file was used for the Fukushima case study. Its creation follows the same principles we used for the tra_20000101_00 file, what needs to be changed is the time range of data we download (2011-03-12T00:00 to P2011-03-20T00:00), the starting position file (provided by startf/startf_fukushima.2), and that we do not need the tracevars file. Additionally we cannot use the scripts/start_caltra.sh script, but have to call scripts/run_caltra.sh manually and provide it with the correct information. After the directory has been setup correctly it can be called using: ./run_caltra_2d.sh startf_fukushima.2 fukushima.4 20110312_00 193 References [1] Rasp, S., Hoyer, S., Merose, A., Langmore, I., Battaglia, P., Russel, T., … Sha, F. (2024). WeatherBench 2: A benchmark for the next generation of data-driven global weather models. arXiv [Physics.Ao-Ph]. Retrieved from http://arxiv.org/abs/2308.15560.\ [2] Sprenger, M. and Wernli, H.: The LAGRANTO Lagrangian analysis tool – version 2.0, Geosci. Model Dev., 8, 2569–2586, https://doi.org/10.5194/gmd-8-2569-2015, 2015.