Data archive for paper "Machine Learning Emulation of 3D Cloud Radiative Effects"

<strong>Overview</strong> This is the data archive for the paper "Machine Learning Emulation of 3D Cloud Radiative Effects". It contains the paper’s data archive with all model outputs (see the <code>results</code> folder) as well as theSingularity image. For the development repository (i.e. only containing source files without generated artefacts), please see https://github.com/dmey/ml-3d-cre. For the Python tool to generate the synthetic data, please refer to the Synthia repository. <strong>Requirements</strong> Linux Singularity &gt;= 3 Portable Batch System (PBS) job scheduler* *Although PBS in not a strict requirement, it is required to run all helper scripts as included in this repository. Please note that depending on your specific system settings and resource availability, you may need to modify PBS parameters at the top of submit scripts stored in the <code>hpc</code> directory (e.g. <code>#PBS -lwalltime=24:00:00</code>). <strong>Initialization</strong> Deflate the data archive with: <pre><code>./init.sh </code></pre> Build the Singularity image with: <pre><code>singularity build --remote tools/singularity/image.sif tools/singularity/image.def </code></pre> Compile ecRad with Singularity: <pre><code>./tools/singularity/compile_ecrad.sh </code></pre> <strong>Usage</strong> To reproduce the results as described in the paper, run the following commands from the <code>hpc</code> folder: <pre><code>qsub -v JOB_NAME=mlp_synthia ./submit_grid_search_synthia.sh qsub -v JOB_NAME=mlp_default ./submit_grid_search_default.sh qsub submit_benchmark.sh </code></pre> then, to plot stats and identify notebooks run: <pre><code>qsub submit_stats.sh </code></pre> <strong>Local development</strong> For local development, notebooks can be run independently. To install the required dependencies, run the following through Anaconda <code>conda env create -f environment.yml</code>. Then, to activate the environment use <code>conda activate radiation</code>. For ecRad, the list of system dependencies are listed in <code>tools\singularity\image.def</code> and can be run with <code>tools\singularity\compile_ecrad.sh</code>. Licence Paper code released under the MIT license. Data released under CC BY 4.0. ecRad released under the Apache 2.0 license.