Community Earth System Model Large Ensemble (CESM LENS)

The Community Earth System Model (CESM) Large Ensemble Numerical Simulation (LENS) dataset includes a 40-member ensemble of climate simulations for the period 1920-2100 using historical data (1920-2005) or assuming the RCP8.5 greenhouse gas concentration scenario (2006-2100), as well as longer control runs based on pre-industrial conditions. The data comprise both surface (2D) and volumetric (3D) variables in the atmosphere, ocean, land, and ice domains. The total data volume of the original dataset is ~500TB, which has traditionally been stored as ~150,000 individual CF/NetCDF files on disk or magnetic tape made available through the NCAR Climate Data Gateway for download or via web services. NCAR has copied a subset (currently ~70 TB) of CESM LENS data to Amazon S3 as part of the AWS Public Datasets Program. To optimize for large-scale analytics we have represented the data as ~275 Zarr stores format accessible through the Python Xarray library. Each Zarr store contains a single physical variable for a given model run type and temporal frequency (monthly, daily, 6-hourly).

社区地球系统模型（CESM）大集合数值模拟（LENS）数据集囊括了40个成员的气候模拟数据，时间跨度为1920年至2100年，其中采用历史数据（1920-2005年）或基于RCP8.5温室气体浓度情景（2006-2100年）进行模拟，同时还包括基于前工业化条件的长期控制运行。数据集包含大气、海洋、陆地和冰域的表面（二维）和体积（三维）变量。原始数据集的总数据量约为500TB，传统上以约150,000个单独的CF/NetCDF文件存储在磁盘或磁带上，并通过NCAR气候数据网关提供下载或通过网络服务获取。作为AWS公共数据集计划的一部分，NCAR已将CESM LENS数据的一个子集（目前约为70TB）复制到Amazon S3。为了优化大规模分析，我们已将数据表示为约275个Zarr存储格式，可通过Python Xarray库访问。每个Zarr存储包含特定模型运行类型和时间频率（月度、日度、六小时度）的单个物理变量。

搜集汇总

背景与挑战

背景概述

Community Earth System Model Large Ensemble (CESM LENS)数据集包含了一个40成员的集合气候模拟，覆盖1920-2100年期间，使用历史数据或RCP8.5温室气体浓度情景。数据集包含大气、海洋、陆地和冰域的表面（2D）和体积（3D）变量，总数据量约为500TB，其中约70TB已复制到Amazon S3，并以Zarr存储格式优化，便于大规模分析。