CaLES: A GPU-accelerated solver for large-eddy simulation of wall-bounded flows

We introduce CaLES, a GPU-accelerated finite-difference solver designed for large-eddy simulations (LES) of incompressible wall-bounded flows in massively parallel environments. Built upon the existing direct numerical simulation (DNS) solver CaNS, CaLES relies on low-storage, third-order Runge-Kutta schemes for temporal discretization, with the option to treat viscous terms via an implicit Crank-Nicolson scheme in one or three directions. A fast direct solver, based on eigenfunction expansions, is used to solve the discretized Poisson/Helmholtz equations. For turbulence modeling, the classical Smagorinsky model with van Driest near-wall damping and the dynamic Smagorinsky model are implemented, along with a logarithmic law wall model. GPU acceleration is achieved through OpenACC directives, following CaNS-2.3.0. Performance assessments were conducted on the Leonardo cluster at CINECA, Italy. Each node is equipped with one Intel Xeon Platinum 8358 CPU (2.60 GHz, 32 cores) and four NVIDIA A100 GPUs (64 GB HBM2e), interconnected via NVLink 3.0 (200 GB/s). The inter-node communication bandwidth is 25 GB/s, supported by a DragonFly+ network architecture with NVIDIA Mellanox InfiniBand HDR. Results indicate that the computational speed on a single GPU is equivalent to approximately 15 CPU nodes, depending on the treatment of viscous terms and the subgrid-scale model, and that the solver efficiently scales across multiple GPUs. The predictive capability of CaLES has been tested using multiple flow cases, including decaying isotropic turbulence, turbulent channel flow, and turbulent duct flow. The high computational efficiency of the solver enables grid convergence studies on extremely fine grids, pinpointing non-monotonic grid convergence for wall-modeled LES.我们提出CaLES：一款面向大规模并行环境下不可压缩壁面流动大涡模拟（Large-Eddy Simulation, LES）的GPU加速有限差分求解器。该求解器基于现有直接数值模拟（Direct Numerical Simulation, DNS）求解器CaNS开发，采用低存储三阶龙格-库塔格式进行时间离散化，支持在单方向或三方向上通过隐式克兰克-尼科尔森（Crank-Nicolson）格式处理粘性项。求解离散后的泊松/亥姆霍兹方程时，采用基于本征函数展开的快速直接求解器。在湍流建模方面，已实现带范德里斯特（van Driest）近壁阻尼的经典Smagorinsky模型、动态Smagorinsky模型，以及对数律壁面模型。GPU加速通过OpenACC指令实现，开发版本基于CaNS-2.3.0。性能评估在意大利CINECA的Leonardo超算集群上开展，每个计算节点配备1颗英特尔至强Platinum 8358 CPU（2.60 GHz，32核）与4块NVIDIA A100 GPU（64 GB HBM2e），节点内通过NVLink 3.0（200 GB/s）实现互联。节点间通信带宽为25 GB/s，由搭载NVIDIA Mellanox InfiniBand HDR的DragonFly+网络架构提供支持。测试结果表明，单GPU的计算速度约等效于15个CPU节点，具体性能取决于粘性项处理方式与亚格子尺度模型；同时该求解器在多GPU环境下可实现高效的并行缩放。CaLES的预测能力已通过多个流动算例验证，包括衰减各向同性湍流、湍流槽道流与湍流管道流。该求解器的高计算效率支持在极精细网格上开展网格收敛性研究，明确了壁面建模大涡模拟的非单调网格收敛特性。