A GPU-accelerated package for simulation of flow in nanoporous source rocks with many-body dissipative particle dynamics

Mesoscopic simulations of hydrocarbon flow in source shales are challenging, in part due to the heterogeneous shale pores with sizes ranging from a few nanometers to a few micrometers. Additionally, the sub-continuum fluid-fluid and fluid-solid interactions in nano- to micro-scale shale pores, which are physically and chemically sophisticated, must be captured. To address those challenges, we present a GPU-accelerated package for simulation of flow in nano- to micro-pore networks with a many-body dissipative particle dynamics (mDPD) mesoscale model. Based on a fully distributed parallel paradigm, the code offloads all intensive workloads on GPUs. Other advancements, such as smart particle packing and no-slip boundary condition in complex pore geometries, are also implemented for the construction and the simulation of the realistic shale pores from 3D nanometer-resolution stack images. Our code is validated for accuracy and compared against the CPU counterpart for speedup. In our benchmark tests, the code delivers nearly perfect strong scaling and weak scaling (with up to 512 million particles) on up to 512 K20X GPUs on Oak Ridge National Laboratory’s (ORNL) Titan supercomputer. Moreover, a single-GPU benchmark on ORNL’s SummitDev and IBM’s AC922 suggests that the host-to-device NVLink can boost performance over PCIe by a remarkable 40%. Lastly, we demonstrate, through a flow simulation in realistic shale pores, that the CPU counterpart requires 840 Power9 cores to rival the performance delivered by our package with four V100 GPUs on ORNL’s Summit architecture. This simulation package enables quick-turnaround and high-throughput mesoscopic numerical simulations for investigating complex flow phenomena in nano- to micro-porous rocks with realistic pore geometries.

烃类在烃源页岩中的介观模拟极具挑战性，部分原因在于页岩孔隙具有非均质性，其尺寸跨度从数纳米至数微米不等。此外，纳米至微米级页岩孔隙内的亚连续流体-流体与流体-固体相互作用具有复杂的物理与化学特性，必须对其进行准确捕捉。为应对上述挑战，我们开发了一款基于多体耗散粒子动力学（many-body dissipative particle dynamics, mDPD）介观模型的GPU加速程序包，用于模拟纳米至微米级孔隙网络中的流体流动。该程序基于全分布式并行范式，将所有计算密集型工作负载均卸载至GPU上执行。此外，程序还集成了多项优化技术，例如智能粒子填充算法以及复杂孔隙几何下的无滑移边界条件，可基于3D纳米分辨率堆叠图像构建并模拟真实页岩孔隙结构。我们对该程序的准确性进行了验证，并与CPU版本程序对比了加速比。在橡树岭国家实验室（Oak Ridge National Laboratory, ORNL）的Titan超级计算机上，使用最多512块K20X GPU开展的基准测试中，该程序实现了近乎完美的强可扩展性与弱可扩展性（粒子数量最高可达5.12亿）。此外，在ORNL的SummitDev与IBM AC922平台上开展的单GPU基准测试显示，主机与设备间采用NVLink互连可使性能相较PCIe提升高达40%，效果显著。最后，通过真实页岩孔隙内的流动模拟实验，我们证实：在ORNL的Summit架构上，若要达到本程序包搭配4块V100 GPU时的性能，CPU版本程序需要动用840颗Power9核心。本模拟程序包可实现快速周转与高吞吐量的介观数值模拟，用于研究具有真实孔隙结构的纳米至微米级多孔岩石内的复杂流动现象。