SensWorkflow: a high-performance framework for remote sensing big data processing on heterogeneous clusters

The increasing volume and complexity of remote sensing data have made high performance computing (HPC) essential, with workflows becoming key to enabling efficient and automated processing. However, challenges remain in heterogeneous distributed computing environments. To address these issues, we propose SensWorkflow, a dynamic and visual framework for high-performance remote sensing applications designed to simplify processing workflows and improve computational efficiency. SensWorkflow integrates efficient large-scale data storage and management, a novel blind dating optimization (BDO) task scheduling algorithm newly proposed in this study to improve load balancing and task assignment, model operator management, and visual workflow and task management. It is deployed on a heterogeneous distributed cluster and integrates the Bee grid file system (BeeGFS) for data storage, a not only structured query language (NoSQL) database for metadata management, and the high throughput Condor (HTCondor) framework for distributed task scheduling. To evaluate its performance, the synergetic retrieval of aerosol properties (SRAP) aerosol optical depth (AOD) retrieval algorithm was used as a case study to process 10-day moderate resolution imaging spectroradiometer (MODIS) data in 2022.