Big data cloud computing for the Surface Water and Ocean Topography project

A new satellite mission for oceanography and hydrology science called Surface Water and Ocean Topography (SWOT) was developed jointly by the U.S. National Aeronautics and Space Administration and France’s Centre National d’Etudes Spatiales (CNES) and launched on December 16, 2022. Using state-of-the-art "radar interferometry" technology to measure the elevation of water, SWOT will observe major lakes, rivers and wetlands while detecting ocean features with unprecedented resolution. SWOT data will provide critical information that is needed to assess water resources on land, track regional sea level changes, monitor coastal processes, and observe small-scale ocean currents and eddies. The SWOT Science Data System (SDS) is the first operational flight project Big Data System with significant data volumes (7.3 Tbits per day, production volume 21 TBytes per day) and orchestration of concurrent jobs. ‘Scalable’ production capacity is developed on Amazon Web Services (AWS) commercial cloud utilizing up to 1300 ‘undedicated’ Amazon Elastic Compute Cloud (EC2) instances scaling up to 55,000 jobs per day. This paper describes the pre-launch preparation, the mission's current status along with some information on the challenges and experiences in developing and operating a large satellite remote sensing data processing system.