Echodataflow: Recipe-based Fisheries Acoustics Workflow Orchestration Proceedings of the Python in Science Conference

With the influx of large data from multiple instruments and experiments, scientists are wrangling complex data pipelines that are context-dependent and non-reproducible. We demonstrate how we leverage Prefect (Prefect, 2024), a modern orchestration framework, to facilitate fisheries acoustics data processing. We built a Python package Echodataflow (Echodataflow, 2024) which 1) allows users to specify workflows and their parameters through editing text “recipes” which provide transparency and reproducibility of the pipelines; 2) supports scaling of the workflows while abstracting the computational infrastructure; 3) provides monitoring and logging of the workflow progress. Under the hood, Echodataflow uses Prefect to execute the workflows while providing a domain-friendly interface to facilitate diverse fisheries acoustics use cases. We demonstrate the features through a typical ship survey data processing pipeline. Grant no. NA21OAR0110201-T1-01