A Distributed Data Processing Scheme Based on Hadoop for Synchrotron Radiation Experiments

This research presents a case study on synchrotron radiation biomolecular crystallography to illustrate a beamline distributed data processing scheme based on the Hadoop ecosystem.We build a distributed file storage system for experimental crystallography data based on Hadoop HDFS. Additionally, we develop a resource scheduling system for the cluster using Hadoop YARN. Furthermore, we design and develop a distributed automated data processing pipeline(Spark-DIALS) for crystallography by combining Hadoop Spark and DIALS. The dials_spot_finder.py and dials_integrate_run.py contain the source code transforming the spots finding and integrate of original DIALS.Moreover, the solution utilizes FastAPI to deploy each functional module in a distributed microservice architecture.There are primarily microservices related to Spark distributed automatic processing jobs and HBase data table operations(sparkJobApi.py&hbaseApi.py).