Benchmarking dynamic analysis methods in diffusion-based single-molecule FRET - Data Repository

Code and data accompanying the publication: [Title and DOI to be added] This repository generates simulated diffusion-based single-molecule FRET (smFRET) datasets, using Brownian motion simulations to accurately reflect translational diffusion through the confocal volume, and analyses them using established methods to benchmark their ability to detect conformational dynamics. It is designed to run on HPC clusters using SLURM, but can be adapted to other job schedulers by modifying the job templates in the generator scripts. The generated Python scripts can also be run directly without a scheduler for smaller datasets or local use. Due to the storage requirements of the full dataset (~1.5 TB), this repository contains a representative subset. Specifically: Diffusion trajectory (stage 00 output): The Brownian motion trajectory file (`pybromo_*.hdf5`, ~271 GB) is not included due to its size. It can be regenerated deterministically using `00_generate_diffusion_jobs.py` with the fixed random seed defined in `config.py`. Photon arrival times (`data/replicate_1/`): The photon arrival times HDF5 file is included (~537 MB). This is the output of stage 01 and input to stage 02. Timestamp files (`data/replicate_1/`): All 21 photon timestamp HDF5 files are included (one per FRET efficiency, ~22 MB each, ~480 MB total). These are the output of stage 02 and input to stage 03. Simulation files (`data/replicate_1/`): *10 of the 20 FRET state pairs are included, spanning the full range from broad to narrow ΔE. For each FRET pair, approximately every 3rd interconversion rate (kobs) is included (14 of 42 values), covering the full timescale range from 0.4 to 2.5×10^5 s-1. All 9 equilibrium constants (K = 0.05-20) are included for each combination. This gives 1,260 HDF5 files compared to 7,560 in the full single-replicate dataset. Analysis results (`data/results/R1`-`R3`): The full set of analysis CSV files is included for all 3 replicates, all 20 FRET pairs, all 42 kobs values, and all 9 equilibrium constants. These are the data used to produce the figures in the publication. Overview The workflow has four stages, each with a script that generates SLURM job files: Script Stage 00_generate_diffusion_jobs.py Brownian motion trajectories 01_generate_timestamp_jobs.py Photon timestamps from trajectories 02_generate_simulation_jobs.py FRET state mixing 03_generate_analysis_jobs.py Burst search and analysis All simulation and analysis parameters are defined in `config.py`. To adapt the workflow, edit this file before generating jobs. Setup Create the conda environment: conda env create -f environment.yml conda activate smfret-benchmarking Approximate runtimes Based on the default configuration (3 replicates, 20 FRET state pairs, 42 interconversion rates, 9 equilibrium constants): Stage Jobs Avg per job Total (sequential) Diffusion 3 ~1 hr ~3 hrs Timestamps 3 ~26 hrs ~78 hrs Simulation 60 ~42 hrs ~2,500 hrs Analysis 2,520 ~20 mins ~840 hrs Total 2,586 ~3,421 hrs Note that each job contains nested loops over equilibrium constants, burst numbers, and analysis methods, so conventional operations such as burst search, burst selection and analysis are each run many times per job. In total, the default configuration corresponds to 272,160 individual analysis instances (3 replicates × 20 FRET pairs × 42 kobs × 9 K values × 12 burst number thresholds). Stages 02–03 are independent per file, so wall time scales with the number of concurrent jobs available on your cluster. Storage requirements With the default configuration (3 replicates, 20 FRET state pairs, 42 interconversion rates, 9 equilibrium constants): Stage Files Size per replicate Total Diffusion trajectories 3 ~270 GB ~810 GB smFRET simulations 22,680 ~230 GB ~690 GB Analysis results (CSV) 2,520 ~1 GB ~3 GB Total ~1.5 TB Citation If you use this code or data, please cite: [Title and DOI to be added] Please also ensure you cite the underlying tools and methods used, such as PyBroMo, FRETBursts, BVA, FRET-2CDE, and burstH2MM.