Benchmarking dynamic analysis methods in diffusion-based single-molecule FRET - Data Repository

Code and data accompanying the publication: [Title and DOI to be added] This repository generates simulated diffusion-based single-molecule FRET (smFRET) datasets, using Brownian motion simulations to accurately reflect translational diffusion through the confocal volume, and analyses them using established methods to benchmark their ability to detect conformational dynamics. It is designed to run on HPC clusters using SLURM, but can be adapted to other job schedulers by modifying the job templates in the generator scripts. The generated Python scripts can also be run directly without a scheduler for smaller datasets or local use. Due to the storage requirements of the full dataset (~1.5 TB), this repository contains a representative subset. Specifically: Diffusion trajectory (stage 00 output): The Brownian motion trajectory file (`pybromo_*.hdf5`, ~271 GB) is not included due to its size. It can be regenerated deterministically using `00_generate_diffusion_jobs.py` with the fixed random seed defined in `config.py`. Photon arrival times (`data/replicate_1/`): The photon arrival times HDF5 file is included (~537 MB). This is the output of stage 01 and input to stage 02. Timestamp files (`data/replicate_1/`): All 21 photon timestamp HDF5 files are included (one per FRET efficiency, ~22 MB each, ~480 MB total). These are the output of stage 02 and input to stage 03. Simulation files (`data/replicate_1/`): *10 of the 20 FRET state pairs are included, spanning the full range from broad to narrow ΔE. For each FRET pair, approximately every 3rd interconversion rate (kobs) is included (14 of 42 values), covering the full timescale range from 0.4 to 2.5×10^5 s-1. All 9 equilibrium constants (K = 0.05-20) are included for each combination. This gives 1,260 HDF5 files compared to 7,560 in the full single-replicate dataset. Analysis results (`data/results/R1`-`R3`): The full set of analysis CSV files is included for all 3 replicates, all 20 FRET pairs, all 42 kobs values, and all 9 equilibrium constants. These are the data used to produce the figures in the publication. Overview The workflow has four stages, each with a script that generates SLURM job files: Script Stage 00_generate_diffusion_jobs.py Brownian motion trajectories 01_generate_timestamp_jobs.py Photon timestamps from trajectories 02_generate_simulation_jobs.py FRET state mixing 03_generate_analysis_jobs.py Burst search and analysis All simulation and analysis parameters are defined in `config.py`. To adapt the workflow, edit this file before generating jobs. Setup Create the conda environment: conda env create -f environment.yml conda activate smfret-benchmarking Approximate runtimes Based on the default configuration (3 replicates, 20 FRET state pairs, 42 interconversion rates, 9 equilibrium constants): Stage Jobs Avg per job Total (sequential) Diffusion 3 ~1 hr ~3 hrs Timestamps 3 ~26 hrs ~78 hrs Simulation 60 ~42 hrs ~2,500 hrs Analysis 2,520 ~20 mins ~840 hrs Total 2,586 ~3,421 hrs Note that each job contains nested loops over equilibrium constants, burst numbers, and analysis methods, so conventional operations such as burst search, burst selection and analysis are each run many times per job. In total, the default configuration corresponds to 272,160 individual analysis instances (3 replicates × 20 FRET pairs × 42 kobs × 9 K values × 12 burst number thresholds). Stages 02–03 are independent per file, so wall time scales with the number of concurrent jobs available on your cluster. Storage requirements With the default configuration (3 replicates, 20 FRET state pairs, 42 interconversion rates, 9 equilibrium constants): Stage Files Size per replicate Total Diffusion trajectories 3 ~270 GB ~810 GB smFRET simulations 22,680 ~230 GB ~690 GB Analysis results (CSV) 2,520 ~1 GB ~3 GB Total ~1.5 TB Citation If you use this code or data, please cite: [Title and DOI to be added] Please also ensure you cite the underlying tools and methods used, such as PyBroMo, FRETBursts, BVA, FRET-2CDE, and burstH2MM.本仓库配套于待补充标题与DOI的发表论文的代码与数据。 本仓库通过布朗运动模拟精准刻画分子通过共焦体积的平动扩散过程，以此生成基于扩散模拟的单分子荧光共振能量转移（single-molecule FRET, smFRET）数据集，并采用成熟分析方法对其进行处理，以基准测试该数据集检测蛋白质构象动态变化的性能。本仓库适配基于SLURM调度器的高性能计算（High Performance Computing, HPC）集群，若需使用其他作业调度器，仅需修改生成脚本中的作业模板即可完成适配。对于小型数据集或本地使用场景，也可直接运行生成的Python脚本，无需依赖调度器。 由于完整数据集存储占用约1.5 TB，本仓库仅提供代表性子集，具体说明如下： 1. 扩散轨迹（00阶段输出）：由于布朗运动轨迹文件（`pybromo_*.hdf5`，约271 GB）体积过大，未随仓库一并提供。用户可通过`config.py`中定义的固定随机种子，运行`00_generate_diffusion_jobs.py`脚本确定性地重新生成该文件。 2. 光子到达时间（`data/replicate_1/`目录下）：已包含光子到达时间的HDF5文件（约537 MB），该文件为01阶段的输出结果，同时作为02阶段的输入数据。 3. 时间戳文件（`data/replicate_1/`目录下）：已包含全部21个光子时间戳HDF5文件（每个FRET效率对应一个文件，单文件约22 MB，总大小约480 MB），该文件为02阶段的输出结果，同时作为03阶段的输入数据。 4. 模拟文件（`data/replicate_1/`目录下）：本仓库提供20组FRET态对中的10组，覆盖了ΔE从宽到窄的全部范围。针对每组FRET态对，选取了42个互变速率（kobs）中约每3个取1个的共14个值，涵盖了0.4至2.5×10^5 s⁻¹的全部时间尺度范围；且针对每组组合，均包含全部9个平衡常数（K=0.05~20）。完整单重复数据集对应7560个HDF5文件，而本仓库仅提供1260个。 5. 分析结果（`data/results/R1`至`R3`目录下）：已包含全部3个重复、20组FRET态对、42个互变速率以及9个平衡常数对应的完整分析CSV文件集，这些数据即为发表论文中绘图所用的原始数据。 ## 工作流概述 本工作流包含四个阶段，每个阶段均配有用于生成SLURM作业文件的脚本： | 脚本路径 | 阶段内容 | | --- | --- | | `00_generate_diffusion_jobs.py` | 布朗运动轨迹生成 | | `01_generate_timestamp_jobs.py` | 基于轨迹生成光子时间戳 | | `02_generate_simulation_jobs.py` | FRET态混合模拟 | | `03_generate_analysis_jobs.py` | 爆发搜索与数据分析 | 所有模拟与分析参数均在`config.py`文件中定义，若需适配自定义工作流，仅需在生成作业前修改该文件即可。 ## 环境搭建 执行以下命令创建Conda环境并激活： conda env create -f environment.yml conda activate smfret-benchmarking ## 近似运行时长 基于默认配置（3个重复、20组FRET态对、42个互变速率、9个平衡常数）的近似运行时长如下： | 阶段 | 作业数 | 单作业平均时长 | 串行总时长 | | --- | --- | --- | --- | | 扩散模拟 | 3 | 约1小时 | 约3小时 | | 时间戳生成 | 3 | 约26小时 | 约78小时 | | FRET模拟 | 60 | 约42小时 | 约2500小时 | | 数据分析 | 2520 | 约20分钟 | 约840小时 | | 总计 | 2586 | - | 约3421小时 | 需注意，每个作业均包含针对平衡常数、爆发数以及分析方法的嵌套循环，因此爆发搜索、爆发筛选与数据分析等常规操作在单个作业中会被多次执行。默认配置下，总共有272160个独立的分析实例（3个重复 × 20组FRET态对 × 42个互变速率 × 9个平衡常数 × 12个爆发数阈值）。 02至03阶段的任务均按文件独立运行，因此实际墙钟时间会随集群可同时运行的作业数量线性缩放。 ## 存储需求 基于默认配置的存储占用情况如下： | 阶段 | 文件数 | 单重复占用大小 | 总占用大小 | | --- | --- | --- | --- | | 扩散轨迹 | 3 | 约270 GB | 约810 GB | | smFRET模拟 | 22680 | 约230 GB | 约690 GB | | 分析结果（CSV格式） | 2520 | 约1 GB | 约3 GB | | 总计 | - | - | 约1.5 TB | ## 引用说明 若您使用本仓库的代码或数据，请引用：[待补充标题与DOI] 同时请务必引用本项目所使用的底层工具与方法，包括PyBroMo、FRETBursts、BVA、FRET-2CDE以及burstH2MM。