Martyn et al. (2025) Variant-EFFECTS Evaluator for GAME API

This Evaluator requests expression predictions for sequences in THP-1 monocytes and Jurkat T cells. It then computes the log2 fold change (log2FC) between alternate and reference sequence predictions, and evaluates performance by calculating the Pearson correlation between the measure Included in Engreitz_evaluator.sif : Scripts to process the data and connect to predictors in the GAME API Scripts to parse the returned predictions and calculate performance metrics All software dependencies /evaluator_data folder contents: /Jurkat all_jurkat_sequences.tsv: SPDI ID; Reference Sequence; Alternate Sequence. ~2000 bp long sequence (depending on indel size) all_Jurkat.tsv: concatenated variants from individual variant files in SPDI format. 332 total variants. /THP1 all_THP1_sequences.tsv: SPDI ID; Reference Sequence; Alternate Sequences. ~2000 bp long sequence (depending on indel size) all_THP1.tsv : concatenated variants from individual variant files in SPDI format. 392 total variants. /SPDI_toseq instructions.tsv : How to run the Rscript to pull sequences from SPDI IDs (hg38) .yaml to create the conda enviroment for the Rscript parse_Engreitz_data.py File Specification_VariantEffectsFiles.png: Details about information in the variant files How to run: apptainer run --containall -B /path_to/evaluator_data/:/evaluator_data -B /path_to/prediction_folder/:/predictions Engreitz_evaluator.sif HOST PORT /predictions Notes: The main evaluator script (Engreitz_evaluator.py)will read in the *_sequences.tsv files from their respective folders and send two seperate requests to the connected Predictor Duplicated SPDI ID sequences are only sent to the Predictors once to minimize duplicated computation and the values are merged with the measured values to calculate the correlation Additional information can be found on GitHub: Genomic API for Model Evaluation Original Publication can be found here: Martyn et al. (2025)本评估器针对THP-1单核细胞（THP-1 monocytes）与Jurkat T细胞（Jurkat T cells）中的序列请求表达预测结果。随后计算变异序列与参考序列预测结果之间的log2倍数变化（log2 fold change, log2FC），并通过计算实测值与对应指标之间的皮尔逊相关系数（Pearson correlation）来评估模型性能。 包含于Engreitz_evaluator.sif文件中的内容包括： - 用于处理数据并连接至GAME API中预测器的脚本 - 用于解析返回的预测结果并计算性能指标的脚本 - 全部软件依赖项 /evaluator_data 文件夹内容如下： /Jurkat all_jurkat_sequences.tsv：包含SPDI标识符、参考序列、变异序列，序列长度约2000 bp（具体取决于插入缺失（indel）的大小） all_Jurkat.tsv：以SPDI格式整合自单个变异文件的所有变异位点，共计332个变异 /THP1 all_THP1_sequences.tsv：包含SPDI标识符、参考序列、变异序列，序列长度约2000 bp（具体取决于插入缺失的大小） all_THP1.tsv：以SPDI格式整合自单个变异文件的所有变异位点，共计392个变异 /SPDI_toseq instructions.tsv：说明如何通过R脚本从SPDI标识符（hg38版本参考基因组）中提取序列 .yaml文件：用于为该R脚本创建Conda运行环境 parse_Engreitz_data.py File Specification_VariantEffectsFiles.png：详细说明变异文件中包含的各类信息 运行指南： apptainer run --containall -B /path_to/evaluator_data/:/evaluator_data -B /path_to/prediction_folder/:/predictions Engreitz_evaluator.sif HOST PORT /predictions 注意事项： 核心评估脚本（Engreitz_evaluator.py）将从对应文件夹中读取*_sequences.tsv文件，并向已连接的预测器发送两组独立请求。 重复的SPDI标识符序列仅会向预测器发送一次，以避免重复计算，随后会将预测结果与实测值合并以计算相关系数。 更多信息可访问GitHub仓库：Genomic API for Model Evaluation 相关原始文献可参阅：Martyn等人（2025年）