CAFA 5 Evaluation Data

# CAFA 5 Evaluation Data This resource contains the files necessary to conduct the CAFA 5 evaluation and benchmarking for CAFA 5 baselines. If you use this resource, please cite this data DOI and the accompanying preprint:```@ARTICLE{depaoliskaluza2026,author={De Paolis Kaluza, M Clara and Ramola, Rashika and Joshi, Parnal and Piovesan, Damiano and Reade, Walter and Orchard, Sandra and Martin, Maria J and Ignatchenko, Alex and Kaggle Competition Participants and Rost, Burkhard and Orengo, Christine A and Robinson-Rechavi, Marc and Durand, Dannie and Brenner, Steven E and Greene, Casey S and Mooney, Sean D and Friedberg, Iddo and Radivojac, Predrag},journal={bioRxiv},title={Advances in protein function prediction from the fifth CAFA challenge.},year={2026},volume={},number={},doi={10.64898/2026.04.27.716980}}``` ## FilesFiles for the main evaluation are in the folder `full_evaluation`: * `go-basic.obo`: Gene Ontology graph used for CAFA 5. release 2023-01-01* `IA_t0.tsv`: Information accretion for each term in GO, used for weighted measures.* `known_t0.tsv`: List of annotations known at submission deadline t0 (2023-08-14). These annotations are excluded from evaluation and used to determine which subset of an antology will be included in the Partial Knowledge evaluation.* `toi_2025_03.tsv`: Terms of interest to use for evaluation. This file specifies which terms in the GO graph from `go-basic.obo` to use in evaluation. This is used to exclude obsolete terms* `eval_terms_[no_knowledge/limited/partial]_2025_03.tsv`: Ground truth for evaluation for each of the three evaluation settings (No Knowledge, Limited, and Partial knowledge, respectively). These are the new annotations on which to assess performance of methods ### Other evaluationsFiles for secondary evaluations can be found in the directory `other_evaluations`. #### Annotations from publications after t0 In CAFA 5, we perform an additional evaluation on the subset of annotations credited to publications after t0. These files are found in the directory `new_publications`. The included files are: * `eval_terms_[no_knowledge/limited/partial]_2025_03_publishedaftert0.tsv`: Ground truth annotations for evaluation for each of the three evaluation settings (No Knowledge, Limited, and Partial knowledge, respectively).* `known_publishedaftert0.tsv`: The known annotations to exclude in Partial Knowledge evaluation. These are all annotations from publications before t0Terms of interest (`toi_2025_03.tsv`), information accretion (`IA_t0.tsv`), and GO (`go-basic.obo`) files from the full evaluation should be used for this evaluation. #### Comparison to previous CAFA versions (head-to-head) We find the intersection of target proteins from CAFA 2-5 and evaluate performance on these proteins for terms that existed in GO since CAFA 2. Because CAFA 2-4 use alias identifiers for proteins rather than UniProt accession IDs, we recreate the ground truth and known annotations files for each CAFA version using the corresponding ID for each protein in this evaluation set. The files are in the directory `previous_cafas_comparison` are as follows: * `h2h_toi_2025_03.tsv`: The list of terms that appear in target GO graphs for CAFA 2-5 * `h2h_cafa[2/3/4/5]_eval_terms_[no_knowledge/limited/partial]_2025_03.tsv`: Ground truth for each evaluation setting using the appropriate ID for each CAFA version* `h2h_cafa[2/3/4/5]_t0_known.tsv`: Known annotations using the appropriate ID for each CAFA version The GO (`go-basic.obo`) and information accretion (`IA_t0.tsv`) files from the full evaluation should be used for these evaluations. For CAFA 5, use the main `t0_known.tsv` file for the known annotations. ## Running evaluation Evaluations are done using the partial knowledge version of the CAFA-Evaluator code base (https://github.com/claradepaolis/CAFA-evaluator-PK). Evaluations are run with the following parameters: ```obo='go-basic.obo'method_dir='' # specify directory with predictions to evaluategt='eval_terms_no_knowledge_2025_03.tsv'ia='IA_t0.tsv'toi='toi_2025_03.tsv'known='known_t0.tsv' cafaeval $obo $method_dir $gt -ia $ia -toi $toi -known $known -prop fill -th_step 0.001 -no_orphans -threads 22 ``` ## Baseline PredictionsBaseline and benchmark predictions are in the directory `baselines`. Baselines included are: * Non-experimental evidence: file: `all_evidence.tsv` * BLAST: predict scores according to similar proteins as defined by BLAST alignment scores. Predictions provided for CAFA 5 baseline and CAFA 2-4 head-to-head comparisons file: `blast_CAFA[2/3/4/5].tsv`* Naive: predict a score for each GO term equal to the frequency observed for that term in experimentally validated annotations. This file must be constructed with the script `naive.py` and the input files `cafa5_targets_toeval.tsv` and `cafa5_naive_frequencies.tsv` (see below).* ProtT5 embedding: Distance in the embedding space of ProtT5 is used to fine N-nearest neighbors for each protein. GO term predictions are distance-weighted scores for any term annotated to the N neighbors. We use cosine similarities and Euclidean distance to find neighbors. We include 1-, 3-, 5-, and 1000-nearest neighbors (files `prott5_[cosine/euclidean]_[1/3/5/1000].tsv`). For 1-nearest neighbors, we also include an unweighted prediction that assigns a score of 1 to any GO terms annoteted to the nearest neighbor protein (files `prott5_[cosine/euclidean]_1_unweighted.tsv`). ### NaiveThe Naive baseline predicts the same scores for all proteins, the prediction file is large (about 29GB for the full CAFA 5 evaluation) evaluation. Instead of packaging this file here, we provide the list of proteins on which we evaluate (in file `cafa5_targets_toeval.tsv`) and the list of GO term frequencies (in file `cafa5_naive_frequencies.tsv`). These files can be used to generate the full Naive prediction file with the script `naive.py`: ```python3 naive.py --freq cafa5_naive_frequencies.tsv --target cafa5_targets_toeval.tsv --out naive.tsv```The script requires python 3.5+ and the packages [pandas](https://pypi.org/project/pandas/) and [tqdm](https://pypi.org/project/tqdm/).