Per-Question Evaluation Outputs for "Graph-Augmented Retrieval for Biomedical Question Answering: When Do Knowledge Graphs Help?"
收藏资源简介:
This deposit contains the raw per-question evaluation outputs, aggregated statistics, and figures behind every numerical claim in the manuscript "Graph-Augmented Retrieval for Biomedical Question Answering: When Do Knowledge Graphs Help?". Reviewers and downstream researchers can verify every reported accuracy, ROUGE score, retrieval recall, and ablation delta directly from the included JSON files without re-running the experiments (a complete evaluation takes ~33 hours of wall-clock time and ~12M OpenAI API tokens). Contents (1.2 GB uncompressed, 60 files): 30 per-question result JSONs covering the FAISS dense-retrieval baseline (matched protocol, n=996), the bridge-based adaptive best configuration (n=1000, 64.10% accuracy), four component ablations (early entity detection, split reranking, adaptive degree thresholding, naive bridge) at n=400 each, the full embedding × hop-depth grid across three biomedical encoders (BiCA, MedCPT, MedTE) with and without graph facts, the BM25 + cross-encoder sparse baseline, the RRF fusion calibration, and BioASQ generalization (707 questions plus a 200-question LLM-as-a-Judge stratified evaluation); 15 aggregated-statistics CSVs corresponding to each table in the paper (comprehensive results, ablation stats, stratified-by-coverage breakdowns, retrieval and latency metrics, BM25 and BioASQ summaries); 10 figures in PDF + PNG (knowledge-graph schema, bridge-retrieval workflow, results comparison, stratum impact, error analysis); and a verify_numbers.py script that recomputes every headline claim directly from the raw JSONs using only the Python standard library. Every JSON includes full reproducibility metadata: configuration hash, random seed (42), library versions (Python 3.12.10, PyTorch 2.6.0+cu124, FAISS, sentence-transformers, OpenAI, Neo4j), platform (Windows 11, RTX 4060 Laptop), and per-question predictions, gold labels, and retrieved PMIDs. The underlying 12.5M-paper, 7.5M-entity, 266M-relationship PubMed knowledge graph and its three FAISS indices (BiCA, MedCPT, MedTE; ~310 GB total) are hosted separately on Hugging Face Datasets at https://huggingface.co/datasets/danielquillanroxas/pubmed-graphrag-data. The split is intentional: Zenodo provides the citable per-DOI evaluation evidence, while Hugging Face hosts the bulk data with native streaming support. Datasets evaluated: PubMedQA (pqa_labeled split, 1,000 questions; Jin et al., 2019) and BioASQ (rag-mini-bioasq test split, 707 questions; Tsatsaronis et al., 2015). License: CC-BY-4.0. Free to use for any purpose including commercial, with attribution to the deposit and the accompanying manuscript. PubMed/PubTator3/MeSH source data retain their respective NLM terms (public-domain or free-with-attribution).



