BiomniBench-DA

# BiomniBench-DA **BiomniBench-DA** is the data-analysis instantiation of **BiomniBench**, a process-level evaluation framework for LLM agents on real-world biomedical research tasks. Each task is a multi-step data analysis derived from a high-impact biomedical publication; agents are graded on the *full analytical trajectory* against an expert-authored rubric, not only the final answer. This repository releases **50 of the 100** BiomniBench-DA tasks; the remaining 50 are held out as a private, contamination-resistant evaluation set. ## Task structure Each task lives in `da-{paper}-{task}/` with: - `instruction.md` — research question, data-file manifest, required output format - `task.toml` — Harbor task configuration (verifier, agent, environment budgets) - `environment/` — public dataset files - `tests/` — reference trace, rubric, and judge harness The default verifier uses **Gemini 3.1 Pro** as the LLM judge. ## Quick start ```bash hf download phylobio/BiomniBench-DA --repo-type dataset --local-dir ./biomnibench-da # Run with the Harbor framework export GEMINI_API_KEY=... # for the LLM judge harbor run --path ./biomnibench-da/da-1-3 --agent terminus-2 --model anthropic/claude-opus-4-7 ``` ## License - Benchmark artifacts (instructions, rubrics, reference traces, judge prompts): CC-BY-4.0. - Underlying datasets retain their original public-release terms; attribution to the source publication is in each task's `instruction.md`. ## Citation ```bibtex @article{qu2026biomnibench, title = {BiomniBench: Process-level Evaluation of LLM Agents for Real-world Biomedical Research}, author = {Qu, Yuanhao and Lu, Yingzhou and Tu, Xinming and Zhang, Serena and She, Tianwei and Shaw, Alexander Glenn and Shih, Jou-Ho and Zhao, Bingqing and Shen, Minjie and Yang, Haochen and Yan, Jielin and Zhang, Rongchuan and Wu, Xinze and Li, Tingting and Zhou, Bin and Wang, Ning and Ma, Adam and Cong, Le and Hu, Xiaobo and Jiang, Yuan and Dong, Jiayun and Peng, Tao and Leskovec, Jure and Huang, Kexin}, year = {2026}, journal = {bioRxiv}, doi = {10.64898/2026.05.12.724604}, url = {https://www.biorxiv.org/content/10.64898/2026.05.12.724604v2} } ```