VAKRA

# 🔷 VAKRA: A Benchmark for Evaluating Multi-Hop, Multi-Source Tool-Calling Capabilities in AI Agents **VAKRA** (e**V**aluating **A**PI and **K**nowledge **R**etrieval **A**gents using multi-hop, multi-source dialogues) is a tool-grounded, executable benchmark designed to evaluate how well AI agents reason end-to-end in enterprise-like settings. Rather than testing isolated skills, **VARKA** measures compositional reasoning across APIs and documents, using full execution traces to assess whether agents can reliably complete multi-step workflows, not just individual steps. **VARKA** provides an executable environment where agents interact with over 8,000 locally hosted APIs (sourced from LiveAPIBench[1]) backed by real databases (sourced from BIRD-SQL[2]) spanning 62 domains, along with domain-aligned document collections (sourced from CLAPnq[3] and Wikidata5M[4]). --- ## ✨ Key Features - 🔧 **8,000+ executable APIs** backed by real databases across **62 domains** - 🔁 **Multi-hop reasoning (3–7 steps)** combining API calls and document retrieval - 📚 **Cross-source grounding** via structured APIs + unstructured documents - 📏 **Trace-level verification** with replayable execution paths - 🔒 **Deterministic evaluation** using locally hosted tools --- ## 🔗 Resources - 🏆 **Leaderboard:** [https://ibm-research-vakra.hf.space/](https://ibm-research-vakra.hf.space/) - 🛠 **Environment & Setup:** [github.com/ibm/vakra/setup.md](https://github.com/IBM/vakra/blob/main/setup.md) - 📊 **Evaluation & Scoring:** [github.com/ibm/vakra](https://github.com/IBM/vakra) --- ## 🧩 Benchmark Structure VAKRA organizes evaluation into four capabilities, which together reflect three progressively complex settings. The dataset viewer shows a snippet of the train dataset. ### 1. Diverse API Interaction Styles These tasks focus on structured tool use over APIs with different interface abstractions. - `capability_1_bi_apis` (API Chaining): nested and compositional API chaining - `capability_2_dashboard_apis` (Tool Selection): large-scale tool selection over query-aligned endpoints ### 2. Multi-hop Reasoning over Structured APIs These tasks require dependent reasoning chains over APIs, where earlier outputs must be interpreted and transformed for later calls. We have single-turn queries that can be answered by a reasoning chain of 1–3 APIs. For example, a sample may be answered by a single API (API), or by two APIs where the output of API₁ is transformed and passed to API₂ (API₁ → API₂), or by three APIs (API₁ → API₂ → API₃). - `capability_3_multihop_reasoning` (Multihop API Reasoning) ### 3. Multi-hop, Multi-source Reasoning with Tool-use Policies These tasks combine reasoning over APIs and document retrieval in a multi-turn setting and also include natural-language constraints about tool use. We have multi-turn dialogues represented as context-response-pairs wherein queries could be answered by a reasoning chain of 1-4 tools (ex., a three-turn dialogue "(API)(RAG)(API-RAG)" wherein using the context from the first two turns, an answer needs to be obtained for the (API-RAG) turn.) - `capability_4_multiturn` (MultiHop MultiSource with Policy Adherence) This represents the most challenging setting, mirroring decision workflows. --- ## 📊 Dataset Statistics ### 🧪 Training Split | Capability | Description | Domains | Samples | Avg Tool Calls | Max Tool Calls | Avg Turns | Max Turns | |------------|------------|---------|---------|----------------|----------------|-----------|-----------| | Capability_1 | API Chaining | 33 | 1,324 | 4.05 | 12 | — | — | | Capability_2 | Tool Selection | 40 | 1,860 | 1.00 | 1 | — | — | | Capability_3 | Multihop API Reasoning | 28 | 346 | 2.05 | 3 | — | — | | Capability_4 | MultiHop MultiSource with Policy Adherence | 36 | 898 | 1.05 | 3 | 2.06 | 5 | --- ### 🧪 Test Split | Capability | Description | Domains | Samples | Avg Tool Calls | Max Tool Calls | Avg Turns | Max Turns | |------------|------------|---------|---------|----------------|----------------|-----------|-----------| | Capability_1 | API Chaining | 54 | 2,077 | 3.96 | 10 | — | — | | Capability_2 | Tool Selection | 17 | 1,597 | 1.00 | 1 | — | — | | Capability_3 | Multihop API Reasoning| 38 | 869 | 2.04 | 5 | — | — | | Capability_4 | MultiHop MultiSource with Policy Adherence | 41 | 644 | 1.34 | 4 | 2.01 | 7 | --- ## 📁 Directory Structure ``` <base_path>/ ├── indexed_documents/ ├── databases/ │ └── <domain>/ │ ├── database_description/ │ └── domain.sqlite ├── test/ │ └── capability-X/ │ └── input/ └── train/ └── capability-X/ ├── input/ └── output/ ``` --- ## 🧪 Dataset Format Following is an example of the input & output structure of the dataset. ### Input Sample ```json { "uuid": "str", "domain": "str", "num_turns": 2, "dialogue": { "turns": [ { "turn_id": 0, "query": "str", "answer": `str | list | int | float` # Only present for historical-turns in capability_4_multiturn }, { "turn_id": 1, "query": "str" }]}, "additional_instructions": "str" # This field is only present for capability_4_multiturn } ``` Expected datatypes: - `uuid`: `str` - `domain`: `str` - `num_turns`: `int` - `dialogue.turns`: `list` - `turn_id`: `int` - `query`: `str` - `answer` in historical turns: `str | list | int | float` - `additional_instructions`: `string` Notes: - In `capability_*/input`, all historical turns can have `answer`, but the last turn must not have `answer`. - In non-multiturn input files, `additional_instructions`, historical-turn would be absent as they are single turn dialogues. --- ### 🔹 Output Schema ```json { "uuid": "str", "domain": "str", "output": [ { "turn_id": 0, "query": "str", "answer": "str", "sequence": { "tool_call": [ { "name": "str", "arguments": { "key": "value" } } ], "tool_response": [ { "name": "str", "response": { "key": "value" }}]}}], "additional_instructions": "str" } ``` Expected datatypes: - `uuid`: `str` - `domain`: `str` - `output`: `list` - `turn_id`: `int` - `query`: `str` - `answer`: `str | list | int | float` - `sequence`: `dict` - `sequence.tool_call`: `list` - `sequence.tool_response`: optional, typically `list` - `additional_instructions`: `string` for `capability_4_multiturn/output` --- ## 🚀 Getting Started Follow the instructions - [github.com/ibm/vakra/setup.md](https://github.com/IBM/vakra/blob/main/setup.md) to download the dataset and set up the repository. --- ## 🏁 Evaluation & Scoring Evaluation code, scoring scripts, and field exclusion lists are maintained at: **[github.com/ibm/vakra](https://github.com/IBM/vakra)** The benchmark uses a waterfall judge consisting of three judges - **PolicyJudge:** Programmatically evaluates the adherence to policy. - **ExactMatchJudge:** Programmatically evaluates an order-invariant exact match (subset-based) between the predicted tool responses and ground truth tool responses. - **GroundednessJudge:** LLM-as-a-Judge evaluating the groundedness of an answer to the tool responses and query. 👉 Full details of judging criteria: **[github.com/ibm/vakra](https://github.com/IBM/vakra)** --- ## 🤝 Contributing To submit results, create a GitHub issue using the 👉 Leaderboard Submission Template: https://github.com/IBM/vakra/issues/new?template=leaderboard_submission.yml To contact the author, create a GitHub issue using https://github.com/IBM/vakra/issues/new?template=leaderboard_submission.md --- ## References - [1] Elder, Benjamin, et al. "Live API-Bench: 2500+ Live APIs for Testing Multi-Step Tool Calling." Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2026. - [2] Li, Jinyang, et al. "Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls." Advances in Neural Information Processing Systems 36 (2023): 42330-42357. - [3] Rosenthal, Sara, et al. "CLAPnq: C ohesive L ong-form A nswers from P assages in Natural Questions for RAG systems." Transactions of the Association for Computational Linguistics 13 (2025): 53-72. - [4] Wang, Xiaozhi, et al. "KEPLER: A unified model for knowledge embedding and pre-trained language representation." Transactions of the Association for Computational Linguistics 9 (2021): 176-194. - [5] Shlomov, Segev, et al. "From Benchmarks to Business Impact: Deploying IBM Generalist Agent in Enterprise Production." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 40. No. 47. 2026. ## Acknowledgments We especially acknowledge Chulaka Gunasekara, Hamid Adebayo, Harold Ship, Himanshu Gupta, Huaiyu Zhu, Jaydeep Sen, Nir Mashkif, Renuka Sindhgatta, Sameep Mehta, Sara Rosenthal, and Segev Shlomov for their contributions and insights. We also thank our interns, Raavi Gupta and Abhinav Jain, for their efforts in benchmark generation and development. ## Citation ``` @misc{vakra, title={VAKRA: A Benchmark for Evaluating Multi-Hop, Multi-Source Tool-Calling Capabilities in AI Agents}, author={Ankita Rajaram Naik*, Anupama Murthi*, Benjamin Elder*, Siyu Huo*, Praveen Venkateswaran, Danish Contractor}, year={2026}, url={https://huggingface.co/spaces/ibm-research/VAKRA}, } ``` _* Equal contributions_