GRAFT_benchmark

# 📊 GRAFT: Graph Reasoning and Alignment For Tasks [](https://huggingface.co/papers/2508.15690) **GRAFT** is a multimodal benchmark designed to rigorously evaluate foundation models on tasks involving **visual reasoning**, **structured data understanding**, and **instruction-following** over synthetic yet realistic charts and tables. The benchmark includes multiple configurations representing different phases of data creation and evaluation. --- ## 📁 Dataset Configurations | Config Name | Purpose | |--------------------------------|-------------------------------------------------------------------------| | `chart_raw_v1.0` | Raw synthetic chart generation (Python code + images + metadata) | | `chart_raw_filtered_v1.0` | Quality-controlled subset after automated visual judgments | | `chart_qna_v1.0` | Chart-based QA with structured YAML answers and core metadata | | `chart_qna_joj_v1.0` | Chart-QnA with Jury-of-Judges (multi-model) evaluation fields | | `chart_qna_joj_filtered_v1.0` | Filtered JoJ Chart-QnA, keeping only high-consensus items | | `table_raw_v1.0` | Raw synthetic table images + detailed rendering/layout metadata | | `table_qna_v1.0` | Table-based QA with conversations and YAML/JSON gold answers | | `table_qna_joj_v1.0` | Table-QnA with JoJ evaluation fields | | `table_qna_joj_filtered_v1.0` | Filtered JoJ Table-QnA, high-consensus only | --- ## 🔧 Dataset Creation Pipeline Each GRAFT configuration is derived through a controlled generation pipeline. The steps include: ### 1. Scenario & Metadata Sampling - Randomized sampling of chart/table genres (e.g., healthcare, finance) - Personas and data topics are selected to ground the scenario - Metadata includes visual attributes, design constraints, and layout ### 2. Data Table Generation - Tables are synthesized with 8–20 rows and realistic schema - Includes multiple numerical/text fields suited to specific plot types ### 3. Code and Chart Rendering - Python code is generated (using Matplotlib/Seaborn) - Plots are rendered and saved as images - Code and rendering metadata are preserved ### 4. Visual QA Authoring - Each chart/table is paired with: - A natural-language **instruction** - A **YAML-formatted golden answer** - Structured metadata such as formatting and correctness criteria ### 5. Quality Control - Auto-evaluation by LLMs for: - Clarity and realism of visualizations - Answerability and visual fidelity - Judgments and explanations stored in metadata fields --- ## Token Length Statistics | Subset | Question (Avg / 90th Pct) | Answer (Avg / 90th Pct) | |---------------|---------------------------|--------------------------| | `chart_qna` | 265.2 / 400 | 163.8 / 425 | | `table_qna` | 461.9 / 848 | 1021.7 / 1623 | --- ## Evaluation Methodology Each prediction is scored across four dimensions using a second model (GPT-4o or equivalent): - **Correctness**: Is the answer factually accurate? - **Completeness**: Does it address all parts of the question? - **Visual Grounding**: Does it properly interpret the image? - **Format Fidelity**: Does it match the expected YAML/JSON format? Scores range from 1 to 5 and are explained with targeted rationales for each error. --- ## Licences and Usage Policies: 1. **Distribution of outputs** If you distribute or use **Llama outputs** in the dataset anywhere for training, you must provide a copy of the **Llama 3.1 license** (https://huggingface.co/meta-llama/Llama-3.1-8B/blob/main/LICENSE) with the outputs. 2. **Use in AI models** If you use **Llama Materials**—or any outputs or results derived from them—to create, train, fine-tune, or otherwise improve an AI model that is then distributed or made available, the model’s name must begin with **“Llama.”** ## Paper If you use GRAFT in your work, please cite our paper: **arXiv:** https://arxiv.org/abs/2508.15690 ## Citation ```bibtex @article{verma2025graft, title = {GRAFT: GRaPH and Table Reasoning for Textual Alignment — A Benchmark for Structured Instruction Following and Visual Reasoning}, author = {Abhigya Verma and Sriram Puttagunta and Seganrasan Subramanian and Sravan Ramachandran}, journal = {arXiv preprint arXiv:2508.15690}, year = {2025}, eprint = {2508.15690}, archivePrefix= {arXiv}, primaryClass = {cs.AI} }