Focus

# Focus Dataset **Focus** is meta-evaluation benchmark designed to assess the robustness of evaluator VLMs across diverse Image-to-Text (I2T) and Text-to-Image (T2I) tasks. Please refer to our [paper](https://arxiv.org/abs/2604.21523) for more details. <p align="center" width="50%"> <img src="images/hero.png" alt="Focus Benchmark" style="width: 50%; min-width: 100px; display: block; margin: auto;"> </p> ## Code The code to generate the perturbations and run evaluations are available on our github repository: [ai4bharat/focus](https://github.com/AI4Bharat/focus) ## Subsets | Subset | Description | Splits | |--------|-------------|--------| | `i2t` | Image-to-Text perturbations | `visual_grounding`, `semantic_interpretation`, `visual_reasoning`, `long_form_generation`, `score_invariant` | **1726 total** | | `t2i` | Text-to-Image perturbations | `visual_fidelity`, `scene_coherence`, `physical_plausibility`, `text_rendering`, `score_invariant` | **2363 total** | ## Usage ```python from datasets import load_dataset # Load a specific split ds = load_dataset("ai4bharat/Focus", "i2t", split="visual_grounding") # Load all splits for a task ds = load_dataset("ai4bharat/Focus", "i2t") ``` ## Schema ### I2T (Image-to-Text) | Field | Type | Description | |-------|------|-------------| | `p_id` | string | Perturbation ID | | `prompt` | string | Question about the image | | `image` | image | Input image | | `gold_answer` | string | Correct answer | | `perturbed_answer` | string | Perturbed (incorrect) answer | ### T2I (Text-to-Image) | Field | Type | Description | |-------|------|-------------| | `p_id` | string | Perturbation ID | | `prompt` | string | Text prompt | | `gold_image` | image | Correctly generated image | | `perturbed_image` | image | Perturbed (edited) image | ## Citation ```bibtex @article{khan2026seeing, title = {Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models}, author = {Mohammed Safi Ur Rahman Khan and Sanjay Suryanarayanan and Tushar Anand and Mitesh M. Khapra}, year = {2026}, journal = {arXiv preprint arXiv: 2604.21523} } ```