CountQA

# **Dataset Summary** **CountQA** is the new benchmark designed to stress-test the Achilles' heel of even the most advanced Multimodal Large Language Models (MLLMs): **object counting**. While modern AI demonstrates stunning visual fluency, it often fails at this fundamental cognitive skill, a critical blind spot limiting its real-world reliability. This dataset directly confronts that weakness with **over 1,500 challenging question-answer pairs** built on real-world images, hand-captured to feature the messy, complex scenes that models struggle with: high object density, distracting clutter, and significant occlusion. Our meticulous *in-situ* annotation process ensures a high-confidence ground truth, far more reliable than standard post-hoc labeling. CountQA is not just a dataset; it's a diagnostic tool engineered to probe the perceptual and numerical limits of AI, paving the way for a new generation of MLLMs that are not only descriptively fluent but also **numerically grounded and spatially aware**. *** # **Paper** For a deep dive into the dataset, methodology, and the surprising results from our evaluation of 15 leading MLLMs, please see our paper: 📄 **CountQA: How Well Do MLLMS Count in the Wild?** - **Arxiv:** [Link to Paper](https://arxiv.org/abs/2508.06585) - **Google Research Link**: [Link to Publication](https://research.google/pubs/pub1042117/) - **Listen to the paper:** [Link to Audio](https://illuminate.google.com/library?play=6oB58I1__Q41h) *** # **Use Cases** CountQA is built for action. Use it for: - **Evaluating and benchmarking** the true counting accuracy of MLLMs in challenging, open-ended VQA scenarios. - **Probing the limits of visual perception**, analyzing how models handle clutter, occlusion, and object density. - **Diagnosing failure modes** with rich metadata, from object recognition errors to flawed attribute binding (e.g., "how many *red* thumb tacks?"). - **Training and fine-tuning** the next generation of multimodally-native models to achieve robust numeracy. - **Analyzing the performance gap** between powerful proprietary models and leading open-source alternatives. *** # **Dataset Structure** - **Split:** A single, powerful `test` split containing all 1,528 QA pairs. - **Granularity:** Each row corresponds to a unique image. We support multiple questions and answers per image to maximize data richness. *** # **Data Fields** - `image` (Image): The high-quality RGB image, ready for processing. - `questions` (list[string]): One or more natural language questions crafted to test enumeration and compositional reasoning. - `answers` (list[string]): The ground-truth answers, aligned with `questions` and stored as strings. - `objects` (list[string]): A detailed list of all countable object types present in the scene, enabling fine-grained analysis. - `categories` (list[string]): High-level scene categories (e.g., "Kitchen & Dining", "Outdoor, Vehicles & Structural") for sliced evaluations. - `is_focused` (bool): A flag to distinguish between images with a clear subject (`true`) and visually "busy" or cluttered scenes (`false`), designed to test model resilience to noise. - `full_config` (string/JSON): The complete, serialized metadata for the sample, ensuring full reproducibility. *** # **Example** Here’s a look at a challenging example from the dataset where many top-tier models stumble, highlighting the difficulty of counting in cluttered scenes: ```json { "image": "<PIL.Image.Image ...>", "questions": ["How many jackets are there?"], "answers": ["15"], "objects": ["jacket", "clothing"], "categories": ["Clothing & Wearables", "Storage & Organization"], "is_focused": false, "full_config": "{\"QA\":[{\"question\":\"How many jackets are there?\",\"answer\":\"15\"}],\"objects\":[\"jacket\", \"clothing\"],\"categories\":[\"Clothing & Wearables\",\"Storage & Organization\"],\"focused\":false}" } ``` # Loading the Dataset Get up and running in seconds with the Hugging Face `datasets` library: ```python from datasets import load_dataset ds = load_dataset("Jayant-Sravan/CountQA") row = ds["test"][0] img = row["image"] # PIL.Image qs = row["questions"] # list[str] ans = row["answers"] # list[str] objs = row["objects"] # list[str] cats = row["categories"] # list[str] focused = row["is_focused"] # bool cfg = row["full_config"] # str (JSON) ``` # Intended Uses This benchmark is specifically **intended for rigorous, quantitative evaluation of MLLM numeracy**. Use it to publish strong, reproducible results and drive the field forward. It is perfect for ablations across object types, scene categories, and visual complexity. # Limitations - **Geographic & Cultural Scope:** The images were manually collected by the authors and are primarily representative of their specific environments. The object distribution may carry a regional bias. We encourage future work to expand this in-situ collection methodology to more diverse global contexts. - **Scale vs. Quality Trade-off:** We deliberately prioritized meticulous, high-confidence ground-truth annotation over massive scale. This labor-intensive approach ensures data reliability but results in a smaller dataset than those created via automated web-scraping. # License - CC BY 4.0 — https://creativecommons.org/licenses/by/4.0/ # Citation Please cite the CountQA paper: ``` @article{tamarapalli2025countqa, title={CountQA: How Well Do MLLMs Count in the Wild?}, author={Tamarapalli, Jayant Sravan and Grover, Rynaa and Pande, Nilay and Yerramilli, Sahiti}, journal={arXiv preprint arXiv:2508.06585}, year={2025} } ```