tiiuae/PBench

--- task_categories: - image-segmentation language: - en tags: - referring-expression-segmentation - perception - segmentation - benchmark pretty_name: PBench size_categories: - 1K<n<10K dataset_info: features: - name: image dtype: image - name: expression dtype: string - name: masks list: - name: size list: int64 - name: counts dtype: string - name: count dtype: int32 - name: id dtype: int64 splits: - name: level_0 num_bytes: 5534494360 num_examples: 1930 - name: level_1 num_bytes: 2783412551 num_examples: 1108 - name: level_2 num_bytes: 3352253137 num_examples: 969 - name: level_3 num_bytes: 3609353809 num_examples: 1089 - name: level_4 num_bytes: 2751869956 num_examples: 861 - name: dense num_bytes: 677685577 num_examples: 381 download_size: 18678114659 dataset_size: 18709069390 configs: - config_name: default data_files: - split: level_0 path: data/level_0-* - split: level_1 path: data/level_1-* - split: level_2 path: data/level_2-* - split: level_3 path: data/level_3-* - split: level_4 path: data/level_4-* - split: dense path: data/dense-* --- # PBench: A Perception Benchmark for Referring Expression Segmentation Referring expression segmentation is increasingly becoming a core building block for real products and real workflows. People want to select objects using language inside creative tools, robots need to ground instructions like "pick up the blue wrench" to pixels, and video analytics systems are starting to support natural language queries like "track the red van" instead of fixed class lists. As models improve, a simple frustration remains: a single aggregate score can look good while hiding very different failure modes. Some models handle clean object categories well but struggle when the expression depends on reading text. Others can follow attributes but break on spatial layout. When those skills are averaged together, it becomes hard to see what a model has actually learned, and what it still misses. PBench is designed to make that diagnostic step easier. We propose a multi-level referring expression segmentation benchmark that evaluates vision-language perception across a structured hierarchy of skills. PBench assigns each sample to a **single complexity level** by construction, so you can measure performance per skill and compare models in a way that is easy to interpret. This release contains **6,338 samples** with **83,977 instance masks** across **5,090 unique expressions**, organized into five levels plus a dense split. ## What PBench measures PBench organizes expressions into five levels plus a dense split. Each level is meant to isolate a primary perceptual capability. By design, a sample belongs to exactly one level, so a per-level score is a closer proxy for the underlying skill than an aggregate score over mixed phenomena. <p align="center"> <img src="level_progression.png" width="90%"/> </p> ### Level 0: General object classes The foundation level tests basic object recognition and mask quality on common object categories. Expressions are short noun phrases like `car`, `person`, `tree`. The average expression length is 1.3 words. What typically breaks here is not language, but vision: boundary quality, small instances, and partial occlusion. ### Level 1: Fine-grained attributes and subtypes Level 1 adds descriptive detail that forces the model to use attributes or subtype distinctions. Expressions include properties (color, size, material), conditions (old, broken, dirty), subtypes (sedan vs SUV), states (open door), and components (cracked windshield). Typical expressions look like `red car` or `dirty white pickup truck`. The average expression length is 3.9 words. These samples often fail when the attribute is subtle, the object is partially visible, or multiple candidates share most attributes. ### Level 2: Text as identifier (OCR) Level 2 tests whether a model can use in-image text to identify or disambiguate an object. Expressions reference brands and product variants (`Diet Coke`), store-specific items (`Starbucks coffee cup`), or visible signage (`Emergency exit door`). The average expression length is 3.7 words. The key failure mode is simple: if the model does not read the text reliably, it will often choose a plausible but wrong instance. ### Level 3: Spatial relationships and layout Level 3 focuses on spatial reasoning. Expressions specify objects by relative position or scene layout: `car on the left`, `bird above the tree`, `third window from left`, `people inside the building`. This level has the longest expressions on average at 6.7 words, which mostly reflects how people naturally describe spatial constraints. Models often struggle when the reference frame is ambiguous (left of what), when depth cues are subtle, or when small spatial errors change the identity of the target. ### Level 4: Relationships and interactions Level 4 targets relational understanding between objects. Expressions describe actions (`person holding umbrella`), functional links (`key for door`), comparisons (`tallest building in the row`), and physical interactions (`book resting on table`). The average expression length is 4.6 words. These samples are often hard because the correct mask depends on understanding who is interacting with what, not just what objects are present. ### Dense split: Dense instance segmentation The dense split uses simple object-class expressions similar to Level 0, but in visually crowded scenes containing many instances of the same class. With an average of 181 masks per sample (up to 679), this split stress-tests whether a model can segment instances exhaustively rather than picking a single easy match. ## Dataset Statistics | Split | Capability | Samples | Total Masks | Unique Expressions | |---|---|---|---|---| | level_0 | General object classes | 1,930 | 7,543 | 939 | | level_1 | Attributes & subtypes | 1,108 | 2,740 | 1,085 | | level_2 | Text / OCR | 969 | 1,361 | 954 | | level_3 | Spatial relationships | 1,089 | 1,751 | 1,083 | | level_4 | Relationships & interactions | 861 | 1,731 | 839 | | dense | Dense instance segmentation | 381 | 68,851 | 264 | | **Total** | | **6,338** | **83,977** | **5,090** | ## Schema Each sample contains the following fields: | Field | Type | Description | |---|---|---| | `id` | `int` | Global ID in range 0 to 6,634 (not necessarily contiguous) | | `image` | `Image` | RGB image (original resolution) | | `expression` | `string` | Referring expression | | `masks` | `[{size: [H,W], counts: str}]` | List of COCO RLE segmentation masks | | `count` | `int` | Number of masks for this expression | ## Usage ```python from datasets import load_dataset ds = load_dataset("tiiuae/PBench") # Access a specific level sample = ds["level_0"][0] print(sample["expression"]) # e.g. "car" print(sample["count"]) # number of masks # Decode masks with pycocotools from pycocotools import mask as mask_utils import numpy as np for m in sample["masks"]: rle = {"size": m["size"], "counts": m["counts"].encode("utf-8")} binary_mask = mask_utils.decode(rle) # H x W numpy array (0/1) ``` ## Notes and limitations - **Ambiguity is real**: some expressions can be underspecified without additional context. We keep the level assignment focused on the dominant skill, but natural language still has edge cases. - **OCR noise**: for Level 2, visual text quality, font, and occlusion can dominate difficulty, even when the rest of the scene is simple. - **Spatial reference frames**: for Level 3, phrases like left, right, and in front of can be sensitive to camera viewpoint and to what the model treats as the reference object. ## Citation If you use PBench in your work, please cite: ```bibtex @article{bevli2026falcon, title = {Falcon Perception}, author = {Bevli, Aviraj and Chaybouti, Sofian and Dahou, Yasser and Hacid, Hakim and Huynh, Ngoc Dung and Le Khac, Phuc H. and Narayan, Sanath and Para, Wamiq Reyaz and Singh, Ankit}, journal = {arXiv preprint arXiv:2603.27365}, year = {2026}, url = {https://arxiv.org/abs/2603.27365} } ```PBench is a perception benchmark for referring expression segmentation, designed to evaluate vision-language perception across a structured hierarchy of skills. The dataset contains 6,338 samples with 83,977 instance masks and 5,090 unique expressions, organized into five levels plus a dense split: Level 0 (general object classes, testing basic object recognition and mask quality), Level 1 (fine-grained attributes and subtypes, testing attribute usage), Level 2 (text as identifier, testing OCR capability), Level 3 (spatial relationships and layout, testing spatial reasoning), Level 4 (relationships and interactions, testing inter-object understanding), and the dense split (dense instance segmentation, testing segmentation in crowded scenes). Each sample includes fields such as image, referring expression, list of masks, and mask count, making it suitable for image segmentation and natural language processing research.

搜集汇总

数据集介绍

构建方式

在指代表达分割日益成为真实产品与工作流核心构件的背景下，PBench 的构建遵循按主导感知技能归类的原则。该基准将每个样本分配至单一复杂度层级，形成 level_0 至 level_4 与 dense 六个划分，分别对应通用物体类别、细粒度属性与子类、文本或 OCR 识别、空间关系与布局、对象间关系与交互，以及拥挤场景中的稠密实例分割。数据包含 6,338 个样本、83,977 个实例掩码与 5,090 条唯一表达式；图像、表达式与 COCO RLE 掩码等字段共同支撑可复现的层级化评测。

特点

从基准诊断能力来看，PBench 的显著特点在于层级化与单归属设计，使各划分得分更接近特定感知技能的代理指标，而非混合现象的聚合结果。其表达式长度随层级变化，Level 3 平均 6.7 词，Level 0 平均 1.3 词；dense 划分则以平均 181 个掩码、最高 679 个实例的拥挤场景，检验模型能否穷尽式分割同类对象。各划分样本量从 381 至 1,930 不等，掩码总量达 83,977，兼顾多项技能的可解释比较与密集实例分割的压力测试。

使用方法

围绕可复现评测流程，PBench 通过 Hugging Face datasets 接口提供直接访问。研究者可调用 load_dataset('tiiuae/PBench') 载入数据，并按 level_0、level_1、level_2、level_3、level_4 或 dense 选择特定划分；读取样本的 expression、count 与 masks 字段后，借助 pycocotools 的 mask_utils.decode 将 COCO RLE 计数解码为 H×W 二值掩码，其中 counts 需编码为 UTF-8 字节串。该流程支持逐层评估模型在物体识别、属性、文本、空间与关系等技能上的表现，并开展模型间诊断比较。

背景与挑战

背景概述

Referring expression segmentation 正从学术评测走向创作工具、机器人与视频分析等真实工作流，语言指代需被精确锚定至像素级实例。PBench 由 TII 团队提出，Aviraj Bevli 等研究者于 2026 年随 Falcon Perception 预印本发布，托管于 tiiuae/PBench，包含 6,338 个样本、83,977 个实例掩码与 5,090 个独特表达式。其核心问题是聚合分数难以揭示模型在对象识别、属性辨析、文本读取、空间推理、交互理解和密集分割中的差异化失效；该基准以五级加密集划分的结构化设计，将每个样本按主导感知能力归类，从而为视觉语言感知提供可解释的诊断框架，并对指代分割评测范式产生推进作用。

当前挑战

PBench 所面对的领域难题在于，指代分割要求模型将自然语言表达式准确映射为像素掩码，同时协调对象类别、细粒度属性、图像内文字、空间布局、对象交互与密集实例等多重能力；传统单一聚合分数易掩盖这些异构技能间的失败模式。构建层面上，样本需依主导技能归入唯一层级，自然语言欠指定与歧义、OCR 字体遮挡噪声、左右前后等参照系对视角的敏感性，均增加层级划分与标注一致性难度；密集划分又以平均 181 个、最高 679 个同类实例掩码考验穷举式实例分割，显著提升标注、存储与评测的复杂度。

常用场景

经典使用场景

在指代表达式分割日益成为视觉语言感知基础设施的语境下，PBench最经典的使用方式，是围绕六级划分开展分层能力诊断与模型比较。研究者将模型分别置于level_0至level_4以及dense split，观察其在一般类别、细粒度属性、图像文本OCR、空间关系、对象交互和密集实例分割上的表现。由于每个样本仅归入一个复杂度层级，逐层得分能够揭示模型究竟掌握何种感知能力，而聚合指标则可能掩盖属性辨识、文本读取或空间推理等专项失败。该用法尤其适合检验模型边界质量、小目标与遮挡处理，以及多实例穷尽分割能力。

实际应用

在真实产品与工作流中，PBench对应语言驱动对象选择、机器人指令落地和视频分析自然语言查询等应用。创意工具需要用户以“红色汽车”或“紧急出口门”选择图层，机器人需将“拿起蓝色扳手”映射至像素，视频系统需追踪“左侧车辆”或“持伞的人”。PBench的分层样本可评估这些系统在属性、OCR、空间关系与交互理解中的可靠性，dense split则检验拥挤场景下实例分割的完备性。其结构化结果有助于识别部署风险，指导模型在视觉grounding、交互式编辑与智能监控中的稳健优化。

衍生相关工作

围绕PBench的学术脉络，最直接的经典衍生工作来自其引用的Falcon Perception研究，该工作将PBench作为多层级感知评测基准，用于呈现视觉语言模型在指代表达式分割中的能力谱系。PBench的分层设计亦启发后续基准构建与模型诊断：研究者可据此开展逐技能消融、跨模型层级比较、OCR与空间推理专项分析，以及密集实例分割评测。其公开样本、RLE掩码与标准化split为可复现实验提供支撑，推动指代分割从单一聚合排名走向能力可解释、错误可定位的评测范式。