LensBench

# LensBench LensBench is a segmentation benchmark for explicit and implicit instructions across general, aerial, driving, and panoramic images. It contains **1,200 human-reviewed paired evaluation units and 4,800 instruction-level samples**. [Paper](https://arxiv.org/abs/2610.23279) · [Project page](https://seerray-lab.github.io/Xiaomi-Lens-0/) · [Code](https://github.com/SeerRay-Lab/Xiaomi-Lens-0) · [Model](https://huggingface.co/SeerRay-Lab/Xiaomi-Lens-0) ## Dataset design Each evaluation unit has four instructions crossing single/multiple targets with explicit/implicit wording. For a given target-cardinality condition, both wordings share the same target mask. Multiple targets are represented by their union mask. Targets and masks are fixed before instruction synthesis, followed by automated consistency checks and human review of each complete unit. | Domain | Paired units | Explicit / single | Explicit / multiple | Implicit / single | Implicit / multiple | Total samples | | --- | ---: | ---: | ---: | ---: | ---: | ---: | | General | 300 | 300 | 300 | 300 | 300 | 1,200 | | Aerial | 300 | 300 | 300 | 300 | 300 | 1,200 | | Driving | 300 | 300 | 300 | 300 | 300 | 1,200 | | Panoramic | 300 | 300 | 300 | 300 | 300 | 1,200 | | **Total** | **1,200** | **1,200** | **1,200** | **1,200** | **1,200** | **4,800** | The release provides a single `test` split for evaluation. ## Files and fields ```text LensBench/ ├── README.md └── test.parquet ``` Images are embedded in the Parquet file; no separate image download is needed. | Field | Type | Description | | --- | --- | --- | | `index` | int64 | Unique instruction-level sample index; not a paired-unit identifier. | | `image` | Hugging Face Image | Embedded image bytes and original filename; decoded to a PIL image by `datasets`. | | `instruction` | string | English instruction describing the segmentation target. | | `segmentation` | struct | COCO compressed RLE with `size` and `counts`. | | `segmentation.size` | list of int32 | Mask dimensions in `[height, width]` order. | | `segmentation.counts` | string | Compressed COCO RLE counts; encode as ASCII bytes before decoding with `pycocotools`. | | `domain` | string | `general`, `aerial`, `driving`, or `panoramic`. | | `explicitness` | string | `explicit` or `implicit`. | | `cardinality` | string | `single` or `multiple`. | The release does not contain a bounding-box field. When a box is needed, derive it from the target mask; for multiple targets, use the enclosing box of the union mask. ## Quick start Install the loading and mask-decoding dependencies: ```bash pip install datasets pillow pycocotools huggingface_hub ``` Load the dataset and decode the first target mask: ```python from datasets import load_dataset from pycocotools import mask as mask_utils dataset = load_dataset("SeerRay-Lab/LensBench", split="test") sample = dataset[0] image = sample["image"] rle = { "size": sample["segmentation"]["size"], "counts": sample["segmentation"]["counts"].encode("ascii"), } mask = mask_utils.decode(rle) assert mask.shape == (image.height, image.width) print(sample["instruction"]) print(sample["domain"], sample["explicitness"], sample["cardinality"]) print(image.size, mask.shape) ``` To download the release and load it locally: ```bash hf download SeerRay-Lab/LensBench --repo-type dataset --local-dir LensBench ``` ```python from datasets import load_dataset dataset = load_dataset( "parquet", data_files={"test": "LensBench/test.parquet"}, split="test" ) implicit_samples = dataset.filter(lambda sample: sample["explicitness"] == "implicit") ``` ## Leaderboard The original baseline results are reported in Table 7 in Section 4.3 of the [Xiaomi-Lens-0 Technical Report (arXiv v1)](https://arxiv.org/html/2610.23279v1#S4.T7). The table below also includes the subsequently added GPT-6-Astra results and is synchronized with the [project results](https://seerray-lab.github.io/Xiaomi-Lens-0/#data). Average reports the arithmetic mean of each metric across General, Aerial, Driving, and Panoramic; Explicit and Implicit are excluded from this calculation. Rows are sorted by unrounded average cIoU, then average Prec@0.5, in descending order; exact ties retain their original order. Averages are displayed to one decimal place. Marginal results pool single and multiple targets. Each cell reports **cIoU / Prec@0.5** (%, higher is better). Bold marks the best score, and underlining marks the second-best score for each metric and column, including ties. | Method | Overall | Explicit | Implicit | General | Aerial | Driving | Panoramic | |:--| :-: | :-: | :-: | :-: | :-: | :-: | :-: | | GPT-6-Astra | **71.8** / **85.1** | **72.6** / **86.3** | **71.0** / **84.0** | 74.1 / **86.2** | **61.7** / <u>76.9</u> | **77.6** / **89.0** | **68.9** / **88.3** | | Qwen3.8-Max | <u>66.0</u> / 66.5 | 67.6 / 69.2 | <u>64.5</u> / 63.8 | <u>74.2</u> / 79.7 | 33.5 / 62.5 | 58.0 / 57.7 | 60.8 / 66.2 | | **Xiaomi-Lens-0_4B** | 62.8 / 77.5 | <u>71.4</u> / 85.0 | 54.7 / 70.0 | 63.9 / 79.1 | <u>57.1</u> / **79.6** | 68.6 / 79.4 | 62.3 / 72.1 | | GPT-5.6-Sol | 62.5 / 71.6 | 63.8 / 73.2 | 61.2 / 70.0 | 68.4 / 67.9 | 42.2 / 58.5 | 68.4 / 81.8 | 57.1 / 78.3 | | Gemini-3.6-Flash | 61.2 / 72.4 | 61.4 / 74.5 | 61.0 / 70.3 | 63.6 / 65.8 | 50.1 / 62.5 | 66.8 / 81.0 | 60.8 / <u>80.3</u> | | Seed-2.1-Pro | 59.0 / 67.3 | 62.1 / 70.2 | 55.9 / 64.5 | 60.6 / 56.7 | 50.1 / 58.5 | 66.8 / 77.4 | 59.9 / 76.7 | | MiMo-V2.5 | 49.3 / 49.5 | 51.4 / 52.2 | 47.4 / 46.8 | 54.2 / 45.2 | 32.6 / 38.6 | 50.5 / 52.8 | 45.4 / 61.4 | | GLM-5.3-Flash | 43.9 / 35.4 | 44.3 / 37.5 | 43.5 / 33.3 | 48.9 / 26.3 | 28.0 / 33.4 | 51.6 / 47.5 | 30.6 / 34.4 | | WeVisOne-VL-2B | 42.0 / 76.9 | 41.4 / 84.5 | 42.8 / 69.2 | 73.9 / 78.9 | 11.6 / 71.3 | 73.3 / 83.9 | 52.7 / 73.3 | | DeepSeek-V4-Flash-Vision-Exp | 41.4 / 29.0 | 41.8 / 30.7 | 41.1 / 27.3 | 46.1 / 28.1 | 25.9 / 28.9 | 43.9 / 27.9 | 32.8 / 31.0 | | WeVisOne-VL-4B | 41.3 / <u>78.8</u> | 50.0 / <u>85.8</u> | 35.0 / <u>71.7</u> | **77.7** / <u>81.9</u> | 11.3 / 73.7 | <u>74.5</u> / <u>84.7</u> | <u>64.2</u> / 74.8 | | Kimi-K3 | 28.8 / 36.0 | 29.4 / 39.1 | 28.3 / 33.0 | 38.5 / 26.1 | 13.8 / 44.9 | 47.2 / 31.1 | 13.4 / 42.1 | | SAM 3 | 16.5 / 28.8 | 17.3 / 38.1 | 15.4 / 19.5 | 23.8 / 24.8 | 4.3 / 27.3 | 27.7 / 35.1 | 11.1 / 27.9 | GPT-6-Astra achieves the highest overall cIoU (71.8) and leads cIoU on explicit and implicit instructions and in aerial, driving, and panoramic imagery. Qwen3.8-Max ranks second in overall cIoU (66.0), while WeVisOne-VL-4B leads cIoU in general imagery (77.7). Xiaomi-Lens-0 ranks third overall (62.8), second in explicit-instruction and aerial cIoU, and first in aerial Prec@0.5 (79.6). Implicit instructions remain challenging: cIoU drops from 71.4 to 54.7, while Prec@0.5 drops from 85.0 to 70.0. The gap is largest for multiple targets in panoramic scenes, where cIoU falls from 71.8 to 48.9 and Prec@0.5 from 81.3 to 48.7. See the [paper](https://arxiv.org/abs/2610.23279) and [project results](https://seerray-lab.github.io/Xiaomi-Lens-0/#data) for the evaluation protocol and comparisons. Preserve the domain, explicitness, and cardinality labels when reporting subgroup results. ## License The images and annotations in this release are licensed under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0). ## Citation If you use LensBench, please cite: ```bibtex @misc{xiaomilens0, title = {Xiaomi-Lens-0 Technical Report}, author = {Pei Fu and Yike Liu and Yue Sun and Hanmo Wang and Xiuwen Xi and Ruoceng Zhang and Shaojie Zhang and Tingwei Huang and Chao Jiang and Hang Li and Bingrui Liu and Zhenbo Luo and Heng Qu and Changjian Xu and Ming Xu and Jian Luan}, year = {2026}, eprint = {2610.23279}, archivePrefix = {arXiv}, primaryClass = {cs.CV}, url = {https://arxiv.org/abs/2610.23279} } ```