xyxy01/ECom-RF-IMMR

# ECom-RF-IMMR and Adapted Public Benchmarks This repository contains four image-to-multimodal item retrieval datasets: - `1_ECom_RF_IMMR_Normal` - `2_ECom_RF_IMMR_Mosaic` - `3_eSSPR` - `4_LookBench` The first two datasets are our constructed ECom-RF-IMMR evaluation datasets. The last two datasets are adapted from public e-commerce retrieval benchmarks. These datasets are designed for **image-to-multimodal item retrieval (IMMR)**, where the query is an image region and each candidate item is represented by an item image and structured item text. This setting reflects practical e-commerce visual search, where a localized product query needs to retrieve item candidates that may appear in full images with background context, multiple objects, or visual distractors. ## Dataset Overview | Dataset | Source | Query Image | Item Image | Box Annotation | Text Annotation | Main Usage | |---|---|---|---|---|---|---| | `1_ECom_RF_IMMR_Normal` | Ours | Original product image | Original product image | Query box and item box | Item title | Clean item retrieval | | `2_ECom_RF_IMMR_Mosaic` | Ours | Original product image | Mosaic image with distractors | Query box and item box | Item title | Cluttered item retrieval | | `3_eSSPR` | Public dataset adapted to IMMR | Product image | Product image | Generated by detector and VLM filtering | Translated item title | Public benchmark evaluation | | `4_LookBench` | Public dataset adapted to IMMR | Street-look or fashion image | Product image | Original box annotation | Generated and translated item title | Noisy and one-to-many retrieval | ## Directory Structure ```text . ├── 1_ECom_RF_IMMR_Normal/ │ ├── dataset.jsonl │ └── img/ │ ├── 0000/ │ │ ├── 00000000_query.jpg │ │ ├── 00000000_item.jpg │ │ └── ... │ └── ... │ ├── 2_ECom_RF_IMMR_Mosaic/ │ ├── dataset.jsonl │ └── img/ │ ├── 0000/ │ │ ├── 00000000_query.jpg │ │ ├── 00000000_item.jpg │ │ └── ... │ └── ... │ ├── 3_eSSPR/ │ ├── esspr.jsonl │ └── img/ │ ├── query/ │ └── item/ │ └── 4_LookBench/ ├── aigen_streetlook.jsonl ├── aigen_studio.jsonl ├── real_streetlook.jsonl ├── real_studio_flat.jsonl ├── look_bench_noise.jsonl └── img/ ├── query/ ├── item/ └── noise/ ``` All image paths in the annotation files are relative paths from the parent directory that contains the four dataset folders. ## Task Definition Given a query image and a query box, the task is to retrieve the matched item from a candidate pool. Each item candidate is represented by an item image and an item title. The box annotations indicate the target region in the query image and item image. The box format is: ```text [x1, y1, x2, y2] ``` where `(x1, y1)` and `(x2, y2)` are the top-left and bottom-right coordinates in pixel space. ## 1. ECom-RF-IMMR-Normal `1_ECom_RF_IMMR_Normal` is our clean evaluation dataset. Both query and item images are original product images. Each sample contains a query-side target box, an item-side target box, and an item title. Only the following five fields are retained: | Field | Type | Description | |---|---|---| | `query_path` | string | Relative path to the query image | | `query_box` | list[float] | Target box in the query image | | `item_path` | string | Relative path to the item image | | `item_box` | list[float] | Target box in the item image | | `item_title` | string | Item title text | Images are stored under `1_ECom_RF_IMMR_Normal/img/`. To avoid having too many images in a single folder, images are split into subfolders. Each query and item image is renamed according to the reading order, padded to eight digits, with `_query` or `_item` as the suffix. Example: ```json { "query_path": "1_ECom_RF_IMMR_Normal/img/0000/00000000_query.jpg", "query_box": [88.277435, 311.30823, 305.92258, 618.63495], "item_path": "1_ECom_RF_IMMR_Normal/img/0000/00000000_item.jpg", "item_box": [74.85748, 191.0392, 426.9335, 479.08618], "item_title": "春款0-1岁宝宝鞋 爱心绣花蝴蝶结公主鞋 软底防滑防掉舒适婴儿鞋" } ``` ## 2. ECom-RF-IMMR-Mosaic `2_ECom_RF_IMMR_Mosaic` is our cluttered evaluation dataset. The query image is the original product image, while the item image is a synthesized Mosaic image containing the target item together with additional distractor items. This dataset evaluates retrieval robustness under complex multi-item layouts. The query image, query box, and item title are kept consistent with the corresponding Normal sample, while the item image is reconstructed as a cluttered scene. Only the following five fields are retained: | Field | Type | Description | |---|---|---| | `query_path` | string | Relative path to the query image | | `query_box` | list[float] | Target box in the query image | | `item_path` | string | Relative path to the Mosaic item image | | `item_box` | list[float] | Target box of the item in the Mosaic image | | `item_title` | string | Item title text | Example: ```json { "query_path": "2_ECom_RF_IMMR_Mosaic/img/0000/00000000_query.jpg", "query_box": [88.277435, 311.30823, 305.92258, 618.63495], "item_path": "2_ECom_RF_IMMR_Mosaic/img/0000/00000000_item.jpg", "item_box": [51.0, 60.0, 172.0, 159.0], "item_title": "春款0-1岁宝宝鞋 爱心绣花蝴蝶结公主鞋 软底防滑防掉舒适婴儿鞋" } ``` ## 3. Adapted eSSPR `3_eSSPR` is adapted from the public eSSPR dataset for the IMMR setting. Duplicate or problematic samples are filtered. Since the original dataset does not provide box annotations, we first use an object detection model to generate candidate boxes, and then use Qwen-VL-Plus to select the box corresponding to the main product. Item titles are translated using DeepSeek. Annotation file: ```text 3_eSSPR/esspr.jsonl ``` Fields: | Field | Type | Description | |---|---|---| | `query_path` | string | Relative path to the query image | | `query_box` | list[int] | Generated target box in the query image | | `query_title` | string | Query-side title | | `item_id` | string | Item identifier | | `item_path` | string | Relative path to the item image | | `item_box` | list[int] | Generated target box in the item image | | `item_catename` | string | Item category name | | `item_title` | string | Translated item title | Example: ```json { "query_path": "3_eSSPR/img/query/H1701c702fb7847a99b53f5438b3042d8e.jpg", "query_box": [392, 93, 749, 703], "query_title": "NS040 亮片羽毛两件套裙装 女装裙 夏季性感夜店连衣裙", "item_id": "85777126353880", "item_path": "3_eSSPR/img/item/Hfe78da61ff4f4f43816161eba56da0b5L.jpg", "item_box": [186, 159, 501, 746], "item_catename": "服装#休闲连衣裙", "item_title": "女士蕾丝羽毛短袖休闲显瘦连衣裙时尚" } ``` ## 4. Adapted LookBench `4_LookBench` is adapted from the public LookBench dataset. The original dataset provides box annotations. For each item, we summarize the category, main attribute, and other attributes into an item title using DeepSeek, and then translate the generated title using Qwen3-VL-32B. LookBench contains four main evaluation subsets: | File | Description | |---|---| | `aigen_streetlook.jsonl` | AI-generated street-look subset | | `aigen_studio.jsonl` | AI-generated studio subset | | `real_streetlook.jsonl` | Real street-look subset | | `real_studio_flat.jsonl` | Real studio-flat subset | Each sample may contain multiple matched items under the `item` field, making this dataset suitable for one-to-many retrieval evaluation. Fields: | Field | Type | Description | |---|---|---| | `query_path` | string | Relative path to the query image | | `query_box` | list[int] | Target box in the query image | | `query_title` | string | Generated query-side title | | `item` | list[dict] | List of matched item candidates | Each element in `item` contains: | Field | Type | Description | |---|---|---| | `item_id` | int | Item identifier within the sample | | `item_path` | string | Relative path to the item image | | `item_box` | list[int] | Target box in the item image | | `item_catename` | string | Item category name | | `item_title` | string | Generated and translated item title | Example: ```json { "query_path": "4_LookBench/img/query/raw_image_1bdb8f5d15ae4600a9bee93965270474.jpg", "query_box": [717, 468, 928, 674], "query_title": "鳄鱼皮 皮革,印花,贴花 包袋", "item": [ { "item_id": 0, "item_path": "4_LookBench/img/item/raw_image_eb689d3192b94dfb80ddee5b7ee7c142.jpg", "item_box": [0, 14, 225, 189], "item_catename": "手提包", "item_title": "新款时尚鳄鱼纹皮革手提包 通勤百搭大容量轻奢女包" } ] } ``` ### Noise Items LookBench also includes noise candidates. These samples do not contain query fields or box annotations, and are used as distractor item candidates. Annotation file: ```text 4_LookBench/look_bench_noise.jsonl ``` Fields: | Field | Type | Description | |---|---|---| | `item_path` | string | Relative path to the noise item image | | `item_title` | string | Generated item title | Example: ```json { "item_path": "4_LookBench/img/noise/image_267c9f0b100d4f2fac9170805439007e.jpg", "item_title": "新款复古印花挂脖连衣裙女时尚百搭显瘦A字裙" } ``` ## Loading Example ```python import json from pathlib import Path def load_jsonl(path): samples = [] with open(path, "r", encoding="utf-8") as f: for line in f: line = line.strip() if line: samples.append(json.loads(line)) return samples root = Path("/path/to/multimodel_datasets") normal_samples = load_jsonl(root / "1_ECom_RF_IMMR_Normal" / "dataset.jsonl") mosaic_samples = load_jsonl(root / "2_ECom_RF_IMMR_Mosaic" / "dataset.jsonl") esspr_samples = load_jsonl(root / "3_eSSPR" / "esspr.jsonl") lookbench_files = [ "aigen_streetlook.jsonl", "aigen_studio.jsonl", "real_streetlook.jsonl", "real_studio_flat.jsonl", ] lookbench_samples = [] for filename in lookbench_files: lookbench_samples.extend(load_jsonl(root / "4_LookBench" / filename)) lookbench_noise = load_jsonl(root / "4_LookBench" / "look_bench_noise.jsonl") print(normal_samples[0]) print(mosaic_samples[0]) print(esspr_samples[0]) print(lookbench_samples[0]) print(lookbench_noise[0]) ``` ## Evaluation Protocol For `1_ECom_RF_IMMR_Normal`, `2_ECom_RF_IMMR_Mosaic`, and `3_eSSPR`, each query is associated with a matched item. Standard retrieval metrics such as Recall@K, MRR@K, and NDCG@K can be used. For `4_LookBench`, each query may be associated with multiple matched items. In addition to Recall@K, MRR@K, and NDCG@K, HitRate@K can also be used for one-to-many retrieval evaluation. ## Notes - `query_path` and `item_path` are relative paths. - `query_box` and `item_box` use `[x1, y1, x2, y2]` pixel coordinates. - For the two ECom-RF-IMMR datasets, only five fields are kept: `query_path`, `query_box`, `item_path`, `item_box`, and `item_title`. - For adapted public datasets, additional fields such as `query_title`, `item_id`, and `item_catename` are retained when available. - Item-side boxes are provided for dataset annotation, supervision, and evaluation. Retrieval models may choose to encode only the full item image and item title during indexing and retrieval. - Noise items in LookBench are used as distractor candidates and do not contain query-side annotations. ## Download and Extraction The image files are stored as tar shards under the `img_shards/` directory of each dataset folder. This is only for upload and storage convenience. After extraction, the directory structure will be restored to the same layout used by the annotation files. After downloading this repository from Hugging Face, enter the repository root directory, which should contain the following dataset folders: ```text 1_ECom_RF_IMMR_Normal/ 2_ECom_RF_IMMR_Mosaic/ 3_eSSPR/ 4_LookBench/ README.md ``` Then run: ```bash find . -path "*/img_shards/*.tar" -print0 | sort -z | xargs -0 -I{} tar -xf "{}" ``` Each tar shard already stores files with dataset-level relative paths, such as: ```text 1_ECom_RF_IMMR_Normal/img/0000/00000000_query.jpg 2_ECom_RF_IMMR_Mosaic/img/0000/00000000_item.jpg 3_eSSPR/img/query/example.jpg 4_LookBench/img/item/example.jpg ``` Therefore, extracting all shards from the repository root will restore the expected image directories directly. After extraction, the dataset structure should be: ```text . ├── 1_ECom_RF_IMMR_Normal/ │ ├── dataset.jsonl │ ├── img/ │ └── img_shards/ ├── 2_ECom_RF_IMMR_Mosaic/ │ ├── dataset.jsonl │ ├── img/ │ └── img_shards/ ├── 3_eSSPR/ │ ├── esspr.jsonl │ ├── img/ │ └── img_shards/ └── 4_LookBench/ ├── aigen_streetlook.jsonl ├── aigen_studio.jsonl ├── real_streetlook.jsonl ├── real_studio_flat.jsonl ├── look_bench_noise.jsonl ├── img/ └── img_shards/ ``` The paths in all annotation files are relative to the repository root. For example, a path such as: ```text 1_ECom_RF_IMMR_Normal/img/0000/00000000_query.jpg ``` can be opened as: ```python from pathlib import Path root = Path("/path/to/downloaded/repository") image_path = root / "1_ECom_RF_IMMR_Normal/img/0000/00000000_query.jpg" ``` The `img_shards/` directories can be kept for reproducibility or removed after successful extraction. ## Citation If you use this dataset or find it helpful for your research, please cite: ```bibtex @article{sun2026tiger, title={TIGER-FG: Text-Guided Implicit Fine-Grained Grounding for E-commerce Retrieval}, author={Sun, Xinyu and Dai, Huangyu and Mao, Lingtao and Zheng, Zexin and Liang, Zihan and Chen, Ben and Lei, Chenyi and Ou, Wenwu}, journal={arXiv preprint arXiv:2605.18434}, year={2026} }This dataset comprises four sub-datasets designed for the Image-to-Multimodal Item Retrieval (IMMR) task, which evaluates retrieval performance in e-commerce visual search scenarios. In IMMR, the query is an image region, and each candidate item is represented by an item image and structured item text (e.g., item title), simulating real-world e-commerce search where localized product queries retrieve items from full images. The sub-datasets include: 1) ECom-RF-IMMR-Normal: a clean evaluation dataset constructed in-house, with both query and item images as original product images, containing query boxes, item boxes, and item titles; 2) ECom-RF-IMMR-Mosaic: a cluttered evaluation dataset constructed in-house, where query images are original product images and item images are synthesized mosaic images with distractors, designed to assess retrieval robustness under complex multi-item layouts; 3) Adapted eSSPR: adapted from the public eSSPR dataset, with box annotations generated via object detection and visual language model filtering, and item titles translated; 4) Adapted LookBench: adapted from the public LookBench dataset, containing real and AI-generated street-look or fashion images, supporting one-to-many retrieval evaluation and including noise candidates. All datasets provide annotation files in JSONL format with fields such as query path, box coordinates (in [x1, y1, x2, y2] format), item path, and text titles, suitable for evaluation using standard retrieval metrics like Recall@K, MRR@K, and NDCG@K.

搜集汇总

数据集介绍

构建方式

在电子商务视觉搜索场景中，用户往往以局部商品区域为查询意图，而候选商品则可能嵌于含有背景干扰或多目标并存的完整图像内，这对检索系统提出了更高的鲁棒性要求。ECom-RF-IMMR数据集围绕图像到多模态商品检索任务构建，包含两个自建评测集与两个改编自公开基准的评测集。自建部分以原始商品图像为基础，保留查询框、商品框与商品标题五元组，并通过图像拼接合成策略将目标商品与干扰项重组为Mosaic图像，形成干净检索与杂乱检索两种对应场景。改编部分则对eSSPR数据集引入目标检测模型生成候选框并以Qwen-VL-Plus筛选主商品框，对LookBench数据集借助DeepSeek与Qwen3-VL-32B生成并翻译商品标题，从而将公开基准统一至IMMR任务范式。

特点

该数据集的核心特点在于其多层次的检索难度设计与丰富的标注结构。ECom-RF-IMMR-Normal与ECom-RF-IMMR-Mosaic共享相同的查询图像、查询框与商品标题，仅商品图像从原始单品图转变为包含额外干扰项的Mosaic合成图，由此形成可控变量的鲁棒性评测条件。改编自公开基准的eSSPR与LookBench则拓展了数据来源的多样性，其中LookBench涵盖AI生成街拍、AI生成影棚、真实街拍与真实影棚平铺四个子集，并支持一对多检索评估，同时提供无查询标注的噪声候选商品以增强检索区分难度。全部数据采用统一JSONL格式存储，图像路径均为相对路径，字段命名一致，便于跨子集联合实验与标准化评测。

使用方法

使用时，研究者需从Hugging Face仓库根目录执行tar解压命令还原图像目录，确保标注文件中的相对路径可正确解析。对于1_ECom_RF_IMMR_Normal、2_ECom_RF_IMMR_Mosaic与3_eSSPR，每条查询对应单一匹配商品，可采用Recall@K、MRR@K与NDCG@K等标准检索指标进行评估；对于4_LookBench，每条查询可能对应多个匹配商品，除上述指标外还可引入HitRate@K以适配一对多检索场景。加载时通过Python的json模块逐行读取JSONL文件即可获得样本列表，查询框与商品框均以[x1, y1, x2, y2]像素坐标表示。检索模型在索引阶段可选择仅编码完整商品图像与商品标题，商品侧边界框则供标注、监督与评测环节使用。LookBench中的噪声商品文件不含查询字段与框标注，仅作为干扰候选参与检索池构建。

背景与挑战

背景概述

随着电子商务视觉搜索技术的迅猛发展，图像到多模态商品检索（Image-to-Multimodal Item Retrieval, IMMR）逐渐成为连接视觉感知与语义理解的关键任务。ECom-RF-IMMR数据集正是面向这一前沿方向而构建，由研究团队精心设计并发布于HuggingFace平台，旨在模拟真实电商场景中以局部图像区域为查询、检索兼具商品图像与结构化文本描述的多模态候选商品。该数据集不仅包含自建的Normal与Mosaic评测子集，还融合了eSSPR与LookBench两个公开基准的适配版本，形成了覆盖干净检索、杂乱场景检索、噪声干扰以及一对多检索等多元评测体系，为电商视觉检索领域提供了兼具真实性与挑战性的基准资源，对推动多模态检索模型的鲁棒性与泛化能力研究具有重要影响。

当前挑战

该数据集所应对的核心领域问题在于：如何在查询仅为图像局部区域且候选商品以图像与文本联合表征的条件下，实现精准且鲁棒的跨模态商品匹配，此问题相较传统图像分类或整体图像检索更具复杂性与现实意义。构建过程中亦面临多重挑战，包括Mosaic合成图像需在保留目标商品语义完整性的同时引入合理干扰项以模拟杂乱货架场景；eSSPR原始数据缺失边界框标注，需依赖目标检测模型生成候选框并借助视觉语言模型进行筛选与验证；LookBench适配则涉及将类别与属性信息通过大语言模型归纳为商品标题并进行多语言翻译，同时处理一对多匹配与噪声候选带来的标注歧义。上述挑战共同构成了该数据集在标注质量、场景多样性与评测公平性方面的核心难点。

常用场景

经典使用场景

在电子商务视觉搜索的典型范式中，图像到多模态商品检索（IMMR）要求系统以查询图像中的局部区域为输入，从候选池中精准定位并召回相匹配的商品条目，而每个条目均由商品图像与结构化文本标题共同表征。ECom-RF-IMMR数据集正是围绕这一核心任务构建，其Normal子集提供原始商品图像及双端边界框标注，Mosaic子集则在商品图像中合成多物品干扰场景以模拟复杂布局，从而为区域级跨模态检索模型的训练与评测提供了标准化的实验场。

衍生相关工作

围绕该数据集的发布，相关研究衍生出若干值得关注的后续工作，包括面向区域级图像与文本联合嵌入的检索模型改进、针对杂乱场景的鲁棒性增强方法、以及在LookBench一对多设定下的一对多检索与去噪策略探索。这些工作进一步拓展了IMMR任务在公开基准上的可比性，并为跨域迁移、噪声候选过滤与多模态预训练等方向提供了新的实验基础与评测参照。

数据集最近研究

最新研究方向

在电子商务视觉搜索向多模态检索范式演进的背景下，ECom-RF-IMMR数据集聚焦于图像区域到多模态商品检索的前沿问题，即查询为图像局部区域，候选商品由商品图像与结构化文本共同表征。当前研究正从单一干净商品图像检索转向复杂场景下的鲁棒性评测，涵盖杂波干扰、多目标布局以及跨域公开基准适配等方向。该数据集通过Normal与Mosaic双版本构建，系统考察模型在背景上下文、多物体与视觉干扰物共现条件下的检索稳定性；同时，eSSPR与LookBench的适配引入检测器与视觉语言模型联合标注流程，并支持一对多检索与噪声候选评估，为电商视觉搜索中的真实应用瓶颈提供了可复现的评测基准，推动了多模态检索在开放场景下的可扩展性与泛化能力研究。