CroQS Benchmark

CroQS (Cross-modal Query Suggestion) v1.0.0 is a benchmark dataset designed to evaluate methods that generate improved textual queries guided by visual results, in the context of text-to-image retrieval. The dataset supports the task of generating query suggestions grounded in visual content, specifically helping users refine or reformulate queries based on result set clusters. This version includes: 50 initial textual queries used to retrieve image sets from the MS COCO 2017 dataset. 295 manually defined semantic clusters, based on visual similarity or common properties in the image results. 8127 unique COCO images (referenced via URL, not redistributed directly). Each query result set is manually grouped into 2 to 10 clusters (average ~5.9 clusters per query). Each cluster is associated with a human-written query suggestion that describes the cluster's shared visual properties. CroQS can be used to train or evaluate models for: Query refinement and expansion Multimodal and cross-modal retrieval Cluster-based query suggestion Vision-language understanding For more details on the benchmark task and evaluation metrics, refer to our paper: Maybe You Are Looking for CroQS: Cross-modal Query Suggestion for Text-to-Image Retrieval.In Proceedings of the 46th European Conference on Information Retrieval (ECIR 2025).Springer Link | arXiv:2412.13834 Website: https://paciosoft.com/CroQS-benchmark