"Tigr100k: A Benchmark for Graded Relevance and Fine-Grained Text-to-Image Retrieval in Compositional Queries"

"Despite the remarkable success of CLIP models in text-to-image retrieval, their performance remains suboptimal when confronted with complex queries encompassing multiple subjects, attributes, or relations. Existing benchmarks and prevailing metrics, like R@k, are fundamentally insufficient, as they treat relevance as a binary state and fail to capture the nuanced ranking quality required for complex search results.To address this critical evaluative gap, we introduce Tigr100k, a novel and robust benchmark dataset specifically designed for rigorous complex query evaluation. Tigr100k surpasses existing resources by comprising 1,044 bilingual complex query pairs and images annotated with fine-grained, multi-level relevance grading (highly, moderately, and irrelevant). Crucially, we couple Tigr100k with hierarchical evaluation metrics that enable a comprehensive assessment of both result coverage and ranking quality.Extensive experiments across multiple CLIP variants using CsdCLIP not only demonstrate significant performance gains, but also confirm the critical value of Tigr100k and its metrics in discerning and accurately measuring true compositional capabilities."