dianetc-OBLIQ-Bench

# OBLIQ-Bench **Exposing Overlooked Bottlenecks in Modern Retrievers with Latent and Implicit Queries** OBLIQ-Bench is a suite of five retrieval benchmarks designed to expose a blind spot in modern search systems: **oblique queries**, where the attributes that determine relevance are latent and have little or no surface expression in the document. Relevant documents are easy to *recognize* when paired with the query (a reasoning LLM can verify them) but extremely hard to *retrieve* from a large corpus using any current retrieval system. OBLIQ-Bench contains five tasks organized by three mechanisms of obliqueness: ### Descriptive Queries Queries seek a latent property that can be inferred from document content but is too nuanced for current retrieval representations. | Task | Corpus | Queries | Avg Positives/Query | Description | |---|---|---|---|---| | **Twitter-Conflict** | 72,122 tweets | 281 | 9.8 | Retrieve tweets expressing a given *implicit* stance on a geopolitical conflict — through irony, hedging, or selective framing — without explicit keywords. | | **WildChat Conversation Errors** | 507,729 conversations | 40 | 18.9 | Retrieve Human–AI conversations exhibiting a specific behavioral failure mode (e.g., formatting corruption the AI never self-corrects), where the failure is visible in the transcript but never acknowledged as a topic. | ### Analogue Queries Queries seek documents sharing a structural archetype with the query content, despite differing in surface topic. | Task | Corpus | Queries | Avg Positives/Query | Description | |---|---|---|---|---| | **Math Meta-Program** | 3,508 problems | 151 | 13.5 | Given a math problem, retrieve other problems requiring the *same abstract proof strategy* across different mathematical fields and notations. | | **Writing-Style** | 10,389 snippets | 512 | 9.0 | Given a text snippet, retrieve other snippets written by the *same author* across unrelated topics, relying on stylistic fingerprints rather than topical overlap. | ### Tip-of-Tongue Queries Queries match a fuzzy, impressionistic recollection to a specific obscure document. | Task | Corpus | Queries | Avg Positives/Query | Description | |---|---|---|---|---| | **Congress Hearings** | 213,650 passages | 254 | 1.0 | Match a hazy, Reddit-style recollection of a congressional hearing moment to the exact transcript passage — with no names, dates, or verbatim phrasing. | ## Repository Structure ``` OBLIQ-Bench/ ├── descriptive/ │ ├── twitter/ │ │ ├── corpus/ │ │ │ └── corpus.jsonl # 72,122 tweets │ │ └── queries+qrels/ │ │ ├── queries.jsonl # 281 queries │ │ ├── qrels.tsv # Gold relevance judgments │ │ └── qrels_pool.tsv # Expanded pooled judgments │ └── wildchat/ │ ├── corpus/ │ │ └── corpus.jsonl # 507,729 conversations │ └── queries+qrels/ │ ├── queries.jsonl # 40 queries │ ├── qrels.tsv │ └── qrels_pool.tsv ├── analogues/ │ ├── math/ │ │ ├── corpus/ │ │ │ └── corpus.jsonl # 3,508 math problems │ │ ├── queries+qrels/ │ │ │ ├── queries.jsonl # 151 queries │ │ │ ├── qrels.tsv │ │ │ ├── qrels_pool.tsv │ │ │ └── per_query_excluded_ids.json │ │ └── final_dataset.json │ └── writing/ │ ├── corpus/ │ │ └── corpus.jsonl # 10,389 snippets │ └── queries+qrels/ │ ├── queries.jsonl # 512 queries │ ├── qrels.tsv │ └── per_query_excluded_ids.json └── tip-of-tongue/ └── congress/ ├── corpus/ │ └── corpus.jsonl # 213,650 hearing passages └── queries+qrels/ ├── queries.jsonl # 254 queries └── qrels.tsv ``` ## File Formats **corpus.jsonl** — One document per line. Each line is a JSON object with at minimum an `_id` field and a `text` field. **queries.jsonl** — One query per line. Each line is a JSON object with a query `_id` and `text`, plus task-specific metadata. **qrels.tsv** — Tab-separated relevance judgments in standard TREC format: `query_id`, `corpus_id`, `score`. **qrels_pool.tsv** — Extended relevance judgments from pooled evaluation, where a reasoning model judged the top-k results from each retrieval system to expand the set of known positives. **per_query_excluded_ids.json** — For tasks where certain documents must be masked at retrieval time. ## Evaluation Standard IR metrics (NDCG@10, NDCG@50, Recall@10/50/100) should be computed using `qrels.tsv` for gold evaluation. Where available, `qrels_pool.tsv` enables pooled evaluation that accounts for unjudged relevant documents discovered during retrieval. For the Writing-Style and Math analogues task, `per_query_excluded_ids.json` lists document IDs that must be excluded from retrieval results for each query (snippets from the same source article as the query). ## License This dataset is released under [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/). ## Issues For any issues or feedback, please create a pull request/start a discussion here or send me (dianetc) an email ! ## Citation If you find our work helpful, please cite us: ```citation @misc{OBLIQ-Bench, title={OBLIQ-Bench: Exposing Overlooked Bottlenecks in Modern Retrievers with Latent and Implicit Queries}, author={Tchuindjo, Diane and Shah, Devavrat and Khattab, Omar}, url={https://arxiv.org/abs/2605.06235}, year={2026}, } ```