eress

# ERESS: E-commerce Relevance Evaluation Scoring Suite ## Dataset Summary **ERESS (E-commerce Relevance Evaluation Scoring Suite)** is an evaluation dataset for **product discovery / e-commerce search reranking**. It contains **~4.7k unique queries** and **~72k labeled (query, product) pairs**, with **graded relevance** designed to reflect real shopping behavior and modern “assistant-style” query intent (e.g., utility/task framing, gift/audience constraints, attribute-heavy queries). ERESS is built to stress-test rerankers under common real-world failure modes: - **Hard negatives** (semantically close but wrong items) - **Lexical confounders** (high token overlap, wrong intent) - **Near-substitutes** (plausible alternatives that violate a key constraint like size/color/compatibility) The dataset is intended primarily for **holistic evaluation** of reranking models using **nDCG** (e.g., nDCG@5, nDCG@10) with graded relevance. --- ## Motivation Public relevance sets such as Amazon-ESCI and WANDS can under-represent modern e-commerce traffic, especially: - Gift and audience-specific constraints (“gift for…”, “for my dad who…”, occasions) - Utility/task queries (“fix squeaky door”, “organize cables”, “reduce glare”) - Highly attribute-constrained queries and conversational phrasing ERESS is explicitly stratified to better cover these intents and to include harder confounders that survive high-recall retrieval. --- ## Supported Tasks and Leaderboard-Style Use **Primary:** Reranking / relevance estimation for product search - Input: a query and a candidate product (title/description/attributes) - Output: a relevance score (graded) **Recommended metrics:** - **nDCG@k** (primary; graded relevance) - Optionally: MRR@k or Precision@k (secondary) --- ## Dataset Composition ### High-level stats - **Unique queries:** ~4.7k - **Labeled pairs:** ~72k (query, product) - **Labels:** graded relevance (scalar in \[0, 1\] as described in the accompanying project materials) ### Query intent families ERESS queries are designed to reflect a spectrum of shopping intents, including: 1. **Attribute-rich** (high feature density; size/material/compatibility constraints) 2. **Navigational** (brand/store/product-line seeking) 3. **Gift & audience-specific** (recipient + occasion) 4. **Generic** (category-level intent with minimal attributes) 5. **Utility** (task/solution framing) 6. **Short & Books** (very short head queries + targeted books) --- ## How ERESS Was Constructed (Overview) ERESS is derived from a modern e-commerce relevance pipeline built for high-recall retrieval + reranking evaluation: 1. **Query generation & coverage control** - Synthetic queries are generated under a constrained protocol and stratified across intent families. - Queries are embedded and clustered to avoid over-representing paraphrases and to preserve semantic diversity. 2. **High-recall candidate retrieval** - Candidates are retrieved via a dual-index strategy to improve recall across query types: - **Index A:** title-focused view (better for short/navigational) - **Index B:** full-text view (better for attribute-rich/utility) - Candidate sets are merged and deduplicated per query. 3. **Graded relevance annotation** - Each (query, product) pair is labeled using an ensemble (“council”) of LLMs. - Annotation uses structured prompts that emit a discrete label as the first token; first-token logits are used as calibrated signals for scoring. 4. **Quality control & leakage prevention** - Candidate-level pruning to remove trivial irrelevance - Deduplication across splits (no duplicate query strings and no duplicate product IDs leaking across train/eval regimes) - Semantic decontamination to reduce benchmark/train overlap via embedding similarity thresholds --- ## Dataset Splits ERESS is intended as an **evaluation suite**. If the hosted dataset includes splits, they typically correspond to evaluation partitions (e.g., `test` and/or multiple slices). If you create your own splits, ensure: - no duplicate queries across splits - no duplicate product IDs across splits (when evaluating generalization) - avoid semantic contamination between training and ERESS queries (nearest-neighbor overlap) --- ## Usage ### Loading ```python from datasets import load_dataset ds = load_dataset("thebajajra/eress") print(ds) print(ds[list(ds.keys())[0]][0])