DSEBench: DSEBench: A Test Collection for Dataset Search with Examples

DSEBench: A Test Collection for Dataset Search with Examples DSEBench is a comprehensive test collection designed to support the evaluation of two complex information retrieval tasks in the dataset domain: Dataset Search with Examples (DSE): A task that requires retrieving candidate datasets ($D_c$) that are both relevant to a textual query ($q$) and similar to a set of known relevant target datasets ($D_t$). Explainable DSE: An extension that requires identifying the specific metadata fields that explain the result dataset's relevance and similarity. This archive includes the essential data files, relevance judgments, and baseline results required to reproduce and benchmark models for DSE and Explainable DSE. Key Components of DSEBench The collection is built upon 46,615 candidate datasets collected from NTCIR, and includes the following data resources: Datasets: Metadata for 46,615 datasets, each with fields including id, title, description, tags, author, and summary. Queries: 3,979 keyword queries, including both NTCIR queries and generated queries. Test and Training Cases: 5,840 cases, each linking a query_id and a target_dataset_id, used to define the DSE input. Relevance Judgments: A total of 130,000+ judgments are provided: 7,415 Human-Annotated Judgments. 122,585 LLM-Annotated Judgments. Each judgment provides scores for query relevance (query_rel) and target similarity (target_sim), along with field-level scores (field_query_rel, field_target_sim) for explaining the judgments. Predefined Splits: Files are included to ensure comparable evaluation, providing 5-Fold splits and an Annotators split for training, validation, and test sets. Baselines and Source Code This release also provides complete experimental results and source code for various baseline models: Retrieval Models (7): TF-IDF, BM25, BGE, GTE, ColBERTv2, coCondenser, and DPR. Reranking Models (4): Stella, SFR, BGE-reranker, and LLM (GLM-4-Plus). Explanation Methods (4): Feature Ablation, LIME, SHAP, and LLM explainer. The complete implementation source code, dependencies, and detailed documentation are available in the associated GitHub repository, ensuring full reproducibility. License and Persistence All data and software source code included in this test collection are licensed under the Apache License 2.0. This permanent Zenodo record, identified by its DOI, provides a stable, persistent, and citable archive of the resource, adhering to the highest standards of the FAIR principles.