Open-Domain Multi-Table Retrieval Benchmark Collection

Exploring Multi-Table Retrieval Through Iterative Search Overview This collection provides pre-processed development sets for five Text-to-SQL benchmarks—Spider, BIRD, Fiben, and Beaver (NW and DW variants).It was curated to streamline the evaluation of multi-table retrieval as presented in the paper: In addition to the standard benchmark assets, each archive includes pre-computed retrieval metadata—such as column uniqueness scores, Jaccard similarities, and semantic embedding similarities—used to reproduce the iterative retrieval and query decomposition experiments in the paper, specifically by the methods Iterative-JAR (ours) and JAR (Chen et al,. 2024). File Structure & Contents Each archive follows a consistent directory layout designed for multi-table retrieval pipelines. 1. Standard Benchmark Files dev_database/Directory containing SQLite .sqlite database files for the dev set. dev.jsonContains natural-language questions, ground-truth SQL queries, and associated database identifiers. dev_tables.jsonSchema definitions: table names, column names, data types, and associated metadata. 2. Augmented Retrieval Metadata These files provide pre-computed metrics describing relationships between tables and columns. They support schema retrieval, table selection, and join inference. decomp.json — Query Decomposition Contains decomposition outputs for each query in dev.json into concepts. dev_uniqueness.json — Column Uniqueness Scores Maps each column to a numerical uniqueness score (0.0–1.0). Computation: (# unique values) / (# total rows) Use: Identifying primary key–like columns, distinguishing categorical vs. high-cardinality attributes. dev_jaccard.json — Content-Based Similarity Stores Jaccard similarity scores for value-based comparison between columns across tables. Computation: |Intersection| / |Union| of column value sets Use: Detecting joinable columns and data-driven foreign key candidates. exact_col_sim.json — Lexical Similarity Contains similarity scores based on exact string-level matching between column headers. Computation: Overlap coefficient of tokenized column names (encoded as db_id table_name column_name). semantic_col_sim.json — Semantic Similarity Provides semantic similarity scores between column names based on vector embeddings. Computation: Cosine similarity of encoder-generated embeddings Use: Capturing contextual similarity beyond lexical overlap. Usage Instructions To use these datasets with the official implementation: git clone https://github.com/Allaa-boutaleb/iterative-jar Download and extract the tar.gz files in the data/folder.