SQL-Schema-Retrieval

# SQL Schema Retrieval The evaluation benchmark from **"Finding the Right Tables and Columns: A Benchmark and Corpus-Adaptive Embeddings for SQL Schema Retrieval"** (Zeng, Yu, Mehta, Zhao, Samdani). **Task — schema linking as retrieval.** Given a natural-language question over a database, retrieve the schema element(s) needed to answer it. Documents are **schema elements**: at *table* granularity, each document is a table schema (columns + types rendered as markdown, with sample rows); at *column* granularity, each document is a single column with its table context. Relevance `R(q)` = the schema elements referenced by the question's gold SQL (obtained by parsing table/column references from the ground-truth query). Retrieval is performed **per database group** (rank the schema at hand). Metrics: **nDCG@10** and **recall@10**. This recasts five text-to-SQL datasets as retrieval, spanning academic → enterprise → large live schemas — a step that pure text-to-SQL accuracy never exercises because it assumes the whole schema fits in context. ## Subsets Counts below are measured from this release (corpus docs = retrieval candidates at the subset's granularity; rel. = relevant query–document judgments). | Subset (this release) | Source | Granularity | Corpus docs | Queries | Rel. judgments | |---|---|---|---:|---:|---:| | `spider` | Spider (academic, cross-domain) | table | 180 | 2,147 | 3,366 | | `bird` | BIRD (realistic, value semantics) | table | 75 | 1,534 | 2,956 | | `beaver` | BEAVER (private enterprise warehouses) | table | 463 | 209 | 928 | | `livesqlbench_table` | LiveSQLBench (base) | table | 244 | 410 | 1,075 | | `livesqlbench_large_table` | LiveSQLBench-Large | table | 971 | 332 | 901 | | `livesqlbench_large_column` | LiveSQLBench-Large | column | 17,709 | 332 | 2,157 | Underlying database sizes (from the paper's Table 1): Spider 40 DB / 785 col; BIRD 11 DB / 798 col; BEAVER 6 DB / 4,238 col; LiveSQLBench 22 DB / 1,942 col; LiveSQLBench-Large 18 DB / 17,708 col. Gold sets average 1.6–4.4 tables (table level) and up to 6.5 columns (column level) per query. The paper additionally studies two document representations (schema-metadata vs. value-only) for Spider/BIRD/BEAVER; this release provides the table/column collections used for evaluation. ## Format (BEIR / MTEB retrieval layout) ``` <subset>/ corpus.jsonl # {"_id": "<schema-element id>", "title": "", "text": "<schema as markdown>"} queries.jsonl # {"_id": "<qid>", "text": "<natural-language question>"} qrels/test.tsv # query-id \t corpus-id \t score (relevant judgments, score>0) ``` `_id` encodes the schema element as provided by the source: `db__table` for table-level subsets, and a column identifier (e.g. `db__table__column`) for the column-level subset. ## Sources & citation This benchmark repackages five public text-to-SQL datasets as retrieval; please cite this work **and** the original datasets, and follow each source's license/terms: - **Spider** — Yu et al., 2018 (EMNLP). https://yale-lily.github.io/spider - **BIRD** — Li et al., 2023 (NeurIPS D&B). https://bird-bench.github.io/ - **BEAVER** — Chen et al., 2024, arXiv:2409.02038. - **LiveSQLBench** (base + large) — BIRD Team, 2024. https://github.com/bird-bench/livesqlbench ```bibtex @inproceedings{zeng2025sqlschemaretrieval, title = {Finding the Right Tables and Columns: A Benchmark and Corpus-Adaptive Embeddings for SQL Schema Retrieval}, author = {Zeng, Qingcheng and Yu, Puxuan and Mehta, Aman and Zhao, Fuheng and Samdani, Rajhans}, year = {2025}, note = {Update venue/URL on publication} } ```