Run archive for "Pricing Execution-Gated Refinement in Text-to-SQL: Cost, Accuracy Across Backbones, and Negative Results"

Version 1.1.0 of the run archive. One data file and its checksum list are added; the existing files are unchanged. Archive of the model requests, model responses, per-question predictions, usage logs and run metadata behind the article "Pricing Execution-Gated Refinement in Text-to-SQL: Cost, Accuracy Across Backbones, and Negative Results" (Li, Song, Wang; submitted to Applied Sciences). Code, analysis plan and result files: https://github.com/vivia-an/nl2sql-gated-refinement (code archive concept DOI 10.5281/zenodo.23739687). What is new in 1.1.0- nl2sql_runs_posthoc_1.1.0.tar.gz (214.6 MB, 352 files): the 48 post hoc runs of analysis plan Sections 28 to 30, all decided after the locked results were known; temperature 0, seed parameter 0. * X2DS (4 runs): the four decomposition arms of the locked run with one changed model, deepseek-v4.1-flash-1017 (a non-Qwen model); BIRD dev. * X2DSR (4): a single call plus the execution gate of the loop (at most three revisions; no critic, no front end) on Qwen3.7-Plus, Qwen3.7-Flash, Qwen3.5-27B and deepseek-v4.1-flash-1017; BIRD dev. * X2DSRR (7), W7RR (4): one rerun of the retry-only arms and of the non-Qwen chain, and of the empty-gate retry; BIRD dev. * W7 (9): single call with thinking on (Qwen), empty-gate retry (four models), MAC-SQL on Flash and 27B; BIRD dev. * W8 (8): the pipeline (gated loop) with the empty gate, s0 and one rerun, four models; BIRD dev. * W9 (12): single call, single call + retry and empty-gate retry on the tuning split HR (988 questions of 7 held-out BIRD training databases), four models. Several arms reuse stored calls of another run so that two arms differ only in the step under study; their llm_cache.jsonl.gz then also holds those imported requests and responses (README_ARCHIVE_posthoc_1.1.0.md, "Imported calls"). Per run: preds.jsonl, cands.jsonl (loop arms), usage.jsonl, llm_cache.jsonl.gz (every request and response) and its manifest, run_meta.json, config.yaml, summary.json. MANIFEST_posthoc_1.1.0.tsv lists every file with its sha256; RUNS_posthoc_1.1.0.tsv lists the runs. The three documents carry a suffix so that unpacking the file next to the 1.0.0 file overwrites nothing.- nl2sql_runs_1.0.0.tar.gz (807.7 MB): unchanged; it still holds every run behind the result files of the code releases 1.0.x, and the 22 runs that the post hoc analysis reads next to the new ones. SHA256SUMS_1.1.0 lists the checksums.- Replay without an API key: code release 1.1.0, README, Level 2 (sh code/tools/x2ds_run_analysis.sh <WORK_DIR> <OUT_DIR>), compared with results/x2ds/. All 5 722 recomputed values equal the released ones (numpy 1.26.4, scipy 1.15.2, B = 10000). Licences and attribution (unchanged)- The files contain benchmark material (questions, evidence strings, schema text with sample values from BIRD dev and train; exemplar questions from Spider). It is shared under CC BY-SA 4.0; please attribute the benchmarks to their authors (BIRD: Li et al., NeurIPS 2023; Spider: Yu et al., EMNLP 2018). The benchmark text was rendered into prompts and model outputs and execution results were attached; this is modified material, shared under CC BY-SA 4.0. The databases underneath are not included and keep their own licences.- Model outputs are shared as research data. The Qwen responses and the deepseek-v4.1-flash-1017 responses were generated through the OpenAI-compatible endpoint of Alibaba Cloud Model Studio with the authors' paid account, under the same account terms. No licence is added to, and none is claimed on, the raw output text; the terms of the service provider (and of the model developer, where they apply) govern its use.- Paths of the machines used have been replaced by <NL2SQL_ROOT> in the small metadata files; the prediction, usage and cache files are byte-identical to the originals. The archive contains no API key, no request header and no machine address. The 20-question pilot runs made before the full runs, interrupted or superseded runs, and live caches are not included.