Extraction Succeeds, Search Fails: replication bundle (generator, solver, harness, results)
收藏资源简介:
Replication bundle for Extraction Succeeds, Search Fails: Offloading Search to a Solver in LLM Planning, a benchmark that separates two steps in language-model planning on river-crossing puzzles: reading the puzzle into a structured state, and searching from that state to a plan. Contains the seeded puzzle generators, the exact breadth-first solver and plan checker, the evaluation harness with every system prompt verbatim (including the July 2025 state-tracking prompt the study tests, byte for byte), the pre-registration with its dated amendments, the analysis scripts, every puzzle set used, and every raw result: one CSV row per reply plus the full model replies in JSONL. Every number in the paper can be recomputed without an API key. paper_tables.py rebuilds the confirmatory tables and results_tables.py rebuilds the pilot tables, both byte for byte from the committed CSVs. Ground truth exists before any text does: each puzzle is sampled as a structure and solved exactly, and only then written in English, so every instance carries a proved shortest plan or a proof that no plan exists. Collecting fresh model responses requires your own API key and incurs cost; REPLICATION.md gives the environment variables, a dry run, and a capped worked command. No language model judges any result at any point. Fully synthetic throughout; no production system, private dataset or user traffic is involved. Released under the MIT license.



