Data for: Whole-Paper Reading versus Retrieval with Open-Weight Language Models for Data Extraction in Systematic Reviews

This record holds everything the study produced: the answers of six open-weight language models to 400 questions about long documents, the verdicts of a three-model LLM jury, automatic similarity metrics, a reasoning-budget sweep, a human validation of the jury and a comparison of the models with the agreement among the QASPER annotators. It also holds the caches that let the whole pipeline be re-run offline, reproducing these files exactly. Questions: a stratified sample (seed 42) of 200 QASPER questions about NLP papers (50 per answer type: extractive, abstractive, yes/no, unanswerable) and 200 QuALITY multiple-choice questions about stories and articles (100 easy, 100 hard), over 272 documents of about 650 to 10,300 words. Generators: Llama-3.2-3B, Qwen3.5-4B, Mistral-7B, Qwen3.5-9B, Gemma4-31B and Qwen3.5-122B, served by Ollama 0.40.8 with the default 4-bit quantisation of each tag, temperature 0, fixed seed, a 32,768-token context and JSON-schema-constrained output. Eleven context conditions: closed book, full document, BM25, dense (BGE-M3) and hybrid retrieval, and four variants of a keyword search chosen by the model. Qwen3.5-4B and Qwen3.5-9B also answered five conditions with reasoning enabled, under generation budgets of 512 to 8192 tokens. Grading: abstentions by rule; other answers by a jury of GPT-OSS-120B, Llama-3.3-70B and Qwen3-32B (strict majority), validated against the QuALITY answer key and against a human annotator on 120 QASPER answers. Files: everything is inside llm-eval-data-v1.0.zip (model outputs, verdicts, metrics, analysis CSVs, the sample, the human validation, the configuration, and gzip-compressed caches), with a SHA-256 manifest. README-v1.0.md describes every file and field. To use the data, clone the code repository and run: python scripts/install_zenodo_data.py llm-eval-data-v1.0.zip [--with-caches] The data are released under CC BY 4.0. They contain text from QASPER and QuALITY (both CC BY 4.0) and outputs of Qwen, Gemma, Mistral and GPT-OSS models (Apache 2.0) and of Llama models (Llama 3.2 / 3.3 Community License; Built with Llama); see NOTICE inside the archive.