Reproduction package for "Question difficulty across models predicts factual errors of language models better than a single model detector"
收藏资源简介:
Stored inputs, analysis scripts, reference results and frozen preregistrations for the article "Question difficulty across models predicts factual errors of language models better than a single model detector". The package covers a main panel of 17 open-weight instruction-tuned models, three replication corpora (Natural Questions, BioASQ, TriviaQA; 18 models), a nine-model design without the hard-question filter, the stored greedy and sampled answers, the question packs with reference aliases, the responses with the answer tokens marked in the extraction pipeline, and the GPU extraction scripts. run_all.sh recomputes 11 result files and compares them with the reference results; run_supplementary.py reproduces the supplementary checks. See README.md. Raw activation tensors (8.1 TB) are not included; they can be regenerated with the extraction scripts and the public model weights referenced in the data files.



