遇见数据集

Replication Data for an Empirical Evaluation of Long-Context Retrieval in Large Language Models under Hard-Negative Interference (1,390 Controlled Trials)

收藏
Zenodo2026-11-01 更新2026-11-05 收录
官方服务:

资源简介:

Replication package for an empirical evaluation of long-context retrieval in nine large language models (GPT-4o-mini, Gemini 3.6 Flash, Grok 4.6, DeepSeek-V3, Kimi K2.6, GLM-5, Qwen3-7-Max, ERNIE-4.5, Doubao) under hard-negative interference. The archive contains 1,390 controlled trials: a single-needle depth scan (Phase 1, 230 trials), a hard-negative distractor gradient with a fitted two-parameter failure model p(N) = a·e−λN and mixed-effects logistic regression (Phase 2, 326 trials), a RAG-versus-long-context cost–accuracy crossover (Phase 3, 129 trials), complementary ablations (soft negatives, temperature, retriever comparison, multi-hop reasoning, 159 trials), five mechanism probes E1–E5 (426 trials: RAG under interference, RAG multi-hop synthesis, unanswerable probes, cross-task generalization, real-corpus validation), and a system-layer intervention pilot E6 (120 trials: upstream disambiguation and a verification loop with within-session baselines). Included are all experiment runners, the core needle-in-a-haystack library, analysis scripts reproducing every reported statistic (summary counts, failure-model fits, mixed-effects regression, exact McNemar tests with Benjamini–Hochberg correction), model configuration templates without credentials, and all trial-level data in JSON Lines format together with derived aggregate statistics. Version 3.2.0 adds the E6 system-layer intervention experiment (120 trials), Wilson 95% score intervals for all reported proportions, a random-slope sensitivity analysis, and knowledge-conflict error-type recoding, and updates the archive to 1,390 controlled trials. Version 3.1.0 adds the homogeneous GPT-4o-mini re-run of the 128K Phase-3 cell (input trimmed to ~119K tokens to fit the 128K-token window including the output budget); both the original substitution records and the re-run records are retained. Haystacks are synthetic; no external corpus is redistributed. Version 3.3.0 corrects the two-sided exact McNemar p-values in the E6 intervention analysis (proper exact binomial test) and regenerates the E6 summary statistics; all trial-level data are unchanged from version 3.2.0.

提供机构:
Zenodo
创建时间:
2026-11-01
二维码
社区交流群
二维码
科研交流群
商业服务