SilentDays: A Benchmark for Silence-Aware, Question-Level Change Detection in News Streams

SilentDays is a benchmark for silence-aware, question-level change detection in news streams. Analystswho follow a developing story keep a list of open questions and want to know, each day, whether any ofthem has a new answer. A change-detection system must decide which questions changed today, rank thedocuments that support each change, and stay silent on the many days when nothing happened. SilentDays gives every question a graded label on every day. It follows 30 TREC 2025 RAGTIME topicsover 1,093 days (2021-08-01 to 2024-07-31) of an English news stream and uses the grade scale of the TREC2026 Change Detection track (0, 1, 5, 10). It contains 13,260 graded document–question items (from1,044 NIST relevance judgments on 1,040 documents), 6,391 graded question-days and the universe of297,296 scored question-days, of which 98.57% are silent. 116 later-emerging questions are held out forchange-detection systems that propose new questions. Labels were produced by an LLM grader (gpt-5.5) that reads the stream in date order and credits onlyinformation not already reported, using the topic narrative, the full document text and the TREC gradedefinitions, without reference answers. The release includes the exact prompts, the grades of twofurther LLM judges (DeepSeek-V4.1-Flash, Qwen3.8-27B) and of a grader re-run, rankings from 15change-detection systems (ordered identically under all three LLM judges' labels), a standalone scorer (Python standard library only), a script that recomputes the paper'snumbers from the released files, the human-validation protocol, and a datasheet. No document text is included. Documents are referenced by id, with a SHA-256 fingerprint of the gradedtext; a script rebuilds and verifies the text from the user's own copy of the TREC RAGTIME Englishcollection. Data are licensed CC BY 4.0 and code MIT. The underlying relevance judgments and questions are releasedby NIST for TREC 2025 RAGTIME; please also cite the RAGTIME overview (Lawrie et al., arXiv:2602.10024)as their source.