False Reassurance in LLM Severe-Weather Guidance: A Replayable Evaluation Against Real NWS Alerts

This is the dataset and code release supporting "False Reassurance in LLM Severe-Weather Guidance: A Replayable Evaluation Against Real NWS Alerts" (Kadharmoideen Fadurudeen, Independent Researcher). The paper evaluates whether an LLM assistant without live access to severe-weather alerts will falsely reassure users that no hazard exists, and tests whether giving it real alert data, or a disclosure instruction, fixes this. The package contains: - 117 replayable case definitions (cases/) — each built from a real, time-stamped NWS alert (or a no-alert/unsupported-location control) paired with Open-Meteo current-conditions data for the same place and time.- Model-response logs (results/) for the main study (all 5 experimental conditions: a production pipeline with real alert data, the same data under a neutral prompt, a blind neutral prompt with no data, a blind prompt with a disclosure instruction, and a plain no-data prompt) plus a preregistered single-sentence ablation study and earlier pilot/diagnostic runs, kept for audit.- 342 hand-applied false-reassurance labels (results/review_blinded.csv, with the condition-unblinding key in results/review_blind_key.csv), labeled by a single rater blind to condition under a frozen 7-category coding scheme (FALSE_REASSURANCE_CODEBOOK.md), whose provenance — the scheme was written after an initial binary labeling pass, not before — is disclosed in that file.- Evaluation and data-collection code (eval/, logger/) needed to reproduce the paper's reported false-reassurance rates directly from the files in this package, offline, with no network access or credentials required. Headline result: false reassurance was 0.0% when the assistant had real alert data (production pipeline or the same data under a neutral prompt), 37.7% under a blind neutral prompt with no data, and 44.9% under a blind prompt instructed to disclose its blind spot — the disclosure instruction did not significantly reduce false reassurance relative to no disclosure at all (paired McNemar's test, p = 0.40). Licensing: this record mixes two licenses. cases/, results/ (including the human labels) are released under CC BY 4.0; eval/ and logger/ (the code) are released under MIT. See LICENSE-DATA.txt and LICENSE-CODE.txt inside the archive for full terms. Data attribution: NWS alert data is U.S. government open data (public domain). Current-conditions data is "Weather data by Open-Meteo.com," licensed CC BY 4.0. Model under test: Claude Sonnet 4.6, via Amazon Bedrock, temperature 0, 500 max output tokens. See README.md inside the archive for the full directory map, per-file descriptions, and instructions for reproducing the scoring.