Know Your Source: A Public Knowledge Store for Media Background Checks

LLM-based retrieval-augmented generation (RAG) is increasingly used forautomated fact-checking (AFC) and related tasks. By grounding LLM outputs inretrieved evidence, RAG-based systems provide transparent justifications whileallowing external information to be updated independently of the underlyingmodel. However, existing approaches often assume retrieved evidence isreliable, although real-world information may be conflicting, outdated, and canoriginate from unreliable or biased sources. Recent work on *source-criticalreasoning* addresses this challenge through media background checks (MBCs)(Schlichtkrull, 2024), which assess the credibility of evidence sources tosupport downstream fact verification. However, generating MBCs relies on costlyproprietary search APIs, limiting reproducibility. To mitigate this issue, weintroduce MEDIAREF, a publicly available knowledge store of web-sourceddocuments that enables reproducible, low-cost evaluation of MBC generationacross 200 media sources. We describe a reproducible methodology forconstructing and updating the collection, assess widely used LLMs on the MBCgeneration task, and demonstrate that MEDIAREF supports higher-quality MBCgeneration through both automatic and qualitative evaluation.