Measurement code and data for "A False Green Light: Three Silent Barriers to Collecting News Data for Research"

Measurement code and raw response logs accompanying the paper A False Green Light: Three Silent Barriers to Collecting News Data for Research. The deposit contains no article text from any publisher. Measurement 1. For each of 44 news outlets the robots.txt file is fetched once. From every Disallow rule in the User-agent: * group containing a wildcard or an end anchor, one concrete URL is constructed that the rule is intended to block. Three parsers are then asked whether that URL may be fetched: robotspy (implementing RFC 9309, used as the reference), urllib.robotparser from the Python standard library, and Protego. Of the 388 cases the reference refuses, the standard library permits 385 and Protego permits 29. Measurement 2. A two-by-two comparison on a bot-protected news site: a persistent HTTP session against a fresh connection for every request, crossed with default python-requests headers against rotated browser header profiles, with pacing held constant and polite throughout. Both sessionless cells returned full content for all 25 requests each; across four independent sessions only the first request was served, and every refusal arrived as HTTP 200 with a short body. Licensing. The Python source files are MIT licensed (see LICENSE). The measurement outputs (CSV, JSON, figures) are CC BY 4.0 (see LICENSE-DATA). The files under robots_cache/ are verbatim copies of robots.txt served by third-party publishers, included so the analysis can be recomputed without re-querying those servers; they remain the work of their respective publishers and robots_cache_provenance.csv records the source URL and retrieval date for each.