Detecting Bot Detection Dataset

This repository includes the literature codings, the instrumentation source, the scan database, and the analysis notebooks and scripts used in the paper "Detecting Bot Detection: Prevalence, Techniques, and Implications for Web Measurement Research".

Structure

01-literature: Literature survey codings (Section 3). Contains codings of all 81 papers: "author1" has the final resolved ratings (after discussion), "author2" has the independent codings of a 20-paper subset for inter-rater reliability. A consolidated spreadsheet ("literature_survey.ods") is included for overview.

02-source-code: Instrumentation demo and database schema (Section 5.1). The "demo" directory contains a self-contained Docker setup that demonstrates the JavaScript instrumentation on any website. Run "cd demo && docker compose build && docker compose run --rm scanner" for a self-test. The database schema ("schema.sql") and an example output file ("demo/example-output.json") are included. See "demo/README.md" for details, including the two instrumentation versions (see notes below).

03-analytics: Jupyter notebooks and R scripts (Sections 3 and 5). Each notebook produces the numbers for its corresponding paper section. All outputs are saved and can be viewed without any dependencies (e.g., on GitHub or in any Jupyter viewer). Re-execution requires the scan database, provided here as a compressed PostgreSQL custom-format dump ("dbd_data.dump"). Restore it with "createdb dbd_data && pg_restore -d dbd_data dbd_data.dump" (requires PostgreSQL 17 or newer), then copy ".env.example" to ".env" and set the connection details. R scripts in "r-paper" cover Cohen's kappa (inter-rater reliability) and the Mann-Whitney U test; pre-rendered ".nb.html" outputs are included so results can be inspected without running R.

Prerequisites

Docker (instrumentation demo)

PostgreSQL 17+ (to restore the scan database dump)

R with "irr" and "effectsize" packages (statistical tests)

Notes

Screenshots excluded: The scan database does not include the per-scan page screenshots. They are omitted for copyright reasons. Screenshots were only used for manual validation and are not required to re-run the released notebooks.

Instrumentation versions: The demo ships two copies of the instrumentation. "init_scripts_old.js" is the exact script used to collect the published data; "init_scripts.js" is a corrected version that the demo runs and that we recommend for reuse. The two differ only in how proxied objects are serialized: the original occasionally emitted spurious tier-0 ("unclassified") events, which are excluded from all analysis. This does not affect any number reported in the paper. See "02-source-code/demo/README.md".

DNS census excluded: The DNS/CDN census (Section 4) is not redistributed in this repository: it derives from the IPinfo Lite dataset, which is released under CC BY-SA 4.0. Omitting it keeps this bundle under CC BY 4.0 and avoids ShareAlike entanglement. The method is fully described in the paper and is reproducible with your own IPinfo Lite snapshot and the Tranco list (ID GVL9K, 2026-02-25). IPinfo attribution is given in the paper's acknowledgements.

License

See "LICENSE". Source code ("02-source-code") is released under the MIT License; data, literature codings, and documentation are released under CC BY 4.0.