Replication Code and Data: Emoji-Based Stylometric Features for Pairwise Authorship Verification in Social Media

# Emoji-Based Authorship Verification - Reproduction Package This archive contains the complete analysis code accompanying the manuscript*"Emoji fingerprints: Topic-independent authorship verification using 20 novelemoji-based stylometric features"* (manuscript FSI-D-26-00293, under review at*Forensic Science International*). Author details are withheld during peerreview and will be added upon acceptance. A single driver script executes theentire study end to end; feature extraction is implemented in two unit-tested,importable modules. --- ## Contents | File | Purpose ||------|---------|| `run_authorship_pipeline.py` | End-to-end pipeline: descriptive statistics -> emoji/classic/hybrid verification systems (49 configurations) -> significance analysis -> ROC -> probabilistic calibration -> score-based likelihood-ratio layer -> cross-subreddit and author-disjoint robustness analyses -> distance-metric and vocabulary-size sensitivity analyses -> result tables and figures. || `emoji_features.py` | Reference implementation of the 20 emoji-based features (F01-F20), including the corpus-aware features F08 (exclusive sequences), F16/F17 (unique / near-unique n-gram patterns) and F20 (mutual best near-unique match), computed exactly via a single-pass `CorpusIndex`. || `classic_features.py` | The 27 classic stylometric baseline features (14 lexical, 11 distributional, 2 overlap). || `test_emoji_features.py` | Unit tests asserting that the corpus-aware features behave according to their definitions (F08 is stricter than F06; F16 differs from F17; F20 is a genuine mutual-best match). || `prepare_reddit_archives.py` | Data-provenance script: streams the 17 subreddit configurations from the public HuggingFace `REDDIT_comments` corpus, applies the source-period filter (January 2021+) and the two per-author inclusion criteria with the canonical emoji detector, and writes `authors_archive_split.zip`/`.xlsx` in the exact layout the pipeline expects. || `features_definition.json` | Machine-readable summary of the 47 features (identities per Table 2 of the manuscript; design-matrix keys extracted from the implementations) and of the modeling choices. || `_zenodo.json` | Zenodo deposit metadata (anonymized). || `requirements.txt`, `CITATION.cff`, `LICENSE` | Dependencies, citation metadata, MIT license. | The pipeline imports the two feature modules directly; the inlined definitionsin the companion single-file notebook version are byte-equivalent to thesemodules (verified by direct output comparison), so both routes yield identicalresults. --- ## What the pipeline computes * **Descriptive statistics.** Corpus overview, emoji frequency and diversity, and the most frequent emoji. Descriptive emoji counts are reported at the grapheme level (each rendered emoji counts once, so flag, keycap, skin-tone and ZWJ sequences are single units); the run-level token counts used by the frequency features are reported separately as the feature vocabulary.* **Three verification systems** over a 7 x 7 grid of profile size *P* and probe size *T* (*P*, *T* in {5, 10, 25, 50, 100, 250, 500}): emoji-only, classic, and hybrid (concatenation). Each system is a Random Forest (500 trees, configuration fixed a priori; no hyperparameter tuning) trained on the same pairs and the same stratified 80/20 train/test split, so the three systems are directly comparable.* **Statistical significance.** Paired McNemar tests on shared held-out predictions (exact binomial form when the number of discordant pairs is below 25, continuity-corrected chi-squared otherwise), a paired t-test on shared cross-validation folds, and Holm-Bonferroni correction over the joint test family.* **ROC / confusion analyses** and probabilistic calibration of the raw Random Forest scores (Brier score, expected calibration error).* **Score-based likelihood-ratio layer.** The Random Forest same-author probabilities are converted to forensic log-likelihood ratios by cross-fitted (5-fold) logistic calibration, which yields the reported system cost Cllr; pool-adjacent-violators (isotonic) recalibration is used only to obtain the discrimination bound Cllr_min (with Cllr_cal = Cllr - Cllr_min). Outputs include Tippett plots, PAV calibration/reliability diagrams, and a paired stratified bootstrap (2,000 resamples) giving 95% confidence intervals for Cllr and for the between-model Cllr differences (reported as the median of the resampled differences). Since v2.1.0 the layer also reports the complementary error summaries of the calibrated systems -- rates of misleading evidence (same-author pairs with LR < 1; different- author pairs with LR > 1) and a weak-evidence rate (comparisons with 0.1 < LR < 10) -- per configuration and model (manuscript Table 14), and exports the per-comparison calibrated log10 LRs. A fully feature-based LR system is left as future work.* **Robustness.** A cross-subreddit analysis (pair types A-D testing topic-independence) and a strict author-disjoint 70/30 re-evaluation (leakage check, split at the author level).* **Sensitivity analyses.** Re-evaluation of the classic and hybrid systems under alternative distance metrics (Manhattan, Burrows' Delta) and n-gram vocabulary sizes (top-100/300), reported in the manuscript's supplementary material. ### Known limitations of the emoji-detection pattern Two properties of the detection pattern are documented here for transparencyand are deliberately left unchanged, because all published results werecomputed with this exact pattern: 1. The broad "enclosed characters" span (U+24C2-U+1F251), common in emoji regular expressions, also matches a small number of enclosed CJK and alphanumeric symbols that are not emoji proper.2. Frequency-type features operate on run-level tokens (maximal consecutive emoji sequences count once); descriptive statistics additionally report grapheme-level counts, as described above. Bare ASCII digits are **not** matched as emoji: keycap emoji are recognizedonly as the complete sequence digit + U+FE0F + U+20E3. --- ## Correspondence to the earlier archive layout (v1) Version 1 of this archive organized the analysis as numbered scripts. All ofthat functionality now lives in clearly labeled cells of the single verifieddriver (which is the code that produced every number in the revisedmanuscript); the mapping is: | v1 script | v2 location (`run_authorship_pipeline.py`) ||-----------|--------------------------------------------|| `00_data_preparation.py` | Cells 2, 5 and 7 (inclusion criteria, loading, caching, shared emoji vocabulary) || `01_descriptive_statistics.py` | Cell 7B (descriptives) and Cell 7C (Fig. 1-2 emoji spectra) || `02_emoji_attribution.py` | Cells 3, 6 and 8 (emoji features + unified runner) || `03_classic_baseline.py` | Cells 4, 6 and 8 (classic features + unified runner) || `04_hybrid_attribution.py` | Cells 6 and 8 (hybrid arm of the unified runner) || `05_significance_analysis.py` | Cell 9 (McNemar, paired t-test, Holm) || `06_roc_confusion_matrix.py` | Cell 10 || `07_calibration_metrics.py` | Cell 11 (Brier/ECE) and Cell 11B (likelihood-ratio layer; new in v2) || `08_author_disjoint_validation.py` | Cell 12B || `config.py` | Cell 2 constants (paths, experiment matrix, seeds) || `features_definition.json` | regenerated in v2 from Table 2 of the revised manuscript | New in v2 (absent from v1): the corrected emoji detection (complete keycapsequences only), the exact corpus-aware features F08/F16/F17/F20, thelikelihood-ratio layer (Cell 11B), the ablation study (Cell 11C), thecross-subreddit Type A-D analysis in its final form (Cell 12), and thedistance-metric / vocabulary-size sensitivity analyses (Cell 12C). Version 1scripts therefore reproduce the pre-correction results and should not be usedto check the revised manuscript. --- ## Data files on this record The v1 record shipped the author-level comment archive as a single ZIP. Forv2, upload the same data alongside the code using the file names the pipelineexpects (legacy names are also recognized): ```authors_archive_split.zip # per-author comment CSVs (v1: "AUTHOR DOCS.zip")authors_archive_split.xlsx # author listmultisub_archive.zip # multi-subreddit archive (cross-subreddit analysis)``` The frozen archive on this record is the authoritative snapshot (HuggingFacecorpus accessed 15 January 2024). `prepare_reddit_archives.py` rebuilds aformat- and criterion-compatible corpus from the live HuggingFace dataset forprovenance and from-scratch replication; the multi-subreddit archive isoptional, because authors active in several subreddits keep one CSV persubreddit inside the main archive and the pipeline reads its multi-subredditpool directly from there. --- ## Data layout Place the input archives in one folder (`BASE_DIR`): ```BASE_DIR/ authors_archive_split.zip # one CSV per unique author (includes the comment text) authors_archive_split.xlsx # author list multisub_archive.zip # multi-subreddit archive (cross-subreddit analysis)``` Legacy file names from earlier internal runs are also recognizedautomatically. The author-level comment data are not redistributed in thisarchive because they contain user-generated content; see the manuscript'sdata-availability statement for access conditions. The underlying publiccorpus is the Reddit comment archive described in the manuscript (Section 2.1). --- ## How to run ### Local ```bashpip install -r requirements.txtexport BASE_DIR=/path/to/your/data # folder containing the files abovepython run_authorship_pipeline.py``` Optional environment-variable overrides (all have sensible defaults):`MAIN_ZIP`, `AUTHOR_XLS`, `OUTPUT_DIR`, `ARCHIVE_DIR`, `EK_ARCHIVE_DIR`(the `EK_` prefix is a historical name for the supplementary multi-subredditarchive and is retained for checkpoint compatibility). ### Google Colab Upload the four `.py` files, then in a cell: ```python!pip -q install regex statsmodels openpyxl%run run_authorship_pipeline.py``` Colab is auto-detected and Google Drive is mounted automatically;`BASE_DIR` defaults to `/content/drive/MyDrive/reddit_authorship_data`(override via the environment variable if your layout differs). ### Clean re-runs and checkpointing The pipeline is checkpoint-resumable. Every checkpoint is stamped with afeature-version string (`FEAT_VERSION`); any checkpoint that does not carrythe current stamp is recomputed automatically. For a guaranteed clean run,either point `OUTPUT_DIR` at an empty folder or set `FORCE_RERUN = True`(Cell 2) for one run to wipe all checkpoints and recompute everyconfiguration. Reproducibility: `RANDOM_SEED = 42` throughout; pair generation, splits,cross-validation folds, bootstrap resampling and the sensitivity analyses areall seeded. --- ## Outputs Everything is written under `OUTPUT_DIR/`: * `supplementary/` - `descriptive_overview.csv`, `top_emojis.csv`, `calibration_metrics.csv`, `significance_summary.csv`, `lr_metrics.csv` (Cllr/Cllr_min/EER with 95% CIs), `lr_cllr_pairwise.csv` (bootstrap Cllr differences and significance), `table14_lr_error_summaries.csv` (misleading-evidence and weak-evidence rates of the calibrated LR systems; manuscript Table 14), `lr_per_pair_llrs.csv` (per-comparison calibrated log10 likelihood ratios underlying Table 14 and the Tippett plots), `cross_subreddit_summary.csv`, `author_disjoint_results.csv`, `supp_D1_distance_sensitivity.csv`, `supp_D2_ngram_sensitivity.csv`, `supp_sensitivity_sanity.csv`, and all figures (accuracy heatmaps, ROC curves, confusion matrices, emoji frequency spectra, Tippett plots, PAV calibration diagrams, sensitivity plots) as PNG and PDF.* `checkpoints*/`, `predictions/` - per-configuration JSON results and the paired predictions used by the significance and likelihood-ratio analyses.* `ALL_RESULTS_TABLES.xlsx` - every result table on its own sheet.* `results_for_manuscript.zip` - the bundled results package. --- ## Verifying the feature implementation ```bashpython test_emoji_features.py``` All four tests should report `PASS`. They check the corpus-aware featuresthat distinguish this implementation: F08 is strictly more exclusive than thecommon-sequence count F06, F16 (unique n-grams) differs from F17(near-unique), and F20 is a true mutual-best near-unique match rather than atrivial disjunction. --- ## Changelog * **v2.1.0 (2026-08)** -- adds the complementary error summaries of the calibrated likelihood-ratio systems requested in peer review: rates of misleading evidence, P(LR < 1 | Hp) and P(LR > 1 | Hd), and a weak-evidence rate, P(0.1 < LR < 10), written per configuration and model to `table14_lr_error_summaries.csv` (manuscript Table 14); additionally exports `lr_per_pair_llrs.csv` with the per-comparison calibrated log10 likelihood ratios underlying Table 14 and the Tippett plots (Fig. 7). No previously reported quantity changed.* **v2.0.0 (2026-07)** -- unified single-driver layout; corrected emoji detection; exact corpus-aware features F08/F16/F17/F20; likelihood-ratio layer; ablation, sensitivity, cross-subreddit and author-disjoint analyses. --- ## Citation See `CITATION.cff`. Please cite both the article and this software archive(the Zenodo DOI will be added once assigned, and author details will becompleted upon acceptance of the manuscript).