DER Commitment Density Audit: An empirical probe of commitment-structured decision events in open human–LLM interaction corpora (v1.0)

This package documents a systematic empirical probe of commitment-structured decision events in publicly available human–LLM interaction corpora. It constitutes the methodological foundation for the DER (Decision-Event Record) empirical validation programme and provides a replicable audit of open corpus suitability for governance research. --- ## Research question > At what density do commitment-structured human–AI decision events occur in open LLM interaction corpora, and does corpus structure (conversation-level vs. turn-level) affect their detectability? --- ## Contents ```DER_Commitment_Density_Audit_v1_0/│├── README.md ← this file├── RIGHTS_NOTE.md ← data rights and licences│├── 01_pipeline/│ ├── commitment_pipeline_v2.py ← full detection pipeline (Python)│ └── pipeline_spec.md ← stage-by-stage specification│├── 02_results/│ ├── corpus_density_table.csv ← primary results table│ ├── wildchat_summaries/ ← per-shard run summaries (11 files)│ ├── sharechat_summaries/ ← per-model run summaries (5 files)│ └── oasst_summary.txt ← OpenAssistant run summary│├── 03_candidates/│ ├── wildchat_candidates_shard00.csv ← 10 candidates from shard 0│ └── manual_review_sheet.csv ← annotated review (der_commitment, der_reconstructable)│└── 04_methodology/ ├── structural_finding_note.md ← ShareChat turn-level vs conversation-level └── corpus_comparison.md ← cross-corpus structural comparison``` --- ## Primary finding Across 84,910 conversations/rows sampled from WildChat-4.8M (55,000), ShareChat (25,000), and OpenAssistant (4,438): | Corpus | Structure | Rows | Signals | Candidates | Rate ||--------|-----------|------|---------|------------|------|| WildChat-4.8M | Conversation-level | 55,000 | 1,471 | 55 | 0.100% || ShareChat (ChatGPT) | Turn-level | 5,000 | 0 | 0 | 0.000% || ShareChat (Claude) | Turn-level | 5,000 | 0 | 0 | 0.000% || ShareChat (Gemini) | Turn-level | 5,000 | 0 | 0 | 0.000% || ShareChat (Grok) | Turn-level | 5,000 | 0 | 0 | 0.000% || ShareChat (Perplexity) | Turn-level | 5,000 | 0 | 0 | 0.000% || OpenAssistant | Message-level | 4,438 | 12 | 0 | 0.000% || **Total** | | **84,910** | **1,483** | **55** | **0.065%** | Manual review of the 10 highest-confidence WildChat candidates confirmed 0–2 genuine DER-reconstructable events. **Estimated true positive rate: < 0.004% (< 4 per 100,000 conversations).** --- ## Structural finding ShareChat (turn-level) yielded zero signal hits despite 25,000 rows. This is not attributable to content sparsity but to structural mismatch: commitment events are conversation-level phenomena, not message-level phenomena. A pipeline operating on individual message rows cannot detect commitment events that span multiple turns. This finding has direct implications for corpus selection in AI governance research: corpora that fragment conversations into row-per-message format systematically suppress commitment event detectability. --- ## Methodological contribution This audit establishes three quantitative reference points for future DER empirical studies: 1. **Open corpus commitment density**: 0.065% overall; 0.100% for WildChat (conversation-level)2. **Signal-to-candidate conversion**: 3.7% (1,483 signals → 55 candidates)3. **Candidate-to-confirmed-event conversion**: < 4% (55 candidates → ≤ 2 confirmed) These figures provide the empirical basis for the design decision to use purpose-built synthetic case banks (DRB-90) rather than open corpora as the primary DER evaluation dataset. --- ## Replication ```bash# Install dependenciespip install pyarrow pandas # Run pipeline (demo)python3 01_pipeline/commitment_pipeline_v2.py --demo # Run on WildChat shard (requires local copy of dataset)python3 01_pipeline/commitment_pipeline_v2.py \ --input /path/to/train-00000-of-00086.parquet \ --dataset wildchat \ --output ./02_results \ --max_rows 5000 # Run on ShareChat (turn-level, expected 0 candidates)python3 01_pipeline/commitment_pipeline_v2.py \ --input /path/to/claude_results_final_language_filtered.csv \ --dataset sharechat \ --model_tag claude \ --output ./02_results \ --max_rows 5000``` --- ## Citation > So, H. B. (2026). DER Commitment Density Audit: An empirical probe of commitment-structured decision events in open human–LLM interaction corpora (v1.0) [Dataset]. Zenodo. https://doi.org/10.5281/zenodo.19054000 --- ## Related outputs - DRB-90 Decision Reconstruction Benchmark (companion empirical dataset)- BTI Rater Pack v1.1: https://doi.org/10.5281/zenodo.18989054- LFP-0 / DER schema: https://doi.org/10.5281/zenodo.17900999- TBT Root Index: https://doi.org/10.5281/zenodo.17992916