CEI Benchmark: Pragmatic Reasoning Dataset

Replication package for "CEI: A Benchmark for Evaluating Pragmatic Reasoning in Language Models" (DMLR  2026).

  The Contextual Emotional Inference (CEI) Benchmark is a dataset of 300 expert-authored scenarios for  evaluating how well language models interpret pragmatically complex utterances in social contexts. Each   scenario presents a communicative exchange involving indirect speech (sarcasm, mixed signals,  strategic politeness, passive aggression, or deflection) where the speaker's literal words diverge from   their actual emotional state. Three trained annotators independently labeled every scenario using  Plutchik's 8 basic emotions and Valence-Arousal-Dominance ratings.

  This archive contains:  • data/human-gold/ — 5 merged annotation CSVs (300 scenarios, 3 annotators each)  • scripts/ — Pipeline, analysis, and HuggingFace upload scripts  • config/ — Model definitions and pricing configuration  • papers/dmlr2026/ — Paper source (LaTeX), bibliography, and figures  • reports/dmlr2026/ — Baseline results (JSON) and LaTeX tables  • LICENSE (MIT for code) and README.md

  The dataset is released under CC-BY-4.0. Code is released under MIT.

  GitHub: https://github.com/jon-chun/cei-tom-dataset-base  HuggingFace: https://huggingface.co/datasets/jonc/cei-benchmark