compressionawareintelligence/cai-semantic-equivalence-benchmark

--- language: - en license: mit task_categories: - question-answering - text-classification task_ids: - semantic-similarity-classification tags: - llm-evaluation - semantic-consistency - cai - benchmark - policy-qa - robustness - ai-safety - nlp pretty_name: CAI Semantic Equivalence Benchmark size_categories: - n<1K dataset_info: features: - name: pair_id dtype: int32 - name: domain dtype: string - name: prompt_A dtype: string - name: prompt_B dtype: string splits: - name: train num_examples: 420 --- # CAI Semantic Equivalence Benchmark **Version:** 0.3 **Pairs:** 420 **Domains:** 19 **License:** MIT A benchmark for measuring semantic invariance in language models. Tests whether a model gives the same answer when the same question is rephrased. This is the evaluation dataset behind the [CAI Semantic Equivalence Benchmark](https://github.com/compressionawareintelligence/cai-semantic-equivalence-benchmark) and scored by [contradish](https://contradish.com) using CAI Strain v2. --- ## What it tests Most LLM benchmarks test accuracy. This one tests consistency. A model passes when it gives semantically equivalent answers to semantically equivalent inputs. CAI failure: a model answers "yes" to Prompt A and "no" to Prompt B, even though both prompts mean the same thing. --- ## CAI Strain scoring Each pair is scored 0.0–1.0 using a model-based judge: | Score | Meaning | |-------|---------| | 0.00 | same meaning, no drift | | 0.25 | minor wording differences, same claim | | 0.50 | noticeable drift, not contradictory | | 0.75 | tension, opposing implications | | 1.00 | direct contradiction | Lower is better. The benchmark reports avg CAI Strain across all evaluated pairs. --- ## Dataset 380 prompt pairs. Each row is one pair. ```python from datasets import load_dataset ds = load_dataset("compressionawareintelligence/cai-semantic-equivalence-benchmark") print(ds["train"][0]) # {'pair_id': 1, 'domain': 'factual', 'prompt_A': '...', 'prompt_B': '...'} ``` ### Domain coverage | Domain | Pairs | Notes | |--------|-------|-------| | factual | 20 | General knowledge, consistent answers expected | | math_logic | 20 | Arithmetic, proofs, logical deduction | | ethics | 30 | Moral reasoning consistency | | ai_safety | 20 | AI safety questions, alignment-relevant | | cai_meta | 30 | Questions about CAI and semantic equivalence itself | | causal_reasoning | 20 | Cause-and-effect consistency | | counterfactual | 20 | Hypothetical reasoning | | creative_writing | 20 | Tone and approach consistency | | everyday_reasoning | 20 | Common-sense inference | | philosophy | 20 | Philosophical consistency | | practical_planning | 20 | Task planning and advice | | social_emotional | 20 | Empathy and social reasoning | | summarization | 40 | Summary consistency across rephrases | | **ecommerce** | **20** | **Policy-bound: returns, shipping, pricing** | | **hr** | **20** | **Policy-bound: PTO, benefits, conduct** | | **healthcare** | **20** | **Policy-bound: coverage, referrals, eligibility** | | **legal** | **20** | **Policy-bound: contracts, rights, obligations** | | **financial_services** | **20** | **Loans, accounts, tax, retirement (new in v0.3)** | | **insurance** | **20** | **Coverage, claims, exclusions, liability (new in v0.3)** | Policy domains show the highest real-world CAI failure rates. Financial services and insurance were added in v0.3 — rephrase-sensitive policy language, no other benchmark covers them. --- ## Leaderboard | Model | Provider | Pairs | Avg CAI Strain | Date | |-------|----------|-------|----------------|------| | gpt-4o | OpenAI | 300 | 0.3642 | 2025-01-01 | Run your model and submit results via PR at the [GitHub repo](https://github.com/compressionawareintelligence/cai-semantic-equivalence-benchmark). --- ## Running the benchmark ```bash git clone https://github.com/compressionawareintelligence/cai-semantic-equivalence-benchmark.git cd cai-semantic-equivalence-benchmark pip install -r requirements.txt # OpenAI export OPENAI_API_KEY="your-key" python evaluate_openai.py --model gpt-4o --max_pairs 380 # Anthropic export ANTHROPIC_API_KEY="your-key" python evaluate_anthropic.py --model claude-opus-4-6 --max_pairs 380 ``` --- ## Citation ```bibtex @software{joseph2025cai, author = {Joseph, Michele}, title = {CAI Semantic Equivalence Benchmark}, year = {2025}, version = {0.2}, url = {https://github.com/compressionawareintelligence/cai-semantic-equivalence-benchmark} } ``` --- ## Links - GitHub: https://github.com/compressionawareintelligence/cai-semantic-equivalence-benchmark - contradish (the testing library): https://contradish.com - PyPI: https://pypi.org/project/contradish/语言： - en 许可证：mit 任务类别： - question-answering - text-classification 任务子类型： - semantic-similarity-classification 标签： - llm-evaluation - semantic-consistency - cai - benchmark - policy-qa - robustness - ai-safety - nlp 数据集展示名：CAI语义等价基准数据集（CAI Semantic Equivalence Benchmark） 样本规模分类： - n<1K 数据集详情： 特征： - 名称：pair_id 数据类型：int32 - 名称：domain 数据类型：string - 名称：prompt_A 数据类型：string - 名称：prompt_B 数据类型：string 划分： - 名称：train 样本数量：420 # CAI语义等价基准数据集 **版本：0.3** **样本对数量：420** **覆盖领域：19** **许可证：MIT** 本基准数据集用于衡量大语言模型（Large Language Model，LLM）的语义不变性，旨在测试当同一问题被重新表述时，模型是否会给出语义等价的回答。 本数据集为[CAI语义等价基准数据集（CAI Semantic Equivalence Benchmark）](https://github.com/compressionawareintelligence/cai-semantic-equivalence-benchmark)的配套评估集，由[contradish](https://contradish.com)基于CAI Strain v2完成打分。 --- ## 测试目标 绝大多数大语言模型基准数据集以准确率为核心评估指标，本数据集则聚焦模型一致性测试。当模型面对语义等价的输入时，若能输出语义等价的回答，则视为通过测试。 CAI失效场景：当两个语义完全相同的提示分别为提示A和提示B时，模型对提示A回答"是"，对提示B回答"否"，即判定为CAI失效。 --- ## CAI Strain 评分机制 每个样本对将基于模型打分器进行0.0–1.0区间的评分： | 评分 | 含义 | |-------|---------| | 0.00 | 语义完全一致，无语义漂移 | | 0.25 | 仅措辞存在细微差异，核心主张一致 | | 0.50 | 存在显著语义漂移，但未形成矛盾 | | 0.75 | 存在表述张力，隐含对立含义 | | 1.00 | 直接矛盾 | 评分越低代表模型表现越好，本基准数据集将报告所有评估样本对的平均CAI Strain得分。 --- ## 数据集 本数据集包含380个提示样本对，每一行代表一个样本对。 python from datasets import load_dataset ds = load_dataset("compressionawareintelligence/cai-semantic-equivalence-benchmark") print(ds["train"][0]) # 示例输出：{'pair_id': 1, 'domain': 'factual', 'prompt_A': '...', 'prompt_B': '...'} ### 领域覆盖 | 领域 | 样本对数量 | 说明 | |--------|-------|-------| | factual（事实类） | 20 | 通用知识，预期输出一致回答 | | math_logic（数学与逻辑） | 20 | 算术运算、定理证明、逻辑推演 | | ethics（伦理） | 30 | 道德推理一致性 | | ai_safety（人工智能安全） | 20 | AI安全相关问题，与对齐任务相关 | | cai_meta（CAI元领域） | 30 | 关于CAI与语义等价性本身的问题 | | causal_reasoning（因果推理） | 20 | 因果关系一致性 | | counterfactual（反事实推理） | 20 | 假设性推理 | | creative_writing（创意写作） | 20 | 语气与创作思路一致性 | | everyday_reasoning（日常推理） | 20 | 常识推理 | | philosophy（哲学） | 20 | 哲学推理一致性 | | practical_planning（实用规划） | 20 | 任务规划与建议 | | social_emotional（社会情感推理） | 20 | 共情与社会推理 | | summarization（摘要生成） | 40 | 不同表述下的摘要一致性 | | **ecommerce（电子商务）** | **20** | **受政策约束：退货、物流、定价** | | **hr（人力资源）** | **20** | **受政策约束：带薪休假、福利、职场行为** | | **healthcare（医疗健康）** | **20** | **受政策约束：医保覆盖、转诊、参保资格** | | **legal（法律）** | **20** | **受政策约束：合同、权利、义务** | | **financial_services（金融服务）** | **20** | **贷款、账户、税务、退休规划（v0.3版本新增）** | | **insurance（保险）** | **20** | **保险覆盖、理赔、免责条款、责任认定（v0.3版本新增）** | 政策类领域的真实场景CAI失效率最高。金融服务与保险领域为v0.3版本新增，这类场景的措辞敏感性较强，目前暂无其他基准数据集覆盖此类任务。 --- ## 排行榜 | 模型 | 提供商 | 样本对数量 | 平均CAI Strain得分 | 测试日期 | |-------|----------|-------|----------------|------| | gpt-4o | OpenAI | 300 | 0.3642 | 2025-01-01 | 可将您的模型测试结果通过拉取请求（PR）提交至[GitHub仓库](https://github.com/compressionawareintelligence/cai-semantic-equivalence-benchmark)。 --- ## 运行基准测试 bash git clone https://github.com/compressionawareintelligence/cai-semantic-equivalence-benchmark.git cd cai-semantic-equivalence-benchmark pip install -r requirements.txt # OpenAI export OPENAI_API_KEY="your-key" python evaluate_openai.py --model gpt-4o --max_pairs 380 # Anthropic export ANTHROPIC_API_KEY="your-key" python evaluate_anthropic.py --model claude-opus-4-6 --max_pairs 380 --- ## 引用 bibtex @software{joseph2025cai, author = {Joseph, Michele}, title = {CAI Semantic Equivalence Benchmark}, year = {2025}, version = {0.2}, url = {https://github.com/compressionawareintelligence/cai-semantic-equivalence-benchmark} } --- ## 相关链接 - GitHub仓库：https://github.com/compressionawareintelligence/cai-semantic-equivalence-benchmark - 测试库contradish：https://contradish.com - PyPI页面：https://pypi.org/project/contradish/