BIG-Bench Hard (BBH) Data Quality Investigation Report
收藏Zenodo2025-12-29 更新2026-05-26 收录
下载链接：
https://zenodo.org/doi/10.5281/zenodo.18090554
下载链接
链接失效反馈官方服务：
资源简介：
44 ground truth labeling errors identified in the BIG-Bench Hard (BBH) benchmark dataset through systematic 6-model cross-verification.
Verification Models
GPT-4o (OpenAI)
GPT-5.2 Pro (OpenAI, manual verification)
Claude Sonnet 4 (Anthropic)
Claude Opus 4.5 (Anthropic)
DeepSeek-Reasoner (DeepSeek)
Qwen3-8B (Alibaba)
Error Summary
date_understanding: 13 errors (calculation/date arithmetic)
geometric_shapes: 31 errors (systematic K mislabeling)
17 cases: (K) ellipse → (A) circle (SVG arc with rx=ry)
14 cases: (K) trapezoid → (H) rectangle (perpendicular sides)
Impact
12.4% error rate in geometric_shapes ground truth causes models with correct reasoning to be penalized, potentially understating published model capabilities by 6-12%.
提供机构：
Zenodo创建时间：
2025-12-29
