遇见数据集

Raising effort didn't catch more edge cases, and Haiku got nearly the same result for 87% less: 504 Claude Code runs on 21 real bugs

收藏
Zenodo2026-10-30 更新2026-11-05 收录
官方服务:

资源简介:

504 pre-registered Claude Code runs on 21 real bugs fixed between August and October 2026 in 12 open-source projects: Claude Haiku 5.5 and Opus 5.5, four conditions (baseline, effort high, an edge-case prompt, edge-case prompt plus reading the existing tests), 3 runs each. Every run is scored by the maintainers' own tests for the fix, split into cases the bug report shows and cases it leaves out. Page with tables and method: https://tokensave.app/data/claude-edge-case-prompts Findings. No condition moved the share of unreported cases passed by more than 4 points (Haiku 84/82/84/86%, Opus 88/86/91/84%; pre-registered threshold 20 points with a bug-level bootstrap interval excluding 0). The edge-case prompt raised the API cost per run by 60% (Opus) to 134% (Haiku, with test reading); effort high by 22% to 60%. Clean fixes of 63 per cell: Haiku 34, 33, 36, 37; Opus 38, 38, 39, 37. Haiku cost about 87% less than Opus per run (USD 0.088 vs USD 0.69). On 12 of 21 bugs both models passed every unreported case in every condition; on a js-yaml bug no run did. Files. claude-edge-case-prompts.csv: one row per run. claude-edge-case-prompts.json: runs plus bug list, conditions, per-bug results and statistics. claude-edge-case-prompts-chart.png. README.md. Method. Claude Code 2.1.295, headless, Windows 11, acceptEdits, project tests allowed, no installs or network. Only the run's library source changes were scored; maintainer tests were run 5 times at parent and fix and unstable tests dropped. Pre-registration, labels, prompts and scoring code: https://github.com/johnisking/token-cost-by-language/tree/main/claude-report-edge-cases Limitations. 21 bugs, 3 runs per cell: a 20-point effect is ruled out, a few points is not. One prompt wording per condition. 12 of 21 bugs were at the ceiling.

提供机构:
Zenodo
创建时间:
2026-10-30
二维码
社区交流群
二维码
科研交流群
商业服务