CoReDD Bench: An Open Benchmark for Defect Detection in Code Review with Quantified Label Error

CoReDD Bench measures how well the defective lines in a proposed change can be identified before it is merged. A detector receives the change as it stood before integration and marks the lines it considers defective. The marks are scored against the lines that caused a later correction. Labels are reconstructed rather than annotated. For every correction observed after integration, language model agents search the commit history for the change that introduced the defect and mark the lines within it that caused the correction. Reconstruction is stochastic and imperfect. A stratified manual audit estimates its error rate, and that estimate is propagated into every reported score as a posterior interval. The estimated error is directional. Marks that miss the root cause dominate, while omissions stay orders of magnitude below, so the benchmark releases positive labels only. Under the propagated noise the worst case resolution is 0.12 in F1 against an attainable ceiling of 0.68. Any score against these labels has to be read on that scale. The release covers 622 correction and defect pairs across 360 defective changes from six Python repositories. The repositories were drawn by stratified sampling from a cohort of 65 with stable review discipline, screened from 9345 Python candidates under an explicit statistical design. The release contains the labels, the audit records, the uncertainty model, and the evaluation software. The repository histories and timestamped GitHub review context the labels were reconstructed from are archived separately as CoReDD Corpus. Results hold within the documented scope. The benchmark makes no claim to represent code review practice at large.