uw-swag/input-bouncer

BouncerBench数据集用于评估自动化软件工程师代理在何种情况下知道不应该采取行动。它包含基于bug报告文本的任务，模型需要判断报告是否因为模糊而无法修复。该数据集基于SWE-bench构建，并利用了OpenAI提供的注释数据来创建SWE-bench Verified。BouncerBench dataset is used to evaluate when automated software engineer agents know not to act. It consists of tasks based on bug-report text, where the model decides if a report is too vague to fix. The dataset is built on top of SWE-bench and utilizes annotation data provided by OpenAI to create SWE-bench Verified.