LangAGI-Lab/SHOR

SHOR（简单harness优化排名）是一个包含人工验证优化场景的集合，旨在直接评估harness优化器。与通过目标代理的最终性能间接评估优化器不同，SHOR使用优先级排名来量化优化器识别哪些harness组件（例如工具、提示、内存或工作流）应被更新以提高性能的能力。数据集包括两个主要配置：SHOR包含182个人工验证的harnesses，覆盖SWE-bench Verified、GAIA、Spider 2.0-lite和τ²-Bench等领域；SHOR-Flaw包含122个有缺陷的harnesses，用于测试优化器识别和纠正错误的能力。关键特点包括直接评估（量化步骤级优化能力，无需昂贵的rollout）、优先级排名（评估优化器优先更新harness组件的能力）和成本效益（平均比传统最终改进观察便宜8倍、快17倍）。

SHOR (Simple Harness Optimization Ranking) is a collection of human-verified optimization scenarios designed to enable the direct evaluation of harness optimizers. Instead of evaluating optimizers indirectly via the end-performance of target agents, SHOR uses Priority Ranking to quantify an optimizers ability to identify which harness components (e.g., tools, prompts, memory, or workflows) should be updated to improve performance. The dataset includes two main configurations: SHOR with 182 human-verified harnesses spanning domains such as SWE-bench Verified, GAIA, Spider 2.0-lite, and τ²-Bench; and SHOR-Flaw with 122 flawed harnesses used to test an optimizers ability to identify and rectify errors. Key features include Direct Evaluation (quantifies step-level optimization ability without expensive rollouts), Priority Ranking (evaluates how well an optimizer prioritizes harness components for updates), and Cost-Efficient (on average 8× cheaper and 17× faster than conventional end-improvement observations).