Autonomous Tester Agent Benchmark

This is a benchmark for Autonomous Tester Agents. This benchmark is composed of a hundred test cases for 3 web applications written by professional testers. Half of the test cases are mutations of the original ones to better evaluate the capacity of agents to identify bugs. The test cases are composed of several steps that need precise assertion.

An evaluation of the SeeAct model can also be found in the repository, as well as the source code to replicate the results. This repository is the replication package of a paper currently submitted to the ISSTA conference. (https://conf.researchr.org/home/issta-2025)

本基准测试集面向自主测试智能体（Autonomous Tester Agents）。该测试集由专业测试人员为3款Web应用编写的100条测试用例构成，其中半数测试用例为原始用例的变异版本，旨在更精准地评估智能体的缺陷识别能力。所有测试用例均包含若干需执行精准断言（assertion）的步骤。

本仓库中还收录了SeeAct模型的评估结果与可复现该实验结果的源代码。本仓库为一篇已提交至ISSTA会议的论文的复现包，相关会议主页为https://conf.researchr.org/home/issta-2025。