Autonomous Tester Agent Benchmark

This is a benchmark for Autonomous Tester Agents. This benchmark is composed of a hundred test cases for 3 web applications written by professional testers. Half of the test cases are mutations of the original ones to better evaluate the capacity of agents to identify bugs. The test cases are composed of several steps that need precise assertion. An evaluation of the SeeAct model can also be found in the repository, as well as the source code to replicate the results. This repository is the replication package of a paper accepted to the ISSTA 2025 conference. (https://conf.researchr.org/home/issta-2025)