VAPU: System for Autonomous Legacy Code Modernization

This dataset includes the extended evaluation resources and data of the evaluation of Verifying Agent Pipeline Updater (VAPU). The system is designed to autonomously update legacy project files based on the given project description to their latest versions. The system is a multi-agent system that uses different roles in a software development team to distribute the update in phases for the given file.

The evaluation of the system is divided into a verification and validation process. Both processes used Zero-Shot Learning (ZSL) and One-shot Learning (OSL) prompts as a comparison for VAPU. The verification process was to update six view files from a legacy web application using five different LLMs in both 1 and 0 temperature as Pass@10. From the results, different error types for updated files were counted for four view files, and for two more complex view files, the successful requirements were counted. LOC and time were counted for most of the updates and correctly Replaced Functions (RF) for view F. For the validation process, 20 Python open-source GitHub repositories were updated with five LLMs as Pass@1, mostly at zero temperature. The updated files were evaluated based on whether the update succeeded, if the original code functionalities worked, and if the new update worked perfectly.

The dataset is divided into four sheets that are explained below:

The sheet “Verification resources” includes the resources of the verification process, listing used view files, LOCs, and used prompts.

The sheet “Verification results” includes the results of the verification process. Each Pass@10 is separated by the used method, LLM, and temperature.

The Sheet “Validation resources” includes the resources of the validation process. The resources include the used GitHub projects, used files, used prompts, and the tasks done to the files.

The Sheet “Validation results” includes the results of the validation process as Pass@1, separated by the used Method and LLM.

The verification results show that VAPU at a lower temperature can provide results at a similar rate of error as OSL/ZSL prompts, with a possibility of higher success in requirements, depending on the task and LLM. The validation results show that the VAPU can provide up to a 22.6% increase in the probability of successful update, depending on the used LLM.

The dataset provides a key resource for understanding how a multi-agent system behaves when updating code compared to OSL/ZSL prompts and with different LLMs and temperatures. Additionally, the validation results serve as a metric to compare different prompt engineering methods and systems for code updates.