Automatically fixing dependency breaking changes

Replication Data for Automated Dependency Update Experiments

Dataset Description

This repository contains a comprehensive collection of data from experiments on automated dependency updates using Large Language Models (LLMs). The dataset consists of two main components:

Execution Traces: Detailed logs of LLM interactions and repair attempts, captured using OpenTelemetry and OpenInference (https://github.com/Arize-ai/openinference). These traces provide insights into the behavior and performance of various LLMs in addressing breaking changes caused by dependency updates in Java projects.

Docker Images: Pre-built Docker images containing the state of Java projects after successful repair attempts. These images allow for direct inspection and verification of the changes made by our automated repair system.

This dataset enables analysis and replication of our study on automated dependency updates, offering both low-level interaction data and high-level repair outcomes.

Data Components

1. Execution Traces

Format: JSON Lines, persisted via a custom OpenTelemetry exporter

Content: Spans capturing various aspects of LLM interactions, including:

Input processing

LLM query generation

LLM response parsing

Code modification attempts

Compilation and test execution results

Purpose: Enables detailed analysis of LLM decision-making processes and performance metrics

2. Docker Images

Format: Compressed Docker image files (.tar.gz) in a .zip

Content: Maven project states after successful repair attempts with patch(es) applied and seperated in the second layer of each image.

Purpose: Allows direct inspection and verification of code changes made during repairs and the successful maven outcome

Research Context and Data Usage

This dataset corresponds to experiments described in our study investigating the effectiveness of zero-shot prompting and agentic approaches in automating dependency updates. It provides valuable insights into:

Performance variations across different LLMs

Influence of various factors on repair success rates

Specific code changes made during successful repairs

This dataset can be used for:

Replicating the experimental results presented in the associated study

Conducting further analysis on LLM behavior in software engineering tasks

Developing and benchmarking new approaches for automated dependency updates

Investigating the decision-making processes of LLMs in code modification tasks

Verifying and inspecting successful repair attempts through Docker images

Repair Replication with Docker
To replicate specific repairs using our pre-built Docker images:

1. Ensure you have Docker installed on your system.
2. Use the included docker-images.zip, which first has to be unzipped 

unzip docker-images.zip

after which the images can be loaded as follows:

docker load < image_name.tar.gz

3. Run the Docker container:

docker run -it [image-name]

These Docker images contain the state of projects after repair attempts, allowing for easy inspection and verification of the changes made by our automated repair system.