Is it generated? A comparative study on human vs generated code

This is the repository for the Python code and dataset used in the paper:

*Title:* Is it generated? A comparative study on human vs generated code

The repository contains the following files and folders:

- [dataset/](./dataset/) - Contains all the datasets used in the study.

- [req.txt](./req.txt) - Contains all the requirements for which are needed to set up the repository.

- [generate_instances.py](./generate_instances.py) - Contains the script which is executed to generate 10 instances for each HumanEval instance

- [create_dataset.py](./create_dataset.py) - Contains the script which is executed to find Gleast and Gmost, as well as the datasets for the ablation study.

- [main.py](./main.py) - Contains the main script, which is executed to iteratively prompt the various LLMs with different prompting strategies.

- [eval.py](./eval.py) - Contains the script which is executed to evaluate the outcome of `main.py'.

- [utils.py](./utils.py) - Contains utility functions which are used in multiple parts of the repository.

## Environment

In order to execute the environment, we suggest creating a virtual environment with Python:

```cli

python -m venv env

source env/bin/activate

pip install -r req.txt

```

Additionally, to the python environment a running instance of Ollama - or any other framework - to run the LLMs locally. PyLint, which is used to assess the code quality for technical debt, is automatically installed with the Python environment, namely version 4.0.4.

## How to generate instances

With the following command, the Python script is executed to generate instances for a specific model (MODEL); note that the <> also needs to be removed. The port for Ollama, or the framework used, needs to be adjusted with the parameter --base-url.

```cli

python generate_instances.py --input-path <PATH/HumanEval> --output-path <PATH/Output> --model <MODEL> --base-url "http://localhost:11434"

```

## Create datasets'

With the following command, the Python script is executed to create from the generated instances the various datasets used in the study. --base-dataset is used to load the human-written solution, to which the generated instances are compared.

```cli

python create_dataset.py --model <MODEL> --input-path <PATH/Input> --output-path <PATH/Output> --base-dataset <PATH/HumanEval>

```

## Main experiment

With the following command, the script is executed to prompt the LLM up to 10 times. The script needs to be executed 10 times, which increases REPETITION_ID from 1 to 10, with various combinations of --variant for human-written (canonical) solution and the LLM-written (generated) solution, --PS indicates the prompting strategy applied. While --similar and --type are used to indicate the similarity which should be inspected or the category (TD or COR). The type is needed only for the ablation study, for Gleast and Gmost, it remains empty ("")

```cli

python main.py --input-path <PATH/Input> --results-path <PATH/Output> --model <MODEL> --repetition <REPETITION_ID> --variant <generated|canonical> --PS <improve|refactor|fix> --similar <"",LS,MS> --type <""|TD|COR>

```

## Evaluation

With the following command, the script is executed to evaluate the outcome of the main experiment. It will compute the Mann-Whitney U test, both two-tailed and one-tailed (m<n), for all three research questions, LLMs, and prompting strategies. For different executions, Gleast and Gmost, the script needs to be executed individually as they are expected to be in separate folders.

```cli

python eval.py --base-path <PATH/Input>

```