Nutanix/cpp_unit_tests_unprocessed_llama3.1_vs_llama3.1_finetuned_gpt_judge

---
dataset_info:
features:
- name: Code
dtype: string
- name: Unit Test_llama3.1
dtype: string
- name: Unit Test_llama3.1_finetuned
dtype: string
- name: Unit Test
dtype: string
- name: Winning Model
dtype: string
- name: Judgement
dtype: string
splits:
- name: train
num_bytes: 10853630
num_examples: 201
download_size: 2697802
dataset_size: 10853630
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
---
# Unit Test Evaluation Results

This repository details the evaluation of unit tests generated by llama models. It compares the unit tests produced by two models: llama3.1 8B Instruct and finetuned llama3.1 8b Instruct against the [groundtruth data](https://huggingface.co/datasets/Nutanix/cpp-unit-test-benchmarking-dataset). In this evaluation, gpt-4o-mini served as the judge, assessing how well the unit tests from both models aligned with the ground truth.

## Models Used

### [Llama3.1 8B Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct)
- **HuggingFace Link**: [Meta-Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct)
- **Precision**: BF16 Precision
- **Description**: Base instruct model to generate the unit tests.

### LLama3.1 8B Instruct - finetuned
- **HuggingFace Link**: [Finetuned LoRA adapter](https://huggingface.co/Nutanix/Meta-Llama-3.1-8B-Instruct_cppunittest_lora_8_alpha_16)
- **Finetune Settings**: LoRaRank = 8, alpha = 16, finetuned llama3.1 8b instruct model for 2 epochs on [this](https://huggingface.co/datasets/Nutanix/cpp_unit_tests_finetuning_dataset_chat_format_less_than_8k) dataset.
- **Description**: A finetuned model whose unit tests were compared against those generated by base model.

## Dataset

The evaluation utilized the [cpp unit test benchmarking dataset](https://huggingface.co/datasets/Nutanix/cpp_unit_tests_eval_dataset_less_than_8k_extracted_code) as the ground truth.

### Dataset Structure

The dataset was loaded using the following structure:

```python
from datasets import Dataset, load_dataset

# Load the dataset
dataset = load_dataset("Nutanix/cpp_unit_tests_unprocessed_llama3.1_vs_llama3.1_finetuned_gpt_judge")

# View dataset structure

Dataset({
features: ['Code', 'Unit Test_llama3.1', 'Unit Test_llama3.1_finetuned', 'Unit Test', 'Winning Model', 'Judgement'],
num_rows: 201
})
```

## Features:

- **Code**: The source code for which the unit tests are written.
- **Unit Test_llama3.1**: Unit test generated by llama3.1 8b instruct model.
- **Unit Test_llama3.1_finetuned**: Unit test generated by finetuned llama3.1 8b instruct model.
- **Unit Test**: The benchmark or ground truth unit test.
- **Winning Model**: The model whose unit test is closer to the ground truth.
- **Judgement**: The evaluation results comparing the unit tests.

The results are summarized in the table below:

## Unit Test Evaluation Results

| Outcome | Count |
|---------------------------------|-------|
| Llama3.1-8b Instruct finetuned | 105 |
| Llama3.1-8b Instruct | 87 |
| Tie | 9 |

### Explanation

1. Llama3.1-8b Instruct finetuned Wins: Llama3.1-8b Instruct finetuned aligned more closely with the ground truth in 105 cases.
2. Llama3.1-8b Instruct Wins: Llama3.1-8b Instruct model aligned more closely with the ground truth in 87 cases.
3. Tie: 9 instances where results were tied between the models.

### Win Rates
- Llama3.1-8b Instruct finetuned Win Percentage: 52.2%
- Llama3.1-8b Instruct Win Percentage: 43.3%
- Tie Percentage: 4.5%

### Framework to generate unit test

<img src="https://cdn-uploads.huggingface.co/production/uploads/6658bb3acf5fc31e3a0bd24a/nFUDNtFeAukk_qLZL24F6.png" alt="image/png" width="600" height="400"/>

### Evaluation Approach

The [gpt-4o-mini](https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/), was used as the judge to evaluate which unit test was closer to the ground truth provided by the benchmark dataset. This evaluation highlights the performance differences between the two models and indicates a higher alignment of finetuned llama3.1 model with the benchmarked unit tests.

Prompt used for evaluation: [Evaluation Prompt](https://huggingface.co/datasets/Nutanix/cpp_unittests_llama8b_vs_llama70b_judge_llama70/blob/main/config_evaluator.yaml)

数据集信息：
特征列表：
- 名称：Code，数据类型：字符串（string）
- 名称：Unit Test_llama3.1，数据类型：字符串（string）
- 名称：Unit Test_llama3.1_finetuned，数据类型：字符串（string）
- 名称：Unit Test，数据类型：字符串（string）
- 名称：Winning Model，数据类型：字符串（string）
- 名称：Judgement，数据类型：字符串（string）
数据集划分：
- 名称：训练集（train），字节占用：10853630，样本数量：201
下载大小：2697802，数据集总大小：10853630
配置项：
- 配置名称：默认（default），数据文件：
- 划分：训练集（train），路径：data/train-*

# 单元测试评估结果

本仓库详细记录了基于Llama系列模型生成的单元测试的评估工作。本次评估对比了两款模型生成的单元测试：Llama3.1 8B Instruct 与微调版 Llama3.1 8B Instruct，并以[基准真值（ground truth）](https://huggingface.co/datasets/Nutanix/cpp-unit-test-benchmarking-dataset)作为评估参照标准。本次评估采用gpt-4o-mini作为评判模型，用于衡量两款模型生成的单元测试与基准真值的对齐程度。

## 所用模型

### Llama3.1 8B Instruct
- **HuggingFace 链接**：[Meta-Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct)
- **精度规格**：BF16 精度
- **描述**：用于生成单元测试的基础指令模型。

### 微调版 Llama3.1 8B Instruct
- **HuggingFace 链接**：[微调LoRA（Low-Rank Adaptation，低秩适配器）适配器](https://huggingface.co/Nutanix/Meta-Llama-3.1-8B-Instruct_cppunittest_lora_8_alpha_16)
- **微调设置**：LoRA秩=8，alpha=16，基于[该数据集](https://huggingface.co/datasets/Nutanix/cpp_unit_tests_finetuning_dataset_chat_format_less_than_8k)对Llama3.1 8B Instruct模型进行了2个训练轮次（epoch）的微调。
- **描述**：一款经过微调的模型，其生成的单元测试将与基础模型生成的单元测试进行对比评估。

## 评估数据集
本次评估采用[cpp单元测试基准数据集](https://huggingface.co/datasets/Nutanix/cpp_unit_tests_eval_dataset_less_than_8k_extracted_code)作为基准真值来源。

### 数据集结构
本次评估使用如下代码加载数据集：
python
from datasets import Dataset, load_dataset

# 加载数据集
dataset = load_dataset("Nutanix/cpp_unit_tests_unprocessed_llama3.1_vs_llama3.1_finetuned_gpt_judge")

# 查看数据集结构

Dataset({
features: ['Code', 'Unit Test_llama3.1', 'Unit Test_llama3.1_finetuned', 'Unit Test', 'Winning Model', 'Judgement'],
num_rows: 201
})

## 特征说明
- **Code**：待编写单元测试的源代码（Code）。
- **Unit Test_llama3.1**：由Llama3.1 8B Instruct基础模型生成的llama3.1单元测试（Unit Test_llama3.1）。
- **Unit Test_llama3.1_finetuned**：由微调版Llama3.1 8B Instruct模型生成的微调版llama3.1单元测试（Unit Test_llama3.1_finetuned）。
- **Unit Test**：基准数据集提供的标准单元测试（Unit Test），即基准真值。
- **Winning Model**：生成的单元测试与基准真值对齐度更高的获胜模型（Winning Model）。
- **Judgement**：两款模型单元测试的详细评估结果（Judgement）。

本次评估结果汇总如下表所示：

## 单元测试评估结果汇总表
| 评估结果 | 样本数 |
|------------------------------|--------|
| 微调版Llama3.1-8B Instruct | 105 |
| 基础版Llama3.1-8B Instruct | 87 |
| 平局 | 9 |

### 结果详解
1. **微调版Llama3.1-8B Instruct获胜**：在105个样本中，微调版模型生成的单元测试与基准真值的对齐程度更高。
2. **基础版Llama3.1-8B Instruct获胜**：在87个样本中，基础版模型生成的单元测试与基准真值的对齐程度更高。
3. **平局**：共有9个样本中两款模型的评估结果持平，对齐程度无显著差异。

### 获胜率统计
- 微调版Llama3.1-8B Instruct 获胜占比：52.2%
- 基础版Llama3.1-8B Instruct 获胜占比：43.3%
- 平局占比：4.5%

### 单元测试生成框架
 alt="PNG格式图像"
*图片宽度：600px，高度：400px*

### 评估方法
本次评估采用[gpt-4o-mini](https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/)作为评判模型，用于对比两款模型生成的单元测试与基准数据集提供的基准真值的贴近程度。本次评估结果凸显了两款模型的性能差异，表明微调后的Llama3.1模型与基准单元测试的对齐度更高。

评估所用提示词：[评估提示词配置文件](https://huggingface.co/datasets/Nutanix/cpp_unittests_llama8b_vs_llama70b_judge_llama70/blob/main/config_evaluator.yaml)