MultiPL-E-completions

# Raw Data from MultiPL-E

**This repository is frozen. See https://huggingface.co/datasets/nuprl/MultiPL-E-completions for a more complete version of this repository.**

Uploads are a work in progress. If you are interested in a split that is not yet available, please contact [e-mail].

This repository contains the raw data -- both completions and executions -- from MultiPL-E that was used to generate several experimental results from the
MultiPL-E, SantaCoder, and StarCoder papers.

The original MultiPL-E completions and executions are stored in JOSN files. We use [the following script](https://github.com/nuprl/MultiPL-E/blob/main/upload_completions.py-)
to turn each experiment directory into a dataset split and upload to this repository.

Every split is named `base_dataset`.`language`.`model`.`temperature`.`variation`

- `base_dataset` is either `humaneval` or `mbpp`.
- `language` is the file extension of the programming language. E.g., `py` for Python or `sh` for Bash.
- `model` is the name of the model. Some model names used by MultiPL-E:
- `bigcode_15b_1000m`: StarCoderBase
- `bigcode_15b_200m`, `bigcode_15b_400m`, `bigcode_15b_600m`, `bigcode_15b_800m`: the 20%, 40%, 60%, 80% checkpoints for StarCoderBase
- `pystarcoder2`: StarCoder
- there are others, just have a look :)
- `temperature`: this is either 0.2 or 0.8
- `variation`: the variation of the MultiPL-E prompt to use, which should be `reworded`. MultiPL-E has several prompt ablations and the `reworded` prompts are the canonical variation.

# MultiPL-E 原始数据集

**本仓库已处于冻结状态，完整版本请访问 https://huggingface.co/datasets/nuprl/MultiPL-E-completions 获取。**

数据上传工作仍在推进中。若您需要尚未开放的数据集拆分版本，请联系 [e-mail]。

本仓库存储了MultiPL-E项目的原始数据（包含代码补全结果与执行结果），这些数据曾用于生成MultiPL-E、SantaCoder及StarCoder相关论文中的多项实验结果。

MultiPL-E的原始补全结果与执行结果均存储于JSON文件中。我们通过[指定脚本](https://github.com/nuprl/MultiPL-E/blob/main/upload_completions.py-)将每个实验目录转换为数据集拆分集，并上传至本仓库。

所有数据集拆分均遵循如下命名格式：`base_dataset`.`language`.`model`.`temperature`.`variation`

- `base_dataset` 为基础数据集名称，仅支持 `humaneval` 或 `mbpp`。
- `language` 为对应编程语言的文件扩展名，例如Python对应 `py`，Bash对应 `sh`。
- `model` 为模型名称，MultiPL-E常用的模型名称包括：
- `bigcode_15b_1000m`：StarCoderBase
- `bigcode_15b_200m`、`bigcode_15b_400m`、`bigcode_15b_600m`、`bigcode_15b_800m`：分别对应StarCoderBase 20%、40%、60%、80%训练进度的模型检查点
- `pystarcoder2`：StarCoder
- 其余模型名称请自行查阅本仓库。
- `temperature` 为生成温度参数，仅支持 `0.2` 或 `0.8`。
- `variation` 为MultiPL-E所用提示词的变体类型，当前仅支持 `reworded`。MultiPL-E包含多种提示词消融实验设置，其中`reworded`提示词为标准基准变体。