The Artifact of the ESEC/FSE 2023 Paper Titled "Natural Language to Code: How Far are We?"

In this online repository, we release the source code of each of the selected techniques as well as the experiment results from each technique (which are stored in the <em>Results.zip</em> file). For each technique, we also provide our scripts to fine this approach on the CodeSearchNet-Python dataset. For example, <em>finetune.sh/inference.sh</em> are used to finetune/evaluate CodeBERT and they are under "CodeBERT/CodeBERT". Our evaluation dataset CodeSearchNet is a well-known benchmark and it can be downloaded on its official webpage. The code to calculate the evaluation metrics are reused from CodeBLEU. Below is a piece of code generated by CodeT5. In this case, CodeT5 generates a statement recurrently, which leads to the syntactic error. Despite that, the code itself fulfills certain functionalities, and that is why it can achieve a CodeBLEU of 24.9%. <pre><code class="language-python">def makeMimiLocal(filename): try: with open(filename, 'rb') as f: data = f.read() except IOError: data = b'' data = data.decode('utf-8') data = data.replace(b'\x00', b'\x00') data = data.replace(b'\x00', b'\x00') data = data.replace(b'\x00', b'\x00') data = data.replace(b'\x00', b'\x00') data = data.replace(b'\x00', b'\x00') data = data.replace(b'\x00', b'\x00') data = data.replace(b'\x00', b'\x00') data = data.replace(b'\x00', b'\x00') data = data.replace(b'\x00', b'\x00') data = data.replace(b'\x00', b'\x00') data = data.replace(b'\ </code></pre> We also release the 100 randomly-selected queries as well as the code generated by ChatGPT in the <em>chatGPT.jsonl</em>.本在线仓库发布了所选取的各技术的源代码，以及各技术对应的实验结果（实验结果存储于<em>Results.zip</em>压缩文件中）。针对每种技术，我们还提供了在代码搜索网络-Python（CodeSearchNet-Python）数据集上微调该方法的脚本。例如，<em>finetune.sh/inference.sh</em>用于对代码BERT（CodeBERT）进行微调/评估，其位于"CodeBERT/CodeBERT"路径下。 本次使用的评估数据集代码搜索网络（CodeSearchNet）是一款广为人知的基准测试集，可从其官方网页下载获取。评估指标的计算代码复用自代码BLEU（CodeBLEU）。 以下为一段由代码T5（CodeT5）生成的代码。在此示例中，代码T5循环生成语句，导致了语法错误。尽管如此，该代码仍实现了部分功能，因此其代码BLEU（CodeBLEU）得分可达24.9%。 <pre><code class="language-python">def makeMimiLocal(filename): try: with open(filename, 'rb') as f: data = f.read() except IOError: data = b'' data = data.decode('utf-8') data = data.replace(b'x00', b'x00') data = data.replace(b'x00', b'x00') data = data.replace(b'x00', b'x00') data = data.replace(b'x00', b'x00') data = data.replace(b'x00', b'x00') data = data.replace(b'x00', b'x00') data = data.replace(b'x00', b'x00') data = data.replace(b'x00', b'x00') data = data.replace(b'x00', b'x00') data = data.replace(b'x00', b'x00') data = data.replace(b'x00', b'x00') data = data.replace(b'x00', b'x00') data = data.replace(b'x00', b'x00') data = data.replace(b' </code></pre> 我们还在<em>chatGPT.jsonl</em>中发布了100条随机选取的查询语句，以及由ChatGPT生成的代码。