AOCG

<pre>### The replication package of AOCG<br><br>The repository is divided into two parts: datasets and the code of our AOCG method.<br><br><br>### Requirements<br>```<br>- python 3.8<br>- Java 1.8.0<br>- transformers 4.5.1<br>- tree-sitter 0.2.2<br>- Pytorch 1.7.1<br>```<br><br>### Data Preprocessing<br>```<br>Experimental datasets contain the API_SUM dataset, the Hearthstone dataset, and the MBPP dataset. We use tree sitter tool to automatically extract the API terms and sketches of programs.<br><br>Take the MBPP dataset as an example:<br><br>To extract API terms, run 'data_process/api_extract.py' and acquire the 'api_terms.jsonl' <br><br>To extract sketches, run 'data_process/sketch_extract.py' and acquire the 'sketches.jsonl' <br><br>Put the API terms, sketches, complete codes, and requirements into the 'final_train.jsonl' and 'final_test.jsonl'.<br><br>```<br><br><br>### Training<br>Given a specific requirement, the APIer predicts API terms, and the Sketcher outputs corresponding the sketch based on the API terms and requirements. And the Coder fills the sketch to a complete program according to the API terms, sketch and requirement.<br><br>```<br>export CUDA_VISIBLE_DEVICES=0<br><br>python AOCG_finetune.py \<br>--stage_1 nl_pp \<br>--stage_2 nl_pp_ss \<br>--stage_3 nl_ss_pp_code \<br>--local_rank -1 <br><br>```<br><br><br>### Inference<br>The AOCG predicts code snippets in a progressive generation manner, and write the predicted codes into 'xx.output'.<br><br>```<br>export CUDA_VISIBLE_DEVICES=0<br><br>python AOCG_inference.py \<br>--stage_1 nl_pp \<br>--stage_2 nl_pp_ss \<br>--stage_3 nl_ss_pp_code \<br>--local_rank -1 <br><br>```<br><br><br>### Evaluation<br>```<br>After acquiring the generated codes, evaluate the programs by running 'evaluator/evaluation.py'.<br><br>```</pre>### AOCG 复现套件 本仓库分为两个部分：数据集与我们提出的AOCG方法的代码实现。 ### 环境依赖 - Python 3.8 - Java 1.8.0 - Transformers 4.5.1 - Tree-sitter 0.2.2 - PyTorch 1.7.1 ### 数据预处理 实验数据集包含API_SUM数据集、炉石传说（Hearthstone）数据集以及MBPP数据集。我们使用Tree-sitter工具自动提取程序的API术语与程序草图。 以MBPP数据集为例： - 提取API术语：运行`data_process/api_extract.py`脚本，将得到`api_terms.jsonl`文件 - 提取程序草图：运行`data_process/sketch_extract.py`脚本，将得到`sketches.jsonl`文件 将API术语、程序草图、完整代码与需求文本整合至`final_train.jsonl`与`final_test.jsonl`文件中。 ### 模型训练 针对给定的需求文本，APIer模块预测所需的API术语，Sketcher模块则基于API术语与需求文本输出对应的程序草图；Coder模块再根据API术语、程序草图与需求文本，将草图补全为完整的可运行程序。 export CUDA_VISIBLE_DEVICES=0 python AOCG_finetune.py --stage_1 nl_pp --stage_2 nl_pp_ss --stage_3 nl_ss_pp_code --local_rank -1 ### 模型推理 AOCG采用渐进式生成的方式预测代码片段，并将预测得到的代码写入`xx.output`文件中。 export CUDA_VISIBLE_DEVICES=0 python AOCG_inference.py --stage_1 nl_pp --stage_2 nl_pp_ss --stage_3 nl_ss_pp_code --local_rank -1 ### 模型评估 在获得生成的代码后，运行`evaluator/evaluation.py`脚本对程序进行性能评估。