AOCG

### The replication package of AOCGThe repository is divided into two parts: datasets and the code of our AOCG method.### Requirements```- python 3.8- Java 1.8.0- transformers 4.5.1- tree-sitter 0.2.2- Pytorch 1.7.1```### Data Preprocessing```Experimental datasets contain the API_SUM dataset, the Hearthstone dataset, and the MBPP dataset. We use tree sitter tool to automatically extract the API terms and sketches of programs.Take the MBPP dataset as an example:To extract API terms, run 'data_process/api_extract.py' and acquire the 'api_terms.jsonl' To extract sketches, run 'data_process/sketch_extract.py' and acquire the 'sketches.jsonl' Put the API terms, sketches, complete codes, and requirements into the 'final_train.jsonl' and 'final_test.jsonl'.```### TrainingGiven a specific requirement, the APIer predicts API terms, and the Sketcher outputs corresponding the sketch based on the API terms and requirements. And the Coder fills the sketch to a complete program according to the API terms, sketch and requirement.```export CUDA_VISIBLE_DEVICES=0python AOCG_finetune.py \--stage_1 nl_pp \--stage_2 nl_pp_ss \--stage_3 nl_ss_pp_code \--local_rank -1 ```### InferenceThe AOCG predicts code snippets in a progressive generation manner, and write the predicted codes into 'xx.output'.```export CUDA_VISIBLE_DEVICES=0python AOCG_inference.py \--stage_1 nl_pp \--stage_2 nl_pp_ss \--stage_3 nl_ss_pp_code \--local_rank -1 ```### Evaluation```After acquiring the generated codes, evaluate the programs by running 'evaluator/evaluation.py'.```### AOCG复现包 本仓库分为两大模块：数据集与我们提出的AOCG方法的实现代码。 ### 环境依赖 - Python 3.8 - Java 1.8.0 - transformers 4.5.1 - tree-sitter 0.2.2 - PyTorch 1.7.1 ### 数据预处理 实验数据集包含API_SUM数据集、炉石传说数据集以及MBPP数据集。我们使用tree-sitter工具自动提取程序的API术语与程序草图。以MBPP数据集为例：若需提取API术语，运行`data_process/api_extract.py`脚本，即可得到`api_terms.jsonl`文件；若需提取程序草图，运行`data_process/sketch_extract.py`脚本，即可得到`sketches.jsonl`文件。将API术语、程序草图、完整代码与需求文本整合至`final_train.jsonl`与`final_test.jsonl`文件中。 ### 模型训练 针对给定的需求文本，APIer模块负责预测API术语，Sketcher模块基于API术语与需求文本输出对应的程序草图，而Coder模块则根据API术语、程序草图与需求文本将草图补全为完整的可执行程序。 export CUDA_VISIBLE_DEVICES=0 python AOCG_finetune.py --stage_1 nl_pp --stage_2 nl_pp_ss --stage_3 nl_ss_pp_code --local_rank -1 ### 模型推理 AOCG以渐进式生成的方式预测代码片段，并将预测得到的代码写入`xx.output`文件中。 export CUDA_VISIBLE_DEVICES=0 python AOCG_inference.py --stage_1 nl_pp --stage_2 nl_pp_ss --stage_3 nl_ss_pp_code --local_rank -1 ### 模型评估 在获取生成的代码后，运行`evaluator/evaluation.py`脚本对生成的程序进行评估。