LiteCoder/LiteCoder_SourceCode

# LiteCoder Experiment Reproducing package - To run the pre-train objective use the following scripts: - Reproduce LiteCoder with all objectives: - Navigate the folder `Pre-training` containing the `LiteCoder.py` file - Then, run `Python LiteCoder.py --train-tt --train-cs --train-pd` - The pretrained model is released on [hugging face](https://huggingface.co/LiteCoder/LiteCoder_pretrained), therefore it automatically loads. - To run the ablation studies: - Ablation 1: `Python LiteCoder.py --train-tt` - Ablation 2: `Python LiteCoder.py --train-tt --train-cs` - Ablation 3: `Python LiteCoder.py --train-tt --train-cs --train-pd` - To `Fine-tuning` LiteCoder on downstream tasks: - Navigate to the `Fine-tuning` folder and then `Downstream task` folder: - Code Clone Detection: - Follow the instruction of `readme.md` file. - Code Translation: - Run `setup.sh` file. - Navigate to the `scripts/finetune` and run `translate.sh` file. - To extract the programming language features (i.e., `token type`, `code sememe`, and `code dependencies`) - We used open source datasets to extract language features. we released the extracted datasets on the Hugging Face: - `LT_Java` : [LiteCoder/LT_Java](https://huggingface.co/datasets/LiteCoder/LT_Java) - `LT_Python` : [LiteCoder/LT_Python](https://huggingface.co/datasets/LiteCoder/LT_Python) - `LT_Java_Dependency` : [LiteCoder/LT_Java_Dependency](https://huggingface.co/datasets/LiteCoder/LT_Java_Dependency) - Navigate to the utils directory: - Use either the `Java` or `Python` notebook file to run over your dataset. - Run the cells, for which, you want to extract the features. - Dependencies: - Feature extraction dependencies: ```bash - pip install ast-comments - pip install ast - pip install javalang - pip install tree-sitter - Model training dependencies: ``` bash - pip install transformers - pip install datasets - pip install pytorch_lightning - pip install torch - Or `pip install -r requirements.txt`# LiteCoder 实验复现包 - 若需运行预训练目标任务，请使用如下脚本： - 全目标复现LiteCoder： - 进入包含`LiteCoder.py`文件的`Pre-training`（预训练）文件夹 - 执行命令`python LiteCoder.py --train-tt --train-cs --train-pd` - 预训练模型已在[Hugging Face](https://huggingface.co/LiteCoder/LiteCoder_pretrained)平台发布，因此可自动加载。 - 若需开展消融实验： - 消融实验1：`python LiteCoder.py --train-tt` - 消融实验2：`python LiteCoder.py --train-tt --train-cs` - 消融实验3：`python LiteCoder.py --train-tt --train-cs --train-pd` - 若需在下游任务（Downstream task）上对LiteCoder执行微调（Fine-tuning）： - 进入`Fine-tuning`（微调）文件夹，随后进入`Downstream task`（下游任务）文件夹： - 代码克隆检测： - 请遵照`readme.md`文件中的说明进行操作。 - 代码翻译： - 执行`setup.sh`脚本 - 进入`scripts/finetune`文件夹后，执行`translate.sh`脚本 - 若需提取编程语言特征（即`Token（Token）`类型、代码义元（code sememe）与代码依赖关系（code dependencies））： - 我们采用开源数据集开展语言特征提取工作，已将提取完成的数据集发布至Hugging Face平台： - `LT_Java`：[LiteCoder/LT_Java](https://huggingface.co/datasets/LiteCoder/LT_Java) - `LT_Python`：[LiteCoder/LT_Python](https://huggingface.co/datasets/LiteCoder/LT_Python) - `LT_Java_Dependency`：[LiteCoder/LT_Java_Dependency](https://huggingface.co/datasets/LiteCoder/LT_Java_Dependency) - 进入`utils`（工具）目录： - 可使用`Java`或`Python`格式的Notebook文件对目标数据集执行特征提取 - 运行您需要提取特征的代码单元即可。 - 依赖项配置： - 特征提取依赖项： bash - pip install ast-comments - pip install ast - pip install javalang - pip install tree-sitter - 模型训练依赖项： bash - pip install transformers - pip install datasets - pip install pytorch_lightning - pip install torch - 或直接执行`pip install -r requirements.txt`安装所有依赖项。

原始信息汇总

数据集概述

数据集内容

• 预训练模型: 预训练模型已发布在Hugging Face，运行时自动加载。
• 编程语言特征提取数据集:
  • LT_Java : LiteCoder/LT_Java
  • LT_Python : LiteCoder/LT_Python
  • LT_Java_Dependency : LiteCoder/LT_Java_Dependency

使用方法

• 预训练:
  • 导航至Pre-training文件夹，运行Python LiteCoder.py --train-tt --train-cs --train-pd。
• 消融研究:
  • Ablation 1: Python LiteCoder.py --train-tt
  • Ablation 2: Python LiteCoder.py --train-tt --train-cs
  • Ablation 3: Python LiteCoder.py --train-tt --train-cs --train-pd
• 微调:
  • 导航至Fine-tuning文件夹的Downstream task文件夹:
    • 代码克隆检测: 遵循readme.md文件的指示。
    • 代码翻译:
      • 运行setup.sh文件。
      • 导航至scripts/finetune并运行translate.sh文件。
• 特征提取:
  • 导航至utils目录:
    • 使用Java或Python笔记本文件运行您的数据集。
    • 运行提取特征所需的单元格。

依赖项

• 特征提取依赖:
  • pip install ast-comments
  • pip install ast
  • pip install javalang
  • pip install tree-sitter
• 模型训练依赖:
  • pip install transformers
  • pip install datasets
  • pip install pytorch_lightning
  • pip install torch
• 或运行: pip install -r requirements.txt