GrayC-AE.zip

# GrayC: Greybox Compiler Fuzzing GrayC is a greybox fuzzer for C compilers and a libfuzzer-based tool. In brief, GrayC works as follows. Starting with an initial corpus of test programs, it uses libfuzzer to perform coverage-guided mutation-based fuzzing of the Clang/LLVM compiler, for a time-limited period. Unconventionally, the purpose of this use of libfuzzer is not to find bugs at this stage, but rather to generate a large corpus of interesting test programs. This is achieved by (a) using a custom mutator to yield an interesting space of compiler test programs that are statically-valid, and (b) saving every test program that libfuzzer produces to an external directory. After the fuzzing run has completed, GrayC processes this external directory of test programs to test compilers (scripts) and code analysers and to extract new UB-free test programs for compilers test suites (via the enhanCer). Implementation Details: we have implemented our approach as a set of tools, the direct coverage fuzzer, GrayC, the program transformer, enhanCer, and a set of bash scripts for crash and differential testing. GrayC and the code analysis part of enhanCer were implemented in the LLVM 12.0.1 Framework with additional C/C++ code implementing our mutators on top of ClangFuzzer/libtooling. The enhanCer code transformation was implemented in python3 with a set of bash scripts. This repository contains the data and code to reproduce the results in the paper "GrayC: Graybox Compiler Fuzzing". Get Started: ------------ - Download the zip file: GrayC-AE.zip. - Follow the instruction below: - Make sure to install teh tools in use - To install the fuzzer follow the instruction in AE folder Tools and compilers in use: ---------------------------- ### GrayC build: 1. Experiments on LLVM 12.0.1 (version from the 4th of October 2021) 2. ninja 1.8.2 3. cmake 3.20.0 4. fdupes 1.6.1 5. remove-parens (Git version: 1b2c68e) 6. flex 2.6.4 7. m4 1.4.18 ### Evaluation with: 1. Csmith 2.4.0 2. ClangFuzzer/LLVM v12.0.1 x86 3. universalmutator v1.0.18 Csmith requires also m4; we used m4 1.4.18. ### enhanCer and testing scripts: 1. Experiments with: (a) LLVM 11,12,13,14,15, (b) GCC 10,11,12,13, and (c) Microsoft ® C/C++ Optimizing Compiler Version 19.28.29915 2. Machines defaults: GCC 10.3.0 LLVM 12.0.1 3. GraphicsFuzz (Git version: de47649) 4. Python 3.9.3 (or above) 5. creduce 2.10.0 and 2.11.0 6. frama-c standart 22.0 (Titanium), 23.0 (Vanadium), 24.0 (Chromium) and 25.0 (Manganese) ### Git Repositories in use: 1. https://github.com/mc-imperial/remove-parens 2. https://github.com/agroce/universalmutator/releases/tag/v1.0.18 3. https://github.com/google/graphicsfuzz.git Installation ------------ GrayC: to install GrayC follow the instructions [here](AE#readme). Initial corpus miner: to install the our corpus miner [here](scripts/2-mining-init-copus/README.md). In addition, we used for our evaluation the seeds listed [here](scripts/1-DATA-set-of-seeds/). The crash testing scripts are [here](scripts/3-crash-testing/). enhanCer: to install the enhanCer follow the instructions [here](scripts/4-diff-testing/). NOTE: [Compilers](scripts/0-install-compilers/README.md): we give full instructions how to install llvm and gcc from source with or without coverage. Evaluation & Data: ------------------ All data and results of our evaluation, including examples in the paper and addition code can be found in the [Evaluation](Evaluation) folder. ### Additional important data (direct links). Controlled experiments (the 24 hours trails): - SECTION V-A: Input corpus [the 24 hours evaluation](AE/data/setA-12-Nov-21) - SECTION V-A: Trails data [here](Evaluation/EVALUATION-VIA-CONTROLLED-EXPERIMENTS/Trials) - SECTION V-B: Scripts, machines setup and results (of running these scripts on our data) for the throughput evaluation [here](AE/throughput/) and [here](Evaluation/EVALUATION-VIA-CONTROLLED-EXPERIMENTS/Coverage/Throughput.png) and [here](Evaluation/EVALUATION-VIA-CONTROLLED-EXPERIMENTS/Bug-finding-trails/data_grayc_paper.xlsx) - SECTION V-C: coverage graphs for GCC, LLVM and LLVM middle- and back-end (line and function coverage) and throughput graph [here](Evaluation/EVALUATION-VIA-CONTROLLED-EXPERIMENTS/Coverage) and [here](Evaluation/EVALUATION-VIA-CONTROLLED-EXPERIMENTS/Bug-finding-trails/data_grayc_paper.xlsx) - SECTION V-D: Bug-finding evaluation for 50 trails [here](Evaluation/EVALUATION-VIA-CONTROLLED-EXPERIMENTS/Bug-finding-trails) and [here](Evaluation/EVALUATION-VIA-CONTROLLED-EXPERIMENTS/Bug-finding-trails/data_grayc_paper.xlsx) Evaluation in the wild: - The initial corpus [up-to-date](AE/data/setA) - The bugs [bugs](Evaluation/USING-GRAYC-IN-THE-WILD/bug-reports) found - Test case contributions [test programs](Evaluation/USING-GRAYC-IN-THE-WILD/test-contribution) to the Clang/LLVM test suite. Evaluation ------------ 1. Mining corpus: to mine your corpus using Csmith programs, follow the instructions [here](scripts/2-mining-init-copus/README.md). 2. Fuzzing: follow the instructions [here](AE#readme). 3. Throughput: follow the instructions [here](AE#readme). 4. Coverage: follow the instructions [here](AE#readme). Bug Finding ------------- Compiler and code analysers bugs found by GrayC: All date in the Evaluation folder.# GrayC：灰盒编译器模糊测试（GrayC: Greybox Compiler Fuzzing） GrayC是一款面向C语言编译器的灰盒模糊测试工具，同时也是一款基于libFuzzer的程序。简而言之，GrayC的工作原理如下：从初始测试程序语料库出发，它会在限定时间内使用libFuzzer对Clang/LLVM编译器执行覆盖引导式基于变异的模糊测试。与常规操作不同，此阶段使用libFuzzer的目的并非在此阶段发现漏洞，而是生成一批包含大量高价值测试程序的语料库。这一目标通过以下两种方式实现：(a) 利用自定义变异器生成静态合法（statically-valid）的编译器测试程序空间，且该空间具备足够的探索价值；(b) 将libFuzzer生成的所有测试程序保存至外部目录。 模糊测试运行结束后，GrayC会处理该外部目录中的测试程序，以测试编译器脚本与代码分析器，并通过enhanCer提取适用于编译器测试套件的新型无未定义行为（Undefined Behavior, UB）测试程序。 ## 实现细节 我们将该方法实现为一组工具：直接覆盖模糊器GrayC、程序转换工具enhanCer，以及一套用于漏洞与差分测试的Bash脚本。GrayC与enhanCer的代码分析部分基于LLVM 12.0.1框架开发，并在ClangFuzzer/libtooling之上新增了我们的变异器相关C/C++代码。enhanCer的代码转换功能则通过Python3与一套Bash脚本实现。 本仓库包含了复现论文《GrayC: Graybox Compiler Fuzzing》中实验结果所需的数据与代码。 ## 快速开始 ------------ - 下载压缩包：GrayC-AE.zip。 - 按照以下步骤操作： - 确保已安装所需工具 - 安装模糊器请参照AE文件夹中的说明文档 ## 所用工具与编译器 ---------------------------- ### GrayC 编译构建： 1. 基于LLVM 12.0.1（2021年10月4日版本） 2. Ninja 1.8.2 3. CMake 3.20.0 4. fdupes 1.6.1 5. remove-parens（Git版本：1b2c68e） 6. Flex 2.6.4 7. M4 1.4.18 ### 评估所用工具： 1. Csmith 2.4.0 2. ClangFuzzer/LLVM v12.0.1 x86架构版本 3. universalmutator v1.0.18 注：Csmith同样依赖M4，本次评估使用的版本为1.4.18。 ### enhanCer与测试脚本： 1. 实验环境包括： (a) LLVM 11、12、13、14、15 (b) GCC 10、11、12、13 (c) Microsoft® C/C++ 优化编译器版本19.28.29915 2. 机器默认环境：GCC 10.3.0、LLVM 12.0.1 3. GraphicsFuzz（Git版本：de47649） 4. Python 3.9.3（或更高版本） 5. C-Reduce 2.10.0与2.11.0 6. Frama-C 标准版本22.0（Titanium）、23.0（Vanadium）、24.0（Chromium）与25.0（Manganese） ### 所用Git仓库： 1. https://github.com/mc-imperial/remove-parens 2. https://github.com/agroce/universalmutator/releases/tag/v1.0.18 3. https://github.com/google/graphicsfuzz.git ## 安装 ------------ GrayC：安装GrayC请参照[此处说明](AE#readme)。 初始语料库挖掘工具：安装我们的语料库挖掘工具请参照[此处](scripts/2-mining-init-copus/README.md)。此外，本次评估所用的种子样本可在[此处](scripts/1-DATA-set-of-seeds/)获取。 漏洞测试脚本位于[此处](scripts/3-crash-testing/)。 enhanCer：安装enhanCer请参照[此处](scripts/4-diff-testing/)。 注意：[编译器安装指南](scripts/0-install-compilers/README.md)：我们提供了完整的从源码编译安装LLVM与GCC的说明，支持带/不带覆盖检测的编译配置。 ## 评估与数据 ------------------ 本次评估的全部数据与结果，包括论文中的示例与附加代码，均可在[Evaluation](Evaluation)文件夹中找到。 ### 额外重要数据（直接链接） 受控实验（24小时测试）： - 第五章A节：输入语料库 [24小时评估数据集](AE/data/setA-12-Nov-21) - 第五章A节：测试数据 [此处](Evaluation/EVALUATION-VIA-CONTROLLED-EXPERIMENTS/Trials) - 第五章B节：吞吐量评估所用的脚本、机器配置与运行结果（基于我们的数据集）[此处](AE/throughput/)、[此处](Evaluation/EVALUATION-VIA-CONTROLLED-EXPERIMENTS/Coverage/Throughput.png)与[此处](Evaluation/EVALUATION-VIA-CONTROLLED-EXPERIMENTS/Bug-finding-trails/data_grayc_paper.xlsx) - 第五章C节：GCC、LLVM以及LLVM中端与后端的覆盖度图表（行覆盖与函数覆盖）与吞吐量图表 [此处](Evaluation/EVALUATION-VIA-CONTROLLED-EXPERIMENTS/Coverage)与[此处](Evaluation/EVALUATION-VIA-CONTROLLED-EXPERIMENTS/Bug-finding-trails/data_grayc_paper.xlsx) - 第五章D节：50次测试的漏洞发现评估 [此处](Evaluation/EVALUATION-VIA-CONTROLLED-EXPERIMENTS/Bug-finding-trails)与[此处](Evaluation/EVALUATION-VIA-CONTROLLED-EXPERIMENTS/Bug-finding-trails/data_grayc_paper.xlsx) 野外评估： - 最新版初始语料库 [AE/data/setA](AE/data/setA) - 发现的漏洞 [漏洞报告](Evaluation/USING-GRAYC-IN-THE-WILD/bug-reports) - 提交至Clang/LLVM测试套件的测试用例贡献 [测试程序](Evaluation/USING-GRAYC-IN-THE-WILD/test-contribution) ## 评估流程 ------------ 1. 语料库挖掘：若需使用Csmith程序挖掘自定义语料库，请参照[此处说明](scripts/2-mining-init-copus/README.md)。 2. 模糊测试：请参照[此处说明](AE#readme)。 3. 吞吐量评估：请参照[此处说明](AE#readme)。 4. 覆盖度评估：请参照[此处说明](AE#readme)。 ## 漏洞发现 ------------- GrayC发现的编译器与代码分析器漏洞：所有相关数据均存储于Evaluation文件夹中。