GrayC-AE.zip

<pre><code># GrayC: Greybox Compiler Fuzzing <br> <br> GrayC is a greybox fuzzer for C compilers and a libfuzzer-based tool. In brief, GrayC works as follows. Starting with an initial corpus of test programs, it uses libfuzzer to perform coverage-guided mutation-based fuzzing of the Clang/LLVM compiler, for a time-limited period. Unconventionally, the purpose of this use of libfuzzer is not to find bugs at this stage, but rather to generate a large corpus of interesting test programs. This is achieved by<br> (a) using a custom mutator to yield an interesting space of compiler test programs that are statically-valid, and<br> (b) saving every test program that libfuzzer produces to an external directory.<br> <br> After the fuzzing run has completed, GrayC processes this external directory of test programs to test compilers (scripts) and code analysers and to extract new UB-free test programs for compilers test suites (via the enhanCer).<br> <br> Implementation Details: we have implemented our approach as a set of tools, the direct coverage fuzzer, GrayC, the program transformer, enhanCer, and a set of bash scripts for crash and differential testing. GrayC and the code analysis part of enhanCer were implemented in the LLVM 12.0.1 Framework with additional C/C++ code implementing our mutators on top of ClangFuzzer/libtooling. The enhanCer code transformation was implemented in python3 with a set of bash scripts.<br> <br> This repository contains the data and code to reproduce the results in the paper "GrayC: Graybox Compiler Fuzzing".<br> <br> <br> Get Started:<br> ------------<br> <br> - Download the zip file: GrayC-AE.zip.<br> - Follow the instruction below:<br> - Make sure to install teh tools in use<br> - To install the fuzzer follow the instruction in AE folder<br> <br> <br> Tools and compilers in use:<br> ----------------------------<br> <br> ### GrayC build:<br> 1. Experiments on LLVM 12.0.1 (version from the 4th of October 2021)<br> 2. ninja 1.8.2<br> 3. cmake 3.20.0<br> 4. fdupes 1.6.1<br> 5. remove-parens (Git version: 1b2c68e)<br> 6. flex 2.6.4<br> 7. m4 1.4.18<br> <br> <br> ### Evaluation with:<br> 1. Csmith 2.4.0<br> 2. ClangFuzzer/LLVM v12.0.1 x86<br> 3. universalmutator v1.0.18<br> <br> Csmith requires also m4; we used m4 1.4.18.<br> <br> ### enhanCer and testing scripts:<br> 1. Experiments with: <br> (a) LLVM 11,12,13,14,15, <br> (b) GCC 10,11,12,13, and<br> (c) Microsoft ® C/C++ Optimizing Compiler Version 19.28.29915<br> 2. Machines defaults: GCC 10.3.0 LLVM 12.0.1<br> 3. GraphicsFuzz (Git version: de47649)<br> 4. Python 3.9.3 (or above)<br> 5. creduce 2.10.0 and 2.11.0<br> 6. frama-c standart 22.0 (Titanium), 23.0 (Vanadium), 24.0 (Chromium) and 25.0 (Manganese)<br> <br> ### Git Repositories in use:<br> 1. https://github.com/mc-imperial/remove-parens <br> 2. https://github.com/agroce/universalmutator/releases/tag/v1.0.18 <br> 3. https://github.com/google/graphicsfuzz.git <br> <br> Installation<br> ------------<br> <br> GrayC: to install GrayC follow the instructions [here](AE#readme).<br> <br> Initial corpus miner: to install the our corpus miner [here](scripts/2-mining-init-copus/README.md). In addition, we used for our evaluation the seeds listed [here](scripts/1-DATA-set-of-seeds/). <br> <br> The crash testing scripts are [here](scripts/3-crash-testing/).<br> <br> enhanCer: to install the enhanCer follow the instructions [here](scripts/4-diff-testing/).<br> <br> NOTE: [Compilers](scripts/0-install-compilers/README.md): we give full instructions how to install llvm and gcc from source with or without coverage.<br> <br> <br> Evaluation &amp; Data:<br> ------------------<br> <br> All data and results of our evaluation, including examples in the paper and addition code can be found in the [Evaluation](Evaluation) folder.<br> <br> ### Additional important data (direct links).<br> <br> Controlled experiments (the 24 hours trails):<br> - SECTION V-A: Input corpus [the 24 hours evaluation](AE/data/setA-12-Nov-21)<br> - SECTION V-A: Trails data [here](Evaluation/EVALUATION-VIA-CONTROLLED-EXPERIMENTS/Trials)<br> - SECTION V-B: Scripts, machines setup and results (of running these scripts on our data) for the throughput evaluation [here](AE/throughput/) and [here](Evaluation/EVALUATION-VIA-CONTROLLED-EXPERIMENTS/Coverage/Throughput.png) and [here](Evaluation/EVALUATION-VIA-CONTROLLED-EXPERIMENTS/Bug-finding-trails/data_grayc_paper.xlsx)<br> - SECTION V-C: coverage graphs for GCC, LLVM and LLVM middle- and back-end (line and function coverage) and throughput graph [here](Evaluation/EVALUATION-VIA-CONTROLLED-EXPERIMENTS/Coverage) and [here](Evaluation/EVALUATION-VIA-CONTROLLED-EXPERIMENTS/Bug-finding-trails/data_grayc_paper.xlsx)<br> - SECTION V-D: Bug-finding evaluation for 50 trails [here](Evaluation/EVALUATION-VIA-CONTROLLED-EXPERIMENTS/Bug-finding-trails) and [here](Evaluation/EVALUATION-VIA-CONTROLLED-EXPERIMENTS/Bug-finding-trails/data_grayc_paper.xlsx)<br> <br> Evaluation in the wild:<br> - The initial corpus [up-to-date](AE/data/setA) <br> - The bugs [bugs](Evaluation/USING-GRAYC-IN-THE-WILD/bug-reports) found<br> - Test case contributions [test programs](Evaluation/USING-GRAYC-IN-THE-WILD/test-contribution) to the Clang/LLVM test suite.<br> <br> <br> Evaluation<br> ------------<br> <br> 1. Mining corpus: to mine your corpus using Csmith programs, follow the instructions [here](scripts/2-mining-init-copus/README.md).<br> 2. Fuzzing: follow the instructions [here](AE#readme).<br> 3. Throughput: follow the instructions [here](AE#readme).<br> 4. Coverage: follow the instructions [here](AE#readme).<br> <br> <br> Bug Finding<br> -------------<br> Compiler and code analysers bugs found by GrayC: All date in the Evaluation folder.</code></pre>