GrayC-AE.zip

# GrayC: Greybox Compiler Fuzzing GrayC is a greybox fuzzer for C compilers and a libfuzzer-based tool. In brief, GrayC works as follows. Starting with an initial corpus of test programs, it uses libfuzzer to perform coverage-guided mutation-based fuzzing of the Clang/LLVM compiler, for a time-limited period. Unconventionally, the purpose of this use of libfuzzer is not to find bugs at this stage, but rather to generate a large corpus of interesting test programs. This is achieved by (a) using a custom mutator to yield an interesting space of compiler test programs that are statically-valid, and (b) saving every test program that libfuzzer produces to an external directory. After the fuzzing run has completed, GrayC processes this external directory of test programs to test compilers (scripts) and code analysers and to extract new UB-free test programs for compilers test suites (via the enhanCer). Implementation Details: we have implemented our approach as a set of tools, the direct coverage fuzzer, GrayC, the program transformer, enhanCer, and a set of bash scripts for crash and differential testing. GrayC and the code analysis part of enhanCer were implemented in the LLVM 12.0.1 Framework with additional C/C++ code implementing our mutators on top of ClangFuzzer/libtooling. The enhanCer code transformation was implemented in python3 with a set of bash scripts. This repository contains the data and code to reproduce the results in the paper "GrayC: Graybox Compiler Fuzzing". Get Started: ------------ - Download the zip file: GrayC-AE.zip. - Follow the instruction below: - Make sure to install teh tools in use - To install the fuzzer follow the instruction in AE folder Tools and compilers in use: ---------------------------- ### GrayC build: 1. Experiments on LLVM 12.0.1 (version from the 4th of October 2021) 2. ninja 1.8.2 3. cmake 3.20.0 4. fdupes 1.6.1 5. remove-parens (Git version: 1b2c68e) 6. flex 2.6.4 7. m4 1.4.18 ### Evaluation with: 1. Csmith 2.4.0 2. ClangFuzzer/LLVM v12.0.1 x86 3. universalmutator v1.0.18 Csmith requires also m4; we used m4 1.4.18. ### enhanCer and testing scripts: 1. Experiments with: (a) LLVM 11,12,13,14,15, (b) GCC 10,11,12,13, and (c) Microsoft ® C/C++ Optimizing Compiler Version 19.28.29915 2. Machines defaults: GCC 10.3.0 LLVM 12.0.1 3. GraphicsFuzz (Git version: de47649) 4. Python 3.9.3 (or above) 5. creduce 2.10.0 and 2.11.0 6. frama-c standart 22.0 (Titanium), 23.0 (Vanadium), 24.0 (Chromium) and 25.0 (Manganese) ### Git Repositories in use: 1. https://github.com/mc-imperial/remove-parens 2. https://github.com/agroce/universalmutator/releases/tag/v1.0.18 3. https://github.com/google/graphicsfuzz.git Installation ------------ GrayC: to install GrayC follow the instructions [here](AE#readme). Initial corpus miner: to install the our corpus miner [here](scripts/2-mining-init-copus/README.md). In addition, we used for our evaluation the seeds listed [here](scripts/1-DATA-set-of-seeds/). The crash testing scripts are [here](scripts/3-crash-testing/). enhanCer: to install the enhanCer follow the instructions [here](scripts/4-diff-testing/). NOTE: [Compilers](scripts/0-install-compilers/README.md): we give full instructions how to install llvm and gcc from source with or without coverage. Evaluation & Data: ------------------ All data and results of our evaluation, including examples in the paper and addition code can be found in the [Evaluation](Evaluation) folder. ### Additional important data (direct links). Controlled experiments (the 24 hours trails): - SECTION V-A: Input corpus [the 24 hours evaluation](AE/data/setA-12-Nov-21) - SECTION V-A: Trails data [here](Evaluation/EVALUATION-VIA-CONTROLLED-EXPERIMENTS/Trials) - SECTION V-B: Scripts, machines setup and results (of running these scripts on our data) for the throughput evaluation [here](AE/throughput/) and [here](Evaluation/EVALUATION-VIA-CONTROLLED-EXPERIMENTS/Coverage/Throughput.png) and [here](Evaluation/EVALUATION-VIA-CONTROLLED-EXPERIMENTS/Bug-finding-trails/data_grayc_paper.xlsx) - SECTION V-C: coverage graphs for GCC, LLVM and LLVM middle- and back-end (line and function coverage) and throughput graph [here](Evaluation/EVALUATION-VIA-CONTROLLED-EXPERIMENTS/Coverage) and [here](Evaluation/EVALUATION-VIA-CONTROLLED-EXPERIMENTS/Bug-finding-trails/data_grayc_paper.xlsx) - SECTION V-D: Bug-finding evaluation for 50 trails [here](Evaluation/EVALUATION-VIA-CONTROLLED-EXPERIMENTS/Bug-finding-trails) and [here](Evaluation/EVALUATION-VIA-CONTROLLED-EXPERIMENTS/Bug-finding-trails/data_grayc_paper.xlsx) Evaluation in the wild: - The initial corpus [up-to-date](AE/data/setA) - The bugs [bugs](Evaluation/USING-GRAYC-IN-THE-WILD/bug-reports) found - Test case contributions [test programs](Evaluation/USING-GRAYC-IN-THE-WILD/test-contribution) to the Clang/LLVM test suite. Evaluation ------------ 1. Mining corpus: to mine your corpus using Csmith programs, follow the instructions [here](scripts/2-mining-init-copus/README.md). 2. Fuzzing: follow the instructions [here](AE#readme). 3. Throughput: follow the instructions [here](AE#readme). 4. Coverage: follow the instructions [here](AE#readme). Bug Finding ------------- Compiler and code analysers bugs found by GrayC: All date in the Evaluation folder.