Data and Analysis for "On the Reliability of Coverage-based Fuzzer Benchmarking"

<pre><strong>Data and Analysis for "On the Reliability of Coverage-based Fuzzer Benchmarking"</strong> <strong>## Cite</strong> </pre> <pre><code>@inproceedings{benchmarking, author = {B{\"o}hme, Marcel and Szekeres, L{\'a}szl{\'o} and Metzman, Jonathan}, title = {On the Reliability of Coverage-based Fuzzer Benchmarking}, year = {2022}, booktitle = {Proceedings of the 44th International Conference on Software Engineering}, series = {ICSE '22}, pages = {1-13}, doi = {10.1145/3510003.3510230} }</code></pre> <pre> <strong>## Data Analysis</strong> The Jupyter notebook generating all tables and figures can be found in fuzzbench.manual.ipynb <strong>## Generated Images and Tables</strong> The generated data analysis artifacts are also available in this artifact. <strong>## Data</strong> All the data is available in the FuzzBench Reports and will be automatically downloaded. * 20 trials of 23 hours with 15 programs and 10 fuzzers. * Experiment name: 2021-02-17-bug-paper * Report: https://www.fuzzbench.com/reports/2021-02-17-bug-paper/index.html * Data: https://www.fuzzbench.com/reports/2021-02-17-bug-paper/data.csv.gz * Fuzzbench Commit: [38e344fef2f1079579391a0d9dcb52319f7051f2](https://github.com/google/fuzzbench/commits/38e344fef2f1079579391a0d9dcb52319f7051f2) * 30 trials of 23 hours with 11 programs and 10 fuzzers. * Experiment name: 2021-08-19-crash-s * Report: https://www.fuzzbench.com/reports/2021-08-19-crash-s/index.html and * Data: https://www.fuzzbench.com/reports/2021-08-19-crash-s/data.csv.gz * Fuzzbench Commit: db192b60815ac87f69ee0f7f3e37aeac71949e1b * 30 trials of 23 hours with 11 programs and 10 fuzzers. * Experiment name: 2021-08-19-crash-s2 * Report: https://www.fuzzbench.com/reports/2021-08-19-crash-s2/index.html and * Data: https://www.fuzzbench.com/reports/2021-08-19-crash-s2/data.csv.gz * Fuzzbench Commit: db192b60815ac87f69ee0f7f3e37aeac71949e1b The deduplicated data can be found in * 2021-02-17-bug-paper-fixed2.csv.gz * 2021-08-19-crash-s-fixed2.csv.gz * 2021-08-19-crash-s2-fixed2.csv.gz <strong>## Reproducibility</strong> </pre> <pre><code class="language-bash"># Download the precise version of FuzzBench used for the experiment git clone https://github.com/google/fuzzbench.git cd fuzzbench git checkout &lt;Fuzzbench Commit&gt; # Download the internal config file. curl https://storage.googleapis.com/[experiment-name]/config/experiment.yaml &gt; /tmp/experiment-config.yaml make install-dependencies # Launch the experiment using paramters from the internal config file. PYTHONPATH=. python experiment/reproduce_experiment.py -c /tmp/experiment-config.yaml -e &lt;new_experiment_name&gt;</code></pre>