ReFuzzer: Feedback-Driven Approach to Enhance Validity of LLM-Generated Test Programs

Abstract Existing LLM-based compiler fuzzers often produce syntactically or semantically invalid testprograms, limiting their effectiveness in exercising compiler optimisations and backend com-ponents. We introduce ReFuzzer, a framework for refining LLM-generated test programs bysystematically detecting and correcting compilation and runtime violations (e.g. division byzero or array out-of-bounds accesses). ReFuzzer employs a feedback loop with a local LLMto validate and filter erroneous programs before execution, improving fuzzing effectiveness be-yond crash detection and enabling the generation of diverse yet valid test programs. We evaluated ReFuzzer’s effectiveness across black-, grey- and white-box fuzzing approaches tar-geting LLVM/Clang. ReFuzzer improved test programs’ validity from 47.0–49.4% to 96.6–97.3%,with an average processing time of 2.9–3.5 s per test program on a dual-GPU machine. Fur-ther, refuzzing significantly increased code coverage in critical optimisation and IR generationcomponents. For example, vectorization coverage had an absolute of 9.2%, 2.3%, and 7.1%improvement in black-, grey-, and white-box fuzzing, enhancing testing effectiveness. Table II Raw Data: The complete CPU version raw data corresponding to Table II can be found in the attached file cpu_data.zip. For results of coverage data in table format please follow: CPU_data_table GPU Results: Single GPU (x1) experimental results are provided in the attached file:gpu(x1)_data.zip. Similarly, for table data for GPU(x1) please follow: GPU(x1)_data_table Dual GPU (x2) experimental results are documented in the accompanying research paper Open-Source Contributions We have extended open-source LLM model support for both Fuzz4All and WhiteFox tools. Pull requests implementing these extensions and support have been submitted to the respective original repositories for review and integration: Fuzz4All Integration: Status: ✅ Accepted and merged Pull Request: https://github.com/fuzz4all/fuzz4all/pull/11 WhiteFox Integration: Status: ✅ Accepted and merged Pull Request: https://github.com/ise-uiuc/WhiteFox/pull/15