llm-fuzz-benchmark

# DIS-Bench: Benchmarking LLMs for System Testing via Directed Input Synthesis A benchmark for evaluating Large Language Models' ability in system testing via Directed Input Synthesis (DIS): that is, generating inputs that cover specific code branches in real-world programs. ## Overview | Dataset | File | Tasks | Description | |---------|------|-------|-------------| | DIS-Bench-Lite | `dataset.jsonl` | 555 | Lightweight evaluation set (15 tasks per program) | | DIS-Bench | `dataset_full.jsonl` | 8267 | Full benchmark | Each task requires generating an input buffer that, when processed by a target program, covers a specific branch direction in the source code. ## Statistics ### By Category | Category | DIS-Bench-Lite | DIS-Bench | |----------|----------------|-----------| | Image | 120 | 798 | | Video/Audio | 60 | 768 | | Documents | 30 | 558 | | Fonts | 45 | 1167 | | Data Formats | 45 | 871 | | Compilers/Interpreters | 60 | 897 | | Binaries | 90 | 1419 | | Network | 45 | 1116 | | Others | 60 | 673 | | **Total** | **555** | **8267** | ### By Program | Program | Format(s) | DIS-Bench-Lite | DIS-Bench | |---------|-----------|----------------|-----------| | binutils (nm) | elf | 15 | 103 | | binutils (objdump) | elf | 15 | 456 | | binutils (readelf) | elf | 15 | 154 | | binutils (size) | elf | 15 | 186 | | binutils (strip) | elf | 15 | 460 | | bloaty | elf,mach-o,pe,wasm | 15 | 60 | | exiv2 | jpeg | 15 | 24 | | ffmpeg | vi,mov,flv,mp4,mp3,h264 | 15 | 430 | | freetype2 | ttf,otf | 15 | 587 | | harfbuzz | ttf,otf | 15 | 522 | | jasper | bmp,mif,pgx,pnm,ras,jpg,png,tga | 15 | 156 | | jhead | jpeg | 15 | 36 | | jq | json | 15 | 23 | | lcms | cms | 15 | 85 | | libjpeg-turbo | jpeg | 15 | 56 | | libpcap | pcap | 15 | 64 | | libpng | png | 15 | 65 | | libsndfile | aifc,aiff,au,caf,mp3,oga,opus,paf,raw,rf64,sds,wav,xi | 15 | 177 | | libtiff (tiff2ps) | tiff | 15 | 198 | | libtiff (tiffsplit) | tiff | 15 | 209 | | libxml2 | xml | 15 | 376 | | lua | lua | 15 | 134 | | mbedtls | dtls | 15 | 100 | | mruby | ruby | 15 | 350 | | mujs | javascript | 15 | 126 | | ncurses | terminfo | 15 | 84 | | openh264 | h264 | 15 | 100 | | openthread | openthread | 15 | 18 | | php | php | 15 | 287 | | proj4 | proj | 15 | 481 | | re2 | re2 | 15 | 23 | | sqlite3 | sql | 15 | 784 | | stb | bmp,gif,jpg,tga,png | 15 | 54 | | tcpdump | pcap | 15 | 998 | | vorbis | vorbis | 15 | 61 | | woff2 | woff2 | 15 | 58 | | xpdf | pdf | 15 | 182 | | **Total** | | **555** | **8267** | ## Target Programs **37 programs** covering **52 file formats**, including popular libraries and utilities from OSS-Fuzz and real-world applications. | Category | Programs | |----------|----------| | Image | libpng, libjpeg-turbo, stb, libtiff, jhead, exiv2, imginfo | | Video/Audio | ffmpeg, openh264, vorbis, lame, sndfile, wav2swf, flvmeta | | Documents | libxml2, poppler, xpdf | | Fonts | woff2, freetype2, harfbuzz | | Data Formats | sqlite3, jsoncpp, jq, libpcap, curl | | Compilers/Interpreters | lua, mruby, mujs, php | | Binaries | binutils (readelf, nm, objdump, strip, size), bloaty | | Network | openssl, mbedtls, tcpdump, openthread | | Others | re2, cflow, proj4, infotocap, pixdata | ## Supported File Formats **52 file formats** are supported across all target programs: | Type | Formats | |------|---------| | Image | png, jpeg, jpg, gif, bmp, tiff, tga, pnm, ras, pgx, mif | | Video/Audio | mp4, flv, mov, h264, mp3, wav, vorbis, aiff, aifc, au, caf, oga, opus, xi, paf, raw, rf64, sds | | Documents | xml, pdf | | Fonts | woff2, otf, ttf | | Data | sql, json, pcap | | Binaries | elf, pe, mach-o, wasm | | Code | lua, ruby, javascript, php | | Network | dtls, openthread | | Others | terminfo, proj, cms, zip, re2, vi | ## Dataset Structure Each line in `dataset.jsonl` / `dataset_full.jsonl` is a JSON object: ```json { "prog": "libpng_read_fuzzer", "src_site_id": 934, "to_take": 1, "source_location": "/src/libpng/libpng/pngrutil.c:4457:11", "branch": true, "format": "png", "src_dir": "/src/libpng/libpng" } ``` | Field | Type | Description | |-------|------|-------------| | `prog` | string | Target program identifier (matches binary name in `/targets/`) | | `src_site_id` | int | Branch site identifier (index into instrumentation data) | | `to_take` | int (0 or 1) | Target branch direction: 0 = false branch, 1 = true branch | | `source_location` | string | Source code location of the branch condition (`file:line:column`) | | `branch` | bool | Same as `to_take` (convenience field: `true`/`false`) | | `format` | string | Expected input file format(s), comma-separated if multiple | | `src_dir` | string | Path to source code directory in the Docker image | ## Dataset Construction The benchmark is constructed by selecting **roadblock branches** from fuzzing campaigns: 1. **Fuzzing Campaigns**: Run multiple 24-hour fuzzing sessions for each target program using AFL 2. **Branch Tracking**: Track when each branch is first reached and when it is covered (both directions exercised) 3. **Roadblock Identification**: A branch is considered a "roadblock" if: - It is reached within the fuzzing campaign - It remains uncovered for at least 1 hour (threshold=3600s) after being reached - This condition holds in at least 50% of the fuzzing runs 4. **Selection**: From all identified roadblock branches, randomly sample to create the benchmark ### Construction Algorithm ```python def select_br_according_to_multiple_runs(prog, fuzz_dirs, threshold=3600, endtime=24*3600, sel_portion=0.5): # For each fuzzing run, track reach_time and cover_time for each branch # A branch is "selected" (roadblock) if: # - It was reached before (endtime - threshold/2) # - Time to cover (cover_time - reach_time) >= threshold # - This condition holds in >= max(n_runs * sel_portion, 1) runs ``` ## Task Description Given a dataset entry, the LLM must generate a Python script containing a `generate_buf()` function that returns a bytes object. When the target program processes this buffer, it should execute the specified branch direction. **Example task:** ``` Program: libpng_read_fuzzer Branch: png_ptr->color_type == PNG_COLOR_TYPE_PALETTE (line 4457) Direction: true (to_take=1) Format: png ``` **Expected LLM output:** ```python def generate_buf(): # Generate a valid PNG with palette color type buf = b'\x89PNG\r\n\x1a\n' # ... IHDR chunk with color_type=3 (palette) ... return buf ``` ## Evaluation ### Docker Environment Run evaluation inside the Docker image for reproducibility: ```bash docker build -t dis_running_env:latest . docker run --rm -it dis_running_env bash ``` ### Single Entry Verification Use `verify_res_repeat_rnr()` to check if a generated buffer covers the target branch: ```python from eval import verify_res_repeat_rnr python_code = ''' def generate_buf(): return b'\\x89PNG\\r\\n\\x1a\\n...' ''' result = verify_res_repeat_rnr( python_code, prog='libpng_read_fuzzer', src_site_id=934, to_take=1, N_repeat=10 # Number of attempts ) # Returns: True if branch covered, False otherwise ``` ### Batch Verification Use `verify_multiple()` to evaluate multiple entries efficiently: ```python from eval import verify_multiple python_codes = [code1, code2, code3, ...] branches = [(src_site_id, to_take), ...] # Same length as python_codes results = verify_multiple( prog='libpng_read_fuzzer', python_codes=python_codes, branches=branches, N_repeat=10, verbose=True ) # Returns: [True, False, True, ...] ``` ## File Structure ``` / ├── dataset.jsonl # DIS-Bench-Lite (555 tasks) ├── dataset_full.jsonl # DIS-Bench (8267 tasks) ├── eval.py # Evaluation utilities ├── config.py # Program and format configurations ├── Dockerfile # Docker environment setup ├── targets/ # Instrumented binaries ├── inst_src_match/ # Branch instrumentation data (*.pkl) ├── src/ # Source code for all programs └── cov_utils/ # Coverage utilities (SeedRunner) ``` ## Notes - All paths inside the Docker image use `/src/` as the source code prefix - Each verification attempt has a 1-second timeout and 100MB memory limit - `N_repeat` allows multiple attempts for non-deterministic generators