Artifact of Search+LLM-based Testing for ARM Simulators

SearchSYS tool can be downloaded here: a239d3c (for this work). This is not the publication version. Please view https://zenodo.org/records/14753604. To process the fuzzed test inputs, post-fuzzing, the *.sh here can run on the output/default/queue and crash folders. The rest are the data collected during experiments: (1) the Excel file contains the full analysis while (2) *.tar.gz are all outputs from the fuzzing stage. The LLM data is taken from here. You can follow the instructions here to install and run SearchSYS for X86. The record here details how to run SearchSYS on an ARM machine (arm64-ubuntu-2204). Experiments Requirements: GCC-11 wget, zip, unzip (any version) LLVM-13+ (we tested with LLVM-13, but any newer version should be okay) Python3 (tested with Python 3.8 and 3.10) Other packages: sudo apt-get update sudo apt-get upgrade sudo add-apt-repository ppa:ubuntu-toolchain-r/test sudo apt-get update sudo apt-get -y install gcc-11 g++-11 cpp-11 wget lsb-release gnupg software-properties-common sudo rm /usr/bin/cpp /usr/bin/gcc /usr/bin/g++ /usr/bin/gcov /usr/bin/c++ sudo rm /usr/bin/cc sudo ln -s /usr/bin/cpp-11 /usr/bin/cpp sudo ln -s /usr/bin/gcc-11 /usr/bin/gcc sudo ln -s /usr/bin/gcc-11 /usr/bin/cc sudo ln -s /usr/bin/g++-11 /usr/bin/g++ sudo ln -s /usr/bin/g++-11 /usr/bin/c++ sudo ln -s /usr/bin/gcov-11 /usr/bin/gcov wget https://apt.llvm.org/llvm.sh sudo chmod +x llvm.sh sudo ./llvm.sh 13 sudo ln -s /usr/bin/llvm-config-13 /usr/bin/llvm-config sudo apt-get install -y lld-13 llvm-13-dev clang-13 sudo apt-get install -y lld llvm llvm-dev clang sudo apt-get install -y build-essential python3-dev automake cmake git flex bison libglib2.0-dev libpixman-1-dev python3-setuptools cargo libgtk-3-dev sudo apt-get install -y ninja-build cmake sudo apt-get install -y git make cmake llvm gdb coreutils sudo apt-get install -y gcc-11-plugin-dev sudo apt -y autoremove sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-13 1300 --slave /usr/bin/clang++ clang++ /usr/bin/clang++-13 sudo update-alternatives --install /usr/bin/llvm-config llvm-config /usr/bin/llvm-config-13 1300 LLVM_CONFIG=/usr/bin/llvm-config-13 We attached all installed packages in list-of-installed-packages.log. Reproduce ICSE-SEIP 2025 Evaluation: To reproduce the experiments follow these commands (Unix system): ## CREATE A NEW USER (you can also use your own user but then you will need to amend /home/debian and debian, accordingly) ## sudo adduser debian # if needed run: # sudo passwd debian # and set some simple password sudo usermod -aG sudo debian sudo mkdir -p /home/debian sudo usermod -d /home/debian debian su - debian ## GET DATA - 7 set of LLM-generated C programs ## cd /home/debian wget "https://zenodo.org/records/10999115/files/LLM_test_inputs-4FuzzExp2.zip" # Get the data unzip -n LLM_test_inputs-4FuzzExp2.zip -d /home/debian/experiment cd /home/debian/experiment/ sed -i "s:/home/ubuntu/experiment-7/:/home/debian/experiment/:g" */input/*.txt cat /home/debian/experiment/TinyLlama-cmin/input/test_input_574567374417673.txt ## COMPILE BINARIES - build the test binaries and tailor the paths in the seeds ## ### Model CodeBooga ### cd /home/debian/experiment/CodeBooga for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done rm binary/fuzz_* || true rm .DS_Store || true cd /home/debian/experiment/ rm -rf CodeBooga-cmin/binary cp -r CodeBooga/binary CodeBooga-cmin/binary cd /home/debian/experiment/CodeBooga-cmin/input sed -i 's\/CodeBooga/\/CodeBooga-cmin/\g' *txt ### Model Llama ### cd /home/debian/experiment/Llama for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done rm binary/fuzz_* || true rm .DS_Store || true cd /home/debian/experiment/ rm -rf Llama-cmin/binary cp -r Llama/binary Llama-cmin/binary cd /home/debian/experiment/Llama-cmin/input sed -i 's\/Llama/\/Llama-cmin/\g' *txt ### Model Magicoder ### cd /home/debian/experiment/Magicoder for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done rm binary/fuzz_* || true rm .DS_Store || true cd /home/debian/experiment/ rm -rf Magicoder-cmin/binary cp -r Magicoder/binary Magicoder-cmin/binary cd /home/debian/experiment/Magicoder-cmin/input sed -i 's\/Magicoder/\/Magicoder-cmin/\g' *txt ### Model TinyLlama ### cd /home/debian/experiment/TinyLlama for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done rm binary/fuzz_* || true rm .DS_Store || true cd /home/debian/experiment/ rm -rf TinyLlama-cmin/binary cp -r TinyLlama/binary TinyLlama-cmin/binary cd /home/debian/experiment/TinyLlama-cmin/input sed -i 's\/TinyLlama/\/TinyLlama-cmin/\g' *txt ### Model Phi ### cd /home/debian/experiment/Phi for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done rm binary/fuzz_* || true rm .DS_Store || true cd /home/debian/experiment/ rm -rf Phi-cmin/binary cp -r Phi/binary Phi-cmin/binary cd /home/debian/experiment/Phi-cmin/input sed -i 's\/Phi/\/Phi-cmin/\g' *txt ### Model gpt3.5-new ### cd /home/debian/experiment/gpt3.5-new for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done rm binary/fuzz_* || true rm .DS_Store || true cd /home/debian/experiment/ rm -rf gpt3.5-new-cmin/binary cp -r gpt3.5-new/binary gpt3.5-new-cmin/binary cd /home/debian/experiment/gpt3.5-new-cmin/input sed -i 's\/gpt3.5-new/\/gpt3.5-new-cmin/\g' *txt ### Model gpt3.5-old ### cd /home/debian/experiment/gpt3.5-old for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done rm binary/fuzz_* || true rm .DS_Store || true cd /home/debian/experiment/ rm -rf gpt3.5-old-cmin/binary cp -r gpt3.5-old/binary gpt3.5-old-cmin/binary cd /home/debian/experiment/gpt3.5-old-cmin/input sed -i 's\/gpt3.5-old/\/gpt3.5-old-cmin/\g' *txt At this point all data for fuzzing is ready. You need now to get SearchSYS and build the SUT (gem5) from source.## GET AFL++ SearchSYS ## cd /home/debian git clone https://github.com/karineek/SearchGEM5.git git clone https://github.com/AFLplusplus/AFLplusplus.git cd AFLplusplus git checkout f596a297c4de6a5e1a6fb9fbb3b4e18124a24f58 cp ../SearchGEM5/src/gem5-afl/afl-fuzz-init.c src/afl-fuzz-init.c AFL_USE_ASAN=0 make ### The compilation shall state that it is ARM architecture: ### ``` debian@node0:~/AFLplusplus$ AFL_USE_ASAN=0 make [*] Compiling AFL++ for OS Linux on ARCH aarch64 [+] ZLIB detected [!] Note: skipping x86 compilation checks (AFL_NO_X86 set). [+] shmat seems to be working. [+] Python 3.8.10 support seems to be working. [+] Everything seems to be working, ready to compile ... ```then you need to check that AFL is build correctly: ```Build Summary: [+] afl-fuzz and supporting tools successfully built [+] LLVM basic mode successfully built [+] LLVM mode successfully built [+] LLVM LTO mode successfully built [+] gcc_mode successfully built``` Then you need to build gem5 with AFL++ instrumentation: cd /home/debian git clone https://github.com/BobbyRBruce/gem5-ssbse-challenge-2023.git cd /home/debian/gem5-ssbse-challenge-2023 cp ../SearchGEM5/src/gem5-afl/SConstruct . cp ../SearchGEM5/src/gem5-afl/SConscript src/ # If needed install: sudo apt install python3-pip sudo apt install scons protobuf-compiler libhdf5-dev libgoogle-perftools-dev libsystemc-dev # Then install the requirements for gem5: pip install -r requirements.txt # And build gem5 with AFL++ instrumentation: CC=afl-cc CXX=afl-c++ scons build/ARM/gem5.opt -j 4 --no-cache ./build/ARM/gem5.opt -C During the configuration, you need to see it built for ARM, hence (1) make sure you use build/ARM not X86, and check the configuration log to see that the right architecture has been properly selected for the build.本研究配套的SearchSYS工具可通过标识符`a239d3c`获取（本版本非正式发布版），正式版本请访问：https://zenodo.org/records/14753604。 针对模糊测试（Fuzzing）后的测试用例，可通过本目录下的`*.sh`脚本处理`output`、`default`、`queue`及`crash`文件夹中的数据。 其余为实验采集的数据集：(1) Excel文件包含完整的分析结果；(2) 所有`*.tar.gz`文件均为模糊测试阶段的输出产物。本数据集包含的大语言模型（LLM）相关数据均可从此处获取。您可参照本目录下的说明文档，在X86架构设备上安装并运行SearchSYS工具。 本记录详细说明了如何在ARM架构设备（arm64-ubuntu-2204）上运行SearchSYS工具。 --- ### 实验 #### 环境依赖要求 - GCC-11 - wget、zip、unzip（任意版本均可） - LLVM-13及以上版本（本研究基于LLVM-13测试，更高版本兼容） - Python3（本研究基于Python 3.8与3.10测试） #### 依赖包安装命令 bash sudo apt-get update sudo apt-get upgrade sudo add-apt-repository ppa:ubuntu-toolchain-r/test sudo apt-get update sudo apt-get -y install gcc-11 g++-11 cpp-11 wget lsb-release gnupg software-properties-common sudo rm /usr/bin/cpp /usr/bin/gcc /usr/bin/g++ /usr/bin/gcov /usr/bin/c++ sudo rm /usr/bin/cc sudo ln -s /usr/bin/cpp-11 /usr/bin/cpp sudo ln -s /usr/bin/gcc-11 /usr/bin/gcc sudo ln -s /usr/bin/gcc-11 /usr/bin/cc sudo ln -s /usr/bin/g++-11 /usr/bin/g++ sudo ln -s /usr/bin/g++-11 /usr/bin/c++ sudo ln -s /usr/bin/gcov-11 /usr/bin/gcov wget https://apt.llvm.org/llvm.sh sudo chmod +x llvm.sh sudo ./llvm.sh 13 sudo ln -s /usr/bin/llvm-config-13 /usr/bin/llvm-config sudo apt-get install -y lld-13 llvm-13-dev clang-13 sudo apt-get install -y lld llvm llvm-dev clang sudo apt-get install -y build-essential python3-dev automake cmake git flex bison libglib2.0-dev libpixman-1-dev python3-setuptools cargo libgtk-3-dev sudo apt-get install -y ninja-build cmake sudo apt-get install -y git make cmake llvm gdb coreutils sudo apt-get install -y gcc-11-plugin-dev sudo apt -y autoremove sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-13 1300 --slave /usr/bin/clang++ clang++ /usr/bin/clang++-13 sudo update-alternatives --install /usr/bin/llvm-config llvm-config /usr/bin/llvm-config-13 1300 LLVM_CONFIG=/usr/bin/llvm-config-13 本数据集已将所有已安装的依赖包清单保存至`list-of-installed-packages.log`文件中。 --- ### 复现ICSE-SEIP 2025评估实验 若需复现本实验，请在Unix类系统中执行如下命令： #### 步骤1：创建新用户（亦可使用现有用户，但需相应修改`/home/debian`与`debian`相关配置） bash sudo adduser debian # 如需设置密码，可执行： # sudo passwd debian # 并设置简易密码即可 sudo usermod -aG sudo debian sudo mkdir -p /home/debian sudo usermod -d /home/debian debian su - debian #### 步骤2：获取数据集（7组大语言模型生成的C语言程序） bash cd /home/debian wget "https://zenodo.org/records/10999115/files/LLM_test_inputs-4FuzzExp2.zip" # 下载数据集 unzip -n LLM_test_inputs-4FuzzExp2.zip -d /home/debian/experiment cd /home/debian/experiment/ sed -i "s:/home/ubuntu/experiment-7/:/home/debian/experiment/:g" */input/*.txt cat /home/debian/experiment/TinyLlama-cmin/input/test_input_574567374417673.txt #### 步骤3：编译二进制文件——构建测试用二进制程序并修改种子文件中的路径配置 ##### 针对CodeBooga模型 bash cd /home/debian/experiment/CodeBooga for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done rm binary/fuzz_* || true rm .DS_Store || true cd /home/debian/experiment/ rm -rf CodeBooga-cmin/binary cp -r CodeBooga/binary CodeBooga-cmin/binary cd /home/debian/experiment/CodeBooga-cmin/input sed -i 's/CodeBooga//CodeBooga-cmin/g' *txt ##### 针对Llama模型 bash cd /home/debian/experiment/Llama for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done rm binary/fuzz_* || true rm .DS_Store || true cd /home/debian/experiment/ rm -rf Llama-cmin/binary cp -r Llama/binary Llama-cmin/binary cd /home/debian/experiment/Llama-cmin/input sed -i 's/Llama//Llama-cmin/g' *txt ##### 针对Magicoder模型 bash cd /home/debian/experiment/Magicoder for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done rm binary/fuzz_* || true rm .DS_Store || true cd /home/debian/experiment/ rm -rf Magicoder-cmin/binary cp -r Magicoder/binary Magicoder-cmin/binary cd /home/debian/experiment/Magicoder-cmin/input sed -i 's/Magicoder//Magicoder-cmin/g' *txt ##### 针对TinyLlama模型 bash cd /home/debian/experiment/TinyLlama for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done rm binary/fuzz_* || true rm .DS_Store || true cd /home/debian/experiment/ rm -rf TinyLlama-cmin/binary cp -r TinyLlama/binary TinyLlama-cmin/binary cd /home/debian/experiment/TinyLlama-cmin/input sed -i 's/TinyLlama//TinyLlama-cmin/g' *txt ##### 针对Phi模型 bash cd /home/debian/experiment/Phi for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done rm binary/fuzz_* || true rm .DS_Store || true cd /home/debian/experiment/ rm -rf Phi-cmin/binary cp -r Phi/binary Phi-cmin/binary cd /home/debian/experiment/Phi-cmin/input sed -i 's/Phi//Phi-cmin/g' *txt ##### 针对gpt3.5-new模型 bash cd /home/debian/experiment/gpt3.5-new for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done rm binary/fuzz_* || true rm .DS_Store || true cd /home/debian/experiment/ rm -rf gpt3.5-new-cmin/binary cp -r gpt3.5-new/binary gpt3.5-new-cmin/binary cd /home/debian/experiment/gpt3.5-new-cmin/input sed -i 's/gpt3.5-new//gpt3.5-new-cmin/g' *txt ##### 针对gpt3.5-old模型 bash cd /home/debian/experiment/gpt3.5-old for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done rm binary/fuzz_* || true rm .DS_Store || true cd /home/debian/experiment/ rm -rf gpt3.5-old-cmin/binary cp -r gpt3.5-old/binary gpt3.5-old-cmin/binary cd /home/debian/experiment/gpt3.5-old-cmin/input sed -i 's/gpt3.5-old//gpt3.5-old-cmin/g' *txt 至此，模糊测试所需的全部数据已准备完毕。接下来请获取SearchSYS工具并从源码编译被测系统（SUT，即gem5）。 ## 获取AFL++与SearchSYS bash cd /home/debian git clone https://github.com/karineek/SearchGEM5.git git clone https://github.com/AFLplusplus/AFLplusplus.git cd AFLplusplus git checkout f596a297c4de6a5e1a6fb9fbb3b4e18124a24f58 cp ../SearchGEM5/src/gem5-afl/afl-fuzz-init.c src/afl-fuzz-init.c AFL_USE_ASAN=0 make 编译过程中应显示目标架构为ARM： debian@node0:~/AFLplusplus$ AFL_USE_ASAN=0 make [*] Compiling AFL++ for OS Linux on ARCH aarch64 [+] ZLIB detected [!] Note: skipping x86 compilation checks (AFL_NO_X86 set). [+] shmat seems to be working. [+] Python 3.8.10 support seems to be working. [+] Everything seems to be working, ready to compile ... 编译完成后，请验证AFL++是否构建成功： 构建总结： [+] afl-fuzz 及配套工具编译成功 [+] LLVM基础模式编译成功 [+] LLVM模式编译成功 [+] LLVM LTO模式编译成功 [+] GCC模式编译成功 接下来请编译搭载AFL++插桩的gem5： bash cd /home/debian git clone https://github.com/BobbyRBruce/gem5-ssbse-challenge-2023.git cd /home/debian/gem5-ssbse-challenge-2023 cp ../SearchGEM5/src/gem5-afl/SConstruct . cp ../SearchGEM5/src/gem5-afl/SConscript src/ # 如需安装依赖，可执行： bash sudo apt install python3-pip sudo apt install scons protobuf-compiler libhdf5-dev libgoogle-perftools-dev libsystemc-dev # 安装gem5所需的Python依赖： bash pip install -r requirements.txt # 编译搭载AFL++插桩的gem5： bash CC=afl-cc CXX=afl-c++ scons build/ARM/gem5.opt -j 4 --no-cache ./build/ARM/gem5.opt -C 编译过程中请确认目标架构为ARM：(1) 确保使用`build/ARM`而非X86架构编译路径；(2) 检查编译日志以确认已正确选择目标架构。