Smart-Contract-Vulnerability-Using-LLM

Smart Contract LLM Evaluation Replication study of Salzano et al. (2026). This project replaces human expert review with locally hosted Large Language Models (LLMs) to evaluate how developers fix smart contract vulnerabilities in Solidity. Requirements Python 3.11 or higher Ollama installed and running (https://ollama.com) Paper dataset cloned from Zenodo Install models: ollama pull qwen2.5-coder:7b ollama pull deepseek-r1:7 Clone dataset: git clone https://zenodo.org/records/17105939 ~/smart-contract-eval/repo Install Python dependencies: pip install requests tqdm --break-system-packages Folder structure ~/smart-contract-eval/ repo/ dataset cloned from Zenodo scripts/ all scripts go here data/ intermediate JSONL files (auto-created) results/ final reports (auto-created) Place all scripts inside ~/smart-contract-eval/scripts/. Step 1 - Verify setup cd ~/smart-contract-eval/scripts python3 01_setup_check.py Checks that the model is downloaded, Ollama is running, and the dataset files are in place. Fix any reported issues before continuing. Step 2 - Load the dataset python3 02_load_dataset.py Reads relevant_commits.csv and new_fixes.json from the repo and writes them to data/ as JSONL. Step 3 - Classify vulnerabilities (Task A) Each commit is assigned one DASP TOP 10 category. Qwen: python3 04_classify_vuln.py DeepSeek: python3 04_classify_deepseek.py Output is compared against the paper human labels automatically. Step 4 - Compare fixes against literature (Task B) Each commit is judged as aligned or non-aligned with documented academic guidelines. Qwen: python3 05_compare_fixes.py DeepSeek: python3 05_compare_fixes_deepseek.py Step 5 - Score novel fixes (Task C) Novel commits are scored on generalizability, sustainability, and effectiveness (scale 1 to 5). Qwen: python3 06_eval_novel_fixes.py DeepSeek: python3 06_eval_novel_fixes_deepseek.py Note: this script requires actual code diffs in the dataset. Step 6 - Discover novel strategies (Extension) For non-aligned commits, the LLM checks whether they match any of the 27 known strategies from the paper or represent something genuinely new. New strategies receive a formal written proposal. Qwen: python3 07_discover_novel_fixes.py --model qwen DeepSeek: python3 07_discover_novel_fixes.py --model deepseek Step 7 - Statistical analysis python3 statistical_analysis.py Performs statistical evaluation and generates summary metrics from the experimental outputs. Resuming after interruption All scripts write results line by line and skip already-processed commits on re-run. If a run is interrupted, just run the same command again. To delete output and start a script from scratch: python3 07_discover_novel_fixes.py --model qwen --fresh Output files python3 final_report.py