Security Vulnerability Profiles of AI-Generated Java Code: A Large-Scale Empirical Study (Dataset and Scripts)

This repository contains the complete replication package for an empirical study comparing the security vulnerability profiles of AI-generated and human-written Java code. The package bundles the full analysis pipeline and all pre-computed data artifacts required to reproduce a 659,183-method, 5,086-repository. Key Features: - Automated GitHub mining of repositories with AI-assisted development indicators - Java method extraction with signature, full source, and line-of-code metadata - Perplexity-based AI code detection using a GPT-2 backbone with DetectGPT-style perturbation ranking - Multi-signal classification combining perplexity, burstiness, composite score, and perturbation rank - Static vulnerability analysis using Semgrep with the OWASP Top 10 rule sets - Statistical cross-analysis producing Fisher's exact tests, odds ratios with 95% confidence intervals, Bonferroni-corrected per-category results, LOC-stratified analyses, and CWE-level breakdowns - Secret sanitization, structural deduplication, and bias analysis utilities - SHA-256 data manifest for end-to-end artifact integrity verification Architecture: The codebase is organized as a modular Python pipeline: - scripts/run_pipeline.py: end-to-end orchestrator for all stages - scripts/download_vibecoded_repos.py: repository collection from GitHub - scripts/build_java_methods_dataset.py: Java method extraction - scripts/ai_code_detector.py: perplexity-based AI classification - scripts/run_semgrep.sh / run_semgrep.ps1: Semgrep OWASP scanning - scripts/ai_detection_owasp_analysis.py: statistical cross-analysis - scripts/pipeline_modules/: reusable components for collection, cleaning, deduplication, secret sanitization, structural statistics, and bias analysis - data/: pre-computed JSONL datasets (Java methods, AI detection results, OWASP-annotated methods) and aggregate summary JSONs - DATA_MANIFEST.md: SHA-256 checksums for every data file Requirements: - Python 3.10+ - Git and Semgrep CLI - CUDA-capable GPU (recommended for the AI detection stage; CPU fallback available) - GitHub API token (required for the repository collection stage only) - Approximately 50 GB of disk space for a full replication Usage: The package supports two workflows. Verification mode uses the pre-computed summary JSONs to check every statistic reported in the paper without re-running the pipeline. Full replication mode re-executes all pipeline stages from scratch via a single orchestrator script, with each stage also runnable independently. All parameters as minimum method length, number of perturbations, classification threshold, input and output paths, batch sizes are configurable via command-line arguments. Full replication takes approximately one to three days depending on hardware; verification completes in minutes. License: Apache-2.0. Author information has been omitted for double-blind review and will be added upon acceptance.