AISmellBench: A Reusable Benchmark of AI Code Smells in ML and LLM-Based Systems

AISmellBench is a reusable benchmark for AI code smells in Machine Learning (ML) and Large Language Model (LLM)-based systems. The dataset is designed to support the empirical evaluation of AI code smell detection approaches by providing manually validated ground truth, multi-level metadata, and traceable provenance artifacts.

The benchmark brings together two large-scale corpora covering 890 ML repositories and 692 LLM repositories, for a total of 1,582 AI-based repositories and 313,941 Python files. It includes 1,265 annotated Python files with 3,714 validated AI code smell instances across 31 smell types. A compact evaluation set containing 567 validated instances is also provided to support efficient and comparable detector evaluation.

AISmellBench was constructed through a reproducible pipeline combining GitHub repository mining, preprocessing and eligibility filtering, tool-assisted identification of potential smell instances, and manual validation with adjudication artifacts. The released package includes repository-level metadata, file-level context, smell-level annotations, validation verdicts, confidence scores, provenance links, construction artifacts, and evaluation files.

This dataset is intended for researchers and practitioners studying AI code smells, static analysis, technical debt, software quality, ML engineering, LLM-based systems, and automated code improvement. It can be used to evaluate new AI code smell detectors, compare existing tools, study smell prevalence and co-occurrence, reproduce empirical analyses, and support future research on technical debt and automated correction in AI-based systems.