AISmellBench: A Reusable Benchmark of AI Code Smells in ML and LLM-Based Systems

AISmellBench is a reusable benchmark for AI code smells in Machine Learning (ML) and Large Language Model (LLM)-based systems. The dataset is designed to support the empirical evaluation of AI code smell detection approaches by providing manually validated ground truth, multi-level metadata, and traceable provenance artifacts.

The benchmark brings together two large-scale corpora covering 890 ML repositories and 692 LLM repositories, for a total of 1,582 AI-based repositories and 313,941 Python files. It includes 1,265 annotated Python files with 3,714 validated AI code smell instances across 31 smell types. A compact evaluation set containing 567 validated instances is also provided to support efficient and comparable detector evaluation.

AISmellBench was constructed through a reproducible pipeline combining GitHub repository mining, preprocessing and eligibility filtering, tool-assisted identification of potential smell instances, and manual validation with adjudication artifacts. The released package includes repository-level metadata, file-level context, smell-level annotations, validation verdicts, confidence scores, provenance links, construction artifacts, and evaluation files.

This dataset is intended for researchers and practitioners studying AI code smells, static analysis, technical debt, software quality, ML engineering, LLM-based systems, and automated code improvement. It can be used to evaluate new AI code smell detectors, compare existing tools, study smell prevalence and co-occurrence, reproduce empirical analyses, and support future research on technical debt and automated correction in AI-based systems.

AISmellBench是一款可复用的人工智能代码异味（AI code smells）基准测试集，面向机器学习（Machine Learning, ML）与基于大语言模型（Large Language Model, LLM）的系统。该数据集通过提供经人工验证的基准真值、多级元数据以及可溯源的来源工件，旨在支持对AI代码异味检测方法的实证评估。

该基准整合了两大规模语料库，覆盖890个机器学习仓库与692个大语言模型仓库，总计1582个基于人工智能的仓库与313941个Python文件。其中包含1265个标注Python文件，涵盖31种异味类型，共计3714个经验证的AI代码异味实例。此外还提供了一套包含567个已验证实例的精简评估集，以支持高效且可对比的检测器性能评估。

AISmellBench通过一套可复现的流水线构建完成，该流程结合了GitHub仓库挖掘、预处理与资格筛选、工具辅助的潜在异味实例识别，以及附带裁决工件的人工验证环节。本次发布的套件包含仓库级元数据、文件级上下文、异味级标注、验证结论、置信度评分、来源链接、构建工件与评估文件。

本数据集面向研究人工智能代码异味、静态分析、技术债务、软件质量、机器学习工程、基于大语言模型的系统以及自动化代码改进的研究人员与从业者。其可用于评估新型AI代码异味检测器、对比现有工具、研究异味的流行度与共现关系、复现实证分析，以及为基于人工智能的系统中的技术债务与自动化校正相关的未来研究提供支撑。