fares-00/codeparrot-clean

CodeParrot数据集是一个从GitHub收集的Python代码文件数据集，它是原始codeparrot数据集的去重和清理版本。该数据集经过处理，包括去重（移除完全匹配的文件）和过滤（如平均行长度小于100、最大行长度小于1000、字母数字字符比例大于0.25，并移除自动生成的文件）。数据集分为训练集和验证集，用于模型训练和评估。结构上，数据集包含约50GB的代码，共有5361373个文件，特征包括仓库名称、文件路径、副本数、大小、内容、许可证、哈希值、平均行长度、最大行长度、字母数字比例和是否自动生成。

A dataset of Python files from Github. This is the deduplicated version of the codeparrot dataset. It was cleaned with steps such as deduplication (removing exact matches) and filtering (e.g., average line length < 100, maximum line length < 1000, alpha numeric characters fraction > 0.25, and removal of auto-generated files). The dataset is split into train and validation sets for training and evaluation, containing approximately 50GB of code with 5361373 files, and features include repo_name, path, copies, size, content, license, hash, line_mean, line_max, alpha_frac, and autogenerated.