Code4ML: a Large-scale Dataset of annotated Machine Learning Code

We present Code4ML: a Large-scale Dataset of annotated Machine Learning Code, a corpus of Python code snippets, competition summaries, and data descriptions from Kaggle. The corpus consists of ≈ 2.5 million snippets of ML code collected from ≈ 100 thousand Jupyter notebooks. A representative fraction of the snippets is annotated by human assessors through a user-friendly interface specially designed for that purpose. The data is organized as a set of tables in CSV format. It includes several central entities: raw code blocks collected from Kaggle (code_blocks.csv), kernels (kernels_meta.csv) and competitions meta information (competitions_meta.csv). Manually annotated code blocks are presented as a separate table (murkup_data.csv). As this table contains the numeric id of the code block semantic type, we also provide a mapping from the id to semantic class and subclass (vertices.csv). Snippets information (code_blocks.csv) can be mapped with kernels meta-data via kernel_id. Kernels metadata is linked to Kaggle competitions information through comp_name. To ensure the quality of the data kernels_meta.csv includes only notebooks with an available Kaggle score. The corpus can help solve various problems, including code synthesis from a prompt in natural language, code autocompletion, and semantic code classification.我们推出Code4ML：一款大规模带标注机器学习代码数据集，其语料库包含从Kaggle平台采集的Python代码片段、竞赛综述与数据说明文本。该语料库包含约250万条机器学习代码片段，均采集自约10万个Jupyter笔记本（Jupyter Notebook）。其中具有代表性的一部分代码片段已由人工标注人员通过专为该场景设计的友好型用户界面完成标注。 该数据集以CSV格式的多张数据表进行组织，包含三类核心实体：从Kaggle采集的原始代码块（对应数据表code_blocks.csv）、Kaggle内核元数据（对应数据表kernels_meta.csv）以及竞赛元信息（对应数据表competitions_meta.csv）。人工标注的代码块单独存储于数据表murkup_data.csv中；由于该数据表包含代码块语义类型的数字编号，我们还附带提供了编号到语义类别与子类的映射表（vertices.csv）。 可通过kernel_id字段将代码片段信息（存储于code_blocks.csv）与Kaggle内核元数据进行关联，而Kaggle内核元数据可通过comp_name字段与竞赛信息建立关联。为保障数据集质量，kernels_meta.csv仅收录了带有Kaggle公开评分的笔记本。 该语料库可用于解决多项任务，包括基于自然语言提示的代码生成、代码自动补全以及语义代码分类。