AISE-TUDelft/leading-comments

---
dataset_info:
- config_name: CodeParrot
features:
- name: comments
dtype: string
splits:
- name: train
num_bytes: 6713883371
num_examples: 14372397
download_size: 1861971823
dataset_size: 6713883371
- config_name: CodeParrotComments
features:
- name: comments
dtype: string
splits:
- name: train
num_bytes: 6740538054
num_examples: 14372397
download_size: 1871456803
dataset_size: 6740538054
- config_name: GitHubCode
features:
- name: comments
dtype: string
splits:
- name: train
num_bytes: 21743488635
num_examples: 45301797
download_size: 6273669540
dataset_size: 21743488635
- config_name: RedPajama
features:
- name: comments
dtype: string
splits:
- name: train
num_bytes: 509970374
num_examples: 2281378
download_size: 232594972
dataset_size: 509970374
- config_name: RedPajamaComments
features:
- name: comments
dtype: string
splits:
- name: train
num_bytes: 510819441
num_examples: 2281378
download_size: 234861060
dataset_size: 510819441
- config_name: ThePile
features:
- name: comments
dtype: string
splits:
- name: train
num_bytes: 3450632528
num_examples: 6794995
download_size: 1229945960
dataset_size: 3450632528
- config_name: ThePileComments
features:
- name: comments
dtype: string
splits:
- name: train
num_bytes: 3462162891
num_examples: 6794995
download_size: 1245174054
dataset_size: 3462162891
- config_name: TheStack
features:
- name: comments
dtype: string
splits:
- name: train
num_bytes: 36267169085
num_examples: 77595559
download_size: 9666361191
dataset_size: 36267169085
configs:
- config_name: CodeParrot
data_files:
- split: train
path: data/CodeParrotComments/train-*
- config_name: CodeParrotComments
data_files:
- split: train
path: data/CodeParrot_Comments/train-*
- config_name: GitHubCode
data_files:
- split: train
path: data/GitHubCodeComments/train-*
- config_name: RedPajama
data_files:
- split: train
path: data/RedPajamaComments/train-*
- config_name: RedPajamaComments
data_files:
- split: train
path: data/RedPajama_Comments/train-*
- config_name: ThePile
data_files:
- split: train
path: data/ThePileComments/train-*
- config_name: ThePileComments
data_files:
- split: train
path: data/ThePile_Comments/train-*
- config_name: TheStack
data_files:
- split: train
path: data/TheStackComments/train-*
---

# Dataset Card for Dataset Name

<!-- Provide a quick summary of the dataset. -->

**Disclaimer:** This dataset will be released shortly.

## Dataset Details

### Dataset Description

<!-- Provide a longer summary of what this dataset is. -->

- **Curated by:** [More Information Needed]
- **Funded by [optional]:** [More Information Needed]
- **Shared by [optional]:** [More Information Needed]
- **Language(s) (NLP):** [More Information Needed]
- **License:** [More Information Needed]

### Dataset Sources [optional]

<!-- Provide the basic links for the dataset. -->

- **Repository:** [More Information Needed]
- **Paper [optional]:** [More Information Needed]
- **Demo [optional]:** [More Information Needed]

## Uses

<!-- Address questions around how the dataset is intended to be used. -->

### Direct Use

<!-- This section describes suitable use cases for the dataset. -->

[More Information Needed]

### Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the dataset will not work well for. -->

[More Information Needed]

## Dataset Structure

<!-- This section provides a description of the dataset fields, and additional information about the dataset structure such as criteria used to create the splits, relationships between data points, etc. -->

[More Information Needed]

## Dataset Creation

### Curation Rationale

<!-- Motivation for the creation of this dataset. -->

[More Information Needed]

### Source Data

<!-- This section describes the source data (e.g. news text and headlines, social media posts, translated sentences, ...). -->

#### Data Collection and Processing

<!-- This section describes the data collection and processing process such as data selection criteria, filtering and normalization methods, tools and libraries used, etc. -->

[More Information Needed]

#### Who are the source data producers?

<!-- This section describes the people or systems who originally created the data. It should also include self-reported demographic or identity information for the source data creators if this information is available. -->

[More Information Needed]

### Annotations [optional]

<!-- If the dataset contains annotations which are not part of the initial data collection, use this section to describe them. -->

#### Annotation process

<!-- This section describes the annotation process such as annotation tools used in the process, the amount of data annotated, annotation guidelines provided to the annotators, interannotator statistics, annotation validation, etc. -->

[More Information Needed]

#### Who are the annotators?

<!-- This section describes the people or systems who created the annotations. -->

[More Information Needed]

#### Personal and Sensitive Information

<!-- State whether the dataset contains data that might be considered personal, sensitive, or private (e.g., data that reveals addresses, uniquely identifiable names or aliases, racial or ethnic origins, sexual orientations, religious beliefs, political opinions, financial or health data, etc.). If efforts were made to anonymize the data, describe the anonymization process. -->

[More Information Needed]

## Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

[More Information Needed]

### Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Users should be made aware of the risks, biases and limitations of the dataset. More information needed for further recommendations.

## Citation [optional]

<!-- If there is a paper or blog post introducing the dataset, the APA and Bibtex information for that should go in this section. -->

**BibTeX:**

[More Information Needed]

**APA:**

[More Information Needed]

## Glossary [optional]

<!-- If relevant, include terms and calculations in this section that can help readers understand the dataset or dataset card. -->

[More Information Needed]

## More Information [optional]

[More Information Needed]

## Dataset Card Authors [optional]

[More Information Needed]

## Dataset Card Contact

[More Information Needed]

This dataset includes multiple configurations, each primarily containing code comments (comments) with data type as string. The dataset is divided into training sets, with each configuration providing data size and number of examples. Specific information such as source, language, license, etc., is not yet provided.

原始信息汇总

数据集概述

CodeParrot

• 特征:
  • comments: 字符串类型
• 分割:
  • train: 14372397个示例，6713883371字节
• 下载大小: 1861971823字节
• 数据集大小: 6713883371字节

CodeParrotComments

• 特征:
  • comments: 字符串类型
• 分割:
  • train: 14372397个示例，6740538054字节
• 下载大小: 1871456803字节
• 数据集大小: 6740538054字节

GitHubCode

• 特征:
  • comments: 字符串类型
• 分割:
  • train: 45301797个示例，21743488635字节
• 下载大小: 6273669540字节
• 数据集大小: 21743488635字节

RedPajama

• 特征:
  • comments: 字符串类型
• 分割:
  • train: 2281378个示例，509970374字节
• 下载大小: 232594972字节
• 数据集大小: 509970374字节

RedPajamaComments

• 特征:
  • comments: 字符串类型
• 分割:
  • train: 2281378个示例，510819441字节
• 下载大小: 234861060字节
• 数据集大小: 510819441字节

ThePile

• 特征:
  • comments: 字符串类型
• 分割:
  • train: 6794995个示例，3450632528字节
• 下载大小: 1229945960字节
• 数据集大小: 3450632528字节

ThePileComments

• 特征:
  • comments: 字符串类型
• 分割:
  • train: 6794995个示例，3462162891字节
• 下载大小: 1245174054字节
• 数据集大小: 3462162891字节

TheStack

• 特征:
  • comments: 字符串类型
• 分割:
  • train: 77595559个示例，36267169085字节
• 下载大小: 9666361191字节
• 数据集大小: 36267169085字节