vikp/starcoder_labeled

---
dataset_info:
features:
- name: code
dtype: string
- name: repo_path
dtype: string
- name: parsed_code
dtype: string
- name: quality_prob
dtype: float64
- name: learning_prob
dtype: float64
splits:
- name: train
num_bytes: 852705076967
num_examples: 65509810
download_size: 0
dataset_size: 852705076967
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
---
# Dataset Card for "starcoder_labeled"

[Starcoder data](https://huggingface.co/datasets/bigcode/starcoderdata), with several popular languages selected, short sequences filtered out, then labeled based on learning quality (educational value) and code quality.

A good heuristic is to take anything with `>.5` code quality and `>.3` learning quality. But you may want to vary the thresholds by language, depending on your target task.

原始信息汇总

数据集卡片 "starcoder_labeled"

数据集信息

特征

• code: 类型为字符串
• repo_path: 类型为字符串
• parsed_code: 类型为字符串
• quality_prob: 类型为浮点数 (float64)
• learning_prob: 类型为浮点数 (float64)

分割

• train: 字节数为 852705076967，样本数为 65509810

大小

• 下载大小: 0
• 数据集大小: 852705076967

配置

• default:
  • 分割: train
  • 路径: data/train-*

数据集描述

Starcoder data，选择了几种流行的编程语言，过滤掉了短序列，并根据学习质量和代码质量进行了标记。

一个好的启发式方法是选择代码质量大于 0.5 且学习质量大于 0.3 的样本。但你可能需要根据目标任务的不同，调整不同语言的阈值。