ytzi/the-stack-dedup-python-scored

---
dataset_info:
config_name: python
features:
- name: hexsha
dtype: string
- name: size
dtype: int64
- name: ext
dtype: string
- name: lang
dtype: string
- name: max_stars_repo_path
dtype: string
- name: max_stars_repo_name
dtype: string
- name: max_stars_repo_head_hexsha
dtype: string
- name: max_stars_repo_licenses
sequence: string
- name: max_stars_count
dtype: int64
- name: max_stars_repo_stars_event_min_datetime
dtype: string
- name: max_stars_repo_stars_event_max_datetime
dtype: string
- name: max_issues_repo_path
dtype: string
- name: max_issues_repo_name
dtype: string
- name: max_issues_repo_head_hexsha
dtype: string
- name: max_issues_repo_licenses
sequence: string
- name: max_issues_count
dtype: int64
- name: max_issues_repo_issues_event_min_datetime
dtype: string
- name: max_issues_repo_issues_event_max_datetime
dtype: string
- name: max_forks_repo_path
dtype: string
- name: max_forks_repo_name
dtype: string
- name: max_forks_repo_head_hexsha
dtype: string
- name: max_forks_repo_licenses
sequence: string
- name: max_forks_count
dtype: int64
- name: max_forks_repo_forks_event_min_datetime
dtype: string
- name: max_forks_repo_forks_event_max_datetime
dtype: string
- name: content
dtype: string
- name: avg_line_length
dtype: float64
- name: max_line_length
dtype: int64
- name: alphanum_fraction
dtype: float64
- name: count_classes
dtype: int64
- name: score_classes
dtype: float64
- name: count_generators
dtype: int64
- name: score_generators
dtype: float64
- name: count_decorators
dtype: int64
- name: score_decorators
dtype: float64
- name: count_async_functions
dtype: int64
- name: score_async_functions
dtype: float64
- name: count_documentation
dtype: int64
- name: score_documentation
dtype: float64
splits:
- name: train
num_bytes: 72919338116
num_examples: 12962249
download_size: 28959409073
dataset_size: 72919338116
configs:
- config_name: python
data_files:
- split: train
path: python/train-*
---

原始信息汇总

数据集概述

数据集配置

• 配置名称: python

数据集特征

• hexsha: 字符串类型
• size: 整数类型 (int64)
• ext: 字符串类型
• lang: 字符串类型
• max_stars_repo_path: 字符串类型
• max_stars_repo_name: 字符串类型
• max_stars_repo_head_hexsha: 字符串类型
• max_stars_repo_licenses: 字符串序列类型
• max_stars_count: 整数类型 (int64)
• max_stars_repo_stars_event_min_datetime: 字符串类型
• max_stars_repo_stars_event_max_datetime: 字符串类型
• max_issues_repo_path: 字符串类型
• max_issues_repo_name: 字符串类型
• max_issues_repo_head_hexsha: 字符串类型
• max_issues_repo_licenses: 字符串序列类型
• max_issues_count: 整数类型 (int64)
• max_issues_repo_issues_event_min_datetime: 字符串类型
• max_issues_repo_issues_event_max_datetime: 字符串类型
• max_forks_repo_path: 字符串类型
• max_forks_repo_name: 字符串类型
• max_forks_repo_head_hexsha: 字符串类型
• max_forks_repo_licenses: 字符串序列类型
• max_forks_count: 整数类型 (int64)
• max_forks_repo_forks_event_min_datetime: 字符串类型
• max_forks_repo_forks_event_max_datetime: 字符串类型
• content: 字符串类型
• avg_line_length: 浮点数类型 (float64)
• max_line_length: 整数类型 (int64)
• alphanum_fraction: 浮点数类型 (float64)
• count_classes: 整数类型 (int64)
• score_classes: 浮点数类型 (float64)
• count_generators: 整数类型 (int64)
• score_generators: 浮点数类型 (float64)
• count_decorators: 整数类型 (int64)
• score_decorators: 浮点数类型 (float64)
• count_async_functions: 整数类型 (int64)
• score_async_functions: 浮点数类型 (float64)
• count_documentation: 整数类型 (int64)
• score_documentation: 浮点数类型 (float64)

数据集分割

• 训练集:
  • 字节数: 72919338116
  • 示例数: 12962249

数据集大小

• 下载大小: 28959409073
• 数据集大小: 72919338116

配置文件

• 配置名称: python
  • 数据文件:
    • 分割: train
      • 路径: python/train-*