ytzi/the-stack-dedup-python-filtered-dec_gen_async

--- dataset_info: config_name: main features: - name: hexsha dtype: string - name: size dtype: int64 - name: ext dtype: string - name: lang dtype: string - name: max_stars_repo_path dtype: string - name: max_stars_repo_name dtype: string - name: max_stars_repo_head_hexsha dtype: string - name: max_stars_repo_licenses sequence: string - name: max_stars_count dtype: int64 - name: max_stars_repo_stars_event_min_datetime dtype: string - name: max_stars_repo_stars_event_max_datetime dtype: string - name: max_issues_repo_path dtype: string - name: max_issues_repo_name dtype: string - name: max_issues_repo_head_hexsha dtype: string - name: max_issues_repo_licenses sequence: string - name: max_issues_count dtype: int64 - name: max_issues_repo_issues_event_min_datetime dtype: string - name: max_issues_repo_issues_event_max_datetime dtype: string - name: max_forks_repo_path dtype: string - name: max_forks_repo_name dtype: string - name: max_forks_repo_head_hexsha dtype: string - name: max_forks_repo_licenses sequence: string - name: max_forks_count dtype: int64 - name: max_forks_repo_forks_event_min_datetime dtype: string - name: max_forks_repo_forks_event_max_datetime dtype: string - name: content dtype: string - name: avg_line_length dtype: float64 - name: max_line_length dtype: int64 - name: alphanum_fraction dtype: float64 - name: original_content dtype: string - name: filtered:remove_decorators dtype: int64 - name: filtered:remove_async dtype: int64 - name: filtered:remove_generators dtype: int64 - name: filtered:remove_delete_markers dtype: int64 splits: - name: train num_bytes: 128851530736 num_examples: 12960052 download_size: 49703710584 dataset_size: 128851530736 configs: - config_name: main data_files: - split: train path: main/train-* --- This is a copy of [bigcode/the-stack-dedup](https://huggingface.co/datasets/bigcode/the-stack-dedup) with some filters applied. The filters filtered in this dataset are: - remove_decorators - remove_async - remove_generators - remove_delete_markers

原始信息汇总

数据集概述

数据集配置

• 配置名称: main

数据集特征

• hexsha: 字符串类型
• size: 整数类型（int64）
• ext: 字符串类型
• lang: 字符串类型
• max_stars_repo_path: 字符串类型
• max_stars_repo_name: 字符串类型
• max_stars_repo_head_hexsha: 字符串类型
• max_stars_repo_licenses: 字符串序列类型
• max_stars_count: 整数类型（int64）
• max_stars_repo_stars_event_min_datetime: 字符串类型
• max_stars_repo_stars_event_max_datetime: 字符串类型
• max_issues_repo_path: 字符串类型
• max_issues_repo_name: 字符串类型
• max_issues_repo_head_hexsha: 字符串类型
• max_issues_repo_licenses: 字符串序列类型
• max_issues_count: 整数类型（int64）
• max_issues_repo_issues_event_min_datetime: 字符串类型
• max_issues_repo_issues_event_max_datetime: 字符串类型
• max_forks_repo_path: 字符串类型
• max_forks_repo_name: 字符串类型
• max_forks_repo_head_hexsha: 字符串类型
• max_forks_repo_licenses: 字符串序列类型
• max_forks_count: 整数类型（int64）
• max_forks_repo_forks_event_min_datetime: 字符串类型
• max_forks_repo_forks_event_max_datetime: 字符串类型
• content: 字符串类型
• avg_line_length: 浮点数类型（float64）
• max_line_length: 整数类型（int64）
• alphanum_fraction: 浮点数类型（float64）
• original_content: 字符串类型
• filtered:remove_decorators: 整数类型（int64）
• filtered:remove_async: 整数类型（int64）
• filtered:remove_generators: 整数类型（int64）
• filtered:remove_delete_markers: 整数类型（int64）

数据集分割

• 训练集:
  • 数据大小: 128851530736字节
  • 样本数量: 12960052

数据集大小

• 下载大小: 49703710584字节
• 数据集大小: 128851530736字节