nayohan/translate_corpus_uniform_313k

---
dataset_info:
features:
- name: english
dtype: string
- name: korean
dtype: string
- name: from
dtype: string
- name: category
dtype: string
- name: en_token_len
dtype: int64
- name: ko_token_len
dtype: int64
splits:
- name: train
num_bytes: 1148575796.079382
num_examples: 312336
- name: valid
num_bytes: 1838686.216253301
num_examples: 500
- name: test
num_bytes: 1838686.216253301
num_examples: 500
download_size: 224624524
dataset_size: 1152253168.5118885
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
- split: valid
path: data/valid-*
- split: test
path: data/test-*
---
# 10개의 명령어(영어-한국어) 병렬 데이터셋 통합 후 공백, 중복데이터, 토큰 4096이상 문장 제거 | (313k)

- [jhflow/orca_ko_en_pair](https://huggingface.co/datasets/jhflow/orca_ko_en_pair) | qeustion, output | 19k *2 = 38k
- [jhflow/platypus_ko_en_pair](https://huggingface.co/datasets/jhflow/platypus_ko_en_pair) | insturction_ko, output_ko | 24k * 2 = 48k
- [jhflow/dolly_ko_en_pair](https://huggingface.co/datasets/jhflow/dolly_ko_en_pair) | instruction, context, response | 15k * 3 = 45k
- [heegyu/OIG-small-chip2-ko](https://huggingface.co/datasets/heegyu/OIG-small-chip2-ko) | user_translated, chip_translated | 210k * 2 % 10 = 42k
- [lemon-mint/en_ko_translation_purified_v0.1](https://huggingface.co/datasets/lemon-mint/en_ko_translation_purified_v0.1) | korean | 1.4k
- [squarelike/sharegpt_deepl_ko_translation](https://huggingface.co/datasets/squarelike/sharegpt_deepl_ko_translation) | korean | 486k % 10 = 48k
- [nayohan/aihub-en-ko-translation-1.2m](https://huggingface.co/datasets/nayohan/aihub-en-ko-translation-1.2m) | 1200k % 25 = 48k
- [amphora/parallel-wiki-koen](https://huggingface.co/datasets/amphora/parallel-wiki-koen) | text_ko | 30k
- [kuotient/gsm8k-ko](https://huggingface.co/datasets/kuotient/gsm8k-ko) | question, answer | 8k * 2 = 16k
- [kuotient/orca-math-word-problems-193k-korean](https://huggingface.co/datasets/kuotient/orca-math-word-problems-193k-korean) | question, answer | 193k *2 % 10 = 38k

**Special Thanks to [jhflow](https://huggingface.co/jhflow), [heegyu](https://huggingface.co/heegyu), [lemon-mint](https://huggingface.co/lemon-mint), [squarelike](https://huggingface.co/squarelike), [kuotient](https://huggingface.co/kuotient), [amphora](https://huggingface.co/amphora).**

原始信息汇总

数据集概述

数据集信息

• 特征:
  • english: 字符串类型
  • korean: 字符串类型
  • from: 字符串类型
  • category: 字符串类型
  • en_token_len: 64位整数类型
  • ko_token_len: 64位整数类型
• 分割:
  • train: 字节数为1148575796.079382，样本数为312336
  • valid: 字节数为1838686.216253301，样本数为500
  • test: 字节数为1838686.216253301，样本数为500
• 下载大小: 224624524字节
• 数据集大小: 1152253168.5118885字节

配置

• 配置名称: default
• 数据文件:
  • train: 路径为data/train-*
  • valid: 路径为data/valid-*
  • test: 路径为data/test-*