zh-tw-llm-dv/zh-tw-pythia-ta8000-v1-e1-tr_sg-301-c1024-sbldt5

---
dataset_info:
dataset_size: 52277349.68449913
download_size: 14950589
features:
- name: input_ids
sequence: int32
- name: attention_mask
sequence: int8
- name: labels
sequence: int64
- dtype: string
name: preview
- dtype: int64
name: length
- dtype: int64
name: messages_count
splits:
- name: train
num_bytes: 51379878.03449913
num_examples: 5950
- name: test
num_bytes: 897471.65
num_examples: 95

---
# zh-tw-pythia-ta8000-v1-e1-tr_sg-301-c1024-sbldt5

This dataset is a part of the `zh-tw-llm` project.

* Tokenizer: `zh-tw-pythia-tokenizer-a8000-v1`
* Built with: `sharegpt`
* Rows: `train` `5950`, `test` `95`
* Max length: `1024`
* Full config:
```json
{"build_with": ["sharegpt"], "preview_length": 128, "sort_by": "length-desc", "translations_settings": {"source_dataset": "zetavg/coct-en-zh-tw-translations-twp-300k", "lang_1_key": "en", "lang_2_key": "ch", "templates": ["English: {lang_1}\nChinese: {lang_2}", "Chinese: {lang_2}\nEnglish: {lang_1}"], "use_template": "random", "rows_limit": 300000, "test_size": 100, "test_split_seed": 42}, "sharegpt_settings": {"source_dataset": "zetavg/ShareGPT-Processed", "train_on_inputs": false, "languages": [{"en": 0.4}, "zh_Hant"], "rows_limit": 8000, "test_size": 0.02, "test_split_seed": 42, "test_rows_limit": 100}}
```

原始信息汇总

数据集概述

基本信息

• 数据集大小: 52277349.68449913 字节
• 下载大小: 14950589 字节

特征信息

• input_ids: 序列类型，数据类型为 int32
• attention_mask: 序列类型，数据类型为 int8
• labels: 序列类型，数据类型为 int64
• preview: 数据类型为 string
• length: 数据类型为 int64
• messages_count: 数据类型为 int64

数据分割

• 训练集:
  • 数据量: 51379878.03449913 字节
  • 样本数: 5950
• 测试集:
  • 数据量: 897471.65 字节
  • 样本数: 95