zh-tw-llm-dv-dv/zh-tw-llm-dev-sample-ta8k-d40d11-only_embeddings-tr_wiki_sg_alp-c17ba7-c2048

--- dataset_info: dataset_size: 3875129.0 download_size: 1187475 features: - name: input_ids sequence: int32 - name: attention_mask sequence: int8 - name: labels sequence: int64 - dtype: string name: preview - dtype: int64 name: length splits: - name: train num_bytes: 3593100.0 num_examples: 500 - name: test num_bytes: 282029.0 num_examples: 50 --- # zh-tw-llm-dev-sample-ta8k-d40d11-only_embeddings-tr_wiki_sg_alp-c17ba7-c2048 This dataset is a part of the `zh-tw-llm-dev` project. * Tokenizer: `zh-tw-llm-dev-tokenizer-a8k-d40d11` * Built with: `translations`, `wikipedia`, `sharegpt`, `alpaca` * Rows: `train` `500`, `test` `50` * Max length: `2048` * Full config: ```json {"build_with": ["translations", "wikipedia", "sharegpt", "alpaca"], "sort_by": "length-desc", "preview_length": 256, "translations_settings": {"source_dataset": "zetavg/coct-en-zh-tw-translations-twp-300k", "lang_1_key": "en", "lang_2_key": "ch", "templates": ["English: {lang_1}\nChinese: {lang_2}", "Chinese: {lang_2}\nEnglish: {lang_1}"], "rows_limit": 100, "test_size": 0.1, "test_split_seed": 42, "test_rows_limit": 10}, "wikipedia_settings": {"source_dataset": "zetavg/zh-tw-wikipedia-dev", "exclude": [{"content_length_longer_than": 512}, {"match": "小行星", "in": "markdown", "in_range": [0, 40]}, {"match": "，是中國", "in": "markdown", "in_range": [0, 20]}, {"match": "中華人民共和國", "in": "markdown", "in_range": [0, 20]}, {"match": "是中華人民共和國", "in": "markdown", "in_range": [0, 40]}], "rows_limit": 100, "test_size": 0.1, "test_split_seed": 42, "test_rows_limit": 10}, "sharegpt_settings": {"source_dataset": "zetavg/ShareGPT-Processed", "train_on_inputs": false, "languages": [{"en": 0.4}, "zh_Hant"], "rows_limit": 100, "test_size": 0.1, "test_split_seed": 42, "test_rows_limit": 10}, "alpaca_settings": {"source_dataset": "zetavg/traditional-chinese-alpaca-en-align", "template": "short", "train_on_inputs": false, "rows_limit": 100, "test_size": 0.1, "test_split_seed": 42, "test_rows_limit": 10}} ```

原始信息汇总

数据集概述

基本信息

• 数据集大小: 3875129.0
• 下载大小: 1187475

特征描述

• input_ids: 序列类型为 int32
• attention_mask: 序列类型为 int8
• labels: 序列类型为 int64
• preview: 数据类型为 string
• length: 数据类型为 int64

数据分割

• 训练集:
  • 数据量: 3593100.0
  • 样本数: 500
• 测试集:
  • 数据量: 282029.0
  • 样本数: 50