duongttr/vi-dataset-for-pretrain

--- dataset_info: features: - name: text dtype: string splits: - name: train num_bytes: 77360702833 num_examples: 23891116 - name: validation num_bytes: 4064634081 num_examples: 1257428 download_size: 2126869688 dataset_size: 81425336914 task_categories: - text-generation language: - vi size_categories: - 10M<n<100M tags: - LM --- # Dataset Card for "vi-dataset-for-pretrain" This is a combination of multiple Vietnamese dataset for pretraining CLMs such as GPT, GPT2, etc. The dataset consists of: - [`vietgpt/covid_19_news_vi`](https://huggingface.co/datasets/vietgpt/covid_19_news_vi) - [`hieunguyen1053/binhvq-news-corpus`](https://huggingface.co/datasets/hieunguyen1053/binhvq-news-corpus) - [`oscar (unshuffled_deduplicated_vi)`](https://huggingface.co/datasets/oscar) - [`vietgpt/wikipedia_vi`](https://huggingface.co/datasets/vietgpt/wikipedia_vi) # Dataset info | Splits | N.o examples | Size | | --- | --- | --- | | Train | 23,891,116 | 77.36 GB | | Validation | 1,257,428 | 4.06 GB | | **Total** | **25,148,544** | **81.43 GB** |

原始信息汇总

数据集概述

基本信息

• 名称: vi-dataset-for-pretrain
• 任务类别: 文本生成
• 语言: 越南语
• 大小类别: 10M<n<100M
• 标签: LM

数据集组成

• vietgpt/covid_19_news_vi
• hieunguyen1053/binhvq-news-corpus
• oscar (unshuffled_deduplicated_vi)
• vietgpt/wikipedia_vi

数据集特征

• 特征名称: text
• 数据类型: string

数据集拆分

拆分	示例数量	大小
训练集	23,891,116	77.36 GB
验证集	1,257,428	4.06 GB
总计	25,148,544	81.43 GB

下载信息

• 下载大小: 2.13 GB
• 数据集总大小: 81.43 GB