tvu-vlinhd11/vi-dataset-for-pretrain

--- dataset_info: features: - name: text dtype: string splits: - name: train num_bytes: 77360702833 num_examples: 23891116 - name: validation num_bytes: 4064634081 num_examples: 1257428 download_size: 2126869688 dataset_size: 81425336914 task_categories: - text-generation language: - vi size_categories: - 10M<n<100M tags: - LM --- # Dataset Card for "vi-dataset-for-pretrain" This is a combination of multiple Vietnamese dataset for pretraining CLMs such as GPT, GPT2, etc. The dataset consists of: - [`vietgpt/covid_19_news_vi`](https://huggingface.co/datasets/vietgpt/covid_19_news_vi) - [`hieunguyen1053/binhvq-news-corpus`](https://huggingface.co/datasets/hieunguyen1053/binhvq-news-corpus) - [`oscar (unshuffled_deduplicated_vi)`](https://huggingface.co/datasets/oscar) - [`vietgpt/wikipedia_vi`](https://huggingface.co/datasets/vietgpt/wikipedia_vi) # Dataset info | Splits | N.o examples | Size | | --- | --- | --- | | Train | 23,891,116 | 77.36 GB | | Validation | 1,257,428 | 4.06 GB | | **Total** | **25,148,544** | **81.43 GB** |数据集信息： 特征字段： - 字段名：text，数据类型：字符串 划分集： - 名称：train（训练集），字节数：77360702833，样本数：23891116 - 名称：validation（验证集），字节数：4064634081，样本数：1257428 下载大小：2126869688，数据集总大小：81425336914 任务类别： - 文本生成 语言： - 越南语 样本规模类别： - 1000万<样本数<1亿 标签： - 语言模型（Language Model, LM） # "用于预训练的越南语数据集"数据集卡片 本数据集整合了多份越南语语料，用于预训练因果语言模型（Causal Language Model, CLM），例如GPT、GPT2等。 本数据集包含以下数据源： - [`vietgpt/covid_19_news_vi`](https://huggingface.co/datasets/vietgpt/covid_19_news_vi)：越南新冠新闻数据集 - [`hieunguyen1053/binhvq-news-corpus`](https://huggingface.co/datasets/hieunguyen1053/binhvq-news-corpus)：binhvq新闻语料库 - [`oscar (unshuffled_deduplicated_vi)`](https://huggingface.co/datasets/oscar)：未洗牌去重越南语子集OSCAR数据集 - [`vietgpt/wikipedia_vi`](https://huggingface.co/datasets/vietgpt/wikipedia_vi)：越南语维基百科数据集 # 数据集详情 | 划分集 | 样本数量 | 大小 | | --- | --- | --- | | 训练集 | 23,891,116 | 77.36 GB | | 验证集 | 1,257,428 | 4.06 GB | | **总计** | **25,148,544** | **81.43 GB** |