jpaulpoliquit/ph-pretrain-v0.2

PH Corpus v0.2 是一个用于预训练的语料库数据集，主要聚焦于菲律宾相关语言。它包含多种语言的文本数据，如宿务语（ceb）、瓦瑞语（war）、他加禄语（tl）、伊洛卡诺语（ilo）、比科尔语（bcl）、邦板牙语（pam）、邦阿西楠语（pag）、希利盖农语（hil）、英语（en）、越南语（vie）、查瓦卡诺语（cbk）和阿塞拜疆语（azj）。数据集源自多个来源，包括维基媒体和其他文本集合，总文档数约为1,111,224个，分为训练集、验证集和测试集。数据以Parquet格式存储，包含丰富的列信息，如文本内容、语言检测、质量评分等，适用于自然语言处理预训练任务。PH Corpus v0.2 is a pre-training corpus dataset focusing on languages associated with the Philippines. It includes text data in multiple languages: Cebuano (ceb), Waray (war), Tagalog (tl), Ilocano (ilo), Bikol (bcl), Pampango (pam), Pangasinan (pag), Hiligaynon (hil), English (en), Vietnamese (vie), Chavacano (cbk), and Azerbaijani (azj). The dataset is sourced from multiple origins including Wikimedia and other text collections, with approximately 1,111,224 total documents split into training, validation, and test subsets. Stored in Parquet format, the dataset contains rich column information such as text content, language detection results, quality scores and other relevant attributes, which is suitable for natural language processing pre-training tasks.