aisingapore/SEA-PILE-v1
收藏Hugging Face2025-12-02 更新2025-05-31 收录
下载链接：
https://hf-mirror.com/datasets/aisingapore/SEA-PILE-v1
下载链接
链接失效反馈官方服务：
资源简介：
SEA-LION-Pile是针对东南亚（SEA）地区预训练的语言模型集合SEA-LION的预训练数据集。该数据集包含了经过清理的mC4语料库部分。数据集由多种语言的数据组成，包括英语、中文、印尼语、马来语、菲律宾语、缅甸语、越南语、泰语、老挝语、高棉语和泰米尔语等。主要的数据源包括 refinedWeb、mC4、WangChanBERTa、the Stack和RedPajama等。数据集可能在互联网上的有毒或偏见内容以及数据过滤和去重方面存在局限性。
SEA-LION-Pile is the pretraining dataset for the SEA-LION collection of Large Language Models (LLMs), which has been pretrained and instruct-tuned for the Southeast Asia (SEA) region. The dataset consists of a cleaned portion of the mC4 corpus. It is composed of various linguistic data including English, Chinese, Indonesian, Malay, Filipino, Burmese, Vietnamese, Thai, Lao, Khmer, and Tamil. The primary data sources are refinedWeb, mC4, WangChanBERTa, the Stack, and RedPajama. The dataset may have limitations related to toxic or biased content on the internet and the possibility of filtering and deduplication errors.
提供机构：
aisingapore
