sudy-super/JetCopper-10B
收藏Hugging Face2024-12-26 更新2024-06-11 收录
下载链接：
https://hf-mirror.com/datasets/sudy-super/JetCopper-10B
下载链接
链接失效反馈官方服务：
资源简介：
JetCopper-10B数据集是通过清洗、过滤和去重来自C4、CC-100、OSCAR-2301、HPLT Datasets v1.2和wiki40b-ja等数据集的日文子集而创建的。该数据集包含了日语、英语和代码三种语言，总大小约为51.4GB，用于Contrail-200m-64k模型的预训练。数据集尚未进行句子结束边界确定或困惑度过滤，因此存在质量提升的空间。
The JetCopper-10B dataset was created by cleaning, filtering, and deduplicating the Japanese subsets from C4, CC-100, OSCAR-2301, HPLT Datasets v1.2, and wiki40b-ja. It contains Japanese, English, and code languages, with a total size of approximately 51.4GB, and was used for pre-training the Contrail-200m-64k model. The dataset has not undergone sentence boundary determination or perplexity filtering, hence there is room for quality improvement.
提供机构：
sudy-super
