PaulPauls/openwebtext-sentences
收藏Hugging Face2024-09-22 更新2024-12-14 收录
下载链接：
https://hf-mirror.com/datasets/PaulPauls/openwebtext-sentences
下载链接
链接失效反馈官方服务：
资源简介：
OpenWebText-Sentences数据集是从流行的OpenWebText数据集派生而来的，包含与原始OpenWebText相同的文本内容，但被分割成单独的句子。句子使用NLTK 3.9.1的预训练Punkt分词器进行分割，并以parquet格式存储以便更快访问。数据集的大小为25.7 GB，包含307,432,490个句子，语言为英语。
The OpenWebText-Sentences dataset is derived from the popular OpenWebText dataset, containing the same text content as the original OpenWebText but split into individual sentences. Sentences were split using NLTK 3.9.1 pre-trained Punkt tokenizer and stored in parquet format for faster access. The dataset size is 25.7 GB, containing 307,432,490 sentences, and the language is English.
提供机构：
PaulPauls
