alexandermorgan/FineWeb-Edu_10B_sample_2_column_word_counts
收藏Hugging Face2024-08-02 更新2024-12-14 收录
下载链接：
https://hf-mirror.com/datasets/alexandermorgan/FineWeb-Edu_10B_sample_2_column_word_counts
下载链接
链接失效反馈官方服务：
资源简介：
该数据集包含一个基于FineWeb-Edu 10B样本数据集的2列CSV文件表示。通过使用Python的`regex`包对文本进行分割，并将分割后的文本块进行计数，生成了一个`str: int`映射的CSV文件。这种处理方式使得文件大小从27GB减少到240MB，便于在内存中处理，主要用于训练分词器。此外，还提到了一个开源的Python分词器BatchBPE，该工具可以加载和保存这种CSV格式的数据集。
This dataset contains a 2-column CSV file representation of the FineWeb-Edu 10B sample dataset. The text chunks from these splits were counted to make the `str: int` mapping of the CSV file. The result is a greater than 100X reduction in file size (27GB -> 240MB) making it easy to fit this representation of the dataset in memory. This is intended for the purposes of training a tokenizer. Additionally, an open-source Python tokenizer called BatchBPE is mentioned, which can load and save datasets in this CSV format.
提供机构：
alexandermorgan
