Beetle-Data/en-for-fi-100M-pretok
收藏官方服务：
资源简介：
该数据集名为Beetle-Data/en-for-fi-100M-pretok，是一个预分块（pretokenized）的文本数据集，每个数据块包含513个token的打包序列，确保没有跨文档混合。数据以分片形式上传，标记文件data/_finalized.json用于指示所有部分已上传完成。数据文件格式为Parquet。
The dataset is named Beetle-Data/en-for-fi-100M-pretok and consists of pretokenized chunks, each containing packed sequences of 513 tokens with no cross-document bleeding. The data is sharded incrementally, and the marker file data/_finalized.json is committed once all parts are uploaded. Data files are in Parquet format.
提供机构：
Beetle-Data
