Beetle-Data/en-for-ar-100M-pretok
收藏官方服务：
资源简介：
Beetle-Data/en-for-ar-100M-pretok是一个预分词的数据集，包含打包的序列，每个序列由513个标记（token）组成，确保没有跨文档混合。数据以分片形式增量上传，并通过标记文件data/_finalized.json来确认所有部分已完整上传。
Beetle-Data/en-for-ar-100M-pretok is a pretokenized dataset consisting of chunks with input_ids as 513-token packed sequences, without cross-document bleeding. The data is sharded incrementally, and the marker file data/_finalized.json is committed once all parts are uploaded.
提供机构：
Beetle-Data
