Beetle-Data/tl-2B-pretok
收藏官方服务：
资源简介：
Beetle-Data/tl-2B-pretok数据集包含预分词的块，这些块是打包的序列，每个序列包含513个令牌（输入ID），且确保没有跨文档渗漏。数据以分片方式增量上传，所有部分上传完成后会通过data/_finalized.json标记来指示完成状态。
The Beetle-Data/tl-2B-pretok dataset consists of pretokenized chunks, which are packed sequences with input_ids of 513 tokens, ensuring no cross-document bleeding. The data is sharded incrementally, and the marker data/_finalized.json is committed once all parts are uploaded.
提供机构：
Beetle-Data
