Beetle-Data/ja-2B-pretok
收藏官方服务：
资源简介：
该数据集是预分词的日语文本数据，包含分块处理，每个块由513个token的打包序列（input_ids）组成，确保无跨文档混合。数据以分片形式逐步上传，并在全部上传完成后通过data/_finalized.json文件标记完成。
This dataset is pretokenized Japanese text data, consisting of chunked processing with each chunk as a packed sequence of 513 tokens (input_ids), ensuring no cross-document bleeding. The data is sharded and uploaded incrementally, with completion marked by the data/_finalized.json file after all parts are uploaded.
提供机构：
Beetle-Data
