kingsidharth/zangei_tokenizer_1M_imgs

这是一个用于Zangei分词器训练的图像数据集，包含约100万张图像。每个图像记录具有稳定的core_id，可用于与DINOv3特征分片以及未来的文本或元数据集进行连接。数据集提供三种视图：原始尺寸的WebP编码图像、中心裁剪并调整大小为256x256的WebP图像，以及直接调整大小为256x256的WebP图像。分片目标为每个分片约10GB，包含核心索引清单、清单文件和分片视图的tar文件。

Packed image dataset for Zangei tokenizer training, containing approximately 1 million images. Each record has a stable core_id that can be used to join with DINOv3 feature shards and future text/metadata datasets. The dataset includes three views: original image dimensions encoded as WebP, center-square crop resized to 256x256 encoded as WebP, and direct resize to 256x256 without cropping encoded as WebP. Shard target is about 10GB per shard, with files including core index manifests, manifest files, and tar files for shard views.