BLIP3o-Pretrain-Long-Caption

## BLIP3o Pretrain Long-Caption Dataset

This collection contains **27 million images**, each paired with a long (~120 token) caption generated by **Qwen/Qwen2.5-VL-7B-Instruct**.

### Download

```python
from huggingface_hub import snapshot_download

snapshot_download(
repo_id="BLIP3o/BLIP3o-Pretrain-Long-Caption",
repo_type="dataset"
)
```

## Load Dataset without Extracting
You don’t need to unpack the .tar archives, use WebDataset support in 🤗datasets instead:

```
from datasets import load_dataset
import glob

data_files = glob.glob("/your/data/path/*.tar")
train_dataset = load_dataset(
"webdataset",
data_files=data_files,
cache_dir="/your/hf/cache",
split="train",
num_proc=128
)
```

Feel free to comment on it when you have any issues.

### BLIP3o 预训练长标题数据集
该数据集涵盖2700万张图像，每张图像均配有由**Qwen/Qwen2.5-VL-7B-Instruct**生成的长标题，单条标题约含120个Token（Token）。

#### 下载方式
可通过以下Python代码完成数据集下载：
python
from huggingface_hub import snapshot_download

snapshot_download(
repo_id="BLIP3o/BLIP3o-Pretrain-Long-Caption",
repo_type="dataset"
)

#### 无需解压加载数据集
无需解压.tar格式归档文件，可直接借助🤗datasets库内置的WebDataset支持完成加载：

from datasets import load_dataset
import glob

data_files = glob.glob("/your/data/path/*.tar")
train_dataset = load_dataset(
"webdataset",
data_files=data_files,
cache_dir="/your/hf/cache",
split="train",
num_proc=128
)

若您在使用过程中遇到任何问题，欢迎留言反馈。