BLIP3o-Pretrain-JourneyDB

## BLIP3o Pretrain JourneyDB Dataset

This collection contains **4 million JourneyDB images**.

### Download

```python
from huggingface_hub import snapshot_download

snapshot_download(
repo_id="BLIP3o/BLIP3o-Pretrain-JourneyDB",
repo_type="dataset"
)
```

## Load Dataset without Extracting
You don’t need to unpack the .tar archives, use WebDataset support in 🤗datasets instead:

```
from datasets import load_dataset
import glob

data_files = glob.glob("/your/data/path/*.tar")
train_dataset = load_dataset(
"webdataset",
data_files=data_files,
cache_dir="/your/hf/cache",
split="train",
num_proc=128
)
```

Feel free to comment on it when you have any issues.

## BLIP3o 预训练JourneyDB数据集

该数据集集合包含400万张JourneyDB图像。

### 下载
python
from huggingface_hub import snapshot_download

snapshot_download(
repo_id="BLIP3o/BLIP3o-Pretrain-JourneyDB",
repo_type="dataset"
)

## 无需解压加载数据集
无需对.tar归档文件进行解压，可直接使用🤗datasets库中的WebDataset支持功能进行加载：

from datasets import load_dataset
import glob

data_files = glob.glob("/your/data/path/*.tar")
train_dataset = load_dataset(
"webdataset",
data_files=data_files,
cache_dir="/your/hf/cache",
split="train",
num_proc=128
)

若遇到任何问题，欢迎留言反馈。

搜集汇总

背景与挑战

背景概述

该数据集名为BLIP3o-Pretrain-JourneyDB，包含400万张JourneyDB图像，适用于预训练任务。它采用Apache License 2.0许可，并提供了通过Hugging Face下载和加载的详细指南。