lance-format/laion-1m

--- license: cc-by-4.0 task_categories: - text-to-image - image-to-text - zero-shot-image-classification language: - en tags: - laion - clip - vision-language - lance pretty_name: laion-1m-lance size_categories: - 1M<n<10M --- # LAION-Subset (Lance Format) A Lance-formatted slice of the [LAION](https://laion.ai/blog/laion-5b/) image-text corpus (~1M rows) with inline JPEG bytes, CLIP image embeddings (`img_emb`), full metadata, and a pre-built ANN index — all available directly from the Hub at `hf://datasets/lance-format/laion-1m/data/train.lance`. ## Key features - **Inline JPEG bytes** in the `image` column — no sidecar files, no image folders. - **Pre-computed CLIP image embeddings** (`img_emb`, 768-dim) with a bundled `IVF_PQ` index for similarity search. - **Full LAION metadata** — captions, URLs, NSFW flags, EXIF, dimensions, similarity scores. - **One columnar dataset** — scan metadata cheaply, then fetch image bytes only for the rows you want. ## Splits `train.lance` ## Schema | Column | Type | Notes | |---|---|---| | `key` | `int64` | Row key (natural join key for merges) | | `image` | `large_binary` | Inline JPEG bytes | | `image_path` | `string` | Original filename | | `caption` | `string` | Image caption | | `url` | `string` | Source URL | | `NSFW` | `int64` | 0 = safe, 1 = NSFW | | `LICENSE` | `string` | Per-row license tag | | `similarity` | `float64` | CLIP image–text cosine similarity | | `width`, `height` | `int64` | Image dimensions | | `original_width`, `original_height` | `int64` | Original dimensions before resize | | `exif`, `md5`, `status`, `error_message` | `string` | Provenance / metadata | | `img_emb` | `fixed_size_list<float32, 768>` | CLIP image embedding | ## Pre-built indices - `IVF_PQ` on `img_emb` — vector similarity search (L2) ## Why Lance? 1. **Blazing Fast Random Access**: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation. 2. **Native Multimodal Support**: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search. 3. **Native Index Support**: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them. 4. **Efficient Data Evolution**: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time. 5. **Versatile Querying**: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes. 6. **Data Versioning**: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history. ## Load with `datasets.load_dataset` You can load Lance datasets via the standard HuggingFace `datasets` interface, suitable if your pipeline already speaks `Dataset` / `IterableDataset` or you want a quick streaming sample without installing anything Lance-specific. ```python import datasets hf_ds = datasets.load_dataset("lance-format/laion-1m", split="train", streaming=True) for row in hf_ds.take(3): print(row["caption"]) ``` ## Load with LanceDB LanceDB is the embedded retrieval library built on top of the Lance format ([docs](https://lancedb.com/docs)), and is the interface most users interact with. It wraps the dataset as a queryable table with search and filter builders, and is the entry point used by the Search, Curate, Evolve, and Versioning examples below. ```python import lancedb db = lancedb.connect("hf://datasets/lance-format/laion-1m/data") tbl = db.open_table("train") print(len(tbl)) ``` ## Load with Lance `pylance` is the Python binding for the Lance format and works directly with the format's lower-level APIs. Reach for it when you want to inspect or operate on dataset internals — schema, scanner, fragments, and the list of pre-built indices. ```python import lance ds = lance.dataset("hf://datasets/lance-format/laion-1m/data/train.lance") print(ds.count_rows(), ds.schema.names) print(ds.list_indices()) ``` > **Tip — for production use, download locally first.** Streaming from the Hub works for exploration, but heavy random access and ANN search are far faster against a local copy: > ```bash > hf download lance-format/laion-1m --repo-type dataset --local-dir ./laion-1m > ``` > Then point Lance or LanceDB at `./laion-1m/data`. ## Search The bundled `IVF_PQ` index on `img_emb` makes approximate-nearest-neighbor search a single call. In production you would encode a user prompt or query image through CLIP at runtime and pass the resulting 768-d vector to `tbl.search(...)`. The example below uses the embedding from row 42 as a runnable stand-in. ```python import lancedb db = lancedb.connect("hf://datasets/lance-format/laion-1m/data") tbl = db.open_table("train") query = ( tbl.search() .select(["img_emb"]) .limit(1) .offset(42) .to_list()[0]["img_emb"] ) hits = ( tbl.search(query) .metric("L2") .select(["caption", "url", "similarity"]) .limit(10) .to_list() ) for r in hits: print(f"{r['similarity']:.3f} | {r['caption'][:80]}") ``` Tune `metric`, `nprobes`, and `refine_factor` to trade recall against latency for your workload. ## Curate Building a focused subset usually means combining similarity with metadata filters. Lance evaluates both inside a single query, so the candidate set comes back already filtered. The example below finds images visually similar to a seed row and restricts the result to safe-rated, high-resolution rows in one call. The bounded `.limit(500)` keeps the output small enough to inspect or hand off. ```python import lancedb db = lancedb.connect("hf://datasets/lance-format/laion-1m/data") tbl = db.open_table("train") seed = ( tbl.search() .select(["img_emb", "caption"]) .limit(1) .offset(42) .to_list()[0] ) candidates = ( tbl.search(seed["img_emb"]) .where('"NSFW" = 0 AND similarity > 0.3 AND width >= 512', prefilter=True) .select(["key", "url", "caption", "similarity"]) .limit(500) .to_list() ) print(f"{len(candidates)} candidates around: {seed['caption'][:60]}") ``` The result is a plain list of dictionaries, ready to inspect, persist as a manifest of row keys, or feed into the Evolve and Train workflows below. ## Evolve Lance stores each column independently, so a new column can be appended without rewriting the existing data. The lightest form is a SQL expression: derive the new column from columns that already exist, and Lance computes it once and persists it. The example below adds a precomputed `aspect_ratio` and an `is_high_res` flag, either of which can then be used directly in `where` clauses without recomputing the predicate on every query. > **Note**: Mutations require a local copy of the dataset, since the Hub mount is read-only. See the Materialize-a-subset section at the end of this card for a streaming pattern that downloads only the rows and columns you need, or use `hf download` to pull the full corpus. ```python import lancedb db = lancedb.connect("./laion-1m/data") # local copy required for writes tbl = db.open_table("train") tbl.add_columns({ "aspect_ratio": "CAST(width AS DOUBLE) / CAST(height AS DOUBLE)", "is_high_res": "width >= 512 AND height >= 512", }) ``` If the values you want to attach already live in another table (offline labels, classifier predictions, aesthetic scores), merge them in by joining on the `key` column: ```python import pyarrow as pa labels = pa.table({ "key": pa.array([0, 1, 2]), "aesthetic_score": pa.array([7.1, 6.4, 8.9]), }) tbl.merge(labels, on="key") ``` The original columns and indices are untouched, so existing code that does not reference the new columns continues to work unchanged. New columns become visible to every reader as soon as the operation commits. For column values that require a Python computation (e.g., running an embedding model over the image bytes), Lance provides a batch-UDF API in the underlying library — see the [Lance data evolution docs](https://lance.org/guide/data_evolution/) for that pattern. ## Train Projection lets a training loop read only the columns each step actually needs. LanceDB tables expose this through `Permutation.identity(tbl).select_columns([...])`, which plugs straight into the standard `torch.utils.data.DataLoader` so prefetch, shuffling, and batching behave as in any PyTorch pipeline. Columns added in the Evolve section above cost nothing per batch until they are explicitly projected. ```python import lancedb from lancedb.permutation import Permutation from torch.utils.data import DataLoader db = lancedb.connect("hf://datasets/lance-format/laion-1m/data") tbl = db.open_table("train") train_ds = Permutation.identity(tbl).select_columns(["image", "caption"]) loader = DataLoader(train_ds, batch_size=256, shuffle=True, num_workers=4) for batch in loader: # batch carries only the projected columns; img_emb / img_emb_dinov3 stay on disk. # decode the JPEG bytes, tokenize the captions, forward, backward... ... ``` Switching feature sets is a configuration change: passing `["img_emb_dinov3", "caption"]` to `select_columns(...)` on the next run reads only those columns, with no data movement or shard reorganization. ## Versioning Every mutation to a Lance dataset, whether it adds a column, merges labels, or builds an index, commits a new version. Previous versions remain intact on disk. You can list versions and inspect the history directly from the Hub copy; creating new tags requires a local copy since tags are writes. ```python import lancedb db = lancedb.connect("hf://datasets/lance-format/laion-1m/data") tbl = db.open_table("train") print("Current version:", tbl.version) print("History:", tbl.list_versions()) print("Tags:", tbl.tags.list()) ``` Once you have a local copy, tag a version for reproducibility: ```python local_db = lancedb.connect("./laion-1m/data") local_tbl = local_db.open_table("train") local_tbl.tags.create("aesthetic-v1", local_tbl.version) ``` A tagged version can be opened by name, or any version reopened by its number, against either the Hub copy or a local one: ```python tbl_v1 = db.open_table("train", version="aesthetic-v1") tbl_v5 = db.open_table("train", version=5) ``` Pinning supports two workflows. A retrieval system locked to `aesthetic-v1` keeps returning stable results while the dataset evolves in parallel; newly added columns or labels do not change what the tag resolves to. A training experiment pinned to the same tag can be rerun later against the exact same data, so changes in metrics reflect model changes rather than data drift. Neither workflow needs shadow copies or external manifest tracking. ## Materialize a subset Reads from the Hub are lazy, so exploratory queries only transfer the columns and row groups they touch. Mutating operations (Evolve, tag creation) need a writable backing store, and a training loop benefits from a local copy with fast random access. Both can be served by a subset of the dataset rather than the full corpus. The pattern is to stream a filtered query through `.to_batches()` into a new local table; only the projected columns and matching row groups cross the wire, and the bytes never fully materialize in Python memory. ```python import lancedb remote_db = lancedb.connect("hf://datasets/lance-format/laion-1m/data") remote_tbl = remote_db.open_table("train") batches = ( remote_tbl.search() .where('"NSFW" = 0 AND similarity > 0.35 AND width >= 512') .select(["key", "image", "caption", "url", "img_emb"]) .to_batches() ) local_db = lancedb.connect("./laion-subset") local_db.create_table("train", batches) ``` The resulting `./laion-subset` is a first-class LanceDB database. Every snippet in the Evolve, Train, and Versioning sections above works against it by swapping `hf://datasets/lance-format/laion-1m/data` for `./laion-subset`. ## Citation ``` @article{schuhmann2022laion5b, title={LAION-5B: An open large-scale dataset for training next generation image-text models}, author={Schuhmann, Christoph and others}, journal={NeurIPS Datasets and Benchmarks Track}, year={2022} } ``` ## License Content inherits LAION's original licensing and safety guidelines. Review [LAION policy](https://laion.ai/blog/laion-5b/) before downstream use.LAION-Subset (Lance Format) is a Lance-formatted slice of the LAION image-text corpus with approximately 1 million rows. It includes inline JPEG bytes, pre-computed CLIP image embeddings (768-dimensional), full metadata (such as captions, URLs, NSFW flags, EXIF, dimensions, similarity scores), and a pre-built approximate nearest neighbor (ANN) index. The dataset supports multiple tasks, including text-to-image, image-to-text, and zero-shot image classification, and is designed for efficient data access, similarity search, and data processing, suitable for machine learning training, retrieval, and interactive applications.

搜集汇总

数据集介绍

构建方式

面向大规模图文预训练与跨模态检索任务，laion-1m 以 LAION-5B 图像—文本语料为来源，通过列式采样与格式重封装形成约百万行切片。其构建将原始图像编码为内联 JPEG 字节，纳入 CLIP 图像嵌入，并保留 caption、URL、NSFW 标记、逐行许可证、图像—文本相似度、EXIF、尺寸及来源校验信息。该切片以 Lance 列式数据集组织，配合 IVF_PQ 近似最近邻索引，使图像、文本、向量与元数据共处单一存储单元，并以 key 列支持自然连接，服务于检索、训练与数据演化场景。

使用方法

在图文建模、零样本分类与跨模态检索应用中，laion-1m 可经 HuggingFace datasets 流式加载，亦可由 LanceDB 或 pylance 直接打开。使用者可执行近似最近邻查询，并以 NSFW、分辨率、相似度等谓词联合过滤候选集；可将结果物化为本地子集，进而完成标签合并、列投影训练与版本固定。通过 select_columns 仅投影训练所需列，借助 DataLoader 组成批次；版本标签可固定快照，便于复现。生产环境宜先下载至本地，以降低随机访问与 ANN 搜索延迟，并保持数据版本可追溯。

背景与挑战

背景概述

在开放多模态学习迅猛演进的脉络中，LAION-5B于2022年由Christoph Schuhmann等研究者与LAION社区发布，旨在以开放大规模图文语料支撑下一代视觉语言模型。laion-1m作为其Lance格式切片，承袭约百万行图文对、内联JPEG字节、CLIP图像嵌入及完整元数据，核心研究问题在于降低大规模图文数据的训练、检索与复现门槛。该数据集面向文本到图像、图像到文本与零样本图像分类等任务，为对比学习、生成模型与跨模态检索提供了可扩展的数据基础设施，对开放多模态研究生态具有显著推动作用。

当前挑战

面向图文语义对齐与跨模态检索，该数据集所对准的领域问题是大规模开放域图文预训练数据的稀缺与治理困难，类似ImageNet之于图像分类，其挑战在于从噪声、偏见、NSFW内容与复杂许可中筛选可用样本，并支撑高召回、低延迟的向量近邻搜索。构建过程中，约百万行内联JPEG、768维CLIP嵌入、ANN索引与列式元数据需统一于Lance格式，须权衡存储膨胀、随机访问效率、索引召回率与查询吞吐，并应对Hub只读挂载、本地写入、版本复现、安全过滤与许可追踪等工程及治理考验。

常用场景

经典使用场景

在跨模态表征学习与图文检索研究持续深化的背景下，laion-1m 以约百万级 LAION 图文对子集、内联 JPEG 字节、CLIP 图像嵌入与 IVF_PQ 索引构成一体化数据基底。其经典使用场景集中于图像—文本对比预训练、零样本图像分类、图文相似检索与文生图或图生文任务的原型验证。研究者可借助预计算嵌入与近似最近邻搜索快速完成查询图像或提示词的近邻召回，亦可按列投影仅读取训练所需字段，从而在有限算力下复现视觉语言模型的关键流程。

解决学术问题

面对大规模图文语料获取成本高、图像文件与元数据割裂、随机访问迟缓以及检索实验难以复现等常见学术难题，laion-1m 以列式 Lance 格式将图像字节、标题、URL、NSFW 标记、分辨率、相似度与 CLIP 嵌入统一存储，并附带磁盘索引与版本化机制。它缓解了多模态研究中数据预处理繁重、索引重算耗时和实验环境漂移的问题，使零样本分类、图文对齐评估与近似最近邻基准能够在相对统一的条件下开展，对提升可复现性和降低资源门槛具有积极意义。

实际应用

在产业场景中，该数据集适用于图文素材检索、内容推荐、电商商品图搜、设计灵感匹配与生成式模型训练数据筛选。借助内联图像与预建向量索引，系统可在单个查询中融合向量相似度、全文检索和 SQL 式元数据过滤，例如限定安全评级、分辨率与相似度阈值后召回候选集。其列式读取与随机访问特性亦利于实时机器学习服务、交互式样例抓取和轻量级多模态应用验证，为从研究原型过渡到可部署检索管线提供支撑。

数据集最近研究

最新研究方向

伴随多模态基础模型对大规模图文语料依赖日深，LAION-1m以Lance列式载体把内联JPEG字节、768维CLIP嵌入与完整元数据共置一处，并预置IVF_PQ索引，令向量近邻检索、属性过滤与版本化快照可于同一查询面协同。其前沿方向聚焦检索增强的多模态训练、跨模态嵌入对齐与数据策展：研究者可借近似最近邻搜索结合NSFW、分辨率与相似度筛选，构建高纯度子集；亦可以投影读取与列式演化支持可复现训练、嵌入模型迭代和标签回填。LAION-5B开放语料引发的版权、隐私与安全争议，使许可追踪、内容治理及版本标签成为热点。该数据集由此降低多模态数据工程门槛，为开放图文语料的合规使用、检索—训练闭环与实验复现提供基础设施。