noakraicer/ID-LoRA-TalkVid

---
configs:
- config_name: default
drop_labels: true
license: cc-by-4.0
task_categories:
- text-to-video
- text-to-audio
tags:
- audio-video-generation
- speaker-identity
- id-lora
- in-context-lora
- multimodal
- ltx-video
pretty_name: "TalkVid Preprocessed for ID-LoRA"
size_categories:
- 1K<n<10K
---

# TalkVid Preprocessed Dataset for ID-LoRA

Preprocessed training data for **[ID-LoRA](https://github.com/ID-LoRA/ID-LoRA)**: identity-driven audio-video personalization with In-Context LoRA ([paper](https://huggingface.co/papers/2603.10256)).

## Overview

| Property | Value |
|----------|-------|
| Source dataset | TalkVid |
| Training pairs | 5,796 |
| Unique videos | 5,803 |
| Speakers | 600 |
| Resolution | Original (1080p/4K); latents computed at 512×512 |
| Frame rate | 25 fps |
| Frames per clip | 121 (~4.84 s) |

## Quick Start

### Browse and load data

`load_dataset` returns training pairs with playable video objects and metadata:

```python
from datasets import load_dataset

ds = load_dataset("noakraicer/ID-LoRA-TalkVid")
print(ds["train"].column_names) # ['target', 'reference', 'speaker_cluster_id', 'target_caption']
print(ds["train"][0]["target_caption"])
```

> **Note:** Loading videos requires the `decord` package (`pip install decord`).

### Download for training

```python
from huggingface_hub import snapshot_download

# Full dataset (videos + precomputed latents)
snapshot_download("noakraicer/ID-LoRA-TalkVid", repo_type="dataset", local_dir="./data/talkvid")

# Precomputed latents only (skip videos)
snapshot_download(
"noakraicer/ID-LoRA-TalkVid",
repo_type="dataset",
allow_patterns=["precomputed/**", "train/metadata.jsonl"],
local_dir="./data/talkvid",
)
```

### Extract precomputed latents

After downloading, extract the `.tar.zst` archives:

```bash
cd ./data/talkvid/precomputed
for f in *.tar.zst; do
tar --use-compress-program=unzstd -xf "$f"
done
```

This creates per-video `.pt` files in `latents/`, `audio_latents/`, `audio_latents_clean/`, and `conditions/`.

## Dataset Structure

```
ID-LoRA-TalkVid/
├── train/
│ ├── metadata.jsonl # Pair metadata (5,796 rows)
│ └── {video_id}.mp4 # 5,803 unique video clips
└── precomputed/
├── latents.tar.zst # Video VAE latents
├── audio_latents.tar.zst # Audio VAE latents (target)
├── audio_latents_clean.tar.zst # Denoised audio VAE latents (reference)
└── conditions.tar.zst # Text + caption embeddings (Gemma 3)
```

Video filenames follow the pattern `{video_id}.mp4`. Latent filenames follow `{video_id}.pt`.

## Metadata Columns

Each row in `train/metadata.jsonl` represents one training pair. When loaded via `load_dataset`, the `*_file_name` columns are resolved to video objects:

| Column | Type | Description |
|--------|------|-------------|
| `target` | Video | Target video clip |
| `reference` | Video | Reference video clip (same speaker, different clip) |
| `speaker_cluster_id` | string | Speaker identity cluster |
| `target_caption` | string | Structured caption with `[VISUAL]`, `[SPEECH]`, `[SOUNDS]`, `[TEXT]` sections |

## Precomputed Latents

Ready-to-train representations stored as `.tar.zst` archives, each containing per-video `.pt` files.

| Archive | Description |
|---------|-------------|
| `latents.tar.zst` | Video VAE latents |
| `audio_latents.tar.zst` | Audio VAE latents — **target** audio (original recording, including environmental sounds) |
| `audio_latents_clean.tar.zst` | Audio VAE latents — **reference** audio (denoised speech, used for IC conditioning) |
| `conditions.tar.zst` | Text embeddings from Gemma 3, computed from the full structured caption |

### Audio Latent Types

Training uses two types of audio latents:

- **`audio_latents`** (target): Audio encoded from the original video. This is what the model learns to generate.
- **`audio_latents_clean`** (reference): Denoised audio with environmental sounds removed, used as in-context conditioning. Clean audio helps the model focus on speaker identity rather than background noise.

See the [training code](https://github.com/ID-LoRA/ID-LoRA) for how reference audio latents are mapped to training pairs.

### Latent File Format

Each `.pt` file is a dict:

- **Video latents**: `latents` (bf16 `[C, F, H, W]`), `num_frames`, `height`, `width`, `fps`
- **Audio latents**: `latents` (fp32 `[C, T, F]`), `num_time_steps`, `frequency_bins`, `duration`
- **Conditions**: `prompt_embeds` (bf16 `[seq_len, hidden]`), `prompt_attention_mask`

## Caption Format

Each caption follows a structured format with four tagged sections:

```
[VISUAL]: <scene description, people, actions, setting>
[SPEECH]: <word-for-word speech transcription>
[SOUNDS]: <speaker vocal style, environmental sounds>
[TEXT]: <on-screen text or "None">
```

## Training

This dataset is designed for training [ID-LoRA](https://github.com/ID-LoRA/ID-LoRA) adapters built on [LTX-2](https://github.com/Lightricks/LTX-Video).
Each pair consists of a target video and a reference video from the **same speaker**.
The model jointly generates audio and video, transferring the speaker's voice identity from the reference
while the visual appearance is controlled via first-frame conditioning and the text prompt.

See the [paper](https://huggingface.co/papers/2603.10256) and
[code](https://github.com/ID-LoRA/ID-LoRA) for training details.

## Citation

```bibtex
@misc{dahan2026idloraidentitydrivenaudiovideopersonalization,
title = {ID-LoRA: Identity-Driven Audio-Video Personalization
with In-Context LoRA},
author = {Aviad Dahan and Moran Yanuka and Noa Kraicer and Lior Wolf and Raja Giryes},
year = {2026},
eprint = {2603.10256},
archivePrefix = {arXiv},
primaryClass = {cs.SD},
url = {https://arxiv.org/abs/2603.10256}
}
```

配置项：
- 配置名称：default
丢弃标签：启用
许可证：CC BY 4.0
任务类别：
- 文本到视频
- 文本到音频
标签：
- 音频视频生成
- 说话人身份
- ID-LoRA
- 上下文LoRA（In-Context LoRA）
- 多模态
- LTX视频
友好名称："适用于ID-LoRA的预处理TalkVid数据集"
数据规模：1000 < 样本数 < 10000

# 适用于ID-LoRA的TalkVid预处理数据集

本数据集为**ID-LoRA（基于上下文LoRA的身份驱动音视频个性化模型，论文链接：https://huggingface.co/papers/2603.10256）**提供预处理后的训练数据。

## 概览

| 数据集属性 | 数值 |
|----------|-------|
| 源数据集 | TalkVid |
| 训练样本对 | 5,796 |
| 唯一视频片段 | 5,803 |
| 说话人数量 | 600 |
| 分辨率 | 原始分辨率（1080p/4K）；隐向量在512×512分辨率下计算 |
| 帧率 | 25 fps |
| 单片段帧数 | 121帧（约4.84秒） |

## 快速上手

### 浏览并加载数据
`load_dataset` 会返回包含可播放视频对象与元数据的训练样本对：
python
from datasets import load_dataset

ds = load_dataset("noakraicer/ID-LoRA-TalkVid")
print(ds["train"].column_names) # ['target', 'reference', 'speaker_cluster_id', 'target_caption']
print(ds["train"][0]["target_caption"])

> **注意：** 加载视频需要安装 `decord` 包（执行 `pip install decord` 完成安装）。

### 下载用于训练
python
from huggingface_hub import snapshot_download

# 完整数据集（包含视频与预计算隐向量）
snapshot_download("noakraicer/ID-LoRA-TalkVid", repo_type="dataset", local_dir="./data/talkvid")

# 仅下载预计算隐向量（跳过视频文件）
snapshot_download(
"noakraicer/ID-LoRA-TalkVid",
repo_type="dataset",
allow_patterns=["precomputed/**", "train/metadata.jsonl"],
local_dir="./data/talkvid",
)

### 解压预计算隐向量
下载完成后，解压 `.tar.zst` 压缩包：
bash
cd ./data/talkvid/precomputed
for f in *.tar.zst; do
tar --use-compress-program=unzstd -xf "$f"
done

该操作将在 `latents/`、`audio_latents/`、`audio_latents_clean/` 与 `conditions/` 目录下生成每个视频对应的 `.pt` 文件。

## 数据集结构

ID-LoRA-TalkVid/
├── train/
│ ├── metadata.jsonl # 训练样本对元数据（共5,796行）
│ └── {video_id}.mp4 # 5,803个唯一视频片段
└── precomputed/
├── latents.tar.zst # 视频变分自编码器（Variational AutoEncoder, VAE）隐向量
├── audio_latents.tar.zst # 目标音频变分自编码器（VAE）隐向量
├── audio_latents_clean.tar.zst # 去噪音频变分自编码器（VAE）隐向量（参考音频）
└── conditions.tar.zst # 文本与字幕嵌入（基于Gemma 3模型）

视频文件名遵循 `{video_id}.mp4` 格式，隐向量文件名遵循 `{video_id}.pt` 格式。

## 元数据列说明
`train/metadata.jsonl` 中的每一行代表一个训练样本对。通过 `load_dataset` 加载时，带 `*_file_name` 后缀的列会被解析为视频对象：
| 列名 | 数据类型 | 描述 |
|--------|------|-------------|
| `target` | 视频 | 目标视频片段 |
| `reference` | 视频 | 参考视频片段（同一说话人，不同片段） |
| `speaker_cluster_id` | 字符串 | 说话人身份聚类ID |
| `target_caption` | 字符串 | 包含 `[VISUAL]`、`[SPEECH]`、`[SOUNDS]`、`[TEXT]` 四个分段的结构化字幕 |

## 预计算隐向量
可直接用于训练的特征表示存储为 `.tar.zst` 压缩包，每个压缩包包含每个视频对应的 `.pt` 文件。
| 压缩包名称 | 描述 |
|---------|-------------|
| `latents.tar.zst` | 视频变分自编码器（VAE）隐向量 |
| `audio_latents.tar.zst` | 音频变分自编码器（VAE）隐向量 —— **目标音频**（原始录制音频，包含环境音） |
| `audio_latents_clean.tar.zst` | 音频变分自编码器（VAE）隐向量 —— **参考音频**（去噪语音，用于上下文调节） |
| `conditions.tar.zst` | 基于Gemma 3模型从完整结构化字幕中提取的文本嵌入 |

### 音频隐向量类型
训练过程使用两种类型的音频隐向量：
- **`audio_latents`（目标音频）**：从原始视频中编码得到的音频，为模型需要学习生成的目标内容。
- **`audio_latents_clean`（参考音频）**：去除环境音后的去噪语音，用作上下文调节输入。纯净音频可帮助模型聚焦于说话人身份而非背景噪声。
请参阅[训练代码](https://github.com/ID-LoRA/ID-LoRA)了解参考音频隐向量与训练样本对的映射方式。

### 隐向量文件格式
每个 `.pt` 文件为一个Python字典，包含以下字段：
- **视频隐向量**：`latents`（bf16格式，维度为 `[C, F, H, W]`）、`num_frames`、`height`、`width`、`fps`
- **音频隐向量**：`latents`（fp32格式，维度为 `[C, T, F]`）、`num_time_steps`、`frequency_bins`、`duration`
- **条件嵌入**：`prompt_embeds`（bf16格式，维度为 `[seq_len, hidden]`）、`prompt_attention_mask`

## 字幕格式
每个字幕遵循包含四个带标签分段的结构化格式：

[VISUAL]: <场景描述、人物、动作、环境>
[SPEECH]: <逐字语音转录>
[SOUNDS]: <说话人发声风格、环境音>
[TEXT]: <屏幕文本或"无">

## 训练说明
本数据集专为训练基于LTX-2（https://github.com/Lightricks/LTX-Video）的ID-LoRA适配器而设计。每个训练样本对包含来自**同一说话人**的目标视频与参考视频。模型可联合生成音频与视频，从参考视频中迁移说话人语音身份，同时通过首帧调节与文本提示控制视觉外观。
请参阅[论文](https://huggingface.co/papers/2603.10256)与[代码](https://github.com/ID-LoRA/ID-LoRA)了解训练细节。

## 引用
bibtex
@misc{dahan2026idloraidentitydrivenaudiovideopersonalization,
title = {ID-LoRA: Identity-Driven Audio-Video Personalization
with In-Context LoRA},
author = {Aviad Dahan and Moran Yanuka and Noa Kraicer and Lior Wolf and Raja Giryes},
year = {2026},
eprint = {2603.10256},
archivePrefix = {arXiv},
primaryClass = {cs.SD},
url = {https://arxiv.org/abs/2603.10256}
}