Pinch-Research/lipsync-hdtf-training-data

--- license: apache-2.0 task_categories: - video-classification tags: - lipsync - talking-head - audio-visual - HDTF - X-Dub pretty_name: Lipsync Training Data (HDTF Teacher Pairs + Landmarks) size_categories: - 1K<n<10K --- # Lipsync Training Data Preprocessed datasets for audio-driven lip-sync model training, generated from the HDTF (High-Definition Talking Face) dataset. ## Contents ### 1. `xdub_teacher_pairs/` (1.3 GB) **1,449 same-speaker lip-synced video pairs** generated by [X-Dub](https://github.com/KlingAIResearch/X-Dub) (Wan2.2-TI2V-5B release). Each pair consists of: - **Source video**: an HDTF clip of speaker S saying utterance A - **Teacher output**: X-Dub's lip-synced version of the same clip with audio from a *different* utterance B by the *same* speaker S These are pseudo-pairs for distillation training: the teacher output has the source's pose/identity but with different lip shapes matching the alternate audio. The alternate audio is muxed into the teacher output mp4. **Generation cost**: ~25 GPU-hours on 4×H100 (X-Dub inference at ~4 min/clip with 30 DDIM steps). Structure: ``` xdub_teacher_pairs/ ├── videos/ │ └── {src_stem}__x__{audio_stem}.mp4 (512×512, 25fps, with alt audio muxed) └── meta/ └── {src_stem}__x__{audio_stem}.json (frame counts, alignment info) ``` ### 2. `xdub_teacher_pairs_manifest.json` (964 KB) Validated manifest of all 1,449 teacher pairs with metadata: - `source`: path to the original HDTF clip - `audio`: path to the alternate audio source clip - `teacher`: path to the X-Dub teacher output - `n_aligned`: min(source_frames, teacher_frames) — safe frame count for training - `src_speaker` / `alt_speaker`: speaker IDs (always same speaker) - `duration_s`: clip duration in seconds All pairs are **same-speaker only** per the X-Dub paper's recommendation (Sec 3.1): *"we sample a_alt from the same speaker as V_real, avoiding instability from unseen data or cross-identity audio-visual combinations."* ### 3. `hdtf_landmarks/` (2.4 GB) **MediaPipe FaceLandmarker landmarks for all 6,965 HDTF clips**, computed per-frame. Each `.npz` file contains: - `landmarks`: `(n_frames, 478, 2)` float16 — normalized [0,1] xy coordinates for 468 face mesh + 10 iris landmarks - `valid`: `(n_frames,)` uint8 — 1 if detection succeeded for that frame Generated using MediaPipe FaceLandmarker (float16 v1 model) on all clips in `data/hdtf/filtered/{WDA,WRA,RD}/`. **Useful for**: face/lip mask generation, face region extraction, head pose estimation, any talking-head research using HDTF. ## Source Data - **HDTF**: 6,965 clips, 341 speakers, ~19 hours total, all 512×512 at 25fps. Face-cropped frontal studio recordings. - **X-Dub teacher**: [KlingAIResearch/X-Dub](https://github.com/KlingAIResearch/X-Dub) (Apache 2.0), Wan2.2-TI2V-5B public release. ## How this data was used This data was created as part of a project to train a proprietary lip-sync model via teacher distillation from X-Dub. The teacher pairs provide (source, alt-audio) → (lip-synced output) training triples where the model learns to imitate X-Dub's lip-sync ability on a smaller/faster architecture. The landmarks were used to generate face/lip region masks for loss weighting during training (X-Dub paper App D: `L_wFM = (1 + w·M_face + w_lip·M_lip) ⊙ L_FM`). ## License - This preprocessed data: Apache 2.0 - HDTF source videos: subject to HDTF's original license (research use) - X-Dub teacher outputs: generated using X-Dub's Apache 2.0 code + released model weights - MediaPipe landmarks: generated using Google's MediaPipe (Apache 2.0)--- 许可证：Apache 2.0 任务类别： - 视频分类 标签： - 唇同步 - 说话人头 - 音视频 - HDTF - X-Dub 展示名称：唇同步训练数据集（HDTF教师对+关键点） 样本规模类别：1K < n < 10K --- # 唇同步训练数据集 本数据集为音频驱动唇同步模型训练的预处理数据集，源自HDTF（高清说话人头，High-Definition Talking Face）数据集。 ## 数据集内容 ### 1. `xdub_teacher_pairs/`（1.3 GB） **1449组同说话人唇同步视频对**，由[X-Dub](https://github.com/KlingAIResearch/X-Dub)（Wan2.2-TI2V-5B公开版本）生成。 每组数据包含： - **源视频**：说话人S说出语句A的HDTF片段 - **教师输出**：X-Dub针对同一片段生成的唇同步版本，其音频为同一说话人S的另一语句B的音频 此类数据为用于蒸馏训练的伪样本对：教师输出保留了源视频的姿态与身份特征，但唇形匹配替代音频，且替代音频已混流至教师输出的MP4文件中。 **生成成本**：在4块H100显卡上耗时约25 GPU小时（X-Dub推理单片段约需4分钟，采用30步DDIM采样）。 文件结构： xdub_teacher_pairs/ ├── videos/ │ └── {src_stem}__x__{audio_stem}.mp4 (512×512, 25fps, 混流替代音频) └── meta/ └── {src_stem}__x__{audio_stem}.json (包含帧数、对齐信息) ### 2. `xdub_teacher_pairs_manifest.json`（964 KB） 包含全部1449组教师对的元数据验证清单： - `source`：原始HDTF片段的路径 - `audio`：替代音频源片段的路径 - `teacher`：X-Dub教师输出文件的路径 - `n_aligned`：取源视频帧数与教师输出帧数的较小值，为训练提供安全的帧数量 - `src_speaker` / `alt_speaker`：说话人ID（始终为同一说话人） - `duration_s`：片段时长，单位为秒 所有数据对**仅包含同说话人组合**，符合X-Dub论文的建议（3.1节）：*"我们从与真实视频V_real同一说话人中采样替代音频a_alt，避免因未知数据或跨身份音视频组合引发的训练不稳定。"* ### 3. `hdtf_landmarks/`（2.4 GB） **所有6965个HDTF片段的逐帧MediaPipe FaceLandmarker（面部关键点检测器）关键点**。 每个`.npz`文件包含： - `landmarks`：`(n_frames, 478, 2)`格式的float16数组，存储468个面部网格关键点与10个虹膜关键点的归一化[0,1] xy坐标 - `valid`：`(n_frames,)`格式的uint8数组，若对应帧的关键点检测成功则取值为1 本关键点数据使用MediaPipe FaceLandmarker（float16 v1版本模型）对`data/hdtf/filtered/{WDA,WRA,RD}/`路径下的所有片段进行逐帧计算生成。 **适用场景**：面部/唇部掩码生成、面部区域提取、头部姿态估计，以及所有基于HDTF的说话人头相关研究。 ## 源数据集 - **HDTF**：包含6965个片段，341位说话人，总时长约19小时，所有片段均为512×512分辨率、25fps帧率的面部裁剪正面棚拍录像。 - **X-Dub教师模型**：[KlingAIResearch/X-Dub](https://github.com/KlingAIResearch/X-Dub)（Apache 2.0许可证），Wan2.2-TI2V-5B公开版本。 ## 数据集使用场景 本数据集为某项目的配套数据，该项目通过基于X-Dub的教师蒸馏技术训练专有唇同步模型。教师对提供了（源视频，替代音频）→（唇同步输出）的训练三元组，使模型能够在更小、更快的架构上复刻X-Dub的唇同步能力。 本数据集的关键点数据用于在训练中生成面部/唇部区域掩码以进行损失加权（详见X-Dub论文附录D：`L_wFM = (1 + w·M_face + w_lip·M_lip) ⊙ L_FM`）。 ## 许可证说明 - 本预处理数据集：采用Apache 2.0许可证 - HDTF源视频：需遵循HDTF原始许可证（仅可用于科研用途） - X-Dub教师输出：基于X-Dub的Apache 2.0开源代码与公开模型权重生成 - MediaPipe关键点数据：基于Google的MediaPipe（Apache 2.0许可证）生成