hallo3_training_data

<h1 align='center'>Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Diffusion Transformer Networks</h1>

<div align='center'>
<a href='https://github.com/cuijh26' target='_blank'>Jiahao Cui</a>¹&emsp;
<a href='https://github.com/crystallee-ai' target='_blank'>Hui Li</a>¹&emsp;
<a href='https://github.com/subazinga' target='_blank'>Yun Zhan</a>¹&emsp;
<a href='https://github.com/NinoNeumann' target='_blank'>Hanlin Shang</a>¹&emsp;
<a href='https://github.com/Kaihui-Cheng' target='_blank'>Kaihui Cheng</a>¹&emsp;
<a href='https://github.com/mayuqi7777' target='_blank'>Yuqi Ma</a>¹&emsp;
<a href='https://github.com/AricGamma' target='_blank'>Shan Mu</a>¹&emsp;
</div>
<div align='center'>
<a href='https://hangz-nju-cuhk.github.io/' target='_blank'>Hang Zhou</a>²&emsp;
<a href='https://jingdongwang2017.github.io/' target='_blank'>Jingdong Wang</a>²&emsp;
<a href='https://sites.google.com/site/zhusiyucs/home' target='_blank'>Siyu Zhu</a>^1✉️&emsp;
</div>

<div align='center'>
¹Fudan University&emsp; ²Baidu Inc&emsp;
</div>

<br>
<div align='center'>
<a href='https://github.com/fudan-generative-vision/hallo3'><img src='https://img.shields.io/github/stars/fudan-generative-vision/hallo3?style=social'></a>
<a href='https://fudan-generative-vision.github.io/hallo3/#/'><img src='https://img.shields.io/badge/Project-HomePage-Green'></a>
<a href='https://arxiv.org/pdf/2412.00733'><img src='https://img.shields.io/badge/Paper-Arxiv-red'></a>
<a href='https://huggingface.co/fudan-generative-ai/hallo3'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20HuggingFace-Model-yellow'></a>
<a href='assets/wechat.jpeg'><img src='https://badges.aleen42.com/src/wechat.svg'></a>
</div>
<br>

## I. Dataset Overview

This dataset serves as the training data for the open - source Hallo3 model, specifically created for the training of video generation models. It is dedicated to promoting the research and development in the field of video generation, providing rich and high - quality data support for relevant practitioners and researchers.

## II. Data Composition

**Pure Talking - Head Videos**: It contains over 70 hours of pure talking - head videos, precisely focusing on the speaker's facial expressions and speech. This can provide effective data for the model to learn human language expressions and facial dynamics.

**Wild - Scene Video Clips**: There are more than 50 wild - scene video clips, covering a wide variety of real - world scenarios, such as bustling market streets and serene natural landscapes. This helps the model learn visual features in different scenarios.

## III. Dataset Download

You can download this training dataset from [HuggingFace Dataset Repo](https://huggingface.co/fudan-generative-ai/hallo3).

## IV. Usage Instructions

**File Extraction**: After downloading, all data is compressed in the `.tgz` file format. You can easily extract these files to obtain the internal data.

**Data Usage**: This dataset is mainly used for academic research and non - commercial model training. During the use process, please ensure compliance with relevant data usage norms and ethical guidelines.

## V. Dataset License

This dataset adopts the Creative Commons Attribution - NonCommercial - NoDerivatives 4.0 International License (CC BY - NC - ND 4.0). This means that you can use this dataset for non - commercial purposes with the original author and source cited, but you are not allowed to modify the data or create derivative works. Please refer to the [CC BY-NC-ND 4.0 Official Documentation](https://creativecommons.org/licenses/by-nc-nd/4.0/) for specific license terms.

## VI. Citation Guidelines

If you use this dataset in your research or projects, to ensure academic integrity and respect for the dataset contributors, please cite this dataset in the following format:

```
@misc{cui2024hallo3,
title={Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Diffusion Transformer Networks},
author={Jiahao Cui and Hui Li and Yun Zhan and Hanlin Shang and Kaihui Cheng and Yuqi Ma and Shan Mu and Hang Zhou and Jingdong Wang and Siyu Zhu},
year={2024},
eprint={2412.00733},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
```

<h1 align='center'>Hallo3：基于扩散Transformer（Diffusion Transformer）网络的高动态写实肖像图像动画</h1>

<div align='center'>
<a href='https://github.com/cuijh26' target='_blank'>崔佳浩</a>¹&emsp;
<a href='https://github.com/crystallee-ai' target='_blank'>李辉</a>¹&emsp;
<a href='https://github.com/subazinga' target='_blank'>詹云</a>¹&emsp;
<a href='https://github.com/NinoNeumann' target='_blank'>尚翰林</a>¹&emsp;
<a href='https://github.com/Kaihui-Cheng' target='_blank'>程开辉</a>¹&emsp;
<a href='https://github.com/mayuqi7777' target='_blank'>马雨琪</a>¹&emsp;
<a href='https://github.com/AricGamma' target='_blank'>穆山</a>¹&emsp;
</div>
<div align='center'>
<a href='https://hangz-nju-cuhk.github.io/' target='_blank'>周航</a>²&emsp;
<a href='https://jingdongwang2017.github.io/' target='_blank'>王京东</a>²&emsp;
<a href='https://sites.google.com/site/zhusiyucs/home' target='_blank'>朱思宇</a>^1✉️&emsp;
</div>

<div align='center'>
¹复旦大学&emsp; ²Baidu Inc&emsp;
</div>

<br>
<div align='center'>
<a href='https://github.com/fudan-generative-vision/hallo3'><img src='https://img.shields.io/github/stars/fudan-generative-vision/hallo3?style=social'></a>
<a href='https://fudan-generative-vision.github.io/hallo3/#/'><img src='https://img.shields.io/badge/项目-主页-Green'></a>
<a href='https://arxiv.org/pdf/2412.00733'><img src='https://img.shields.io/badge/论文-Arxiv-red'></a>
<a href='https://huggingface.co/fudan-generative-ai/hallo3'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20HuggingFace-模型-yellow'></a>
<a href='assets/wechat.jpeg'><img src='https://badges.aleen42.com/src/wechat.svg'></a>
</div>
<br>

## 一、数据集概览

本数据集作为开源Hallo3模型的训练数据，专为视频生成模型训练打造，致力于推动视频生成领域的研究与发展，为相关从业者与科研人员提供丰富且高质量的数据支撑。

## 二、数据构成

**纯头部讲话视频**：包含超过70小时的纯头部讲话视频，精准聚焦于讲话者的面部表情与语音表达，可有效助力模型学习人类语言表达与面部动态特征。

**野外场景视频片段**：包含50余条野外场景视频片段，涵盖多样化的真实世界场景，如繁华集市街道与静谧自然景观，可帮助模型学习不同场景下的视觉特征。

## 三、数据集下载

您可从[HuggingFace 数据集仓库](https://huggingface.co/fudan-generative-ai/hallo3)下载本训练数据集。

## 四、使用说明

**文件解压**：下载完成后，所有数据均以`.tgz`格式压缩，您可轻松解压以获取内部数据。

**数据用途**：本数据集主要用于学术研究与非商业性模型训练，使用过程中请务必遵守相关数据使用规范与伦理准则。

## 五、数据集许可

本数据集采用知识共享署名-非商业性使用-禁止演绎4.0国际许可协议（CC BY-NC-ND 4.0）。这意味着您可在标注原作者与来源的前提下，将本数据集用于非商业用途，但不得对数据进行修改或创建衍生作品。具体许可条款请参阅[CC BY-NC-ND 4.0 官方文档](https://creativecommons.org/licenses/by-nc-nd/4.0/)。

## 六、引用规范

若您在研究或项目中使用本数据集，为确保学术严谨性并尊重数据集贡献者，请按照以下格式引用本数据集：

@misc{cui2024hallo3,
title={Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Diffusion Transformer Networks},
author={Jiahao Cui and Hui Li and Yun Zhan and Hanlin Shang and Kaihui Cheng and Yuqi Ma and Shan Mu and Hang Zhou and Jingdong Wang and Siyu Zhu},
year={2024},
eprint={2412.00733},
archivePrefix={arXiv},
primaryClass={cs.CV}
}