VACE-Benchmark

<p align="center">

<h1 align="center">VACE: All-in-One Video Creation and Editing</h1>
<p align="center">
<strong>Zeyinzi Jiang^*</strong>
·
<strong>Zhen Han^*</strong>
·
<strong>Chaojie Mao^*&dagger;</strong>
·
<strong>Jingfeng Zhang</strong>
·
<strong>Yulin Pan</strong>
·
<strong>Yu Liu</strong>
<br>
<b>Tongyi Lab - <a href="https://github.com/Wan-Video/Wan2.1"><img src='https://ali-vilab.github.io/VACE-Page/assets/logos/wan_logo.png' alt='wan_logo' style='margin-bottom: -4px; height: 20px;'></a> </b>
<br>
<br>
<a href="https://arxiv.org/abs/2503.07598"><img src='https://img.shields.io/badge/VACE-arXiv-red' alt='Paper PDF'></a>
<a href="https://ali-vilab.github.io/VACE-Page/"><img src='https://img.shields.io/badge/VACE-Project_Page-green' alt='Project Page'></a>
<a href="https://huggingface.co/collections/ali-vilab/vace-67eca186ff3e3564726aff38"><img src='https://img.shields.io/badge/VACE-HuggingFace_Model-yellow'></a>
<a href="https://modelscope.cn/collections/VACE-8fa5fcfd386e43"><img src='https://img.shields.io/badge/VACE-ModelScope_Model-purple'></a>
<br>
</p>

## Introduction
<strong>VACE</strong> is an all-in-one model designed for video creation and editing. It encompasses various tasks, including reference-to-video generation (<strong>R2V</strong>), video-to-video editing (<strong>V2V</strong>), and masked video-to-video editing (<strong>MV2V</strong>), allowing users to compose these tasks freely. This functionality enables users to explore diverse possibilities and streamlines their workflows effectively, offering a range of capabilities, such as Move-Anything, Swap-Anything, Reference-Anything, Expand-Anything, Animate-Anything, and more.

<img src='https://raw.githubusercontent.com/ali-vilab/VACE/refs/heads/main/assets/materials/teaser.jpg'>

## 🎉 News
- [x] Mar 31, 2025: 🔥VACE-Wan2.1-1.3B-Preview and VACE-LTX-Video-0.9 models are now available at [HuggingFace](https://huggingface.co/collections/ali-vilab/vace-67eca186ff3e3564726aff38) and [ModelScope](https://modelscope.cn/collections/VACE-8fa5fcfd386e43)!
- [x] Mar 31, 2025: 🔥Release code of model inference, preprocessing, and gradio demos.
- [x] Mar 11, 2025: We propose [VACE](https://ali-vilab.github.io/VACE-Page/), an all-in-one model for video creation and editing.

## 🪄 Models
| Models | Download Link | Video Size | License |
|--------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------|-----------------------------------------------------------------------------------------------|
| VACE-Wan2.1-1.3B-Preview | [Huggingface](https://huggingface.co/ali-vilab/VACE-Wan2.1-1.3B-Preview) 🤗 [ModelScope](https://modelscope.cn/models/iic/VACE-Wan2.1-1.3B-Preview) 🤖 | ~ 81 x 480 x 832 | [Apache-2.0](https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B/blob/main/LICENSE.txt) |
| VACE-Wan2.1-1.3B | [To be released](https://github.com/Wan-Video) <img src='https://ali-vilab.github.io/VACE-Page/assets/logos/wan_logo.png' alt='wan_logo' style='margin-bottom: -4px; height: 15px;'> | ~ 81 x 480 x 832 | [Apache-2.0](https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B/blob/main/LICENSE.txt) |
| VACE-Wan2.1-14B | [To be released](https://github.com/Wan-Video) <img src='https://ali-vilab.github.io/VACE-Page/assets/logos/wan_logo.png' alt='wan_logo' style='margin-bottom: -4px; height: 15px;'> | ~ 81 x 720 x 1080 | [Apache-2.0](https://huggingface.co/Wan-AI/Wan2.1-T2V-14B/blob/main/LICENSE.txt) |
| VACE-LTX-Video-0.9 | [Huggingface](https://huggingface.co/ali-vilab/VACE-LTX-Video-0.9) 🤗 [ModelScope](https://modelscope.cn/models/iic/VACE-LTX-Video-0.9) 🤖 | ~ 97 x 512 x 768 | [RAIL-M](https://huggingface.co/Lightricks/LTX-Video/blob/main/ltx-video-2b-v0.9.license.txt) |

- The input supports any resolution, but to achieve optimal results, the video size should fall within a specific range.
- All models inherit the license of the original model.

## ⚙️ Installation
The codebase was tested with Python 3.10.13, CUDA version 12.4, and PyTorch >= 2.5.1.

### Setup for Model Inference
You can setup for VACE model inference by running:
```bash
git clone https://github.com/ali-vilab/VACE.git && cd VACE
pip install torch==2.5.1 torchvision==0.20.1 --index-url https://download.pytorch.org/whl/cu124 # If PyTorch is not installed.
pip install -r requirements.txt
pip install wan@git+https://github.com/Wan-Video/Wan2.1 # If you want to use Wan2.1-based VACE.
pip install ltx-video@git+https://github.com/Lightricks/LTX-Video@ltx-video-0.9.1 sentencepiece --no-deps # If you want to use LTX-Video-0.9-based VACE. It may conflict with Wan.
```
Please download your preferred base model to `<repo-root>/models/`.

### Setup for Preprocess Tools
If you need preprocessing tools, please install:
```bash
pip install -r requirements/annotator.txt
```
Please download [VACE-Annotators](https://huggingface.co/ali-vilab/VACE-Annotators) to `<repo-root>/models/`.

### Local Directories Setup
It is recommended to download [VACE-Benchmark](https://huggingface.co/datasets/ali-vilab/VACE-Benchmark) to `<repo-root>/benchmarks/` as examples in `run_vace_xxx.sh`.

We recommend to organize local directories as:
```angular2html
VACE
├── ...
├── benchmarks
│ └── VACE-Benchmark
│ └── assets
│ └── examples
│ ├── animate_anything
│ │ └── ...
│ └── ...
├── models
│ ├── VACE-Annotators
│ │ └── ...
│ ├── VACE-LTX-Video-0.9
│ │ └── ...
│ └── VACE-Wan2.1-1.3B-Preview
│ └── ...
└── ...
```

## 🚀 Usage
In VACE, users can input **text prompt** and optional **video**, **mask**, and **image** for video generation or editing.
Detailed instructions for using VACE can be found in the [User Guide](https://github.com/ali-vilab/VACE/blob/main/UserGuide.md).

### Inference CIL
#### 1) End-to-End Running
To simply run VACE without diving into any implementation details, we suggest an end-to-end pipeline. For example:
```bash
# run V2V depth
python vace/vace_pipeline.py --base wan --task depth --video assets/videos/test.mp4 --prompt 'xxx'

# run MV2V inpainting by providing bbox
python vace/vace_pipeline.py --base wan --task inpainting --mode bbox --bbox 50,50,550,700 --video assets/videos/test.mp4 --prompt 'xxx'
```
This script will run video preprocessing and model inference sequentially,
and you need to specify all the required args of preprocessing (`--task`, `--mode`, `--bbox`, `--video`, etc.) and inference (`--prompt`, etc.).
The output video together with intermediate video, mask and images will be saved into `./results/` by default.

> 💡**Note**:
> Please refer to [run_vace_pipeline.sh](https://github.com/ali-vilab/VACE/blob/main/run_vace_pipeline.sh) for usage examples of different task pipelines.

#### 2) Preprocessing
To have more flexible control over the input, before VACE model inference, user inputs need to be preprocessed into `src_video`, `src_mask`, and `src_ref_images` first.
We assign each [preprocessor](https://raw.githubusercontent.com/ali-vilab/VACE/refs/heads/main/vace/configs/__init__.py) a task name, so simply call [`vace_preprocess.py`](https://raw.githubusercontent.com/ali-vilab/VACE/refs/heads/main/vace/vace_preproccess.py) and specify the task name and task params. For example:
```angular2html
# process video depth
python vace/vace_preproccess.py --task depth --video assets/videos/test.mp4

# process video inpainting by providing bbox
python vace/vace_preproccess.py --task inpainting --mode bbox --bbox 50,50,550,700 --video assets/videos/test.mp4
```
The outputs will be saved to `./proccessed/` by default.

> 💡**Note**:
> Please refer to [run_vace_pipeline.sh](https://github.com/ali-vilab/VACE/blob/main/run_vace_pipeline.sh) preprocessing methods for different tasks.
Moreover, refer to [vace/configs/](https://github.com/ali-vilab/VACE/blob/main/vace/configs/) for all the pre-defined tasks and required params.
You can also customize preprocessors by implementing at [`annotators`](https://github.com/ali-vilab/VACE/blob/main/vace/annotators/__init__.py) and register them at [`configs`](https://github.com/ali-vilab/VACE/blob/main/vace/configs).

#### 3) Model inference
Using the input data obtained from **Preprocessing**, the model inference process can be performed as follows:
```bash
# For Wan2.1 single GPU inference
python vace/vace_wan_inference.py --ckpt_dir <path-to-model> --src_video <path-to-src-video> --src_mask <path-to-src-mask> --src_ref_images <paths-to-src-ref-images> --prompt "xxx"

# For Wan2.1 Multi GPU Acceleration inference
pip install "xfuser>=0.4.1"
torchrun --nproc_per_node=8 vace/vace_wan_inference.py --dit_fsdp --t5_fsdp --ulysses_size 1 --ring_size 8 --ckpt_dir <path-to-model> --src_video <path-to-src-video> --src_mask <path-to-src-mask> --src_ref_images <paths-to-src-ref-images> --prompt "xxx"

# For LTX inference, run
python vace/vace_ltx_inference.py --ckpt_path <path-to-model> --text_encoder_path <path-to-model> --src_video <path-to-src-video> --src_mask <path-to-src-mask> --src_ref_images <paths-to-src-ref-images> --prompt "xxx"
```
The output video together with intermediate video, mask and images will be saved into `./results/` by default.

> 💡**Note**:
> (1) Please refer to [vace/vace_wan_inference.py](https://github.com/ali-vilab/VACE/blob/main/vace/vace_wan_inference.py) and [vace/vace_ltx_inference.py](https://github.com/ali-vilab/VACE/blob/main/vace/vace_ltx_inference.py) for the inference args.
> (2) For LTX-Video and English language Wan2.1 users, you need prompt extension to unlock the full model performance.
Please follow the [instruction of Wan2.1](https://github.com/Wan-Video/Wan2.1?tab=readme-ov-file#2-using-prompt-extension) and set `--use_prompt_extend` while running inference.

### Inference Gradio
For preprocessors, run
```bash
python vace/gradios/preprocess_demo.py
```
For model inference, run
```bash
# For Wan2.1 gradio inference
python vace/gradios/vace_wan_demo.py

# For LTX gradio inference
python vace/gradios/vace_ltx_demo.py
```

## Acknowledgement

We are grateful for the following awesome projects, including [Scepter](https://github.com/modelscope/scepter), [Wan](https://github.com/Wan-Video/Wan2.1), and [LTX-Video](https://github.com/Lightricks/LTX-Video).

## BibTeX

```bibtex
@article{vace,
title = {VACE: All-in-One Video Creation and Editing},
author = {Jiang, Zeyinzi and Han, Zhen and Mao, Chaojie and Zhang, Jingfeng and Pan, Yulin and Liu, Yu},
journal = {arXiv preprint arXiv:2503.07598},
year = {2025}
}

<p align="center">
<h1 align="center">VACE：一体化视频创作与编辑</h1>
<p align="center">
<strong>姜泽印<sup>*</sup></strong>
·
<strong>韩震<sup>*</sup></strong>
·
<strong>毛超杰<sup>*&dagger;</sup></strong>
·
<strong>张景峰</strong>
·
<strong>潘玉林</strong>
·
<strong>刘钰</strong>
<br>
<b>通义实验室 - <a href="https://github.com/Wan-Video/Wan2.1"><img src='https://ali-vilab.github.io/VACE-Page/assets/logos/wan_logo.png' alt='wan_logo' style='margin-bottom: -4px; height: 20px;'></a> </b>
<br>
<br>
<a href="https://arxiv.org/abs/2503.07598"><img src='https://img.shields.io/badge/VACE-arXiv-red' alt="论文PDF"></a>
<a href="https://ali-vilab.github.io/VACE-Page/"><img src='https://img.shields.io/badge/VACE-Project_Page-green' alt="项目主页"></a>
<a href="https://huggingface.co/collections/ali-vilab/vace-67eca186ff3e3564726aff38"><img src='https://img.shields.io/badge/VACE-HuggingFace_Model-yellow' alt="HuggingFace 模型"></a>
<a href="https://modelscope.cn/collections/VACE-8fa5fcfd386e43"><img src='https://img.shields.io/badge/VACE-ModelScope_Model-purple' alt="ModelScope 模型"></a>
<br>
</p>

## 简介
<strong>VACE</strong> 是一款专为视频创作与编辑打造的一体化模型，涵盖参考视频生成（reference-to-video generation，<strong>R2V</strong>）、视频到视频编辑（video-to-video editing，<strong>V2V</strong>）以及掩码式视频到视频编辑（masked video-to-video editing，<strong>MV2V</strong>）等多种任务，支持用户自由组合各类任务。该功能不仅助力用户探索多样化创作可能，还能有效简化工作流程，提供包括任意移动（Move-Anything）、任意替换（Swap-Anything）、任意参考（Reference-Anything）、任意拓展（Expand-Anything）、任意动画化（Animate-Anything）等在内的丰富能力。

<img src='https://raw.githubusercontent.com/ali-vilab/VACE/refs/heads/main/assets/materials/teaser.jpg'>

## 🎉 最新动态
- [x] 2025年3月31日：🔥VACE-Wan2.1-1.3B-Preview与VACE-LTX-Video-0.9模型现已在[HuggingFace](https://huggingface.co/collections/ali-vilab/vace-67eca186ff3e3564726aff38)和[ModelScope](https://modelscope.cn/collections/VACE-8fa5fcfd386e43)平台上线！
- [x] 2025年3月31日：🔥发布模型推理、预处理及Gradio演示相关代码。
- [x] 2025年3月11日：我们推出了[VACE](https://ali-vilab.github.io/VACE-Page/)，一款一体化视频创作与编辑模型。

## 🪄 模型列表
| 模型名称 | 下载链接 | 视频分辨率 | 许可证 |
|--------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------|-----------------------------------------------------------------------------------------------|
| VACE-Wan2.1-1.3B-Preview | [Huggingface](https://huggingface.co/ali-vilab/VACE-Wan2.1-1.3B-Preview) 🤗 [ModelScope](https://modelscope.cn/models/iic/VACE-Wan2.1-1.3B-Preview) 🤖 | ~ 81 x 480 x 832 | [Apache-2.0](https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B/blob/main/LICENSE.txt) |
| VACE-Wan2.1-1.3B | [即将发布](https://github.com/Wan-Video) <img src='https://ali-vilab.github.io/VACE-Page/assets/logos/wan_logo.png' alt='wan_logo' style='margin-bottom: -4px; height: 15px;'> | ~ 81 x 480 x 832 | [Apache-2.0](https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B/blob/main/LICENSE.txt) |
| VACE-Wan2.1-14B | [即将发布](https://github.com/Wan-Video) <img src='https://ali-vilab.github.io/VACE-Page/assets/logos/wan_logo.png' alt='wan_logo' style='margin-bottom: -4px; height: 15px;'> | ~ 81 x 720 x 1080 | [Apache-2.0](https://huggingface.co/Wan-AI/Wan2.1-T2V-14B/blob/main/LICENSE.txt) |
| VACE-LTX-Video-0.9 | [Huggingface](https://huggingface.co/ali-vilab/VACE-LTX-Video-0.9) 🤗 [ModelScope](https://modelscope.cn/models/iic/VACE-LTX-Video-0.9) 🤖 | ~ 97 x 512 x 768 | [RAIL-M](https://huggingface.co/Lightricks/LTX-Video/blob/main/ltx-video-2b-v0.9.license.txt) |

- 输入支持任意分辨率，但为获得最佳效果，视频尺寸应处于特定范围内。
- 所有模型均继承自原始模型的许可证。

## ⚙️ 安装指南
本代码库在Python 3.10.13、CUDA 12.4及PyTorch >=2.5.1环境下完成测试。

### 模型推理环境配置
你可以通过以下命令配置VACE模型推理环境：
bash
git clone https://github.com/ali-vilab/VACE.git && cd VACE
pip install torch==2.5.1 torchvision==0.20.1 --index-url https://download.pytorch.org/whl/cu124 # 若尚未安装PyTorch，请执行此命令。
pip install -r requirements.txt
pip install wan@git+https://github.com/Wan-Video/Wan2.1 # 若需使用基于Wan2.1的VACE，请执行此命令。
pip install ltx-video@git+https://github.com/Lightricks/LTX-Video@ltx-video-0.9.1 sentencepiece --no-deps # 若需使用基于LTX-Video-0.9的VACE，请执行此命令，该依赖可能与Wan的依赖冲突。

请将你选择的基础模型下载至`<repo-root>/models/`目录下。

### 预处理工具配置
若需使用预处理工具，请执行以下命令安装依赖：
bash
pip install -r requirements/annotator.txt

请将[VACE-Annotators](https://huggingface.co/ali-vilab/VACE-Annotators)下载至`<repo-root>/models/`目录下。

### 本地目录配置
推荐将[VACE-Benchmark](https://huggingface.co/datasets/ali-vilab/VACE-Benchmark)下载至`<repo-root>/benchmarks/`目录下，作为`run_vace_xxx.sh`中的示例使用。
我们建议的本地目录结构如下：
angular2html
VACE
├── ...
├── benchmarks
│ └── VACE-Benchmark
│ └── assets
│ └── examples
│ ├── animate_anything
│ │ └── ...
│ └── ...
├── models
│ ├── VACE-Annotators
│ │ └── ...
│ ├── VACE-LTX-Video-0.9
│ │ └── ...
│ └── VACE-Wan2.1-1.3B-Preview
│ └── ...
└── ...

## 🚀 使用方法
在VACE中，用户可输入**文本提示词（text prompt）**以及可选的**视频**、**掩码**和**图像**来完成视频生成或编辑。VACE的详细使用说明请参见[用户指南](https://github.com/ali-vilab/VACE/blob/main/UserGuide.md)。

### 命令行推理
#### 1) 端到端运行
若希望无需深入了解实现细节即可快速运行VACE，我们推荐使用端到端流水线。示例如下：
bash
# 运行V2V深度任务
python vace/vace_pipeline.py --base wan --task depth --video assets/videos/test.mp4 --prompt 'xxx'

# 通过提供边界框运行MV2V修复任务
python vace/vace_pipeline.py --base wan --task inpainting --mode bbox --bbox 50,50,550,700 --video assets/videos/test.mp4 --prompt 'xxx'

该脚本将依次执行视频预处理与模型推理，你需要指定预处理所需的全部参数（如`--task`、`--mode`、`--bbox`、`--video`等）以及推理所需参数（如`--prompt`等）。默认情况下，输出视频及中间视频、掩码、图像将保存至`./results/`目录。

> 💡**注意**：
> 不同任务流水线的使用示例请参见[run_vace_pipeline.sh](https://github.com/ali-vilab/VACE/blob/main/run_vace_pipeline.sh)。

#### 2) 预处理
若希望对输入拥有更灵活的控制权，在执行VACE模型推理前，需先将用户输入预处理为`src_video`、`src_mask`和`src_ref_images`。
我们为每个[预处理程序](https://raw.githubusercontent.com/ali-vilab/VACE/refs/heads/main/vace/configs/__init__.py)分配了任务名称，你只需调用[`vace_preprocess.py`](https://raw.githubusercontent.com/ali-vilab/VACE/refs/heads/main/vace/vace_preproccess.py)并指定任务名称及任务参数即可。示例如下：
angular2html
# 处理视频深度信息
python vace/vace_preproccess.py --task depth --video assets/videos/test.mp4

# 通过提供边界框处理视频修复任务
python vace/vace_preproccess.py --task inpainting --mode bbox --bbox 50,50,550,700 --video assets/videos/test.mp4

默认情况下，预处理结果将保存至`./proccessed/`目录。

> 💡**注意**：
> 不同任务的预处理方法请参见[run_vace_pipeline.sh](https://github.com/ali-vilab/VACE/blob/main/run_vace_pipeline.sh)。
此外，所有预定义任务及所需参数请参见[vace/configs/](https://github.com/ali-vilab/VACE/blob/main/vace/configs/)。你也可以在[`annotators`](https://github.com/ali-vilab/VACE/blob/main/vace/annotators/__init__.py)中实现自定义预处理程序，并在[`configs`](https://github.com/ali-vilab/VACE/blob/main/vace/configs)中完成注册。

#### 3) 模型推理
使用**预处理**得到的输入数据，可按以下方式执行模型推理：
bash
# Wan2.1单GPU推理
python vace/vace_wan_inference.py --ckpt_dir <path-to-model> --src_video <path-to-src-video> --src_mask <path-to-src-mask> --src_ref_images <paths-to-src-ref-images> --prompt "xxx"

# Wan2.1多GPU加速推理
pip install "xfuser>=0.4.1"
torchrun --nproc_per_node=8 vace/vace_wan_inference.py --dit_fsdp --t5_fsdp --ulysses_size 1 --ring_size 8 --ckpt_dir <path-to-model> --src_video <path-to-src-video> --src_mask <path-to-src-mask> --src_ref_images <paths-to-src-ref-images> --prompt "xxx"

# LTX推理请执行以下命令
python vace/vace_ltx_inference.py --ckpt_path <path-to-model> --text_encoder_path <path-to-model> --src_video <path-to-src-video> --src_mask <path-to-src-mask> --src_ref_images <paths-to-src-ref-images> --prompt "xxx"

默认情况下，输出视频及中间视频、掩码、图像将保存至`./results/`目录。

> 💡**注意**：
> (1) 推理参数详情请参见[vace/vace_wan_inference.py](https://github.com/ali-vilab/VACE/blob/main/vace/vace_wan_inference.py)和[vace/vace_ltx_inference.py](https://github.com/ali-vilab/VACE/blob/main/vace/vace_ltx_inference.py)。
> (2) 对于LTX-Video及使用英文提示的Wan2.1用户，你需要使用提示词扩展功能以充分发挥模型性能。请遵循[Wan2.1的说明](https://github.com/Wan-Video/Wan2.1?tab=readme-ov-file#2-using-prompt-extension)，并在推理时添加`--use_prompt_extend`参数。

### Gradio推理
若需使用预处理程序的Gradio演示，请执行：
bash
python vace/gradios/preprocess_demo.py

若需使用模型推理的Gradio演示，请执行：
bash
# Wan2.1的Gradio推理演示
python vace/gradios/vace_wan_demo.py

# LTX的Gradio推理演示
python vace/gradios/vace_ltx_demo.py

## 致谢
我们感谢以下优秀开源项目，包括[Scepter](https://github.com/modelscope/scepter)、[Wan](https://github.com/Wan-Video/Wan2.1)以及[LTX-Video](https://github.com/Lightricks/LTX-Video)。

## 引用格式
bibtex
@article{vace,
title = {VACE: All-in-One Video Creation and Editing},
author = {Jiang, Zeyinzi and Han, Zhen and Mao, Chaojie and Zhang, Jingfeng and Pan, Yulin and Liu, Yu},
journal = {arXiv preprint arXiv:2503.07598},
year = {2025}
}

搜集汇总

背景与挑战

背景概述

VACE-Benchmark是VACE模型用于视频创建和编辑的基准测试数据集，支持参考视频生成、视频到视频编辑和掩码视频编辑等多种任务，旨在评估模型性能并提供相关工具。该数据集采用Apache License 2.0许可，包含预处理和推理流程的示例代码。