EvalAnything-Selection_Synergy

# All-Modality Generation (Modality Selection and Modality Synergy Part)

<span style="color: red;">All-Modality Generation benchmark evaluates a model's ability to follow instructions, automatically select appropriate modalities, and create synergistic outputs across different modalities (text, visual, audio) while avoiding redundancy.</span>

[🏠 Homepage](https://github.com/PKU-Alignment/align-anything) | [👍 Our Official Code Repo](https://github.com/PKU-Alignment/align-anything)

[🤗 All-Modality Understanding Benchmark](https://huggingface.co/datasets/PKU-Alignment/EvalAnything-AMU)

[🤗 All-Modality Generation Benchmark (Instruction Following Part)](https://huggingface.co/datasets/PKU-Alignment/EvalAnything-InstructionFollowing)

[🤗 All-Modality Generation Benchmark (Modality Selection and Synergy Part)](https://huggingface.co/datasets/PKU-Alignment/EvalAnything-Selection_Synergy)

[🤗 All-Modality Generation Reward Model](https://huggingface.co/PKU-Alignment/AnyRewardModel)

## Data Example

<div align="center">
<img src="example-amg.png" width="100%"/>
</div>

## Load dataset

The loading method for modality selection and synergy data is

```python
dataset = load_dataset(
'PKU-Alignment/EvalAnything-Selection_Synergy',
trust_remote_code=True
)
```

## Model Evaluation

### Modality Selection

Use [eval_anything/amg/selection/example.py](https://github.com/PKU-Alignment/align-anything/tree/main/align_anything/evaluation/eval_anything/amg/selection/example.py) for modality selection evaluation. Note that you need to implement the code related to generating responses.

### Modality Synergy

Since there isn't currently a true all-modality generation model, you can simulate the all-modality generation process using Agent-related technologies. Reference the Agent code in [eval_anything/amg/agent](https://github.com/PKU-Alignment/align-anything/tree/main/align_anything/evaluation/eval_anything/amg/agent).

For evaluation:
1. Use [eval_anything/amg/synergy/example.py](https://github.com/PKU-Alignment/align-anything/tree/main/align_anything/evaluation/eval_anything/amg/synergy/example.py) to generate relevant instructions.
2. Use [eval_anything/amg/generate.sh](https://github.com/PKU-Alignment/align-anything/tree/main/align_anything/evaluation/eval_anything/amg/generate.sh) to call the agent to simulate the all-modality generation process.
3. Format the generated results as shown in [eval_anything/amg/synergy](https://github.com/PKU-Alignment/align-anything/tree/main/align_anything/evaluation/eval_anything/amg/synergy).
4. Use [eval_anything/amg/synergy/reward_eval.py](https://github.com/PKU-Alignment/align-anything/tree/main/align_anything/evaluation/eval_anything/amg/synergy/reward_eval.py) to evaluate modality synergy.

We've trained a multi-modal input model for Modality Synergy scoring. For model details, refer to [PKU-Alignment/AnyRewardModel](https://huggingface.co/PKU-Alignment/AnyRewardModel).

**Note:** The current code is a sample script for the All-Modality Generation subtask of Eval Anything. In the future, we will integrate Eval Anything's evaluation into the framework to provide convenience for community use.

## Citation
Please cite our work if you use our benchmark or model in your paper.
```bibtex
@inproceedings{ji2024align,
title={Align Anything: Training All-Modality Models to Follow Instructions with Language Feedback},
author={Jiaming Ji and Jiayi Zhou and Hantao Lou and Boyuan Chen and Donghai Hong and Xuyao Wang and Wenqi Chen and Kaile Wang and Rui Pan and Jiahao Li and Mohan Wang and Josef Dai and Tianyi Qiu and Hua Xu and Dong Li and Weipeng Chen and Jun Song and Bo Zheng and Yaodong Yang},
year={2024},
url={https://arxiv.org/abs/2412.15838}
}
```

# 全模态生成（模态选择与模态协同子任务）

<span style="color: red;">全模态生成基准（All-Modality Generation benchmark）用于评估模型遵循指令、自动选择适配模态，并在文本、视觉、音频等多模态间生成协同输出且避免冗余的能力。</span>

[🏠 主页](https://github.com/PKU-Alignment/align-anything) | [👍 官方代码仓库](https://github.com/PKU-Alignment/align-anything)

[🤗 全模态理解基准](https://huggingface.co/datasets/PKU-Alignment/EvalAnything-AMU)

[🤗 全模态生成基准（指令遵循子任务）](https://huggingface.co/datasets/PKU-Alignment/EvalAnything-InstructionFollowing)

[🤗 全模态生成基准（模态选择与协同子任务）](https://huggingface.co/datasets/PKU-Alignment/EvalAnything-Selection_Synergy)

[🤗 全模态生成奖励模型](https://huggingface.co/PKU-Alignment/AnyRewardModel)

## 数据示例

<div align="center">
<img src="example-amg.png" width="100%"/>
</div>

## 数据集加载

模态选择与协同数据的加载方式如下：
python
dataset = load_dataset(
'PKU-Alignment/EvalAnything-Selection_Synergy',
trust_remote_code=True
)

## 模型评估

### 模态选择
可使用 [eval_anything/amg/selection/example.py](https://github.com/PKU-Alignment/align-anything/tree/main/align_anything/evaluation/eval_anything/amg/selection/example.py) 开展模态选择评估。请注意，您需自行实现与响应生成相关的代码逻辑。

### 模态协同
鉴于当前尚未存在真正意义上的全模态生成模型，您可借助AI智能体（AI Agent）相关技术模拟全模态生成流程。可参考 [eval_anything/amg/agent](https://github.com/PKU-Alignment/align-anything/tree/main/align_anything/evaluation/eval_anything/amg/agent) 中的智能体代码。
评估流程如下：
1. 使用 [eval_anything/amg/synergy/example.py](https://github.com/PKU-Alignment/align-anything/tree/main/align_anything/evaluation/eval_anything/amg/synergy/example.py) 生成相关指令；
2. 使用 [eval_anything/amg/generate.sh](https://github.com/PKU-Alignment/align-anything/tree/main/align_anything/evaluation/eval_anything/amg/generate.sh) 调用智能体以模拟全模态生成流程；
3. 将生成结果按照 [eval_anything/amg/synergy](https://github.com/PKU-Alignment/align-anything/tree/main/align_anything/evaluation/eval_anything/amg/synergy) 中所示的格式进行整理；
4. 使用 [eval_anything/amg/synergy/reward_eval.py](https://github.com/PKU-Alignment/align-anything/tree/main/align_anything/evaluation/eval_anything/amg/synergy/reward_eval.py) 开展模态协同效果评估。

我们已针对模态协同评分训练了一款多模态输入模型。如需了解模型细节，请参考 [PKU-Alignment/AnyRewardModel](https://huggingface.co/PKU-Alignment/AnyRewardModel)。

**注意：** 当前代码仅为 Eval Anything 全模态生成子任务的示例脚本。未来我们将把 Eval Anything 的评估流程集成至该框架中，以方便社区用户使用。

## 引用声明
若您在论文中使用本基准或模型，请引用我们的工作：
bibtex
@inproceedings{ji2024align,
title={Align Anything: Training All-Modality Models to Follow Instructions with Language Feedback},
author={Jiaming Ji and Jiayi Zhou and Hantao Lou and Boyuan Chen and Donghai Hong and Xuyao Wang and Wenqi Chen and Kaile Wang and Rui Pan and Jiahao Li and Mohan Wang and Josef Dai and Tianyi Qiu and Hua Xu and Dong Li and Weipeng Chen and Jun Song and Bo Zheng and Yaodong Yang},
year={2024},
url={https://arxiv.org/abs/2412.15838}
}