AS-V2

# The All-Seeing Project V2

We release the training data utilized for the All-Seeing Project V2 in this repository.

- `llava_v1_5_mix665k_asmv2_format.json`: the instruction tuning data used in Stage 1.
- `as_pretrain_10m.json`: the filtered 10M samples in AS-1B, which are used in the pretraining phase of Stage 2.
- `as_mix_4m.json`: the instruction tuning data used in Stage 2.
- `rec_conversation_22k.json`: the conversation data of AS-V2.
- `rec_detailed_description.json`: the detailed description data of AS-V2.
- `rec_region_captioning.json`: the region description data of AS-V2.

***NOTE***:
- AS-V2 has been intergrated into `as_mix_4m.json`.
- the bounding boxes in `rec_conversation_22k.json`, `rec_detailed_description.json`, and `rec_region_captioning.json` have been preprocessed to fit square pad. See `rec_conversation_22k_wo_square_pad.json`, `rec_detailed_description_wo_square_pad.json`, and `rec_region_captioning_wo_square_pad.json` for data without square pad preprocess.

See our [paper](https://arxiv.org/abs/2402.19474) and [projects](https://github.com/OpenGVLab/all-seeing) for more details!

# Citation

If you find our work useful in your research, please consider cite:

```BibTeX
@article{wang2023allseeing,
title={The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open World},
author={Wang, Weiyun and Shi, Min and Li, Qingyun and Wang, Wenhai and Huang, Zhenhang and Xing, Linjie and Chen, Zhe and Li, Hao and Zhu, Xizhou and Cao, Zhiguo and others},
journal={arXiv preprint arXiv:2308.01907},
year={2023}
}
@article{wang2024allseeing_v2,
title={The All-Seeing Project V2: Towards General Relation Comprehension of the Open World},
author={Wang, Weiyun and Ren, Yiming and Luo, Haowen and Li, Tiantong and Yan, Chenxiang and Chen, Zhe and Wang, Wenhai and Li, Qingyun and Lu, Lewei and Zhu, Xizhou and others},
journal={arXiv preprint arXiv:2402.19474},
year={2024}
}
```

# 全视项目V2（The All-Seeing Project V2）
本仓库发布了全视项目V2所使用的训练数据集。

- `llava_v1_5_mix665k_asmv2_format.json`：第一阶段（Stage 1）所使用的指令微调数据集。
- `as_pretrain_10m.json`：AS-1B中经过筛选的1000万条样本，用于第二阶段（Stage 2）的预训练阶段。
- `as_mix_4m.json`：第二阶段（Stage 2）所使用的指令微调数据集。
- `rec_conversation_22k.json`：AS-V2的对话数据集。
- `rec_detailed_description.json`：AS-V2的详细描述数据集。
- `rec_region_captioning.json`：AS-V2的区域描述数据集。

***注意事项***：
- AS-V2的相关数据集已整合至`as_mix_4m.json`中。
- `rec_conversation_22k.json`、`rec_detailed_description.json`与`rec_region_captioning.json`中的边界框（bounding boxes）已完成预处理，以适配方形填充要求。如需获取未经过方形填充预处理的原始数据集，请参阅`rec_conversation_22k_wo_square_pad.json`、`rec_detailed_description_wo_square_pad.json`及`rec_region_captioning_wo_square_pad.json`。

如需了解更多细节，请参阅我们的[论文](https://arxiv.org/abs/2402.19474)与[项目主页](https://github.com/OpenGVLab/all-seeing)！

# 引用
若您的研究工作中用到了本项目的相关成果，请考虑引用以下文献：
BibTeX
@article{wang2023allseeing,
title={The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open World},
author={Wang, Weiyun and Shi, Min and Li, Qingyun and Wang, Wenhai and Huang, Zhenhang and Xing, Linjie and Chen, Zhe and Li, Hao and Zhu, Xizhou and Cao, Zhiguo and others},
journal={arXiv preprint arXiv:2308.01907},
year={2023}
}
@article{wang2024allseeing_v2,
title={The All-Seeing Project V2: Towards General Relation Comprehension of the Open World},
author={Wang, Weiyun and Ren, Yiming and Luo, Haowen and Li, Tiantong and Yan, Chenxiang and Chen, Zhe and Wang, Wenhai and Li, Qingyun and Lu, Lewei and Zhu, Xizhou and others},
journal={arXiv preprint arXiv:2402.19474},
year={2024}
}

搜集汇总

背景与挑战

背景概述

AS-V2是一个用于视觉识别和理解的开源数据集，包含多阶段训练数据和多种数据类型，适用于开放世界的全景视觉研究。