AS-Core

# AS-Core

AS-Core is the human-verified subset of AS-1B.

- `semantic_tag_1m.json`: the human verified annotations for semantic tags.
- `region_vqa_1m.jsonl`: the human verified annotations for region VQA.
- `region_caption_400k.jsonl`: the region captions generated base on paraphrasing the region question-answer pairs.

***NOTE***: The bbox format is `x1y1x2y2`.

## Introduction
We present the All-Seeing Project with:

[***All-Seeing 1B (AS-1B) dataset***](https://huggingface.co/datasets/Weiyun1025/AS-100M): we propose a new large-scale dataset (AS-1B) for open-world panoptic visual recognition and understanding, using an economical semi-automatic data engine that combines the power of off-the-shelf vision/language models and human feedback.

[***All-Seeing Model (ASM)***](https://huggingface.co/Weiyun1025/All-Seeing-Model-FT): we develop a unified vision-language foundation model (ASM) for open-world panoptic visual recognition and understanding. Aligning with LLMs, our ASM supports versatile image-text retrieval and generation tasks, demonstrating impressive zero-shot capability.

<img width="820" alt="image" src="https://github.com/OpenGVLab/all-seeing/assets/8529570/e43ab8db-6437-46f1-8aa1-c95f012e9147">

Figure 1: Overview and comparison of our All-Seeing project with other popular large foundation models.

<!-- ## Online Demo
**All-Seeing Model demo** is available [here](https://openxlab.org.cn/apps/detail/wangweiyun/All-Seeing-Model-Demo).

**Dataset Browser** is available [here](https://openxlab.org.cn/apps/detail/wangweiyun/All-Seeing-Dataset-Browser).

https://github.com/OpenGVLab/all-seeing/assets/47669167/9b5b32d1-863a-4579-b576-b82523f2205e -->

## Dataset Overview
AS-1B with over 1 billion regions annotated with semantic tags, question-answering pairs, and detailed captions. It covers a wide range of 3.5 million common and rare concepts in the real world, and has 132.2 billion tokens that describe the concepts and their attributes.

<img width="800" alt="image" src="https://github.com/OpenGVLab/all-seeing/assets/8529570/adac37ed-312f-4f11-ba8a-6bc62067438f">

Some examples

<img width="800" alt="image" src="https://github.com/OpenGVLab/all-seeing/assets/8529570/fcf6ab07-c4ba-441c-aa6c-111c769f75b1">

Please see our [paper](https://arxiv.org/abs/2308.01907) to learn more details.

## Model Architecture

The All-Seeing model (ASM) is a unified framework for panoptic visual recognition and understanding, including image/region-text retrieval, image/region recognition, captioning, and question-answering.
<img width="820" alt="image" src="https://github.com/OpenGVLab/all-seeing/assets/8529570/8995e88c-6381-452f-91e4-05d68a2795fc">

## License

This project is released under the [Apache 2.0 license](LICENSE).

# Citation

If you find our work useful in your research, please consider cite:

```BibTeX
@article{wang2023allseeing,
title={The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open World},
author={Wang, Weiyun and Shi, Min and Li, Qingyun and Wang, Wenhai and Huang, Zhenhang and Xing, Linjie and Chen, Zhe and Li, Hao and Zhu, Xizhou and Cao, Zhiguo and others},
journal={arXiv preprint arXiv:2308.01907},
year={2023}
}
@article{wang2024allseeing_v2,
title={The All-Seeing Project V2: Towards General Relation Comprehension of the Open World},
author={Wang, Weiyun and Ren, Yiming and Luo, Haowen and Li, Tiantong and Yan, Chenxiang and Chen, Zhe and Wang, Wenhai and Li, Qingyun and Lu, Lewei and Zhu, Xizhou and others},
journal={arXiv preprint arXiv:2402.19474},
year={2024}
}
```

# AS-Core
AS-Core是经人工验证的AS-1B子集。

- `semantic_tag_1m.json`：用于语义标签的人工验证标注文件。
- `region_vqa_1m.jsonl`：用于区域视觉问答（Visual Question Answering, VQA）的人工验证标注文件。
- `region_caption_400k.jsonl`：基于对区域问答对进行释义生成的区域描述文件。

***注意***：边界框（bounding box, bbox）格式为`x1y1x2y2`。

## 简介
我们推出的全视项目（All-Seeing Project）包含：
[***全视1B数据集（All-Seeing 1B, AS-1B）***](https://huggingface.co/datasets/Weiyun1025/AS-100M)：我们构建了一款面向开放世界全景视觉识别与理解的大规模数据集AS-1B，采用结合了现成视觉语言模型能力与人工反馈的低成本半自动数据引擎进行构建。

[***全视模型（All-Seeing Model, ASM）***](https://huggingface.co/Weiyun1025/All-Seeing-Model-FT)：我们开发了一款用于开放世界全景视觉识别与理解的统一视觉语言基础模型（ASM）。该模型与大语言模型（Large Language Model, LLM）对齐，支持多样化的图文检索与生成任务，展现出优异的零样本（Zero-shot）能力。

<img width="820" alt="示意图" src="https://github.com/OpenGVLab/all-seeing/assets/8529570/e43ab8db-6437-46f1-8aa1-c95f012e9147">

图1：本全视项目与其他主流大型基础模型的概览与对比。

<!-- ## 在线演示
**全视模型演示** 可点击[此处](https://openxlab.org.cn/apps/detail/wangweiyun/All-Seeing-Model-Demo)访问。

**数据集浏览器** 可点击[此处](https://openxlab.org.cn/apps/detail/wangweiyun/All-Seeing-Dataset-Browser)访问。

https://github.com/OpenGVLab/all-seeing/assets/47669167/9b5b32d1-863a-4579-b576-b82523f2205e -->

## 数据集概览
AS-1B包含超过10亿个经过标注的区域，涵盖语义标签、问答对与详细描述。该数据集覆盖现实世界中350万个常见与稀有概念，包含1322亿个用于描述这些概念及其属性的Token。

<img width="800" alt="示意图" src="https://github.com/OpenGVLab/all-seeing/assets/8529570/adac37ed-312f-4f11-ba8a-6bc62067438f">

部分示例

<img width="800" alt="示例图片" src="https://github.com/OpenGVLab/all-seeing/assets/8529570/fcf6ab07-c4ba-441c-aa6c-111c769f75b1">

如需了解更多细节，请参阅我们的[论文](https://arxiv.org/abs/2308.01907)。

## 模型架构
全视模型（ASM）是一款用于全景视觉识别与理解的统一框架，支持图像/区域-文本检索、图像/区域识别、图像描述生成与问答任务。
<img width="820" alt="示意图" src="https://github.com/OpenGVLab/all-seeing/assets/8529570/8995e88c-6381-452f-91e4-05d68a2795fc">

## 许可证
本项目采用[Apache 2.0许可证](LICENSE)进行开源。

# 引用
若您的研究中用到了本项目的工作，请引用以下文献：

BibTeX
@article{wang2023allseeing,
title={The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open World},
author={Wang, Weiyun and Shi, Min and Li, Qingyun and Wang, Wenhai and Huang, Zhenhang and Xing, Linjie and Chen, Zhe and Li, Hao and Zhu, Xizhou and Cao, Zhiguo and others},
journal={arXiv preprint arXiv:2308.01907},
year={2023}
}
@article{wang2024allseeing_v2,
title={The All-Seeing Project V2: Towards General Relation Comprehension of the Open World},
author={Wang, Weiyun and Ren, Yiming and Luo, Haowen and Li, Tiantong and Yan, Chenxiang and Chen, Zhe and Wang, Wenhai and Li, Qingyun and Lu, Lewei and Zhu, Xizhou and others},
journal={arXiv preprint arXiv:2402.19474},
year={2024}
}