tgokhale/sr2d_visor

---
license: cc-by-nc-nd-4.0
viewer: false
---
# Benchmarking Spatial Relationships in Text-to-Image Generation
*Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vibhav Vineet, Eric Horvitz, Ece Kamar, Chitta Baral, Yezhou Yang*

- We introduce a large-scale challenge dataset SR_2D that contains sentences describing two objects and the spatial relationship between them.
- We introduce a metric called VISOR (short for **V**erify**I**ng **S**patial **O**bject **R**elationships) to quantify spatial reasoning performance.
- VISOR and SR_2D can be used off-the-shelf with any text-to-image model.

## SR_2D Dataset
Our dataset is hosted as [here](https://huggingface.co/datasets/tgokhale/sr2d_visor). This contains
1. The text prompt dataset in `.json` format (`text_spatial_rel_phrases.json`)
2. Images generated using 7 models (GLIDE, CogView2, DALLE-mini, Stable Diffusion, GLIDE + Stable Diffusion + CDM, and Stable Diffusion v2.1)

Alternatively, the text prompt dataset can also accessed from [`text_spatial_rel_phrases.json`](https://github.com/microsoft/VISOR/blob/main/text_spatial_rel_phrases.json). It contains all examples from the current version of the dataset (31680 text prompts) accompanied by the corresponding metadata.
This dataset can also be generated by running the script `python create_spatial_phrases.py`

## GitHub repository
The GitHub repository for [VISOR](https://github.com/microsoft/VISOR/) contains code for generating images with prompts from the SR_2D dataset and evaluating the generated images using VISOR.

## References
Code for text-to-image generation:
1. GLIDE: https://github.com/openai/glide-text2im
2. DALLE-mini: https://github.com/borisdayma/dalle-mini
3. CogView2: https://github.com/THUDM/CogView2
4. Stable Diffusion: https://github.com/CompVis/stable-diffusion
5. Composable Diffusion Models: https://github.com/energy-based-model/Compositional-Visual-Generation-with-Composable-Diffusion-Models-PyTorch
6. OpenAI API for DALLE-2: https://openai.com/api/

## Citation
If you find SR_2D or VISOR useful in your research, please use the following citation:
```
@article{gokhale2022benchmarking,
title={Benchmarking Spatial Relationships in Text-to-Image Generation},
author={Gokhale, Tejas and Palangi, Hamid and Nushi, Besmira and Vineet, Vibhav and Horvitz, Eric and Kamar, Ece and Baral, Chitta and Yang, Yezhou},
journal={arXiv preprint arXiv:2212.10015},
year={2022}
}
```

---
许可证：CC BY-NC-ND 4.0
查看器：禁用
---
# 文本生成图像中的空间关系基准测试
*Tejas Gokhale、Hamid Palangi、Besmira Nushi、Vibhav Vineet、Eric Horvitz、Ece Kamar、Chitta Baral、Yezhou Yang*

- 我们构建了大规模挑战数据集SR₂D，该数据集包含描述两个物体及其间空间关系的语句。
- 我们提出了一种名为VISOR（全称为**V**erify**I**ng **S**patial **O**bject **R**elationships）的指标，用于量化空间推理性能。
- VISOR与SR₂D均可开箱即用，适配任意文本生成图像模型。

## SR₂D 数据集
本数据集托管于 [此处](https://huggingface.co/datasets/tgokhale/sr2d_visor)，包含以下内容：
1. 以`.json`格式存储的文本提示数据集（`text_spatial_rel_phrases.json`）
2. 由7种模型生成的图像，分别为GLIDE、CogView2、DALLE-mini、Stable Diffusion、GLIDE + Stable Diffusion + CDM以及Stable Diffusion v2.1。

此外，文本提示数据集也可通过 [`text_spatial_rel_phrases.json`](https://github.com/microsoft/VISOR/blob/main/text_spatial_rel_phrases.json) 获取，该文件包含当前版本数据集的全部样本（共31680条文本提示）及其对应的元数据。用户也可通过运行脚本`python create_spatial_phrases.py`生成本数据集。

## GitHub 仓库
VISOR的GitHub仓库 [VISOR](https://github.com/microsoft/VISOR/) 包含基于SR₂D数据集的提示生成图像，以及使用VISOR对生成图像进行评估的代码。

## 参考文献
文本生成图像相关代码：
1. GLIDE：https://github.com/openai/glide-text2im
2. DALLE-mini：https://github.com/borisdayma/dalle-mini
3. CogView2：https://github.com/THUDM/CogView2
4. Stable Diffusion：https://github.com/CompVis/stable-diffusion
5. 组合式扩散模型（Composable Diffusion Models）：https://github.com/energy-based-model/Compositional-Visual-Generation-with-Composable-Diffusion-Models-PyTorch
6. DALLE-2的OpenAI API：https://openai.com/api/

## 引用
若您在研究中使用SR₂D或VISOR，请采用以下引用格式：

@article{gokhale2022benchmarking,
title={Benchmarking Spatial Relationships in Text-to-Image Generation},
author={Gokhale, Tejas and Palangi, Hamid and Nushi, Besmira and Vineet, Vibhav and Horvitz, Eric and Kamar, Ece and Baral, Chitta and Yang, Yezhou},
journal={arXiv preprint arXiv:2212.10015},
year={2022}
}

原始信息汇总

SR_2D 数据集概述

数据集介绍

• 名称：SR_2D
• 描述：包含描述两个对象及其空间关系的句子的挑战数据集。
• 用途：可与任何文本到图像生成模型配合使用。

数据集内容

• 文本提示数据：以 .json 格式提供，文件名为 text_spatial_rel_phrases.json，包含 31680 个文本提示及其相应元数据。
• 生成图像：使用 7 种模型（GLIDE, CogView2, DALLE-mini, Stable Diffusion, GLIDE + Stable Diffusion + CDM, 和 Stable Diffusion v2.1）生成的图像。

数据集访问

• 链接：SR_2D 数据集
• 文本提示数据：也可通过 text_spatial_rel_phrases.json 访问。
• 生成脚本：可通过运行脚本 python create_spatial_phrases.py 生成数据集。

相关工具

• VISOR：用于量化空间推理性能的指标，可与 SR_2D 数据集配合使用。
• GitHub 仓库：VISOR 包含生成图像和评估生成图像的代码。

引用

@article{gokhale2022benchmarking, title={Benchmarking Spatial Relationships in Text-to-Image Generation}, author={Gokhale, Tejas and Palangi, Hamid and Nushi, Besmira and Vineet, Vibhav and Horvitz, Eric and Kamar, Ece and Baral, Chitta and Yang, Yezhou}, journal={arXiv preprint arXiv:2212.10015}, year={2022} }

搜集汇总

背景与挑战

背景概述

SR2D是一个用于评估文本到图像生成模型中空间关系理解能力的数据集，包含31680个文本提示和由7种模型生成的图像，并引入了VISOR指标进行性能评估。