FudanCVL/gRefCOCO

# gRefCOCO - Dataset for [CVPR2023 Highlight] GRES: Generalized Referring Expression Segmentation [](https://pytorch.org/) [](https://www.python.org/downloads/) [](https://paperswithcode.com/sota/generalized-referring-expression-segmentation?p=gres-generalized-referring-expression-1) **[🏠[Project page]](https://henghuiding.github.io/GRES/)** &emsp; **[📄[GRES Arxiv]](https://arxiv.org/abs/2306.00968)** &emsp; **[📄[GREC Arxiv]](https://arxiv.org/abs/2308.16182)** This repository contains information and tools for the [gRefCOCO](https://henghuiding.github.io/GRES/) dataset, proposed by the **CVPR2023 Highlight** paper: > [GRES: Generalized Referring Expression Segmentation](https://arxiv.org/abs/2306.00968) > Chang Liu, Henghui Ding, Xudong Jiang > CVPR 2023 Highlight, Acceptance Rate 2.5% <div align="center"> <img src="https://github.com/henghuiding/ReLA/blob/main/imgs/fig1.png?raw=true" width="100%" height="100%"/> </div><br/> ## Usage - Like RefCOCO, gRefCOCO also should be used together with images from the `train2014` of [MS COCO](https://cocodataset.org/#download). - An example of dataloader [grefer.py](https://github.com/henghuiding/gRefCOCO/blob/main/grefer.py) is provided. - We will update this repository with full API package and documentation soon. Please follow the usage in the [baseline code](https://github.com/henghuiding/ReLA) for now. ## Task 1 - GREC: Generalized Referring Expression Comprehension - The GREC evaluation metric code is [here](https://github.com/henghuiding/gRefCOCO/blob/main/mdetr/datasets/refexp.py). - We provide code based on [MDETR](https://github.com/ashkamath/mdetr), its training and inference are as follows: ### Training (Finetuning) 1. Process grefcoco to coco format. ``` python scripts/fine-tuning/grefexp_coco_format.py --data_path xxx --out_path mdetr_annotations/ --coco_path xxx ``` 2. Training and download `pretrained_resnet101_checkpoint.pth` from [MDETR](https://github.com/ashkamath/mdetr) ``` python -m torch.distributed.launch --nproc_per_node=2 --use_env main.py --dataset_config configs/grefcoco.json --batch_size 4 --load pretrained_resnet101_checkpoint.pth --ema --text_encoder_lr 1e-5 --lr 5e-5 --output-dir grefcoco ``` ### Inference 1. Obtain `checkpoint.pth` after training or download trained model [ here ☁️ Google Drive](https://drive.google.com/file/d/14OrM3n_Oap7xCT6nxj9QEnkJOUMpBGjB/view?usp=drive_link) 2. For test results, pass --test and --test_type test or testA or testB according to the dataset. ``` python -m torch.distributed.launch --nproc_per_node=2 --use_env main.py --dataset_config configs/grefcoco.json --batch_size 4 --resume grefcoco/checkpoint.pth --ema --eval ``` ## Task 2 - GRES: Generalized Referring Expression Segmentation Please refer to [ReLA](https://github.com/henghuiding/ReLA) for more details. ## Acknowledgement Our project is built upon [refer](https://github.com/lichengunc/refer) and [cocoapi](https://github.com/cocodataset/cocoapi). Many thanks to the authors for their great works! ## BibTeX Please consider to cite GRES/GREC if it helps your research. ```bibtex @inproceedings{GRES, title={{GRES}: Generalized Referring Expression Segmentation}, author={Liu, Chang and Ding, Henghui and Jiang, Xudong}, booktitle={CVPR}, year={2023} } @article{GREC, title={{GREC}: Generalized Referring Expression Comprehension}, author={He, Shuting and Ding, Henghui and Liu, Chang and Jiang, Xudong}, journal={arXiv preprint arXiv:2308.16182}, year={2023} } ``` We also recommend other highly related works: ```bibtex @article{VLT, title={{VLT}: Vision-language transformer and query generation for referring segmentation}, author={Ding, Henghui and Liu, Chang and Wang, Suchen and Jiang, Xudong}, journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, year={2023}, volume={45}, number={6}, publisher={IEEE} } @inproceedings{MeViS, title={{MeViS}: A Large-scale Benchmark for Video Segmentation with Motion Expressions}, author={Ding, Henghui and Liu, Chang and He, Shuting and Jiang, Xudong and Loy, Chen Change}, booktitle={ICCV}, year={2023} } ```# gRefCOCO——对应CVPR 2023高亮论文的数据集：GRES：广义指称表达式分割（Generalized Referring Expression Segmentation） [](https://pytorch.org/) [](https://www.python.org/downloads/) [](https://paperswithcode.com/sota/generalized-referring-expression-segmentation?p=gres-generalized-referring-expression-1) **[🏠[项目页面]](https://henghuiding.github.io/GRES/)** &emsp; **[📄[GRES Arxiv]](https://arxiv.org/abs/2306.00968)** &emsp; **[📄[GREC Arxiv]](https://arxiv.org/abs/2308.16182)** 本仓库包含**录用率仅2.5%的CVPR 2023高亮论文**提出的[gRefCOCO](https://henghuiding.github.io/GRES/)数据集的相关信息与工具： > [GRES：广义指称表达式分割（Generalized Referring Expression Segmentation）](https://arxiv.org/abs/2306.00968) > 刘畅，丁恒辉，江旭东 > CVPR 2023 高亮论文，录用率2.5% <div align="center"> <img src="https://github.com/henghuiding/ReLA/blob/main/imgs/fig1.png?raw=true" width="100%" height="100%"/> </div><br/> ## 使用说明 - 与RefCOCO类似，使用gRefCOCO时需搭配[MS COCO](https://cocodataset.org/#download)的`train2014`数据集图像。 - 我们提供了数据加载器示例[grefer.py](https://github.com/henghuiding/gRefCOCO/blob/main/grefer.py)。 - 我们将尽快在此仓库中更新完整的API包与文档，当前请参考[基线代码](https://github.com/henghuiding/ReLA)中的使用方法。 ## 任务1——GREC：广义指称表达式理解（Generalized Referring Expression Comprehension） - GREC的评估指标代码见[此处](https://github.com/henghuiding/gRefCOCO/blob/main/mdetr/datasets/refexp.py)。 - 我们提供基于[MDETR](https://github.com/ashkamath/mdetr)的实现代码，其训练与推理流程如下： ### 训练（微调） 1. 将gRefCOCO转换为COCO格式。 python scripts/fine-tuning/grefexp_coco_format.py --data_path xxx --out_path mdetr_annotations/ --coco_path xxx 2. 执行训练并从[MDETR](https://github.com/ashkamath/mdetr)下载预训练模型`pretrained_resnet101_checkpoint.pth` python -m torch.distributed.launch --nproc_per_node=2 --use_env main.py --dataset_config configs/grefcoco.json --batch_size 4 --load pretrained_resnet101_checkpoint.pth --ema --text_encoder_lr 1e-5 --lr 5e-5 --output-dir grefcoco ### 推理 1. 获取训练完成后得到的`checkpoint.pth`，或从[此处 ☁️ Google Drive](https://drive.google.com/file/d/14OrM3n_Oap7xCT6nxj9QEnkJOUMpBGjB/view?usp=drive_link)下载预训练模型。 2. 如需生成测试结果，请根据数据集传入`--test`参数，并通过`--test_type`指定`test`/`testA`/`testB`。 python -m torch.distributed.launch --nproc_per_node=2 --use_env main.py --dataset_config configs/grefcoco.json --batch_size 4 --resume grefcoco/checkpoint.pth --ema --eval ## 任务2——GRES：广义指称表达式分割（Generalized Referring Expression Segmentation） 更多细节请参考[ReLA](https://github.com/henghuiding/ReLA)。 ## 致谢 本项目基于[refer](https://github.com/lichengunc/refer)与[cocoapi](https://github.com/cocodataset/cocoapi)开发，在此向上述项目的作者致以诚挚谢意。 ## BibTeX引用 如本项目对您的研究有所帮助，请引用GRES/GREC相关论文： bibtex @inproceedings{GRES, title={{GRES}: Generalized Referring Expression Segmentation}, author={Liu, Chang and Ding, Henghui and Jiang, Xudong}, booktitle={CVPR}, year={2023} } @article{GREC, title={{GREC}: Generalized Referring Expression Comprehension}, author={He, Shuting and Ding, Henghui and Liu, Chang and Jiang, Xudong}, journal={arXiv preprint arXiv:2308.16182}, year={2023} } 我们同时推荐以下高度相关的研究工作： bibtex @article{VLT, title={{VLT}: Vision-language transformer and query generation for referring segmentation}, author={Ding, Henghui and Liu, Chang and Wang, Suchen and Jiang, Xudong}, journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, year={2023}, volume={45}, number={6}, publisher={IEEE} } @inproceedings{MeViS, title={{MeViS}: A Large-scale Benchmark for Video Segmentation with Motion Expressions}, author={Ding, Henghui and Liu, Chang and He, Shuting and Jiang, Xudong and Loy, Chen Change}, booktitle={ICCV}, year={2023} }