Created by Zhaoqing Wang*, Yu Lu*, Qiang Li*, Xunqiang Tao, Yandong Guo, Mingming Gong and Tongliang Liu

This is an official PyTorch implementation of the CRIS

CLIP-Driven Referring Image Segmentation (CRIS) framework is proposed to transfer the image-level semantic knowledge of the CLIP model to dense pixel-level referring image segmentation. More specifically, we design a vision-language decoder to propagate fine-grained semantic information from textual representations to each pixel-level activation, which promotes consistency between the two modalities. In addition, we present text-to-pixel contrastive learning to explicitly enforce the text feature similar to the related pixel-level features and dissimilar to the irrelevances.

🍻CRIS actives new state-of-the-art performance on RefCOCO, RefCOCO+ and G-Ref with simple framework!

• 🔧 [Jun 6, 2022] Pytorch implementation of CRIS are released.
• ☀️ [Mar 2, 2022] Our paper was accepted by CVPR-2022.

Main results on RefCOCO

Main results on RefCOCO+

Main results on G-Ref

1. Environment
   • PyTorch (e.g. 1.10.0)
   • Other dependencies in requirements.txt
2. Datasets
   • The detailed instruction is in prepare_datasets.md

This implementation only supports multi-gpu, DistributedDataParallel training, which is faster and simpler; single-gpu or DataParallel training is not supported. Besides, the evaluation only supports single-gpu mode.

Before training, please login in your wandb via wandb login or wandb login --anonymously. To do training of CRIS with 8 GPUs, run:

# e.g., Evaluation on the val-set of the RefCOCO dataset
python -u train.py --config config/refcoco/cris_r50.yaml

To do evaluation of CRIS with 1 GPU, run:

# e.g., Evaluation on the val-set of the RefCOCO dataset
CUDA_VISIBLE_DEVICES=0 python -u test.py \
 --config config/refcoco/cris_r50.yaml \
 --opts TEST.test_split val-test \
 TEST.test_lmdb datasets/lmdb/refcocog_g/val.lmdb

This project is under the MIT license. See LICENSE for details.

If you find our work useful in your research, please consider citing:

@inproceedings{wang2021cris,
 title={CRIS: CLIP-Driven Referring Image Segmentation},
 author={Wang, Zhaoqing and Lu, Yu and Li, Qiang and Tao, Xunqiang and Guo, Yandong and Gong, Mingming and Liu, Tongliang},
 booktitle={Proceedings of the IEEE/CVF conference on computer vision and pattern recognition},
 year={2022}
}