Visual Delta Generator with Large Multi-modal Models for Semi-supervised Composed Image Retrieval

1Meta AI 2Korea University 3University of Central Florida
Intro Image A
CIR triplet generation with human supervision
Intro Image B
Utilize Visual Delta Generator


Composed Image Retrieval (CIR) is a task that retrieves images similar to a query, based on a provided textual modification. Current techniques rely on supervised learning for CIR models using labeled triplets of the (reference image, text, target image). These specific triplets are not as commonly available as simple image-text pairs, limiting the widespread use of CIR and its scalability. On the other hand, zero-shot CIR can be relatively easily trained with image-caption pairs without considering the image-to-image relation, but this approach tends to yield lower accuracy. We propose a new semi-supervised CIR approach where we search for a reference and its related target images in auxiliary data and learn our large language model-based Visual Delta Generator (VDG) to generate text describing the visual difference (i.e., visual delta) between the two. VDG, equipped with fluent language knowledge and being model agnostic, can generate pseudo triplets to boost the performance of CIR models. Our approach significantly improves the existing supervised learning approaches and achieves state-of-the-art results on the CIR benchmarks.

Method Image A
VDG training process
Method Image B
Generated examples
Method Image A
Retrieval results on CIRR
Method Image B
Retrieval results on Deepfashion


  author    =  {Jang, Young Kyun and Kim, Donghyun and Meng, Zihang and Huynh, Dat and Lim, Ser-Nam},
  title     = {Visual Delta Generator with Large Multi-modal Models for Semi-supervised Composed Image Retrieval},
  journal   = {CVPR},
  year      = {2024},