Research map / Vision-language and multimodal / Human and embodied modeling
Missing Target-Relevant Information Prediction with World Model for Accurate Zero-Shot Composed Image Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PrediCIR, which trains a JEPA-style world model on cropped image-caption pairs to predict missing target-relevant visual content in latent space and maps the result to a pseudo-word token for zero-shot composed image retrieval.
Problem
Zero-shot composed image retrieval fails when the reference image lacks essential target content specified by the manipulation text, because existing image-to-word mapping networks ignore missing target content and CLIP embeddings are coarse-grained.
Approach
Construct pseudo-triplets (cropped source view, caption action, original target view) from image-caption pairs, train a JEPA-style predictor to infer masked target patch features guided by the action, and gate-combine predicted and source features into a pseudo-word token via contrastive alignment with a frozen CLIP.
Key results
- On CIRR with ViT-L/14, PrediCIR reaches R@1 27.2 vs 25.6 for Context-I2W.
- On ImageNet domain conversion with ViT-L/14, PrediCIR averages R@10 16.4 / R@50 33.0, about 4.45% over Context-I2W.
- With ViT-G/14, PrediCIR reaches 47.2 R@10 on Fashion-IQ average and 17.2 R@1 on COCO, surpassing LinCIR and CompoDiff.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PrediCIR instead of LinCIR | ZS-CIR with ViT-G/14 · Fashion-IQ average | R10: 45.1 -> 47.2 | Helped |
| use PrediCIR instead of LinCIR | ZS-CIR with ViT-G/14 · CIRR | R1: 35.3 -> 37.0 | Helped |
| use PrediCIR instead of LinCIR | ZS-CIR with ViT-G/14 · CIRCO | mAP@5: 19.7 -> 23.7 | Helped |
| use PrediCIR instead of LinCIR | ZS-CIR with ViT-G/14 · COCO | R1: 14.8 -> 17.2 | Helped |
| use PrediCIR instead of LinCIR | ZS-CIR with ViT-G/14 · ImageNet | R10: 15.5 -> 19.0 | Helped |
| use PrediCIR instead of CompoDiff | ZS-CIR with ViT-G/14 · GeneCLS | R1: 15.5 -> 17.7 | Helped |
| add cropped source views rather than using the entire target image (ablation) | PrediCIR · CIRR | R1: 23.5 -> 27.2 | Helped |
| add the action embedding (ablation) | PrediCIR · CIRR | R1: 22.4 -> 27.2 | Helped |
| add the Target Content Predictor (ablation) | PrediCIR · CIRR | R1: 20.2 -> 27.2 | Helped |
| add the L2 prediction loss (ablation) | PrediCIR · CIRR | R1: 22.0 -> 27.2 | Helped |
| add the gating strategy to combine predicted and source features (ablation) | PrediCIR · CIRR | R1: 25.9 -> 27.2 | Helped |
| replace a typical self-attention network with the PrediCIR mapping strategy (ablation) | PrediCIR · CIRR | R1: 18.2 -> 27.2 | Helped |
Surprising findings
- Predicting the entire target image instead of partial blocks reduces performance, which the authors attribute to computation and overfitting.
- Random masking of source images instead of cropping degrades results by about 6.2% on average, attributed to the frozen CLIP encoder struggling with masked inputs.
Limitations
- Struggles with complex or redundant manipulation texts involving multiple objects and attributes.
- Limited by the CLIP language encoder's ability to interpret abstract or redundant intentions.
- Model is larger than simple MLP mapping, increasing training time relative to Pic2Word and SEARLE.
Experimental setup
Datasets: Conceptual Captions COCO GeneCIS ImageNet CIRR CIRCO Fashion-IQ
Benchmarks: CIRR CIRCO Fashion-IQ COCO ImageNet GeneCIS
Metrics: Recall@K mAP@K
Baselines: Pic2Word SEARLE-XL LinCIR Context-I2W CompoDiff
Builds on: Pic2Word I-JEPA CLIP LinCIR Context-I2W
Closest papers by meaning
- Context-I2W: Mapping Images to Context-Dependent Words for Accurate Zero-Shot Composed Image Retrieval (AAAI 2024 · similarity 0.93)
- Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrieval (CVPR 2025 · similarity 0.91)
- Language-only Training of Zero-shot Composed Image Retrieval (CVPR 2024 · similarity 0.89)
- Knowledge-Enhanced Dual-stream Zero-shot Composed Image Retrieval (CVPR 2024 · similarity 0.89)
- Vision-by-Language for Training-Free Compositional Image Retrieval (ICLR 2024 · similarity 0.89)
- Generative Zero-Shot Composed Image Retrieval (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.