Research map / Vision-language and multimodal / Human and embodied modeling
ICAR: Image-Based Complementary Auto Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ICAR, a self-supervised two-stage framework for scene-aware cross-domain complementary item retrieval that pairs a ResNet50 visual-similarity embedding with a Flexible Bidirectional Transformer (FBT) that predicts item categories and next-item embeddings, plus a new SFID metric.
Problem
Scene-aware complementary item retrieval requires generating a set of items across domains that are both visually similar in style to a scene and complementary to each other; existing evaluation (FITB, human-in-the-loop) does not measure set-level compatibility well.
Approach
Separates visual similarity learning (ResNet50 retrieval embedding) from complementarity learning, and models outputs as unordered sets via a transformer with random shuffling, random-length masking, a category prediction arm and an embedding prediction arm trained self-supervised from scene images; also introduces SFID, a Frechet-distance metric over composed item images.
Key results
- ICAR reaches 87.1 FITB on DeepRooms with category given (vs 77.6 for OutfitTransformer) and 86.6 FITB on STL-Home vs 77.0 for OutfitTransformer.
- SFID on DeepRooms drops to 36.8 from 48.0 for OutfitTransformer (23.3% improvement).
- Human expert ratings show about 0.7 average Pearson correlation between SFID and human judgement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ICAR with given category rather than predict category | ICAR · DeepRooms | FITB accuracy: 83.9 -> 87.1 | Helped |
| use ICAR instead of OutfitTransformer | Scene-aware CIR · STL-Fashion | FITB accuracy: 70.0 -> 75.3 | Helped |
| use ICAR instead of OutfitTransformer | Scene-aware CIR · STL-Home | FITB accuracy: 77.0 -> 86.6 | Helped |
| use ICAR instead of OutfitTransformer | Scene-aware CIR · Street2Shop | FITB accuracy: 67.2 -> 71.4 | Helped |
| use ICAR instead of OutfitTransformer | Scene-aware CIR · DeepRooms | SFID: 48.0 -> 36.8 | Helped |
| use ICAR instead of OutfitTransformer | Scene-aware CIR · STL-Fashion | SFID: 15.2 -> 11.8 | Helped |
| use ICAR instead of OutfitTransformer | Scene-aware CIR · STL-Home | SFID: 9.1 -> 6.2 | Helped |
| use ICAR instead of OutfitTransformer | Scene-aware CIR · Street2Shop | SFID: 8.7 -> 7.1 | Helped |
| use random length masking instead of fixed length masking (ablation) | ICAR · DeepRooms (predict category) | FITB accuracy: 82.2 -> 83.9 | Helped |
| use random length masking instead of fixed length masking (ablation) | ICAR · DeepRooms (given category) | FITB accuracy: 84.8 -> 87.1 | Helped |
| use visual similarity retrieval features instead of VQGAN reconstruction features (ablation) | ICAR · DeepRooms | FITB accuracy: 73.1 -> 87.1 | Helped |
| use visual similarity retrieval features instead of Swin classification features (ablation) | ICAR · DeepRooms | FITB accuracy: 80.5 -> 87.1 | Helped |
Surprising findings
- Visual similarity embeddings outperform classification, reconstruction, generation, and contrastive embeddings as the first-stage features for complementary retrieval (87.1 vs 80.5 for Swin).
Limitations
- Subjective notion of complementarity makes objective evaluation difficult; SFID relies on composed images and a feature extractor.
- Evaluated on a limited set of domains (home and fashion) with a single retrieval pool per dataset.
Experimental setup
Datasets: DeepRooms STL-Home STL-Fashion Exact Street2Shop
Benchmarks: DeepRooms STL-Home STL-Fashion Street2Shop
Metrics: FITB accuracy SFID
Baselines: OutfitTransformer CSA-Net Complete the Look Siamese Nets BPR-DAE IBR Swin BEiT VQGAN
Builds on: OutfitTransformer Vision Transformer FID normalized softmax loss
Closest papers by meaning
- Beyond Instance-Level Image Retrieval: Leveraging Captions to Learn a Global Visual Representation for Semantic Retrieval (CVPR 2017 · similarity 0.84)
- CrossOver: 3D Scene Cross-Modal Alignment (CVPR 2025 · similarity 0.84)
- Heterogeneous Prompt-Guided Entity Inferring and Distilling for Scene-Text Aware Cross-Modal Retrieval (AAAI 2025 · similarity 0.84)
- RMIR: A Benchmark Dataset for Reasoning-Intensive Multimodal Image Retrieval (CVPR 2026 · similarity 0.83)
- Composing Object Relations and Attributes for Image-Text Matching (CVPR 2024 · similarity 0.83)
- Context-I2W: Mapping Images to Context-Dependent Words for Accurate Zero-Shot Composed Image Retrieval (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.