Research map / Vision-language and multimodal / Cross-modal learning
IDEA: Inverted Text with Cooperative Deformable Aggregation for Multi-modal Object Re-Identification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds text-enhanced multi-modal (RGB/NIR/TIR) ReID benchmarks with MLLM-generated structured captions and proposes IDEA, which uses modal prefixes, an InverseNet (text-to-pseudo-image-token) and cooperative deformable aggregation over CLIP features.
Problem
Multi-modal object re-identification ignores text semantics, lacks text annotations for multi-spectral images, and suffers from conflicting cross-modal information and noisy aggregation of local features.
Approach
Reverse text into pseudo-image tokens (InverseNet) inside CLIP's vision encoder with modality-specific prefixes, and adaptively generate shared deformable sampling positions to aggregate discriminative local features with global tokens via cross attention.
Key results
- IDEA achieves 80.2% mAP and 82.1% Rank-1 on RGBNT201, surpassing DeMo (79.0% mAP) and TOP-ReID (72.3% mAP).
- IDEA reaches 87.2% mAP on RGBNT100 and 47.0% mAP / 62.4% Rank-1 on MSVR310.
- Ablation: removing text drops RGBNT201 mAP from 80.2% to 74.5%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add text guidance (ablation) | IDEA · RGBNT201 | mAP: 74.5 -> 80.2 | Helped |
| add text guidance (ablation) | IDEA · RGBNT201 | Rank-1: 75.0 -> 82.1 | Helped |
| add the inverted multi-modal feature extractor (ablation) | three-branch vision baseline with text · RGBNT201 | mAP: 73.4 -> 77.2 | Helped |
| add cooperative deformable aggregation (ablation) | IMFE · RGBNT201 | mAP: 77.2 -> 80.2 | Helped |
| add Modal Prefixes to parallel text encoders (ablation) | IMFE · RGBNT201 | mAP: 72.6 -> 73.4 | Helped |
| use inverted structure instead of parallel text encoders (ablation) | IMFE without prefixes · RGBNT201 | mAP: 72.6 -> 73.7 | Helped |
| add Modal Prefixes to the inverted structure (ablation) | IMFE · RGBNT201 | mAP: 73.7 -> 75.4 | Helped |
| add learnable tokens to Modal Prefixes (ablation) | IMFE with inverted structure and prefixes · RGBNT201 | mAP: 75.4 -> 77.2 | Helped |
| add cross attention without sampling (ablation) | CDA · RGBNT201 | mAP: 76.3 -> 77.0 | Helped |
| add sampling without cross attention or shared offsets (ablation) | CDA · RGBNT201 | mAP: 76.3 -> 76.8 | Helped |
| share offsets across modalities (ablation) | CDA with sampling and no cross attention · RGBNT201 | mAP: 76.8 -> 77.6 | Helped |
| add cross attention to sampling without shared offsets (ablation) | CDA · RGBNT201 | mAP: 76.8 -> 79.5 | Helped |
Limitations
- Caption generation relies on API-based MLLM (Qwen-VL) with randomness and possible hallucination.
- Text length limited by text encoder input length, causing truncation with longer prompts.
Experimental setup
Datasets: RGBNT201 MSVR310 RGBNT100
Benchmarks: RGBNT201 RGBNT100 MSVR310
Metrics: mAP Rank-1 Rank-5 Rank-10
Baselines: HAMNet PFNet IEEE DENet LRMM UniCat HTT TOP-ReID EDITOR RSCNet WTSF-ReID MambaPro
Builds on: CLIP-ReID TOP-ReID EDITOR CLIP
Closest papers by meaning
- MDReID: Modality-Decoupled Learning for Any-to-Any Multi-Modal Object Re-Identification (NeurIPS 2025 · similarity 0.87)
- DeMo: Decoupled Feature-Based Mixture of Experts for Multi-Modal Object Re-Identification (AAAI 2025 · similarity 0.87)
- Magic Tokens: Select Diverse Tokens for Multi-modal Object Re-Identification (CVPR 2024 · similarity 0.87)
- Chain-of-Thought Guided Multi-Modal Object Re-Identification (CVPR 2026 · similarity 0.87)
- MambaPro: Multi-Modal Object Re-identification with Mamba Aggregation and Synergistic Prompt (AAAI 2025 · similarity 0.86)
- Multi-Modal Object Re-identification via Sparse Mixture-of-Experts (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.