Research map / Vision-language and multimodal / Cross-modal learning
All in One Framework for Multimodal Re-identification in the Wild
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AIO, a framework that tokenizes RGB, IR, sketch and text into a shared space and feeds them through a frozen LAION-2B-pretrained ViT with three cross-modal heads for zero-shot multimodal person re-identification.
Problem
Person re-identification across uncertain combinations of RGB, infrared, sketch and text modalities, especially zero-shot generalization to unseen modalities and domains, without costly fine-tuning of large models.
Approach
Freeze a large pre-trained ViT as a modality-shared encoder and train only lightweight per-modality tokenizers plus cross-modal heads (classification, vision-guided masked attribute modeling, and a multimodal feature binding loss centered on RGB), with synthetic IR (channel augmentation) and sketch (Lineart) data and progressive synthetic-to-real training.
Key results
- Zero-shot R→R Rank-1 of 79.6 on Market1501 with the LAION-2B ViT backbone, competitive with unimodal generalization methods.
- Zero-shot S+T→R Rank-1 of 92.1 on Tri-CUHK-PEDES, versus 91.4 for UNIReID on PKU-Sketch multimodal setting (Table 8 reports 92.1 for AIO vs 91.4 for UNIReID).
- Zero-shot T→R Rank-1 of 53.4 on CUHK-PEDES and I→R Rank-1 of 57.6 on SYSU-MM01, outperforming listed cross-modal baselines in the zero-shot setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the conventional classification head (ablation) | AIO · Market1501, R→R | Rank-1: 3.2 -> 78.0 | Helped |
| Add the conventional classification head (ablation) | AIO · SYSU-MM01, I→R | Rank-1: 1.5 -> 43.7 | Helped |
| Add the conventional classification head (ablation) | AIO · PKU-Sketch, S→R | Rank-1: 1.6 -> 46.8 | Helped |
| Add the conventional classification head (ablation) | AIO · CUHK-PEDES, T→R | Rank-1: 21.6 -> 53.5 | Helped |
| Add the conventional classification head (ablation) | AIO · Tri-CUHK-PEDES, S+T→R | Rank-1: 0.5 -> 91.8 | Helped |
| Add Vision Guided Masked Attribute Modeling head (ablation) | AIO · CUHK-PEDES, T→R | Rank-1: 21.6 -> 52.4 | Helped |
| Add Vision Guided Masked Attribute Modeling head (ablation) | AIO · SYSU-MM01, I→R | Rank-1: 1.5 -> 0.9 | Hurt |
| Add the feature binding head (ablation) | AIO · Market1501, R→R | Rank-1: 78.0 -> 79.6 | Helped |
| Add the feature binding head (ablation) | AIO · SYSU-MM01, I→R | Rank-1: 43.7 -> 57.6 | Helped |
| Add the feature binding head (ablation) | AIO · PKU-Sketch, S→R | Rank-1: 46.8 -> 70.2 | Helped |
| Add the feature binding head (ablation) | AIO · CUHK-PEDES, T→R | Rank-1: 53.5 -> 53.4 | No clear effect |
| Use the LAION-pretrained foundation model rather than the tested smaller-pretraining alternatives (ablation) | AIO · Market1501, R→R | Rank-1: 74.2; 76.3; 77.7; 79.6 | Helped |
Surprising findings
- Large-scale ReID-specific pre-trained models (e.g., APTM, LuPerson-based models) perform poorly in the zero-shot setting, with the exception of PLIP.
Limitations
- Evaluation of IR and sketch in multimodal settings relies on synthetic images (CA and Lineart) rather than real paired data.
- Cross-modal performance on some tasks (e.g., T→R on CUHK-PEDES) remains modest relative to the text-based state of the art in some comparisons.
- Frozen backbone limits adaptation; detailed limitations are deferred to the supplementary material.
Experimental setup
Datasets: SYNTH-PEDES LLCM MaSk1K Market1501 SYSU-MM01 PKU-Sketch CUHK-PEDES Tri-CUHK-PEDES
Benchmarks: Market1501 SYSU-MM01 PKU-Sketch CUHK-PEDES Tri-CUHK-PEDES
Metrics: Rank-1 Rank-5 Rank-10 mAP mINP
Baselines: UNIReID IRRA AGW APTM PLIP OSNet-IBN M3L OSNet-AIN TriReID LuPerson-NL
Builds on: ViT CLIP UNIReID ViT-ICS Channel Augmentation Lineart
Closest papers by meaning
- IDEA: Inverted Text with Cooperative Deformable Aggregation for Multi-modal Object Re-Identification (CVPR 2025 · similarity 0.86)
- MambaPro: Multi-Modal Object Re-identification with Mamba Aggregation and Synergistic Prompt (AAAI 2025 · similarity 0.86)
- MDReID: Modality-Decoupled Learning for Any-to-Any Multi-Modal Object Re-Identification (NeurIPS 2025 · similarity 0.86)
- Magic Tokens: Select Diverse Tokens for Multi-modal Object Re-Identification (CVPR 2024 · similarity 0.86)
- DeMo: Decoupled Feature-Based Mixture of Experts for Multi-Modal Object Re-Identification (AAAI 2025 · similarity 0.85)
- Empowering Visible-Infrared Person Re-Identification with Large Foundation Models (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.