Research map / Vision-language and multimodal / Cross-modal learning
Empowering Visible-Infrared Person Re-Identification with Large Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TVI-LFM, a text-enhanced visible-infrared person re-ID framework that uses fine-tuned BLIP captioners, Vicuna-7B text augmentation, and CLIP text-encoder fine-tuning to create semantically filtered fusion features and ensemble queries, and releases three expanded VI-ReID datasets with generated captions.
Problem
Visible-infrared person re-identification suffers from the missing color information in infrared images, and prior text-based auxiliary methods rely on costly manual annotation and hand-crafted priors.
Approach
Automatically generate modality-specific captions with fine-tuned BLIP models, augment them with an LLM, and incrementally fine-tune a CLIP text encoder so that text-derived complementary features (visible minus infrared text features) are added to infrared image features to form parameter-free fusion features aligned with the visible modality; at inference, an ensemble of infrared, text, and fusi
Key results
- On Tri-SYSU-MM01 (All Search), TVI-LFM reaches 84.90% Rank-1 and 81.47% mAP with the fusion query, versus 77.78% Rank-1 for PartMix (infrared query).
- On Tri-RegDB, TVI-LFM achieves 91.38% Rank-1 and 85.92% mAP, above YYDS (90.95% / 84.22%).
- On Tri-LLCM, TVI-LFM achieves 58.19% Rank-1 and 65.08% mAP, slightly above YYDS (58.13% / 64.91%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add generated-text enhancement with SFF, MJL, and MER | VI-ReID Backbone · Tri-SYSU-MM01 All Search | Rank-1: 69.89 -> 84.90 | Helped |
| add generated-text enhancement with SFF, MJL, and MER | VI-ReID Backbone · Tri-RegDB | Rank-1: 89.51 -> 91.38 | Helped |
| add generated-text enhancement with SFF, MJL, and MER | VI-ReID Backbone · Tri-LLCM | Rank-1: 53.53 -> 58.19 | Helped |
| add semantic filtered fusion (SFF) (ablation) | TVI-LFM baseline · Tri-SYSU-MM01 | Rank-1: 72.52 -> 77.00 | Helped |
| add semantic filtered fusion (SFF) (ablation) | TVI-LFM baseline · Tri-LLCM | Rank-1: 52.63 -> 54.73 | Helped |
| add modality joint learning (MJL) after SFF (ablation) | TVI-LFM baseline + SFF · Tri-SYSU-MM01 | Rank-1: 77.00 -> 83.97 | Helped |
| add modality joint learning (MJL) after SFF (ablation) | TVI-LFM baseline + SFF · Tri-LLCM | Rank-1: 54.73 -> 56.76 | Helped |
| add LLM-based text augmentation (ablation) | TVI-LFM with SFF and MJL · Tri-SYSU-MM01 | Rank-1: 83.97 -> 84.17 | Helped |
| add LLM-based text augmentation (ablation) | TVI-LFM with SFF and MJL · Tri-LLCM | Rank-1: 56.76 -> 57.13 | Helped |
| add MER without LLM augmentation (ablation) | TVI-LFM with SFF and MJL · Tri-SYSU-MM01 | Rank-1: 83.97 -> 84.88 | Helped |
| add MER without LLM augmentation (ablation) | TVI-LFM with SFF and MJL · Tri-LLCM | Rank-1: 56.76 -> 57.09 | Helped |
| add MER to the version with SFF, MJL, and LLM augmentation (ablation) | TVI-LFM with SFF, MJL, and LLM augmentation · Tri-SYSU-MM01 | Rank-1: 84.17 -> 84.90 | Helped |
Surprising findings
- Training the whole framework from scratch instead of incrementally fine-tuning the text encoder on a frozen backbone lowered the visual backbone's Rank-1 by 5.43% on Tri-SYSU-MM01.
Limitations
- Retrieval accuracy depends on the quality of generated text.
- Performance on the hard LLCM dataset still has room for improvement (58.19% Rank-1).
- Gains over the strongest prior method on Tri-LLCM and Tri-RegDB are marginal.
Experimental setup
Datasets: SYSU-MM01 LLCM RegDB pedestrian image-text dataset [40]
Benchmarks: Tri-SYSU-MM01 Tri-RegDB Tri-LLCM
Metrics: Rank-1 mAP mINP
Baselines: AGW DDAG CAJ DART DEEN SAAI PartMix YYDS
Builds on: BLIP CLIP CAJ AGW Vicuna
Closest papers by meaning
- Learning Progressive Modality-Shared Transformers for Effective Visible-Infrared Person Re-identification (AAAI 2023 · similarity 0.93)
- Implicit Discriminative Knowledge Learning for Visible-Infrared Person Re-Identification (CVPR 2024 · similarity 0.92)
- Shallow-Deep Collaborative Learning for Unsupervised Visible-Infrared Person Re-Identification (CVPR 2024 · similarity 0.90)
- High-Order Structure Based Middle-Feature Learning for Visible-Infrared Person Re-identification (AAAI 2024 · similarity 0.89)
- TokenMatcher: Diverse Tokens Matching for Unsupervised Visible-Infrared Person Re-Identification (AAAI 2025 · similarity 0.89)
- ReID5o: Achieving Omni Multi-modal Person Re-identification in a Single Model (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.