Research map / Vision-language and multimodal / Human and embodied modeling
Visual Perturbation for Text-Based Person Search
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ViPer, a CLIP-driven end-to-end text-based person search network that uses spatial, attentive, and fine-grained visual perturbations during training to narrow the train-inference discrepancy.
Problem
Text-based person search trains on image-text pairs where the image covers all described clues, but at inference the scene image often captures only part of the described clues due to occlusion, clutter, and misaligned boundaries.
Approach
Simulate inference-time visual perturbations during training: shift and scale person proposals (Spatial ViPer), remove or exchange attention-selected tokens before global pooling (Attentive ViPer), and mask visual tokens then recover them via a full cross-attention transformer from language (Fine-grained ViPer).
Key results
- On PRW-TBPS, ViPer reaches 22.07% mAP and 35.62% top-1, versus 18.18% and 33.25% for the prior best MACA.
- On CUHK-SYSU-TBPS, ViPer reaches 62.13% mAP and 55.82% top-1, versus 57.77% and 52.03% for MACA.
- Ablation on PRW-TBPS: adding all three ViPer modules raises mAP from 19.41 (baseline) to 22.07 and top-1 from 31.25 to 35.62.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ViPer visual perturbation modules to the CLIP-driven baseline | CLIP ResNet50 TBPS model · PRW-TBPS | mAP: 18.18 -> 22.07 | Helped |
| Add ViPer visual perturbation modules to the CLIP-driven baseline | CLIP ResNet50 TBPS model · PRW-TBPS | top-1: 33.25 -> 35.62 | Helped |
| Add ViPer visual perturbation modules to the CLIP-driven baseline | CLIP ResNet50 TBPS model · CUHK-SYSU-TBPS | mAP: 57.77 -> 62.13 | Helped |
| Add ViPer visual perturbation modules to the CLIP-driven baseline | CLIP ResNet50 TBPS model · CUHK-SYSU-TBPS | top-1: 52.03 -> 55.82 | Helped |
| Add Spatial ViPer by perturbing person proposals (ablation) | CLIP ResNet50 TBPS baseline · PRW-TBPS | mAP: 19.41 -> 20.26 | Helped |
| Add Spatial ViPer by perturbing person proposals (ablation) | CLIP ResNet50 TBPS baseline · PRW-TBPS | top-1: 31.25 -> 32.26 | Helped |
| Add attentive token removal and exchange (ablation) | Spatial ViPer · PRW-TBPS | mAP: 20.26 -> 21.42 | Helped |
| Add attentive token removal and exchange (ablation) | Spatial ViPer · PRW-TBPS | top-1: 32.26 -> 34.94 | Helped |
| Replace attentive token removal with random token removal (ablation) | Spatial ViPer · PRW-TBPS | mAP: 20.75 -> 20.15 | Hurt |
| Replace attentive token removal with random token removal (ablation) | Spatial ViPer · PRW-TBPS | top-1: 34.10 -> 33.53 | Hurt |
| Replace attentive token exchange with random token exchange (ablation) | Spatial ViPer · PRW-TBPS | mAP: 20.82 -> 20.59 | Hurt |
| Add Fine-grained ViPer with the proposed full cross-attention transformer instead of the IRRA Encoder (ablation) | Spatial + Attentive ViPer · PRW-TBPS | mAP: 21.75 -> 22.07 | Helped |
Surprising findings
- Random token removal or exchange gives only marginal gains, while attention-guided manipulation gives clear improvements.
- The CLIP-based baseline alone already outperforms MACA on PRW-TBPS, which the authors attribute to CLIP pretraining.
Limitations
- TBPS performance remains far below image-based person search methods, indicating large room for improvement.
- Gains are reported on the same benchmarks with a single-run setup, with no variance or seeds reported.
Experimental setup
Datasets: PRW-TBPS CUHK-SYSU-TBPS RSTPReid
Benchmarks: PRW-TBPS CUHK-SYSU-TBPS RSTPReid
Metrics: mAP top-1 mINP
Baselines: OIM NAE BSL SDPG MACA IRRA
Builds on: CLIP OIM loss Faster R-CNN IRRA SDPG
Closest papers by meaning
- An Empirical Study of CLIP for Text-Based Person Search (AAAI 2024 · similarity 0.86)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.85)
- Causality-Inspired Invariant Representation Learning for Text-Based Person Retrieval (AAAI 2024 · similarity 0.85)
- Exploring the Potential of Large Vision-Language Models for Unsupervised Text-Based Person Retrieval (AAAI 2025 · similarity 0.85)
- Adaptive Uncertainty-Based Learning for Text-Based Person Retrieval (AAAI 2024 · similarity 0.85)
- Same or Not? Enhancing Visual Perception in Vision-Language Models (CVPR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.