Research map / Adversarial robustness and security / Attacks on text and graphs
LRR: Language-Driven Resamplable Continuous Representation against Adversarial Tracking Attacks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LRR, a language-guided spatial-temporal implicit representation that reconstructs incoming tracking frames to remove adversarial perturbations while keeping CLIP-text semantic consistency with the object template.
Problem
Visual object trackers (e.g., SiamRPN++) are highly vulnerable to adversarial perturbations in video frames, and existing defenses (adversarial training, image preprocessing, diffusion purification) are costly or ignore temporal and template semantics.
Approach
Extends local implicit image functions to a spatial-temporal implicit representation (STIR) that reconstructs pixels from neighboring frames, and adds a language-driven resampling network (LResampleNet) that predicts coordinate offsets guided by a CLIP text embedding of the object template, so reconstructed frames stay semantically consistent with the target.
Key results
- On OTB100 with SiamRPN++ ResNet50, LRR raises precision under CSA attack from 47.2% to 89.4% and under SPARK from 69.8% to 89.3%, close to the 87.8% clean accuracy.
- On VOT2019 with SiamRPN++ ResNet50 under SPARK, LRR achieves EAO 0.269 versus 0.079 without defense.
- LRR runs at about 25 fps versus 29 fps for STIR alone, while DiffPure with T=50 takes 3391 ms per frame.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add LRR frame reconstruction and language-guided resampling | SiamRPN++ ResNet50 · OTB100 under RTAA, IoU Attack, CSA, and SPARK | precision (%): 32.7 -> 86.9; 75.9 -> 85.3; 47.2 -> 89.4; 69.8 -> 89.3 | Helped |
| add LRR frame reconstruction and language-guided resampling | SiamRPN++ ResNet50 · VOT2019 under RTAA, IoU Attack, CSA, and SPARK | EAO: 0.080 -> 0.255; 0.153 -> 0.217; 0.089 -> 0.237; 0.079 -> 0.269 | Helped |
| add LRR frame reconstruction and language-guided resampling | SiamRPN++ ResNet50 · UAV123 under RTAA, IoU Attack, CSA, and SPARK | precision (%): 41.2 -> 77.7; 70.5 -> 78.6; 46.5 -> 81.8; 40.8 -> 79.3 | Helped |
| add LResampleNet resampling to STIR (ablation) | STIR on SiamRPN++ ResNet50 · OTB100, VOT2019, and UAV123 under attacks | tracking score (precision or EAO): 85.9 -> 86.9; 0.247 -> 0.255; 77.0 -> 77.7 | Helped |
| add template language guidance to ResampleNet (ablation) | LRR on SiamRPN++ ResNet50 · OTB100 and UAV123 under RTAA, CSA, and SPARK | precision (%): 86.1 -> 86.9; 86.0 -> 89.4; 87.8 -> 89.3; 77.4 -> 77.7; 79.8 -> 81.8; 78.1 -> 79.3 | Helped |
| add template language guidance to ResampleNet (ablation) | LRR on SiamRPN++ ResNet50 · VOT2019 IoU Attack | EAO: 0.219 -> 0.217 | Hurt |
| increase STIR input length from N=1 to N=5 (ablation) | STIR on SiamRPN++ ResNet50 · OTB100 IoU Attack | precision (%): 78.7 -> 84.0 | Helped |
| increase STIR input length from N=1 to N=5 (ablation) | STIR on SiamRPN++ ResNet50 · VOT2019 RTAA | EAO: 0.240 -> 0.247 | Helped |
| add feature-consistency loss between reconstructed frame and object template (ablation) | LRR training · Tracking defense | tracking performance | No clear effect |
| apply LRR to ToMP-50 | ToMP-50 · OTB100 RTAA | precision (%): 61.8 -> 90.0 | Helped |
| apply LRR to ToMP-50 | ToMP-50 · VOT2019 RTAA | EAO: 0.337 -> 0.552 | Helped |
| apply LRR to ToMP-50 | ToMP-50 · UAV123 RTAA | precision (%): 58.5 -> 88.0 | Helped |
Surprising findings
- Adversarial fine-tuning (FGSM, PGD) decreased clean-data accuracy and provided little defense against RTAA and SPARK.
- Under the paper's settings, LRR's adversarial accuracy matches or exceeds its clean accuracy on several attacks.
Limitations
- As an extra preprocessing module, LRR adds computational and time cost.
- Slight clean-data accuracy drop (under 2% on OTB100/UAV123 and about 0.01 EAO on VOT2019) and generalization to non-noise attacks such as motion blur are untested.
- Evaluation covers only template-based trackers, not natural-language-specified trackers.
Experimental setup
Datasets: ImageNet-DET ImageNet-VID YouTube-BoundingBoxes
Benchmarks: OTB100 VOT2019 UAV123 LaSOT NFS30 NFS240 TrackingNet
Metrics: precision (%) EAO time per frame
Baselines: adversarial fine-tuning (FGSM, PGD, CSA) DISCO STIR (without resampling) DiffPure RTAA defense image resizing JPEG compression
Builds on: LIIF DISCO CLIP SiamRPN++
Closest papers by meaning
- LVPTrack: High Performance Domain Adaptive UAV Tracking with Label Aligned Visual Prompt Tuning (AAAI 2025 · similarity 0.84)
- Event Stream-based Visual Object Tracking: A High-Resolution Benchmark Dataset and A Novel Baseline (CVPR 2024 · similarity 0.83)
- OVTR: End-to-End Open-Vocabulary Multiple Object Tracking with Transformer (ICLR 2025 · similarity 0.82)
- Learning Occlusion-Robust Vision Transformers for Real-Time UAV Tracking (CVPR 2025 · similarity 0.82)
- SUTrack: Towards Simple and Unified Single Object Tracking (AAAI 2025 · similarity 0.82)
- Modeling Continuous Motion for 3D Point Cloud Object Tracking (AAAI 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.