Research map / Image generation / Autoregressive and latent generation
MIMTrack: In-Context Tracking via Masked Image Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reformulates single-object visual tracking as masked image modeling with in-context learning over a visual prompt image, predicting the target state as an RGB target image decoded back into a bounding box, with a latent memory module for appearance updates.
Problem
Classification and regression subtask branches in Siamese and Transformer trackers produce location and scale misalignment and optimization divergence when predicting object states.
Approach
Encodes bounding boxes as RGB target images (corner heatmaps in green, center heatmap in red, box mask in blue) and treats tracking as masked pixel generation of the search target image conditioned on a template and its target image as in-context example, with a latent memory of learnable tokens for appearance changes.
Key results
- MIMTrack-L reaches 75.3% AO on GOT-10k, above SeqTrack-L (74.5%) and OSTrack (71.0%).
- MIMTrack-B surpasses OSTrack by 1.6% AO on GOT-10k (72.6 vs 71.0) and runs at 51.2 FPS.
- MIMTrack-B (LM) with latent memory reaches 70.0% AUC on LaSOT and 48.9% AUC on LaSOText.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add latent memory | MIMTrack-B · GOT-10k | AO: 72.6 -> 73.5 | Helped |
| Use a ViT-Large backbone instead of ViT-Base | MIMTrack · GOT-10k | AO: 72.6 -> 75.3 | Helped |
| Use one convolutional decoder layer instead of the baseline decoder (ablation) | MIMTrack-B · GOT-10k | AO: 72.6 -> 71.4 | Hurt |
| Replace the baseline decoder with two linear layers (ablation) | MIMTrack-B · GOT-10k | AO: 72.6 -> 71.0 | Hurt |
| Replace the baseline decoder with 8 ViT blocks (ablation) | MIMTrack-B · GOT-10k | AO: 72.6 -> 73.0 | Helped |
| Use random masking instead of the fixed search-target mask (ablation) | MIMTrack-B · GOT-10k | AO: 72.6 -> 59.8 | Hurt |
| Fuse fixed and random masking strategies (ablation) | MIMTrack-B · GOT-10k | AO: 72.6 -> 64.5 | Hurt |
| Keep only one encoder feature layer, removing two early features (ablation) | MIMTrack-B · GOT-10k | AO: 72.6 -> 71.6 | Hurt |
| Use one in-context example instead of two in training and testing (ablation) | MIMTrack-B · GOT-10k | AO: 72.6 -> 70.8 | Hurt |
| Use three test examples instead of two, with two training examples (ablation) | MIMTrack-B · GOT-10k | AO: 72.6 -> 72.1 | Hurt |
| Use SmoothL1 loss alone instead of SmoothL1 plus SSIM (ablation) | MIMTrack-B · GOT-10k | AO: 72.6 -> 72.2 | Hurt |
| Replace the baseline combined loss with L1 loss (ablation) | MIMTrack-B · GOT-10k | AO: 72.6 -> 72.2 | Hurt |
Surprising findings
- Random masking as in MAE drops GOT-10k AO sharply (to 59.8%), indicating that fixed target-aware occlusion is needed rather than random masking.
Limitations
- Dynamic prompts with more test examples than training examples degrade performance due to memory limits and forgetting.
- Adding an 8-ViT decoder improves accuracy slightly but drops speed by 7.4 FPS and doubles training time.
Experimental setup
Datasets: COCO LaSOT GOT-10k TrackingNet
Benchmarks: GOT-10k TrackingNet LaSOT LaSOText UAV123
Metrics: AO SR0.5 SR0.75 AUC Precision Normalized Precision FPS
Baselines: OSTrack SeqTrack ARTrack STARK AiATrack MixFormer-22k SwinTrack-T TransT DiMP SiamRPN++ KeepTrack EVPTrack
Builds on: Masked Autoencoders (MAE) Visual Prompting (Bar et al. 2022) SimMIM OSTrack Compressive Transformer-style memory
Closest papers by meaning
- Less Is More: Token Context-Aware Learning for Object Tracking (AAAI 2025 · similarity 0.87)
- SPMTrack: Spatio-Temporal Parameter-Efficient Fine-Tuning with Mixture of Experts for Scalable Visual Tracking (CVPR 2025 · similarity 0.87)
- Explicit Visual Prompts for Visual Object Tracking (AAAI 2024 · similarity 0.87)
- UTPTrack: Towards Simple and Unified Token Pruning for Visual Tracking (CVPR 2026 · similarity 0.86)
- MVLM: Template-Free Tracking via Vision-Language Margin Confidence and Memory-Gated Tracking (CVPR 2026 · similarity 0.86)
- ODTrack: Online Dense Temporal Token Learning for Visual Tracking (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.