Research map / Video / Video segmentation and tracking

When Trackers Fail: VLM-Guided Verification and Recovery for Robust Video Object Segmentation

NeurIPS 2026 · Video › Video segmentation and tracking · approach: Matching-based video object segmentation

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Dual-Agent adds a zero-shot VLM verifier (Qwen3-VL-8B-Thinking) and an SFT-trained VLM corrector (Qwen3-VL-4B, LoRA) on top of frozen SAM3 to detect and recover tracking failures in video object segmentation and long-term tracking.

Problem

Memory-based SAM-style trackers propagate errors recursively, so once identity drift or target loss occurs, the wrong state persists; existing methods lack explicit failure detection and recovery.

Approach

Keep the propagator frozen and add a control loop: a VLM verifier judges sparse-triggered frames from a 7-image spatio-temporal prompt, a failure-trained corrector re-localizes the target, and a commit gate (box IoU below 0.5 and low SAM3 confidence) decides whether to overwrite tracker state.

Key results

• On SecVOS-160, Dual-Agent improves SAM3 from 74.21 to 76.94 J&F, with the largest gain on multi-scene-change subset (69.27 to 73.39).
• On LVOS-v2 validation, Dual-Agent reaches 89.4 J&F vs SAM3 88.2, with best unseen-object score (85.8 J, 94.1 F).
• Dual-Agent improves SAM3 on LaSOT-ext AUC (63.7 to 64.3), TNL2K AUC (65.2 to 66.5), and VOT-LT2022 F-score (60.1 to 61.2); the framework also transfers to Cutie, SAM2.1, and SeC without retraining.

Surprising findings

• Using the corrector at every trigger without the verifier returns SecVOS performance to the SAM3 baseline (74.10 vs 74.21), showing the verifier is essential.
• The unconditional corrector slightly degrades already-accurate predictions (mean IoU change -0.043 on IoU>=0.7 samples), motivating selective intervention.

Limitations

• Not real-time; runtime is 4-6x slower than SAM3 per video.
• Incorrect verifier decisions or corrections can override accurate tracks, e.g., 19 of 88 committed corrections on SecVOS worsen results and 8 of 202 seen LVOS objects regress.
• Relies on a large VLM verifier and LoRA-tuned corrector, adding compute and latency overhead.

Experimental setup

Datasets: SA-V LVOS-v2 SecVOS LaSOT-ext TNL2K VOT-LT2022

Benchmarks: LVOS-v2 SecVOS LaSOT-ext TNL2K VOT-LT2022

Metrics: J&F J F AUC precision normalized precision recall F-score

Baselines: SAM3 SAM2.1 SAM2.1Long SeC SAMURAI Cutie XMem DAM4SAM

Builds on: SAM3 SAM2 SAMURAI SeC

Closest papers by meaning

• A Distractor-Aware Memory for Visual Object Tracking with SAM2 (CVPR 2025 · similarity 0.88)
• SAM2MOT: A Novel Paradigm of Multi-Object Tracking by Segmentation (AAAI 2026 · similarity 0.87)
• D3S - A Discriminative Single Shot Segmentation Tracker (CVPR 2020 · similarity 0.86)
• RTracker: Recoverable Tracking via PN Tree Structured Memory (CVPR 2024 · similarity 0.85)
• SAM-I2V: Upgrading SAM to Support Promptable Video Segmentation with Less than 0.2% Training Cost (CVPR 2025 · similarity 0.84)
• Track-On: Transformer-based Online Point Tracking with Memory (ICLR 2025 · similarity 0.84)