Research map / 3D vision / 3D generation
Mono3DVLT: Monocular-Video-Based 3D Visual Language Tracking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Mono3DVLT, a monocular-video 3D visual language tracking task with the LLM-generated Mono3DVLT-V2X dataset (79,158 sequences) and a memory-augmented transformer baseline, Mono3DVLT-MT.
Problem
Visual language tracking is limited to 2D; 3D tracking relies on expensive sensors (LiDAR, radar, depth), and there is no language-described 3D tracking task for monocular video.
Approach
Extends VLT to 3D by pairing monocular video with ChatGPT-generated, human-verified attribute-based language descriptions, and proposes a transformer with language-guided visual/depth encoders and a memory-improved Token Turing Machine decoder predicting 2D and 3D boxes.
Key results
- Mono3DVLT-MT reaches SR@0.5 of 81.63 and AOR of 85.12 on Mono3DVLT-V2X, versus 71.75 and 79.13 for the best baseline Mono3DVG-TR.
- ACE drops to 0.521 from 0.594 for Mono3DVG-TR.
- Swin Transformer extractor improves SR@0.9 from 49.13 to 53.91 over ResNet50 in the ablation; adding memory-improved TTM further raises SR@0.9 to 58.86.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Mono3DVLT-MT instead of Mono3DVG-TR | Mono3DVLT-V2X tracking · Mono3DVLT-V2X test | SR@0.9: 49.13 -> 58.86 | Helped |
| Use Mono3DVLT-MT instead of Mono3DVG-TR | Mono3DVLT-V2X tracking · Mono3DVLT-V2X test | AOR: 79.13 -> 85.12 | Helped |
| Use Mono3DVLT-MT instead of Mono3DVG-TR | Mono3DVLT-V2X tracking · Mono3DVLT-V2X test | ACE: 0.594 -> 0.521 | Helped |
| Replace ResNet50 with Swin Transformer features (ablation) | Mono3DVLT-MT without memory augmentation · Mono3DVLT-V2X test | SR@0.9: 49.13 -> 53.91 | Helped |
| Add Memory-improved TTM (ablation) | Mono3DVLT-MT with ResNet50 features · Mono3DVLT-V2X test | SR@0.9: 49.13 -> 55.38 | Helped |
| Add Memory-improved TTM (ablation) | Mono3DVLT-MT with Swin Transformer features · Mono3DVLT-V2X test | SR@0.9: 53.91 -> 58.86 | Helped |
Limitations
- Dataset descriptions are generated by ChatGPT, with only manual verification, so language quality depends on the LLM pipeline.
- Baselines are 2D grounding/VLT methods adapted with back-projection, making comparisons less direct.
- Compute resources and training details are sparsely reported.
Experimental setup
Datasets: V2X-Seq Mono3DVLT-V2X
Benchmarks: Mono3DVLT-V2X
Metrics: SR@0.5-0.9 AOR PR@1.0 ACE
Baselines: ZSGNet FAOA ReSC TransVG JointNLT Mono3DVG-TR
Builds on: Mono3DVG Token Turing Machine V2X-Seq Grounding DINO
Closest papers by meaning
- MonoVLM: Monocular 3D Visual Grounding with Vision Language Models (CVPR 2026 · similarity 0.89)
- Unifying Visual and Vision-Language Tracking via Contrastive Learning (AAAI 2024 · similarity 0.87)
- MemVLT: Vision-Language Tracking with Adaptive Memory-based Prompts (NeurIPS 2024 · similarity 0.87)
- ChatTracker: Enhancing Visual Tracking Performance via Chatting with Multimodal Large Language Model (NeurIPS 2024 · similarity 0.87)
- GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models (ICLR 2026 · similarity 0.86)
- Beyond Human Perception: Understanding Multi-Object World from Monocular View (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.