Research map / Vision-language and multimodal / Visual question answering
JTD-UAV: MLLM-Enhanced Joint Tracking and Description Framework for Anti-UAV Systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the UAV Tracking and Intent Understanding (UTIU) task and JTD-UAV, an MLLM-based framework that jointly tracks UAVs and describes their motion/intent via VQA, plus the TDUAV thermal benchmark.
Problem
Anti-UAV systems focus on position tracking and do not capture UAV motion states or intent; there is no joint tracking and intent-description benchmark or model.
Approach
A dual-branch framework pairing a UAV detection-and-tracking module (YOLOv8 global detector plus local tracker) with an MLLM that receives compressed visual features of the tracked UAV, trained jointly with a tracking loss and an LLM text loss to produce motion descriptions and answer VQA questions.
Key results
- JTD-UAV reaches BLEU-4 of 12.75, 11.23, and 14.66 on TDUAV subsets 1-3, above GPT-4o's 2.32, 2.39, and 3.00.
- JTD-UAV achieves SPICE of 0.3264/0.3012/0.3157 and ACC of 92.52%/89.94%/91.20% on TDUAV subsets, the highest among compared MLLMs.
- On TDUAV tracking, JTD-UAV gets 0.631 validation and 0.593 test accuracy, and 0.612 on DUT Anti-UAV and 0.633 on Anti-UAV.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the JTD-UAV pipeline (tracking, feature compression, LLM with joint training) versus plain Qwen2-VL trained on same data (ablation) | Qwen2-VL base model · TDUAV subsets 1-3 | BLEU-4, SPICE, ACC: BLEU-4 9.98 -> 12.75 (Subset 1); SPICE 0.2812 -> 0.3264; ACC 83.12 -> 92.52 | Helped |
| Joint training of tracking and description with the LLM, versus a description-only model (ablation) | JTD-UAV · TDUAV subset 2 | BLEU-4: 10.12 -> 11.23 | Helped |
| Use JTD-UAV tracking (detector + tracker) for UAV tracking instead of prior anti-UAV trackers | Tracking on DUT Anti-UAV · DUT Anti-UAV | ACC: 0.612 vs best baseline KeepTrack 0.592 | Helped |
| Use JTD-UAV tracking instead of prior trackers | Tracking on Anti-UAV · Anti-UAV | ACC: 0.633 vs best baseline OSTrack 0.549 | Helped |
| Use JTD-UAV versus general MLLMs for description and VQA | Caption and VQA on TDUAV · TDUAV subsets 1-3 | BLEU-4: 12.75 vs GPT-4o 2.32 (Subset 1); 14.66 vs best open baseline Qwen2-VL-7B 10.36 (Subset 3) | Helped |
Limitations
- The description of the LLM backbone and its size is not specified in the text.
- Evaluation relies on n-gram and SPICE metrics and ACC without human or LLM-judge evaluation.
- The paper does not report compute requirements or training hyperparameters beyond optimizer and learning rate.
Experimental setup
Datasets: TDUAV AntiUAV LaSOT TrackingNet GOT10K COCO
Benchmarks: TDUAV DUT Anti-UAV Anti-UAV
Metrics: BLEU-1 BLEU-2 BLEU-3 BLEU-4 SPICE ACC tracking accuracy
Baselines: GPT-4o Qwen-Max Qwen-Turbo InternVL2-8B KOSMOS-2 Qwen2-VL-7B-Instruct OSTrack MixFormer Stark SwinTrack-Base Qwen2-VL (same data)
Builds on: YOLOv8 LLaVA Qwen2-VL
Closest papers by meaning
- Towards Realistic UAV Vision-Language Navigation: Platform, Benchmark, and Methodology (ICLR 2025 · similarity 0.84)
- Dynamic Semantic-Aware Correlation Modeling for UAV Tracking (NeurIPS 2025 · similarity 0.84)
- LVPTrack: High Performance Domain Adaptive UAV Tracking with Label Aligned Visual Prompt Tuning (AAAI 2025 · similarity 0.82)
- Open-World Drone Active Tracking with Goal-Centered Rewards (NeurIPS 2025 · similarity 0.82)
- UAV-Flow Colosseo: A Real-World Benchmark for Flying-on-a-Word UAV Imitation Learning (NeurIPS 2025 · similarity 0.82)
- Drones Help Drones: A Collaborative Framework for Multi-Drone Object Trajectory Prediction and Beyond (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.