Research map / Video / Video-language understanding
VIoTGPT: Learning to Schedule Vision Tools Towards Intelligent Video Internet of Things
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds VIoTGPT, an LLM agent fine-tuned with ReAct-style instruction tuning (LoRA) to schedule 11 vision tools and query video knowledge for Video Internet of Things tasks, along with the VIoT-Tool dataset and benchmark.
Problem
Scheduling many domain-specific perceiving models and analyzing collected surveillance videos uniformly and intelligently for complex VIoT tasks, including fine-grained tool differentiation and multi-step use of interrelated tools.
Approach
Fine-tune open LLMs (Llama 7B, Vicuna 7B) with LoRA on semi-automatically annotated ReAct instructions covering 11 surveillance vision tools organized in a three-category taxonomy, so the agent decides tool use, selects tools, and forms tool inputs from video knowledge base queries.
Key results
- VIoTGPT (Vicuna) reaches 94.13% decision accuracy and 66.29% tool accuracy on the test set for all responses, versus 71.64% and 48.37% for zero-shot Vicuna 7B.
- VIoTGPT (Llama) achieves 47.20% whole-response accuracy on all test responses, while zero-shot Llama 7B scores 0%.
- ChatGPT few-shot reaches 83.16% whole-response accuracy overall but drops to 28.25% on interrelated tool responses, below VIoTGPT (Vicuna) at 52.00%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Instruction-tune Llama on VIoT-Tool ReAct instructions | Llama 7B · VIoT-Tool test set, all responses | whole response accuracy: 0.00 -> 47.20 | Helped |
| Instruction-tune Vicuna on VIoT-Tool ReAct instructions | Vicuna 7B · VIoT-Tool test set, all responses | whole response accuracy: 0.00 -> 39.59 | Helped |
| Use ChatGPT few-shot prompting rather than zero-shot prompting | ChatGPT (gpt-3.5-turbo) · VIoT-Tool test set, all responses | whole response accuracy: 72.03 -> 83.16 | Helped |
| Instruction-tune Llama on VIoT-Tool ReAct instructions | Llama 7B · VIoT-Tool test set, single tool responses | whole response accuracy: 0.00 -> 41.11 | Helped |
| Instruction-tune Vicuna on VIoT-Tool ReAct instructions | Vicuna 7B · VIoT-Tool test set, single tool responses | whole response accuracy: 0.00 -> 35.79 | Helped |
| Use ChatGPT few-shot prompting rather than zero-shot prompting | ChatGPT (gpt-3.5-turbo) · VIoT-Tool test set, single tool responses | whole response accuracy: 93.92 -> 99.85 | Helped |
| Instruction-tune Llama on VIoT-Tool ReAct instructions | Llama 7B · VIoT-Tool test set, interrelated tools responses | whole response accuracy: 0.00 -> 67.25 | Helped |
| Instruction-tune Vicuna on VIoT-Tool ReAct instructions | Vicuna 7B · VIoT-Tool test set, interrelated tools responses | whole response accuracy: 0.00 -> 52.00 | Helped |
| Use ChatGPT few-shot prompting rather than zero-shot prompting | ChatGPT (gpt-3.5-turbo) · VIoT-Tool test set, interrelated tools responses | whole response accuracy: 0.00 -> 28.25 | Helped |
Surprising findings
- Vicuna 13B fine-tuned on GPT4Tools gives no significant gains over original Vicuna 7B on the VIoT benchmarks.
- Instruction-tuned Vicuna often invokes tools and queries videos correctly but fails to return the final response.
Limitations
- Large performance gap between validation and test sets due to semantic differences in queries and video diversity.
- Fine-grained tool confusion (e.g., license plate recognition vs vehicle re-identification, gait vs person re-identification) limits whole-response accuracy.
- Only 7B-scale open LLMs were fine-tuned; the paper suggests more powerful LLMs are needed.
Experimental setup
Datasets: VIoT-Tool training set Market-1501 CASIA-B VeRi CCPD UCF-Crime Sherlock VSCrowd FDST
Benchmarks: VIoT-Tool test set
Metrics: decision accuracy tool name accuracy tool input accuracy whole response accuracy
Baselines: Llama 7B zero-shot Llama 7B few-shot Vicuna 7B zero-shot Vicuna 7B few-shot ChatGPT zero-shot ChatGPT few-shot Vicuna 13B GPT4Tools
Builds on: ReAct Toolformer GPT4Tools Langchain FastChat
Closest papers by meaning
- Real-time Core-Periphery Guided ViT with Smart Data Layout Selection on Mobile Devices (NeurIPS 2024 · similarity 0.83)
- VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuning (NeurIPS 2025 · similarity 0.82)
- Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition (ICML 2024 · similarity 0.82)
- Act2See: Emergent Active Visual Perception for Video Reasoning (CVPR 2026 · similarity 0.81)
- DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent) (ICML 2024 · similarity 0.81)
- HydraViT: Stacking Heads for a Scalable ViT (NeurIPS 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.