Research map / Large language models / Agents, code and math
ContextAgent: Context-Aware Proactive LLM Agents with Open-world Sensory Perceptions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ContextAgent, a context-aware proactive LLM agent that uses wearable egocentric video/audio and persona contexts to predict when to proactively help and call tools, fine-tuned with distilled CoT traces, plus the 1,000-sample ContextAgentBench.
Problem
Existing proactive LLM agents only observe enclosed environments (e.g., desktop UIs), rely on direct inference or rule-based notifications, and cannot use open-world sensory context from wearables to decide when and how to proactively assist with tools.
Approach
A proactive-oriented context extraction pipeline (ICL-prompted VLM for egocentric video, Whisper for audio, notifications, personas) feeds a fine-tuned LLM reasoner that thinks before acting, outputs a proactive score, and plans tool chains.
Key results
- ContextAgent with Llama-3.1-8B-Instruct reaches 0.874 Acc-P, up to 8.5% above the best baseline, and +6.0% Acc-Args.
- ContextAgent with DeepSeek-R1-7B reaches 0.888 Acc-P and 0.647 F1 on ContextAgentBench, comparable to GPT-4o ICL-All (0.921 Acc-P, 0.715 F1) and approaching 70B baselines.
- Removing persona context lowers Llama-3.1-8B Acc-P by up to 9.0% and F1 by up to 12.3%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ContextAgent instead of ICL-All | Llama-3.1-8B-Instruct · ContextAgentBench | Acc-P: 0.757 -> 0.874 | Helped |
| Use ContextAgent instead of ICL-All | Llama-3.1-8B-Instruct · ContextAgentBench | F1-score: 0.582 -> 0.626 | Helped |
| Use ContextAgent instead of Vanilla SFT | Llama-3.1-8B-Instruct · ContextAgentBench | Acc-Args: 0.405 -> 0.448 | Helped |
| Use ContextAgent instead of ICL-P | Qwen2.5-7B-Instruct · ContextAgentBench-Lite | Acc-P: 0.840 -> 0.850 | Helped |
| Use ContextAgent instead of CoT | Qwen2.5-7B-Instruct · ContextAgentBench-Lite | F1-score: 0.564 -> 0.624 | Helped |
| Use ContextAgent instead of SFT-P | Qwen2.5-7B-Instruct · ContextAgentBench-Lite | Acc-Args: 0.374 -> 0.479 | Helped |
| Add visual context (ablation) | ContextAgent · ContextAgentBench-Lite | Acc-P: 0.709 -> 0.888 | Helped |
| Add visual context (ablation) | ContextAgent · ContextAgentBench-Lite | F1-score: 0.414 -> 0.647 | Helped |
| Add audio context (ablation) | ContextAgent · ContextAgentBench-Lite | Acc-P: 0.720 -> 0.888 | Helped |
| Add audio context (ablation) | ContextAgent · ContextAgentBench-Lite | F1-score: 0.493 -> 0.647 | Helped |
| Use proactive-oriented sensory context extraction instead of zero-shot Qwen-2.5-VL extraction (ablation) | ContextAgent · ContextAgentBench-Lite | Acc-P: 3.0% | Helped |
| Add persona context during training and testing (ablation) | ContextAgent · ContextAgentBench-Lite | Acc-P: 9.0% | Helped |
Surprising findings
- Zero-shot Qwen2.5-VL for sensory context extraction degrades performance, showing that proactive-oriented context extraction matters.
- Under SFT the benefit of thought traces is reduced, although they help ICL substantially.
Limitations
- Benchmark samples are largely synthetic LLM-generated and annotator-written, with only 300 Lite samples having raw sensor data.
- Fine-tuning was only applied to models under 7B parameters; larger models were evaluated only as baselines.
Experimental setup
Datasets: ContextAgentBench PersonaHub Pexels egocentric videos
Benchmarks: ContextAgentBench ContextAgentBench-Lite
Metrics: Acc-P MD FD RMSE Precision Recall F1-score Acc-Args
Baselines: Proactive Agent Vanilla ICL CoT ICL-P ICL-All Vanilla SFT SFT-P
Builds on: Proactive Agent ProactiveBench chain-of-thought LoRA API-Bank
Closest papers by meaning
- Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance (ICLR 2025 · similarity 0.89)
- FingerTip 20K: A Benchmark for Proactive and Personalized Mobile LLM Agents (ICLR 2026 · similarity 0.89)
- ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence On Mobile Devices (CVPR 2026 · similarity 0.88)
- HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses Through Reasoning MLLMs (AAAI 2026 · similarity 0.87)
- ProAct: A Benchmark and Multimodal Framework for Structure-Aware Proactive Response (ICML 2026 · similarity 0.87)
- UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.