Research map / Multi-agent systems and games / LLM agents
Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ProactiveBench, a simulated and human-annotated benchmark plus reward model for training LLM agents to proactively predict and offer tasks from user activity, with fine-tuned 7-8B models outperforming GPT-4o on F1.
Problem
Most LLM agents are reactive and wait for explicit user instructions; there is no data-driven way to train or evaluate agents that anticipate user needs and proactively offer assistance from observed activity.
Approach
Combine an LLM-driven environment gym that simulates user activities, events and agent-task execution with a reward model trained on human accept/reject annotations of predicted tasks, which serves as an automatic user simulator for generating training data and evaluating proactiveness.
Key results
- Fine-tuned Qwen2-7B-Instruct reaches 66.47% F1 on ProactiveBench test, versus 64.60% for GPT-4o and 60.74% for base Qwen2-7B.
- The reward model achieves 91.80% F1 and 91.67% accuracy against human majority-vote judgments, versus 67.04% F1 for GPT-4o as reward model.
- Fine-tuned LLaMA-3.1-8B improves F1 from 55.06% to 66.25% on ProactiveBench.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune LLaMA-3.1-8B on ProactiveBench | LLaMA-3.1-8B proactive agent · ProactiveBench test set | F1-Score: 55.06% -> 66.25% | Helped |
| Fine-tune Qwen2-7B on ProactiveBench | Qwen2-7B proactive agent · ProactiveBench test set | F1-Score: 60.74% -> 66.47% | Helped |
| Fine-tune LLaMA-3.1-8B-Instruct on human-annotated judgments | LLaMA-3.1-8B reward model · Reward model test set | F1-Score: 58.46% -> 91.80% | Helped |
| Predict three candidate tasks instead of one (ablation) | GPT-4o-mini · ProactiveBench | F1-Score: 52.15% -> 78.80% | Helped |
| Predict three candidate tasks instead of one (ablation) | GPT-4o · ProactiveBench | F1-Score: 64.60% -> 77.72% | Helped |
| Predict three candidate tasks instead of one (ablation) | LLaMA-3.1-8B · ProactiveBench | F1-Score: 55.06% -> 69.10% | Helped |
| Add reward-model feedback (ablation) | GPT-4o-mini · ProactiveBench | F1-Score: 52.15% -> 59.22% | Helped |
| Add reward-model feedback (ablation) | GPT-4o · ProactiveBench | F1-Score: 64.60% -> 56.00% | Hurt |
| Add reward-model feedback (ablation) | LLaMA-3.1-8B · ProactiveBench | F1-Score: 55.06% -> 54.81% | No clear effect |
| Predict three candidate tasks and add reward-model feedback (ablation) | GPT-4o-mini · ProactiveBench | F1-Score: 52.15% -> 79.05% | Helped |
| Predict three candidate tasks and add reward-model feedback (ablation) | GPT-4o · ProactiveBench | F1-Score: 64.60% -> 77.53% | Helped |
| Predict three candidate tasks and add reward-model feedback (ablation) | LLaMA-3.1-8B · ProactiveBench | F1-Score: 55.06% -> 74.76% | Helped |
Surprising findings
- Existing models, including GPT-4o, tend to offer assistance too readily, yielding high false-alarm rates when the user needs no help.
- Providing feedback from the reward model made models stay silent more often, which reduced false alarms but hurt recall and F1 for GPT-4o.
Limitations
- Limited environment settings (coding, writing, daily life) and scenarios.
- Still high false-alarm rates; models offer help even when unnecessary.
- Reward model is a proxy for user judgment; evaluation is relative to simulated and small test set (233 events).
Experimental setup
Datasets: ProactiveBench training split human-annotated reward model dataset collected real-world user activity logs
Benchmarks: ProactiveBench
Metrics: Recall Precision Accuracy False-Alarm F1-Score agreement ratio
Baselines: GPT-4o GPT-4o-mini Claude-3-Sonnet Claude-3.5-Sonnet LLaMA-3.1-8B LLaMA-3.1-70B Qwen2-7B
Builds on: ActivityWatch
Closest papers by meaning
- ProAct: A Benchmark and Multimodal Framework for Structure-Aware Proactive Response (ICML 2026 · similarity 0.91)
- Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimization (ICML 2026 · similarity 0.91)
- UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks (NeurIPS 2026 · similarity 0.90)
- ProAgent: Building Proactive Cooperative Agents with Large Language Models (AAAI 2024 · similarity 0.90)
- Scaling Autonomous Agents via Automatic Reward Modeling And Planning (ICLR 2025 · similarity 0.89)
- ContextAgent: Context-Aware Proactive LLM Agents with Open-world Sensory Perceptions (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.