Research map / Multi-agent systems and games / LLM agents

PPT-Eval: A Benchmark for Computer-Use Agents on PowerPoint Tasks

ICML 2026 · Multi-agent systems and games › LLM agents · approach: GUI agent memory and trajectories · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces PPT-Eval, a 120-task GUI benchmark for computer-use agents editing PowerPoint Online, scored by tree-structured rubrics with partial credit that correlate with human judgment (Kendall's tau_b 0.77).

Problem

Existing computer-use and presentation benchmarks either lack depth in PowerPoint GUI editing or rely on binary success metrics and programmatic APIs, so realistic slide editing with partial-progress evaluation is underexplored.

Approach

A GUI-based PowerPoint Online benchmark of 120 tasks over 12 decks, scored by task-specific hierarchical rubrics with critical/non-critical nodes, a modified aggregation rule that gives partial credit and penalizes extraneous edits, and LLM/VLM-generated natural-language feedback.

Key results

• Claude-4.5-Opus achieves 45% success rate and 0.57 average rubric score on PPT-Eval, versus 80% SR and 0.90 for human baseline.
• Rubric scores reach Kendall's tau_b 0.77 and Spearman's rho 0.84 with human expectation categories in meta-evaluation on 30 tasks.
• Best open-weights model OpenCUA-32B reaches 28% SR; an API-based Claude Code baseline with Opus 4.5 reaches 62% SR overall.

Surprising findings

• The API/CLI baseline (Claude Code with pptx skill) outperforms the best GUI agent overall (62% vs 45% SR), despite GUI agents having richer feature access.
• The 8B Qwen3-VL model slightly outperforms its 32B variant (16% vs 14% SR).

Limitations

• Over 81% of model-generated rubric drafts required human fixes, and rubric development took about 150 hours, limiting scalability.
• Rubrics rely on LLM/VLM calls that can hallucinate, and perfect-completion mismatches arose from VLM errors and human error.
• Evaluation focuses on PowerPoint Online only and single-app tasks.

Experimental setup

Datasets: 12 openly licensed PowerPoint decks from Internet Archive (404 slides)

Benchmarks: PPT-Eval

Metrics: success rate average rubric score Kendall's tau_b Spearman's rho category accuracy

Baselines: Claude-4.5-Opus Claude-4-Sonnet Computer-Use-Preview OpenCUA-32B OpenCUA-7B Qwen3-VL-8B Qwen3-VL-32B Claude Code API baseline

Builds on: Mind2Web 2 RubricTree OSWorld Go-Browse python-pptx SheetAgent

Closest papers by meaning

• AutoPresent: Designing Structured Visuals from Scratch (CVPR 2025 · similarity 0.85)
• GTA: A Benchmark for General Tool Agents (NeurIPS 2024 · similarity 0.84)
• Presenting a Paper is an Art: Self-Improvement Aesthetic Agents for Academic Presentations (ICLR 2026 · similarity 0.84)
• PerfGuard: A Performance-Aware Agent for Visual Content Generation (ICLR 2026 · similarity 0.84)
• AgentVista: Evaluating Multimodal Agent in Ultra-Challenging Realistic Visual Scenarios (NeurIPS 2026 · similarity 0.84)
• VideoWebArena: Evaluating Long Context Multimodal Agents with Video Understanding Web Tasks (ICLR 2025 · similarity 0.84)