Research map / Multi-agent systems and games / LLM agents

Graph2Eval: Automatic Multimodal Task Generation for Agents via Knowledge Graphs

CVPR 2026 · Multi-agent systems and games › LLM agents · approach: Robust and efficient agent execution · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Graph2Eval, a knowledge-graph-driven pipeline that generates document-understanding (RAG) and web-interaction agent tasks via subgraph sampling, templates, and meta-paths, and releases the 1,319-task Graph2Eval-Bench.

Problem

Automatically generated agent tasks suffer from semantic inconsistency and poor solvability because LLMs generate them from preprocessed text/images without explicit entity-relation modeling, and web task generation fails to model page relationships.

Approach

Treat a knowledge graph built from documents and web pages as a latent task space: sample subgraphs guided by task templates (documents) or seed nodes and meta-path patterns (web), then have LLMs instantiate tasks, followed by reachability, LLM-scoring, and similarity-based coverage filtering.

Key results

• Graph2Eval improves task semantic consistency by 20% and solvability by 17% on average over the KG-free baseline (human evaluation: consistency 0.95 vs 0.74 doc; solvability 0.93 vs 0.73 doc).
• On web tasks, Agent S 2.5 with Gemini 2.5 Flash reaches 69.20% overall SR versus 14.51% for SoM Agent with the same model.
• Graph2Eval-Bench contains 1,319 tasks (1,002 document understanding, 317 web interaction) generated in about 34.87 s per doc task and 95.51 s per web task.

Surprising findings

• Agent S 2.5 vastly outperforms SoM Agent on the same models, suggesting tasks emphasize multi-step reasoning over visual grounding.
• On document tasks, GPT-4o leads on F1/ROUGE-L while Deepseek-V3 leads on LLM Judge, showing metric disagreement.

Limitations

• Depends on accurate knowledge graph construction; incomplete or erroneous graphs degrade task quality.
• Focuses on structured knowledge and predefined relations, limiting unstructured or novel content coverage.
• Web task success is judged by an LLM evaluator, validated only on 50 trajectories with 88% agreement.

Experimental setup

Datasets: academic papers (16 documents) web pages from 8 websites

Benchmarks: Graph2Eval-Bench

Metrics: F1 ROUGE-L LLM Judge Success Rate Consistency Solvability

Baselines: KG-free Graph2Eval variant TaskCraft OSWorld MMBench

Builds on: Set-of-Marks prompting Agent S TaskCraft all-MiniLM-L6-v2

Closest papers by meaning

• Graph-of-Agents: A Graph-based Framework for Multi-Agent LLM Collaboration (ICLR 2026 · similarity 0.88)
• InfoMosaic-Bench: Evaluating Multi-Source Information Seeking in Tool-Augmented Agents (ICLR 2026 · similarity 0.87)
• KARMA: Leveraging Multi-Agent LLMs for Automated Knowledge Graph Enrichment (NeurIPS 2025 · similarity 0.87)
• MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chains (ICLR 2026 · similarity 0.86)
• Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge (NeurIPS 2025 · similarity 0.86)
• TaskBench: Benchmarking Large Language Models for Task Automation (NeurIPS 2024 · similarity 0.86)