Research map / Multi-agent systems and games / LLM agents

GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging

AAAI 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces GitTaskBench, a 54-task benchmark for code agents that must leverage GitHub repositories (including environment setup) to solve real-world multimodal tasks, plus an alpha-value metric for economic benefit, evaluated across Aider, OpenHands, and SWE-Agent with multiple LLMs.

Problem

Existing code-agent benchmarks focus on isolated code generation or program repair and rarely test whether agents can autonomously set up and repurpose open-source repositories to complete end-to-end, user-centric tasks.

Approach

A benchmark of 54 repository-grounded daily-life tasks across 7 modalities, each with human-curated automated test scripts (ECR and TPR), plus an alpha-score that nets agent success, quality, and API cost against freelance market value of the task.

Key results

• OpenHands with Claude 3.7 achieves the best ECR of 72.22% and TPR of 48.15% among the original evaluated settings.
• Error analysis attributes 65.04% of failures to environment setup (E1).
• Raising OpenHands (GPT-4o) timeout from 120s to 1800s increases TPR from 12.96% to 35.19%, with cost rising from $0.68 to $6.52.

Surprising findings

• Replacing humans with agents is not always cost-effective; DeepSeek V3 and GPT-4.1 offer better cost-performance than Claude 3.7 despite lower success.
• Qwen3-32B in think mode reaches up to about 60% of Claude 3.5's performance with far fewer tokens.

Limitations

• OpenHands does not strictly enforce max_iterations, complicating step control.
• Market values for alpha are estimated from freelance platform fees and may be imprecise.
• Benchmark is small at 54 tasks across 18 repositories.

Experimental setup

Datasets: GitHub repositories (18 Python projects)

Benchmarks: GitTaskBench

Metrics: Execution Completion Rate (ECR) Task Pass Rate (TPR) alpha-score token cost

Baselines: Aider SWE-Agent OpenHands

Builds on: SWE-Bench OpenHands SWE-Agent Aider

Closest papers by meaning

• RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades (NeurIPS 2026 · similarity 0.91)
• SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks (NeurIPS 2026 · similarity 0.90)
• Automated Benchmark Generation for Repository-Level Coding Tasks (ICML 2025 · similarity 0.90)
• NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents (ICML 2026 · similarity 0.90)
• SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution (NeurIPS 2026 · similarity 0.90)
• RepoMaster: Autonomous Exploration and Understanding of GitHub Repositories for Complex Task Solving (NeurIPS 2025 · similarity 0.90)