Research map / Multi-agent systems and games / LLM agents

Toward Interactive Understanding of Code APIs

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces PAU, a benchmark for interactive black-box understanding of Python methods, and adapts asymmetric actor-critic RL (GRPO with a privileged-information critic) to tune Qwen3-8B so it explores APIs better, matching GPT-5-mini.

Problem

Language model agents are evaluated mostly with read access to code, but external APIs are black boxes; it is unclear whether LMs can understand a method's functionality purely through querying it with inputs and observing outputs.

Approach

A benchmark of 741 test Python methods with black-box API access, scored by an LM judge and a description-to-code match; plus an asymmetric actor-critic RL setup where a frozen critic with the gold description (same backbone as actor) gives step-level rewards for exploration, trained with GRPO.

Key results

• Best frontier model Claude-4-Opus fails (score 2 or lower) to describe over 45% of PAU test methods in the interactive setting.
• AAC-tuned Qwen3-8B reaches description score 2.591, rivalling GPT-5-mini at 2.595; AAC improves average score by +0.13 and code match by +6.27% across base models.
• Supervised fine-tuning on PAU training data performed worse than base models in the in-context setting.

Surprising findings

• Most models show no significant improvement from interactive API access over static in-context examples; only Claude models improve significantly.
• Failures are dominated by overconfidence and insufficient exploration, with 43.3% of failures from hypotheses contradicted by already-observed examples.

Limitations

• Only stateless Python methods are covered; memory and stateful APIs are not evaluated.
• Test methods come from popular benchmarks and may be contaminated in LM pretraining data.
• Some test instances may be impossible to recover via black-box interaction; semantic equivalence metrics are imperfect.

Experimental setup

Datasets: HumanEval MBPP CruxEval CodeAlpaca

Benchmarks: PAU

Metrics: description score (1-5 LM-as-a-judge) code match score statement and branch coverage

Baselines: GPT-5-mini Claude-4-Opus Claude-4-Sonnet Gemini-3-Flash GPT-4o DeepSeek-V3 Gemma4-31B Llama3-70B

Builds on: GRPO Asymmetric Actor Critic ReAct PPO

Closest papers by meaning

• BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.88)
• Frontier Coding Agents Use Metaprogramming to Adapt to Unfamiliar Programming Languages (NeurIPS 2026 · similarity 0.88)
• R2E: Turning any Github Repository into a Programming Agent Environment (ICML 2024 · similarity 0.87)
• ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents (ICLR 2025 · similarity 0.87)
• LibEvoBench: Probing Temporal Knowledge Stratification in Code Generation Models (NeurIPS 2026 · similarity 0.87)
• ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (ICLR 2024 · similarity 0.86)