Research map / Large language models / Agents, code and math
QAagent: A Multiagent System for Unit Test Generation via Natural Language Pseudocode (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QAagent, a multi-agent LLM system that first generates natural-language pseudocode for a function and then uses it to generate unit tests covering base and edge cases, evaluated on HumanEval and MBPP for coverage and accuracy.
Problem
Automated unit test generation is time-consuming and current LLM-based test creation lacks accuracy and code coverage for reliable test-driven development and code generation pipelines.
Approach
Insert an LLM-written natural-language pseudocode plan (code architect agent, chain-of-thought) into the context before a test generator agent produces base and edge-case tests, so the tests better cover the function's logic without the model writing the implementation.
Key results
- QAagent reaches 96.2% (first five tests) / 96.6% (all tests) line coverage on HumanEval, vs 84.7% / 87.5% for AgentCoder.
- On MBPP QAagent gets 87.8% / 87.9% coverage, but only 52.7% test accuracy vs 89.9% for AgentCoder.
- QAagent's test accuracy on HumanEval is 88.6%, slightly above AgentCoder's 87.8%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use QAagent instead of AgentCoder | unit-test generation · HumanEval, first five test cases | line coverage: 84.7 -> 96.2 | Helped |
| Use QAagent instead of AgentCoder | unit-test generation · HumanEval, all test cases | line coverage: 87.5 -> 96.6 | Helped |
| Use QAagent instead of AgentCoder | unit-test generation · MBPP, first five test cases | line coverage: 85.3 -> 87.8 | Helped |
| Use QAagent instead of AgentCoder | unit-test generation · MBPP, all test cases | line coverage: 89.5 -> 87.9 | Hurt |
| Use QAagent instead of AgentCoder | unit-test generation · HumanEval | accuracy: 87.8 -> 88.6 | Helped |
| Use QAagent instead of AgentCoder | unit-test generation · MBPP | accuracy: 89.9 -> 52.7 | Hurt |
Surprising findings
- QAagent's test accuracy on MBPP (52.7%) is far below AgentCoder (89.9%) and GPT-4 Turbo (42.6%) only barely lower, despite higher coverage.
Limitations
- Test accuracy on MBPP is substantially lower than competing methods.
- Evaluation uses code-generation benchmarks rather than dedicated test-generation benchmarks.
- Relies on the canonical solution for execution-based evaluation and a single base LLM (GPT-4 Turbo).
Experimental setup
Datasets: HumanEval MBPP
Benchmarks: HumanEval MBPP
Metrics: line coverage test accuracy
Baselines: GPT-4 Turbo (zero-shot) CodeCoT AgentCoder
Builds on: chain-of-thought prompting AgentCoder CodeCoT
Closest papers by meaning
- SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents (NeurIPS 2024 · similarity 0.88)
- Rethinking Verification for LLM Code Generation: From Generation to Testing (NeurIPS 2025 · similarity 0.86)
- R2E: Turning any Github Repository into a Programming Agent Environment (ICML 2024 · similarity 0.86)
- Training Language Models to Generate Quality Code with Program Analysis Feedback (NeurIPS 2025 · similarity 0.86)
- CodeChemist: Test-Time Scaling for Low-Resource Code Generation via Functional Knowledge Transfer (ICML 2026 · similarity 0.85)
- WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.