Research map / Large language models / Agents, code and math
An LLM Compiler for Parallel Function Calling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LLMCompiler, a compiler-inspired framework (planner, task fetching unit, executor) that plans DAGs of function calls and runs independent ones in parallel, reducing latency and cost versus ReAct.
Problem
Sequential reason-act loops such as ReAct make multi-function LLM calling slow, costly, and sometimes inaccurate due to repeated calls and premature stopping.
Approach
A Planner LLM emits a task DAG with placeholder variables for dependencies; a Task Fetching Unit substitutes outputs and dispatches ready tasks to an Executor that runs them concurrently, with optional replanning from executor feedback and streamed planning.
Key results
- On HotpotQA and Movie Recommendation, LLMCompiler achieves up to 3.74x latency speedup and 6.73x cost reduction over ReAct with gpt-3.5-turbo.
- On ParallelQA with LLaMA-2 70B, LLMCompiler improves accuracy by about 9% over ReAct (68.14 vs 59.59) with 2.27x speedup.
- On WebShop with gpt-3.5-turbo, success rate rises from 19.8 (ReAct) to 48.2 for LLMCompiler (500 examples).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Execute independent function calls in parallel with LLMCompiler rather than sequentially with ReAct | HotpotQA comparison dev, GPT · HotpotQA comparison dev | latency: 7.12 -> 3.95; 1.80× | Helped |
| Execute independent function calls in parallel with LLMCompiler rather than sequentially with ReAct | Movie Recommendation, GPT · Movie Recommendation | cost reduction: 3.37× | Helped |
| Use LLMCompiler instead of ReAct† | ParallelQA, LLaMA-2 70B · ParallelQA | accuracy: 59.59 -> 68.14 | Helped |
| Use LLMCompiler replanning and parallel execution instead of Tree-of-Thoughts | Game of 24, GPT-4 · Game of 24 | latency: 241.2 -> 83.6; 2.89× | Helped |
| Explore all ten search results using parallel function calls | WebShop, gpt-3.5-turbo · WebShop | success rate: 19.8 -> 48.2 | Helped |
| Add ReAct-specific prompting to reduce early stopping (ablation) | ReAct on Movie Recommendation, GPT · Movie Recommendation | accuracy: 68.60 -> 72.47 | Helped |
| Add planner streaming so tasks execute as soon as produced (ablation) | LLMCompiler on ParallelQA · ParallelQA | latency: 21.72 -> 16.69; 1.30× | Helped |
| Add planner streaming so tasks execute as soon as produced (ablation) | LLMCompiler on HotpotQA · HotpotQA | latency: 4.00 -> 3.95; 1.01× | Helped |
| Add planner streaming so tasks execute as soon as produced (ablation) | LLMCompiler on Movie Recommendation · Movie Recommendation | latency: 5.64 -> 5.47; 1.03× | Helped |
| Avoid repetitive function calls with LLMCompiler instead of ReAct | HotpotQA examples with more than four ReAct calls, LLaMA-2 70B · HotpotQA | accuracy: less than 10% -> around 50% | Helped |
| Replace ReAct with LLMCompiler on HotpotQA comparison dev | HotpotQA comparison dev, LLaMA-2 70B · HotpotQA comparison dev | accuracy: 54.40 -> 57.83 | Helped |
| Use LLMCompiler instead of ReAct† on WebShop | WebShop, gpt-3.5-turbo · WebShop | latency: 5.98 -> 10.48 | Hurt |
Surprising findings
- On WebShop, LLMCompiler is slightly slower than ReAct (10.48s vs 5.98s), attributed to planner overhead.
- Prompting ReAct to avoid early stopping and repeated calls helps but does not fully fix these failures.
Limitations
- Parallel speedup limited by planner and final-answer overhead and straggler tasks.
- Planner errors and executor/tool errors (e.g., math tool) still cause failures; ParallelQA failures mostly from executor.
- Latency slightly worse than ReAct on WebShop due to planning overhead.
Experimental setup
Datasets: HotpotQA Movie Recommendation ParallelQA Game of 24 WebShop
Benchmarks: HotpotQA Movie Recommendation ParallelQA Game of 24 WebShop
Metrics: accuracy end-to-end latency speedup token cost success rate score
Baselines: ReAct OpenAI parallel function calling Tree-of-Thoughts LATS LASER ReWOO TPTU HuggingGPT ViperGPT
Builds on: ReAct Tree-of-Thoughts OpenAI parallel function calling Skeleton-of-Thought
Closest papers by meaning
- The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models (ICML 2025 · similarity 0.87)
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.86)
- REASONING COMPILER: LLM-Guided Optimizations for Efficient Model Serving (NeurIPS 2025 · similarity 0.85)
- Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuning (ICLR 2025 · similarity 0.85)
- EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code (NeurIPS 2025 · similarity 0.84)
- Hogwild! Inference: Parallel LLM Generation via Concurrent Attention (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.