Research map / Large language models / Agents, code and math
REASONING COMPILER: LLM-Guided Optimizations for Efficient Model Serving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Reasoning Compiler, which combines LLM-generated transformation proposals (with parent/grandparent schedule history and cost-model feedback) with Monte Carlo tree search to optimize tensor programs in TVM MetaSchedule with far fewer samples than TVM's evolutionary search, without any retraining.
Problem
Compiler autotuning for neural network layers has a huge, interdependent transformation space, and existing stochastic/evolutionary search methods are sample-inefficient and ignore contextual dependencies between transformations.
Approach
Cast compiler optimization as a finite-horizon MDP searched by UCT-based MCTS, where an off-the-shelf LLM (no fine-tuning) acts as a context-aware proposal engine that reads the code, schedule trace, and predicted costs of the node and its ancestors and proposes the next transformation sequence.
Key results
- Across 25 platform-operator pairs, the Reasoning Compiler achieves 5.0x geomean speedup using 5.8x fewer samples than TVM, a 10.8x sample efficiency gain.
- On end-to-end Llama-3-8B, it uses 3.9x fewer samples to reach 4.0x geomean speedup, a 5.6x sample efficiency improvement over TVM.
- On the Llama-4-Scout MLP layer on Intel Core i9, it reaches 12.7x speedup at 20 samples while TVM reaches 5.6x with 230 samples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LLM-guided MCTS instead of TVM search | Layer-wise neural compiler optimization · Layer-wise benchmarks, geomean across 25 platform-operator pairs | speedup: 2.7× -> 5.0× | Helped |
| Use LLM-guided MCTS instead of TVM search | End-to-end Llama 3 optimization · Llama 3, geomean across five hardware platforms | speedup: 2.8× -> 4.0× | Helped |
| Add the great-grandparent schedule to the prompt context (ablation) | REASONING COMPILER with GPT-4o mini · Llama 3 attention layer | speedup: 11.33 -> 11.87 | Helped |
| Add the great-grandparent schedule to the prompt context (ablation) | REASONING COMPILER with GPT-4o mini · DeepSeek MoE layer | speedup: 9.45 -> 10.65 | Helped |
| Add the great-grandparent schedule to the prompt context (ablation) | REASONING COMPILER with GPT-4o mini · Flux attention layer | speedup: 5.43 -> 5.61 | Helped |
| Add the great-grandparent schedule to the prompt context (ablation) | REASONING COMPILER with GPT-4o mini · Flux convolution layer | speedup: 2.47 -> 2.50 | Helped |
| Increase MCTS branching factor from 2 to 4 (ablation) | REASONING COMPILER · Llama 3 attention layer | speedup: 11.33 -> 10.99 | Hurt |
| Increase MCTS branching factor from 2 to 4 (ablation) | REASONING COMPILER · DeepSeek MoE layer | speedup: 9.45 -> 9.10 | Hurt |
| Increase MCTS branching factor from 2 to 4 (ablation) | REASONING COMPILER · Flux attention layer | speedup: 5.43 -> 5.55 | Mixed |
| Increase MCTS branching factor from 2 to 4 (ablation) | REASONING COMPILER · Flux convolution layer | speedup: 2.47 -> 2.43 | Hurt |
| Use Llama3.3-Instruct (70B) as the proposal model instead of GPT-4o mini (ablation) | REASONING COMPILER · Llama 3 attention layer | speedup: 11.33 -> 9.81 | Mixed |
| Use DeepSeek-Distill-Qwen (32B) as the proposal model instead of GPT-4o mini (ablation) | REASONING COMPILER · DeepSeek MoE layer | speedup: 9.45 -> 12.02 | Helped |
Surprising findings
- Smaller open-source LLMs (e.g., 8B) show fallback rates of 10.5% and 17.2% for invalid proposals, while larger and proprietary models show near zero.
Limitations
- Depends on external LLM APIs, raising cost and reproducibility concerns.
- Evaluated only on five kernels and one end-to-end model; performance varies across LLMs.
- Smaller open-source models have high fallback rates to non-LLM expansion.
Experimental setup
Datasets: Llama-3-8B self-attention layer kernel DeepSeek-R1 MoE layer kernel FLUX attention layer kernel FLUX convolution layer kernel Llama-4-Scout MLP layer kernel end-to-end Llama-3-8B
Benchmarks: Llama-3-8B attention layer DeepSeek-R1 MoE layer FLUX attention layer FLUX convolution layer Llama-4-Scout MLP layer end-to-end Llama-3-8B
Metrics: speedup over unoptimized code number of samples sample efficiency (speedup per sample)
Baselines: TVM MetaSchedule (evolutionary search) MCTS without LLM guidance
Builds on: Monte Carlo tree search UCT TVM MetaSchedule chain-of-thought prompting
Closest papers by meaning
- HYSYNTH: Context-Free LLM Approximation for Guiding Program Synthesis (NeurIPS 2024 · similarity 0.90)
- Puzzle: Distillation-Based NAS for Inference-Optimized LLMs (ICML 2025 · similarity 0.90)
- MILES: Modular Instruction Memory with Learnable Selection for Self-Improving LLM Reasoning (NeurIPS 2026 · similarity 0.90)
- The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility (NeurIPS 2026 · similarity 0.89)
- Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference (NeurIPS 2025 · similarity 0.89)
- xLSTM 7B: A Recurrent LLM for Fast and Efficient Inference (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.