Research map / Large language models / Agents, code and math
Lemur: Harmonizing Natural Language and Code for Language Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Lemur and Lemur-Chat, Llama-2-70B continually pre-trained on a 10:1 code-to-text corpus (90B tokens) and instruction-tuned on ~300K text and code examples to balance natural language and coding for language agents.
Problem
Open-source LLMs specialize in either natural language or code, but language agents need both to reason, communicate, and execute grounded actions in environments.
Approach
Continually pre-train Llama-2-70B on a code-intensive corpus (about 90% code, mostly scripting languages) mixed with text, then instruction-tune on text and code data, and evaluate balanced capability across text, code, and multi-turn agent benchmarks.
Key results
- Lemur-70B reports the highest average (47.9) across text and code benchmarks, versus 43.6 for Llama-2-70B and 46.0 for Code Llama-34B.
- Lemur-Chat-70B reaches an average of 55.0 versus 45.6 for Code Llama-34B-Instruct and 40.2 for Llama-2-70B-Chat.
- On InterCode-SQL, Lemur-Chat-70B reaches 73.79 success rate, above GPT-3.5-turbo (72.82) and far above Llama-2-70B-Chat (67.89); on RoboCodeGen it reaches 75.68 versus 83.78 for GPT-3.5-turbo.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| continue-pre-train Llama-2-70B on code-intensive text-and-code data | Llama-2-70B · Text and code benchmarks (average) | average: 43.6 -> 47.9 | Helped |
| instruction fine-tune Lemur on text and code instructions | Lemur · Text and code benchmarks (average) | average: 40.2 -> 55.0 | Helped |
| use Lemur-Chat rather than Llama-2-70B-Chat | Llama-2-70B-Chat · M-GSM8K | success rate: 33.33 -> 58.33 | Helped |
| use Lemur-Chat rather than Llama-2-70B-Chat | Llama-2-70B-Chat · M-HumanEval | success rate: 8.89 -> 46.67 | Helped |
| add GPT-4 natural-language feedback | Lemur-Chat · MINT reasoning and code generation | micro average: 30.31 -> 38.50 | Helped |
| map WebArena actions to Python representations instead of directly predicting predefined actions | Lemur-Chat · WebArena | performance | Helped |
| use Lemur-Chat rather than other open-source models | open-source language agents · 13 agent benchmarks | performance: 12 of the 13 agent benchmarks | Helped |
| increase interaction turns to incorporate environment feedback | Lemur-Chat · InterCode-SQL | success rate | Helped |
Surprising findings
- Code Llama-34B-Instruct fails to follow environment feedback on M-HumanEval and M-MBPP due to overfitting to the [PYTHON] tag, scoring near 2%.
Limitations
- Data mixture ratio (10:1) chosen from limited continued pre-training study due to computational limits; not systematically optimized.
- Smaller models may suffer more forgetting of natural language knowledge when harmonizing text and code.
- Still well behind GPT-4 on agent tasks, especially multi-turn interactive and domain-specific tasks such as cybersecurity CTF.
Experimental setup
Datasets: The Stack RefinedWeb RedPajama CommonCrawl Wikipedia Books ArXiv StackExchange DM Mathematics Open Assistant 1 OpenOrca ShareGPT
Benchmarks: MMLU BBH GSM8K HumanEval MBPP Spider MultiPL-E DS-1000 MINT-GSM8K MINT-MATH MINT-TheoremQA MINT-HotpotQA
Metrics: accuracy pass@1 success rate score delta feedback
Baselines: Llama-2-70B Llama-2-70B-Chat Code Llama-34B Code Llama-34B-Instruct StarCoder-15B StarCoderPlus-15B WizardCoder-15B GPT-3.5-turbo GPT-4 text-bison-001
Builds on: Llama 2 The Stack EasyLM MINT InterCode
Closest papers by meaning
- Large Language Models Miss the Multi-agent Mark (NeurIPS 2025 · similarity 0.87)
- Communicating Activations Between Language Model Agents (ICML 2025 · similarity 0.87)
- Building Cooperative Embodied Agents Modularly with Large Language Models (ICLR 2024 · similarity 0.87)
- \$OneMillion-Bench: How Far are Language Agents from Human Experts? (NeurIPS 2026 · similarity 0.87)
- PACE: Two-Timescale Self-Evolution for Small Language Model Agents (NeurIPS 2026 · similarity 0.87)
- RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Code (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.