Research map / Large language models / Language, safety and interpretability

LLM Augmented LLMs: Expanding Capabilities through Composition

ICLR 2024 · Large language models › Language, safety and interpretability · approach: Multilingual LLM data and culture · 10 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CALM, which composes a frozen anchor LLM (PaLM2-S) with a frozen augmenting model via trainable projections and cross-attention over intermediate layers, enabling new capabilities in low-resource translation, math, and code tasks.

Problem

Augmenting a large monolithic foundation model with new skills from specialized models is expensive, and existing routing, ensembling, merging, or further pre-training approaches either fail when neither model has the capability or cause catastrophic forgetting.

Approach

Keep both anchor and augmenting models frozen and learn a small set of linear projections plus cross-attention layers that let the anchor attend over the augmenting model's intermediate representations, trained on a small amount of composition data (about 5-20% of the domain data).

Key results

• On FLORES-200 XX-En across 10 low-resource languages, CALM averages 46.1 chrF1 vs 41.7 for PaLM2-S, and improves over PaLM2-S on 175 of 192 languages.
• On the code tasks, CALM reaches 22.5 Pass@1 on HumanEval (vs 16.4 for PaLM2-S, a 6.1-point absolute gain) and 32.2 on MBPP (vs 28.6).
• On the synthetic key-value arithmetic task, the composed model reaches 84.3% accuracy on KV-MATH where both base models score near 0%.

Measured effects reported in the paper

Surprising findings

• Fine-tuning PaLM2-S on code data causes catastrophic forgetting on code-to-text (chrF1 drops to 18.9 on Python), while CALM retains and slightly improves it.
• An un-specialized vanilla PaLM2-XXS used as the augmenting model still gives a slight gain over PaLM2-S on FLORES-200 (102 languages improved vs 175 for the specialized model).

Limitations

• Experiments use proprietary PaLM2 models, limiting reproducibility.
• Only one augmenting model is composed in experiments; multi-model composition is left to future work.
• Silver-standard low-resource GSM-8K was produced by machine translation and only quality-checked via back-translation.

Experimental setup

Datasets: Key-value synthetic data (KV-SUBS, KV-MATH, NUM-MATH) Next Thousand Languages (NTL) corpora GitHub open-source code corpus FLORES-200 GSM-8K (multilingual and silver low-resource translations) HumanEval MBPP CodeXGLUE

Benchmarks: FLORES-200 GSM-8K HumanEval MBPP CodeXGLUE Key-value arithmetic synthetic task

Metrics: chrF1 accuracy Pass@1 number of languages improved over base

Baselines: PaLM2-XXS + NTL PaLM2-S + NTL (continued pre-training) LoRA PaLM2-S + Code (continued pre-training) vanilla PaLM2-XXS augmenting random PaLM2-XXS augmenting encoder-only augmenting variant

Builds on: LoRA encoder-decoder cross-attention Flamingo-style cross-attention

Closest papers by meaning

• CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets (ICLR 2024 · similarity 0.90)
• Knowledge Fusion of Large Language Models (ICLR 2024 · similarity 0.90)
• Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.90)
• Self-MoE: Towards Compositional Large Language Models with Self-Specialized Experts (ICLR 2025 · similarity 0.89)
• metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.89)
• ELICIT: LLM Augmentation Via External In-context Capability (ICLR 2025 · similarity 0.89)