Research map / Large language models / Instruction and fine-tuning

ESSAM: A Novel Competitive Evolution Strategies Approach to Reinforcement Learning for Memory Efficient LLMs Fine-Tuning

NeurIPS 2026 · Large language models › Instruction and fine-tuning · approach: Fine-tuning for reasoning data · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ESSAM, a zeroth-order full-parameter fine-tuning method that adds a sharpness-aware perturbation step to evolution strategies, reaching RL-level GSM8K accuracy at inference-level GPU memory.

Problem

RL fine-tuning of LLMs for math reasoning needs very large GPU memory, while standard evolution strategies (ES) fine-tuning is memory-cheap but underperforms RL and tends toward sharp minima with poor generalization.

Approach

Combines ES with Sharpness-Aware Maximization: a first stage perturbs parameters along the reward-weighted noise direction (reverse step), then a second stage estimates the ES update at that perturbed point and applies it to the original parameters; an accelerated variant ESSAM-F uses Frobenius-normalized perturbation with a smaller SAM population.

Key results

• ESSAM averages 78.27% accuracy across 7 models on GSM8K vs PPO 77.72%, GRPO 78.34%, and ES 75.97%.
• ESSAM reduces average GPU memory 18x vs PPO and 10x vs GRPO (e.g., 2.99 GiB vs 57.32 GiB PPO on Qwen2.5-0.5B).
• ESSAM-F reaches 78.02% average accuracy with nearly 2x speedup over ESSAM at the same memory.

Surprising findings

• ESSAM-F with a smaller SAM population gives nearly 2x faster training than ESSAM at almost no accuracy cost.
• ESSAM's training time is about 2x GRPO on most models and about 3.5x on Qwen2.5-0.5B, so the memory savings come with longer wall-clock time.

Limitations

• ESSAM wall-clock training time is roughly 2x GRPO on most models and up to 3.5x on the 0.5B model.
• Evaluation is limited to GSM8K training with out-of-distribution transfer on a few math/code benchmarks and a single reward function.
• Sensitive to population size, noise scale and SAM radius hyperparameters, per the ablation on Qwen2.5-0.5B.

Experimental setup

Datasets: GSM8K

Benchmarks: GSM8K SVAMP ASDiv GSM-Hard AMC MATH code generation benchmark

Metrics: accuracy peak GPU memory training time

Baselines: PPO GRPO ES Original (no fine-tuning)

Builds on: Evolution Strategies Sharpness-Aware Minimization PPO GRPO vLLM

Closest papers by meaning

• On Predictability of Reinforcement Learning Dynamics for Large Language Models (ICLR 2026 · similarity 0.89)
• Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.89)
• Diversity-Aware Policy Optimization for Large Language Model Reasoning (NeurIPS 2025 · similarity 0.89)
• Evolution Fine-Tuning: Learning to Discover Across 371 Optimization Tasks (NeurIPS 2026 · similarity 0.89)
• STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens (NeurIPS 2026 · similarity 0.88)
• SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning (ICML 2026 · similarity 0.88)