Research map / Large language models / Language, safety and interpretability

Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minima

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Pruning and circuits in LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Nexus, a dual-loop gradient approximator that maximizes cross-source gradient cosine similarity on top of AdamW, yielding near-identical pretraining loss but better downstream benchmark and OOD loss for LLM pretraining.

Problem

Standard LLM pretraining minimizes the averaged loss over heterogeneous data sources, but converged models with identical pretraining loss can generalize very differently downstream; the paper asks how to steer pretraining toward minimizers close to each source's task minimizer.

Approach

Use gradient cosine similarity across data sources as a tractable upper bound on parameter closeness to per-task minimizers; approximate the gradient of this similarity with an inner loop of normalized SGD steps over mini-batches whose displacement serves as a pseudo-gradient for an outer optimizer (AdamW).

Key results

• On the 3B model, Nexus matches AdamW's pretraining loss (1.602 vs 1.606) while improving GSM8k accuracy by +15.0 (59.0 vs 44.0), MATH by +8.0, and HumanEval by +4.0; average accuracy 40.3 vs 37.1.
• Average benchmark gain grows with model size: +0.8% (130M), +1.5% (300M), +1.7% (520M), +2.6% (1.2B), +3.2% (2.3B).
• At 4x Chinchilla tokens on 3B, Nexus averages 41.7 vs 38.0 for AdamW, with the gap persisting.

Surprising findings

• Nexus-Dot (unnormalized dot product) substantially slows pretraining loss minimization, so cosine similarity was used instead.
• Muon does not show a favorable implicit bias: its curve nearly overlaps AdamW on the pretraining-vs-downstream loss plot, and Muon combined with Nexus underperforms AdamW-Nexus.

Limitations

• Nexus is incompatible with Muon in current experiments; Muon+Nexus underperforms AdamW-Nexus on downstream tasks.
• Nexus does not drive all task minima to intersect; achieves only moderate reduction in distance to task minimizers.
• Main experiments use a proprietary in-house corpus, and public-dataset results are contaminated with benchmark data.

Experimental setup

Datasets: in-house strictly decontaminated pretraining corpus public pretraining datasets (appendix experiments)

Benchmarks: MMLU GPQA GPQA Diamond BBH GSM8k MATH500 HumanEval MBPP

Metrics: accuracy loss pretraining validation loss OOD loss gradient similarity parameter closeness

Baselines: AdamW Muon Nexus-Dot NSGD

Builds on: AdamW Muon Reptile-style meta-learning inner-outer loops mu-P Chinchilla scaling

Closest papers by meaning

• Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoning (NeurIPS 2025 · similarity 0.89)
• Data Engineering for Scaling Language Models to 128K Context (ICML 2024 · similarity 0.88)
• The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.88)
• An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence (NeurIPS 2026 · similarity 0.88)
• Chameleon: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuning (ICML 2025 · similarity 0.88)
• Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining (ICLR 2025 · similarity 0.88)