Research map / Label-efficient and robust learning / Domain adaptation and generalization

Nemotron-CLIMB: Clustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training

NeurIPS 2025 · Label-efficient and robust learning › Domain adaptation and generalization · approach: Conditional distribution alignment · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CLIMB, an embedding-clustering plus iterative proxy-model-and-LightGBM-predictor search for pre-training data mixtures, and releases Nemotron-ClimbLab (1.2T tokens) and Nemotron-ClimbMix (400B tokens).

Problem

Finding an optimal pre-training data mixture when web-scale corpora lack explicit domain labels, and the mixture-performance relationship is nonlinear and expensive to search.

Approach

Cluster documents in embedding space (stella embeddings, FAISS k-means) to create unlabeled domains, then iteratively sample mixture weights from top predicted configurations and refit a LightGBM predictor on proxy-model results, rather than a single-round uniform sampling plus regression as in RegMix.

Key results

• The 1B model trained on 40B tokens with CLIMB mixture reaches 60.41% average on general reasoning vs 57.93% for Random and 59.37% for RegMix.
• The CLIMB 950M model trained on 400B tokens averages 53.54% vs 51.56% for Llama-3.2-1B (about 2.0% gain).
• Domain-specific search on MMLU social sciences yields about 5% improvement over Random at 350M (39.36% vs. random) per the paper's text; predictor reaches 94% Spearman correlation.

Measured effects reported in the paper

Surprising findings

• Pre-training from scratch favors a more balanced cluster distribution than continued pre-training, which concentrates weight on a few domains.
• Highly similar clusters to a downstream task are not always the most important; using only in-domain data does not necessarily give the best performance.

Limitations

• Proxy model training still incurs non-trivial compute cost.
• Domain evaluation relies on MMLU's coarse categories and does not cover high-stakes domains such as finance or healthcare.
• Phase-1 pre-training on 10T tokens is over-trained relative to scaling laws, as acknowledged by the authors.

Experimental setup

Datasets: Nemotron-CC smollm-corpus DCLM TxT360 MMLU PIQA ARC HellaSwag WinoGrande SIQA

Benchmarks: PIQA ARC-C ARC-E HellaSwag WinoGrande SIQA MMLU OBQA BoolQ RACE LAMBADA TruthfulQA

Metrics: accuracy accuracy_norm perplexity Spearman rank correlation

Baselines: Random DoReMi RegMix Llama-3.2-1B Qwen2.5-0.5B SmolLM-360M TinyLlama-1.1B AMD-OLMo-1.2B

Builds on: RegMix DoReMi WSD learning rate schedule LightGBM

Closest papers by meaning

• Chameleon: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuning (ICML 2025 · similarity 0.87)
• RegMix: Data Mixture as Regression for Language Model Pre-training (ICLR 2025 · similarity 0.87)
• Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training (ICML 2026 · similarity 0.87)
• DEPT: Decoupled Embeddings for Pre-training Language Models (ICLR 2025 · similarity 0.87)
• Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minima (NeurIPS 2026 · similarity 0.87)
• Domain2Vec: Vectorizing Datasets to Find the Optimal Data Mixture without Training (ICML 2025 · similarity 0.86)