Research map / Large language models / Language, safety and interpretability

Paloma: A Benchmark for Evaluating Language Model Fit

NeurIPS 2024 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Paloma, a decontaminated, stratified perplexity benchmark over 546 English and code domains from 16 sources, plus six controlled 1B baseline LMs pretrained on different corpora.

Problem

Language model fit is usually reported as perplexity on one monolithic held-out set, which hides how models fit many distinct domains; existing perplexity evaluations also lack decontamination, fixed training order, stratified subsampling, and fixed vocabulary.

Approach

A fine-grained perplexity benchmark that treats domains as separate evaluation units, with standardized controls (sub-document decontamination against pretraining corpora, fixed training order, stratified per-domain subsampling, fixed GPT-NeoX vocabulary, no concatenation at inference), and released 1B baseline models trained under these controls.

Key results

• Baselines trained only on Common Crawl (C4, mC4-en, Falcon RefinedWeb) show extreme per-domain gaps, e.g., C4 model reaches perplexity up to 391,171 on arXiv in RedPajama.
• Few vocabulary types dominate loss: 5% of types cover roughly 50% of overall perplexity.
• No single Paloma perplexity source correlates well with all eight downstream tasks in Spearman rank analysis.

Measured effects reported in the paper

Surprising findings

• Models trained on Common Crawl only show erratic, non-monotonic perplexity on some domains despite lower aggregate perplexity.
• Perplexity on the same model differs hugely across overlapping domains from different sources (arXiv in RedPajama vs peS2o in Dolma), so perplexity reflects preprocessing as well as model knowledge.

Limitations

• Scope is restricted to English and code data.
• Domains are defined heuristically by metadata and overlapping domains across sources are treated as distinct.
• Decontamination removes whole documents and code is not decontaminated.

Experimental setup

Datasets: C4 mC4-en WikiText-103 Penn Treebank RedPajama Falcon RefinedWeb Dolma M2D2 S2ORC M2D2 Wikipedia C4-100-domains TwitterAAE Manosphere Corpus

Benchmarks: Paloma

Metrics: perplexity macro-averaged perplexity bits per byte average likelihood per vocabulary type Spearman rank correlation with downstream tasks

Baselines: C4-trained 1B LM mC4-en-trained 1B LM Falcon RefinedWeb-trained 1B LM The Pile-trained 1B LM RedPajama-trained 1B LM Dolma-trained 1B LM Pythia-7B

Builds on: The Pile M2D2 Dolma OLMo

Closest papers by meaning

• Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages (AAAI 2025 · similarity 0.90)
• metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.89)
• Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.89)
• MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures (NeurIPS 2024 · similarity 0.89)
• DataMan: Data Manager for Pre-training Large Language Models (ICLR 2025 · similarity 0.88)
• Measuring what Matters: Construct Validity in Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.88)