Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning

Epiplexity Guided Data Selection and Generation for Out-of-Distribution Generalization

NeurIPS 2026 · Label-efficient and robust learning › Zero-/few-shot and meta-learning · approach: Fine-tuning strategies for few-shot · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Operationalizes epiplexity (prequential estimate of learnable structure) as an online signal for domain data selection via fitted cross-domain scaling laws (EpiSelect) and for REINFORCE-trained synthetic data generation (EpiGen).

Problem

How to select or generate training data that transfers to out-of-distribution tasks, and how to tractably estimate epiplexity during training to use it as a signal.

Approach

Estimate epiplexity as the area between per-domain training loss and the current loss; fit cross-domain scaling laws to forecast the marginal epiplexity gain per domain to set sampling weights (EpiSelect), and use the step-wise drop in learner loss over a replay buffer of generated data as a REINFORCE reward for a generator (EpiGen).

Key results

• EpiSelect achieves 0.394 average zero-shot accuracy at 124M (vs 0.379 ADO, 0.377 Natural) and 0.431 at 1.3B (vs 0.425 ADO).
• EpiGen raises average GLUE score from 0.743 (Pretrained GPT-2) to 0.770, while the PPL-reward and NoBuffer ablations reach 0.756 and 0.757.
• Epiplexity correlates with mean zero-shot accuracy across five Pile domain models (Pearson r=0.88, p≈0.05), while weight norm does not (r=0.01).

Surprising findings

• Training only on the PileCC domain of The Pile outperforms the state-of-the-art ADO selector, so the Pile is saturated as a selection benchmark.
• EpiGen gives essentially no GLUE gain (-0.003) when generator and learner are randomly initialized, so a pretrained initialization is necessary.

Limitations

• Experiments are limited to small models (124M, 1.3B for selection; GPT-2 117M for generation).
• The prequential epiplexity proxy is used rather than the true epiplexity; no theoretical guarantee that maximizing the proxy maximizes epiplexity.
• EpiGen learner drifts from natural-language distribution, with much higher perplexity on OpenWebText, SlimPajama and FineWeb for the PPL and NoBuffer variants.

Experimental setup

Datasets: The Pile (uncopyrighted subset) Common Pile OpenWebText GLUE SlimPajama FineWeb

Benchmarks: LM Evaluation Harness zero-shot (ARC, BBQ, BoolQ, CSQA, HellaSwag, LAMBADA, OBQA, PIQA, SciQ, WinoGrande) GLUE (CoLA, SST-2, MRPC, QQP, MNLI, QNLI, RTE) OpenWebText perplexity SlimPajama and FineWeb perplexity

Metrics: zero-shot accuracy average accuracy GLUE score (MCC for CoLA, accuracy otherwise) perplexity Pearson correlation

Baselines: Natural ADO DoReMi Pretrained FrozenGen PPL reward NoBuffer reward

Builds on: epiplexity ADO DoReMi REINFORCE prequential coding

Closest papers by meaning

• Understanding Generalization Requires Universal Induction (NeurIPS 2026 · similarity 0.85)
• Slower Generalization, Faster Memorization: A Sweet Spot in Algorithmic Learning (NeurIPS 2026 · similarity 0.85)
• Self-Consuming Generative Models with Co-Evolving Human Preferences (NeurIPS 2026 · similarity 0.84)
• The two clocks and the innovation window: When and how generative models learn rules (NeurIPS 2026 · similarity 0.84)
• Towards Generalization beyond Pointwise Learning: A Unified Information-theoretic Perspective (ICML 2024 · similarity 0.84)
• G-Zero: Self-Play for Open-Ended Generation from Zero Data (NeurIPS 2026 · similarity 0.84)