Research map / Large language models / Language, safety and interpretability

EMO: EARTH MOVER DISTANCE OPTIMIZATION FOR AUTO-REGRESSIVE LANGUAGE MODELING

ICLR 2024 · Large language models › Language, safety and interpretability · approach: Linguistic encoding in LLM representations · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes EMO, which replaces or augments MLE with a differentiable upper bound of earth mover distance using embedding-based cosine transport costs, improving open-ended generation (MAUVE) and downstream NLU after light continual fine-tuning.

Problem

Maximum likelihood (forward cross-entropy) training of autoregressive LMs suffers from recall-prioritization, negative diversity ignorance, and train-test mismatch, leading to degenerate text generation.

Approach

Use earth mover distance between the model's and data's next-token distributions with a semantically informed cosine cost from the LM head embeddings, and optimize a tractable differentiable upper bound Q^T C P (combined with MLE) so it trains end-to-end without external OT solvers.

Key results

• On open-ended generation, EMO improves MAUVE over MLE for GPT-2 across six datasets (e.g., WikiText2 87.5 vs 77.5), averaging about 6.2 points gain.
• Continual fine-tuning of LLaMa-13B on WikiText-103 with EMO raises the average accuracy across 8 NLU tasks, with TE going from 58.6 (MLE) to 70.4.
• With the oracle GPT-2-Large setup, EMO lowers PPL oracle to 55.85 versus 114.46 for MLE while raising MAUVE to 83.4.

Measured effects reported in the paper

Surprising findings

• MLE continual fine-tuning often only marginally improves or even hurts downstream performance relative to the pre-trained LLM.
• EMO-tuned models reach performance of MixCE trained on 100M tokens with only 4M tokens.

Limitations

• Gradient scaling from bounded cosine-based costs can slow convergence for weak models, requiring a dynamic weighting heuristic.
• The transport cost is fixed and derived from a pre-trained MLE model's embedding space, so it depends on that model's quality.
• Evaluation of instruction tuning relies on LLM-based judges (GPT-4, Auto-J, PandaLM), which may be biased.

Experimental setup

Datasets: WikiText-2 WikiText-103 WebText Penn Tree Bank WritingPrompts AG News GPT-2-output dataset Alpaca-GPT4 Recycled Evol-Instruct-70K OpenPlatypus Tweet Emotion TREC

Benchmarks: WikiText-2 WikiText-103 WebText test PTB WritingPrompts AG News Tweet Emotion TREC SST-2 Subj Customer Review Rotten Tomatoes

Metrics: MAUVE PPL test PPL oracle ROUGE-1 ROUGE-L accuracy win rate

Baselines: MLE TaiLr MixCE

Builds on: maximum likelihood estimation Wasserstein GAN TaiLr MixCE

Closest papers by meaning

• Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark (ICML 2024 · similarity 0.86)
• The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.85)
• Scaling Law with Learning Rate Annealing (NeurIPS 2025 · similarity 0.85)
• Test-Time Training on Nearest Neighbors for Large Language Models (ICLR 2024 · similarity 0.85)
• Low-Rank Adapting Models for Sparse Autoencoders (ICML 2025 · similarity 0.85)
• SparseLLM: Towards Global Pruning of Pre-trained Language Models (NeurIPS 2024 · similarity 0.85)