Research map / Large language models / Instruction and fine-tuning

Less is More: Local Intrinsic Dimensions of Contextual Language Models

NeurIPS 2025 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measures local intrinsic dimension (TwoNN on L-nearest-neighbor neighborhoods) of contextual token embeddings to track fine-tuning shifts, grokking, training saturation, and overfitting without labels.

Problem

Understanding how fine-tuning and training affect LLM internal geometry and whether unsupervised geometric signals can diagnose training dynamics, generalization, grokking, and overfitting.

Approach

Apply a localized TwoNN intrinsic dimension estimator to contextual token embeddings sampled from a dataset, and use the mean local dimension (on training data alone) as an unsupervised signal of fine-tuning effects and training dynamics.

Key results

• Fine-tuning RoBERTa-base lowers local dimension only on the fine-tuning dataset: standardized mean difference 1.19 for MultiWOZ versus 0.08 for Reddit and 0.1 for Wikipedia.
• In TripPy-R dialogue state tracking on MultiWOZ 2.1, mean local dimension correlates with joint-goal accuracy at -0.982 (train), -0.958 (validation), -0.905 (test) Spearman.
• In grokking runs with training fraction at least 20%, Spearman correlation between training mean local dimension and validation accuracy is negative, while runs at 10% and 15% (no grokking) show positive correlation (0.880/0.076 and 0.922/0.257).

Measured effects reported in the paper

Surprising findings

• In TripPy-R, validation loss reaches a minimum around 7,500 batches long before JGA converges, so the loss would mislead about training completion, while the mean local dimension was still decreasing.
• In the EmoWOZ overfitting experiment, the mean local dimension drops after the first epoch and then rises while validation loss increases, indicating memorization.

Limitations

• TwoNN assumptions (Poisson point process on uniform-dimensional subspace, locally constant density) hold only approximately for transformer embeddings.
• Absolute mean local dimension values depend on sampling and neighborhood hyperparameters, so comparisons across architectures are relative.
• Computationally expensive nearest-neighbor based estimation, and the relationship between LID reduction and generalization is correlational, not causal.

Experimental setup

Datasets: MultiWOZ 2.1 Schema-Guided Dialogue Reddit TSLA comments (2022) ICLR 2024 submissions WikiText-103 EmoWOZ synthetic modular addition

Benchmarks: MultiWOZ 2.1 EmoWOZ modular addition grokking

Metrics: mean local TwoNN dimension joint-goal accuracy Spearman rank correlation cross-entropy loss weighted F1 macro F1 standardized mean difference training and validation accuracy

Baselines: ERToD

Builds on: TwoNN grokking TripPy-R

Closest papers by meaning

• Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.89)
• Tracing the Representation Geometry of Language Models from Pretraining to Post-training (NeurIPS 2025 · similarity 0.89)
• Geometry of Decision Making in Language Models (NeurIPS 2025 · similarity 0.88)
• Remarkable Robustness of LLMs: Stages of Inference? (NeurIPS 2025 · similarity 0.88)
• Where does In-context Learning Happen in Large Language Models? (NeurIPS 2024 · similarity 0.88)
• Exploring Activation Patterns of Parameters in Language Models (AAAI 2025 · similarity 0.88)