Research map / Label-efficient and robust learning / Recognition and long-tail

Optimization Dynamics Imprint Semantic Specificity in Contrastive Embedding Norms

NeurIPS 2026 · Label-efficient and robust learning › Recognition and long-tail · approach: Contrastive representation theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Gives a stationary-equilibrium theory (bi-quadratic balance of radial drift, stochastic heat, and weight decay) explaining why unnormalized embedding norms in scale-invariant contrastive models encode specificity/uncertainty, and proposes norm-discounted retrieval scoring.

Problem

Contrastive embedding models trained with scale-invariant losses ignore embedding norm, yet norms empirically correlate with concept specificity, token frequency, and human uncertainty; no mechanism predicts norm equilibrium for individual inputs.

Approach

Derives a zero-drift equilibrium for the squared norm of a fixed input under scale-invariant loss, weight decay, and NTK-transmitted task updates, yielding R_eq^2 = (D + sqrt(D^2 + 2V^2))/2 where D is radial drift and V^2 is stochastic heat; uses norm-discounted scoring cos(q,d)*||d||^alpha as a free inference-time signal.

Key results

• Predicted vs observed norm for fine-tuned DistilBERT gives log-log slope 0.88 with Pearson r=0.97.
• CLIP ViT-B/32 norm-weighted retrieval gains +32.3% NDCG@10 on TREC CAR overall, while text-only models stay near cosine (<=1.5% gain).
• Across 11 standard contrastive VL models on CIFAR-10H, V^2 correlates positively with model entropy in every case (e.g., CLIP ViT-B/32 r=+0.234).

Surprising findings

• Text-only embedding models show near-zero benefit from norm discounting, while CLIP with projection head gains substantially, so the signal is regime-specific.
• Removing the CLIP projection head flips the best norm exponent to positive, boosting high-norm documents.

Limitations

• Derivation assumes SGD-style updates and a loss that does not match mean pooling exactly; momentum and AdamW are only heuristically handled.
• Drift and damping estimates are noisy (sign disagreement on 85% of tokens for D), while V^2 is more stable.
• Retrieval gains are model-regime specific: text-only encoders show little benefit.

Experimental setup

Datasets: MultiNLI BEIR TREC CAR CIFAR-10H synthetic concept datasets synthetic general/specific Q/A triples

Benchmarks: BEIR TREC CAR CIFAR-10H

Metrics: NDCG@10 Pearson correlation log-log slope entropy correlation

Baselines: cosine similarity dot product MagFace AdaFace

Builds on: NormFace alignment-uniformity Neural Tangent Kernel AdamW

Closest papers by meaning

• On the Origins of Linear Representations in Large Language Models (ICML 2024 · similarity 0.85)
• The "Law'' of the Unconscious Contrastive Learner: Probabilistic Alignment of Unpaired Modalities (ICLR 2025 · similarity 0.85)
• Less is More: Local Intrinsic Dimensions of Contextual Language Models (NeurIPS 2025 · similarity 0.85)
• Tracing the Representation Geometry of Language Models from Pretraining to Post-training (NeurIPS 2025 · similarity 0.85)
• GRACE: Generative Representation Learning via Contrastive Policy Optimization (ICLR 2026 · similarity 0.85)
• Diversity Is All You Need for Contrastive Learning: Spectral Bounds on Gradient Magnitudes (NeurIPS 2025 · similarity 0.85)