Research map / Label-efficient and robust learning / Recognition and long-tail

Towards a learning theory of representation alignment

ICLR 2025 · Label-efficient and robust learning › Recognition and long-tail · approach: Contrastive representation theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a learning-theoretic framework unifying representation alignment metrics (kernel alignment, distance, independence-testing, measure-based) and bounds linear/Lipschitz stitching error by kernel alignment.

Problem

Representation alignment across models and modalities is studied empirically with fragmented metrics and lacks theoretical guarantees linking alignment to task performance under realistic assumptions.

Approach

Connects kernel alignment, distance alignment, HSIC/MI/Wasserstein-based measures in a unified RKHS and spectral framework, and proves that excess stitching risk (linear stitching, Lipschitz output maps) is bounded by a misalignment term derived from kernel alignment.

Key results

• Shows empirical kernel alignment concentrates to its population value with an O(1/sqrt(n)) deviation bound via McDiarmid's inequality.
• Proves that for linear g_q, the linear stitching risk equals the risk of H_1, and for Lipschitz g_2 the stitching risk is bounded by R_2(H_2) + kappa_2^2 A~_2 + 2 kappa_2 sqrt(A~_2 R_2(H_2)).
• Establishes that D(d_1,d_2) = 8C(1 - A(K_1,K_2)) under unit-norm kernel assumptions, showing distance alignment and kernel alignment are equivalent.

Limitations

• Stitching analysis restricted to linear stitching maps and Lipschitz output maps.
• Results are theoretical; no experiments validate the bounds.
• Bounds involve unnormalized/assumption-heavy quantities such as bounded kernels and equal output spaces.

Experimental setup

Builds on: kernel alignment centered kernel alignment SVCCA HSIC stitching

Closest papers by meaning

• Functional Alignment Can Mislead: Examining Model Stitching (ICML 2025 · similarity 0.91)
• Latent Functional Maps: a spectral framework for representation alignment (NeurIPS 2024 · similarity 0.88)
• Position: The Platonic Representation Hypothesis (ICML 2024 · similarity 0.87)
• Objective drives the consistency of representational similarity across datasets (ICML 2025 · similarity 0.86)
• Bridging Critical Gaps in Convergent Learning: How Representational Alignment Evolves Across Layers, Training, and Distribution Shifts (NeurIPS 2025 · similarity 0.86)
• Concept frustration: Aligning human concepts and machine representations (NeurIPS 2026 · similarity 0.86)