Research map / Learning theory and algorithms / Kernels and regression theory

Towards an Explainable Comparison and Alignment of Feature Embeddings

ICML 2025 · Learning theory and algorithms › Kernels and regression theory · approach: Kernel regression under shift · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SPEC, a spectral method that eigendecomposes the kernel difference matrix of two embeddings to find sample clusters captured differently, with a linear-time implementation via random Fourier features and a SPEC-align objective that aligns one embedding to another.

Problem

Embeddings are usually compared by downstream classification accuracy, which is not interpretable about which samples are clustered differently by two embedding models; existing comparisons do not localize such mismatches.

Approach

Eigendecompose the normalized difference of two embeddings' kernel matrices so top eigenvectors reveal sample clusters grouped by one embedding but not the other; reduce cost to linear in sample size via a differential covariance matrix (plus RFF for Gaussian kernels), and use the SPEC-diff spectral radius as a differentiable alignment penalty (SPEC-align).

Key results

• SPEC-align raises frozen-backbone ImageNet-1K top-1 accuracy of CLIP from 67.20% to 73.93%, compared with DINOv2 at 78.99%.
• Cholesky-based SPEC computation scales to d=10000 RFF features in 85 seconds, while the direct eig approach runs only up to d=4000 (72 s) in the reported table.
• In typographic-attack experiments, SPEC clusters for CLIP follow overlaid text while DINOv2 clusters follow visual content.

Measured effects reported in the paper

Limitations

• Relies on the assumption that differently clustered samples are detectable by spectral (principal eigendirection) methods; non-spectral clustering is not covered.
• Framework is demonstrated for pairwise comparisons of same-modality embeddings; cross-modal comparison (e.g., CLIP vs BLIP) left as future work.
• Random Fourier feature proxies require many features (e.g., m=2000 to 6000) for clean clusters; small m gives inconsistent clusters.

Experimental setup

Datasets: AFHQ FFHQ ImageNet-1K MS-COCO 2017 ImageWoof

Benchmarks: AFHQ MS-COCO 2017 ImageNet-1K FFHQ

Metrics: adjusted mutual information top-1 accuracy (linear probe on frozen backbone) cosine similarity sanity check

Baselines: DINOv2 vs CLIP comparison (no separate baseline method) standard CLIP (alignment baseline)

Builds on: kernel PCA random Fourier features spectral clustering

Closest papers by meaning

• Beyond Supervised vs. Unsupervised: Representative Benchmarking and Analysis of Image Representation Learning (CVPR 2022 · similarity 0.88)
• When Kernels Multiply, Clusters Unify: Fusing Embeddings with the Kronecker Product (NeurIPS 2025 · similarity 0.87)
• Stabilizing Sample Similarity in Representation via Mitigating Random Consistency (ICML 2025 · similarity 0.86)
• When is an Embedding Model More Promising than Another? (NeurIPS 2024 · similarity 0.84)
• $\mathbb{X}$-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs (ICLR 2025 · similarity 0.84)
• Latent Functional Maps: a spectral framework for representation alignment (NeurIPS 2024 · similarity 0.84)