Research map / Vision-language and multimodal / Cross-modal learning

AKRMap: Adaptive Kernel Regression for Trustworthy Visualization of Cross-Modal Embeddings

ICML 2025 · Vision-language and multimodal › Cross-modal learning · approach: Unimodal-to-multimodal representation transfer · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AKRMap, a supervised neural dimensionality-reduction method that jointly learns a 2D projection, Nadaraya-Watson kernel regression with a learnable generalized t-kernel, and a t-SNE neighborhood loss to visualize cross-modal embedding metrics (CLIPScore, HPSv2, PickScore, Aesthetic Score).

Problem

Existing dimensionality-reduction visualizations (PCA, t-SNE, UMAP, autoencoders) model only single-modality feature structure and do not faithfully map cross-modal embedding metric values (e.g., CLIPScore, HPS) onto 2D contour maps, producing occlusion and inaccurate contours.

Approach

Train a parametric projection network with a post-projection kernel regression loss (train/validation split for in- and out-of-sample accuracy), plus a perplexity-free t-SNE KL term, and an adaptive generalized t-kernel whose alpha and beta are learned jointly with the projection, so the 2D kernel regression map is optimized together with the embedding layout.

Key results

• On HPD out-of-sample points, AKRMap reduces HPSv2 MAE to 0.8108 versus 1.4361 for t-SNE (about 43% lower), and CLIPScore MAE to 1.8707 versus 4.0042 for PCA.
• Removing kernel regression supervision (w/o KR) makes AKRMap nearly indistinguishable from baselines (e.g., CLIPScore MAE 4.0164), showing KR supervision is the most impactful component.
• Neuro-Visualizer performs worst on cross-modal metrics, with CLIPScore MAE 12.4201 on test points.

Measured effects reported in the paper

Surprising findings

• Neuro-Visualizer, an autoencoder-based method effective on loss landscapes, performs worst on cross-modal metrics such as CLIPScore.
• The kernel regression supervision alone accounts for most of the gain; without it, AKRMap behaves like t-SNE in mapping accuracy.

Limitations

• Neighborhood preservation is weaker than traditional DR methods (trustworthiness 0.8284 vs. 0.8945 for t-SNE at n=20).
• Unclear how to handle metrics computed over sequences of embeddings (e.g., CodeBertScore) that lack a single vector per instance.
• Theoretical convergence conditions (integrability, alpha growth with N) are only argued heuristically and empirically.

Experimental setup

Datasets: Human Preference Dataset (HPD) MS-COCO PartiPrompts MSR-VTT Flickr 8k Audio Caption Corpus

Benchmarks: HPD test set

Metrics: mae mape rmse trustworthiness

Baselines: PCA t-SNE UMAP SAE Neuro-Visualizer

Builds on: t-SNE Nadaraya-Watson kernel regression supervised t-SNE parametric t-SNE Neuro-Visualizer

Closest papers by meaning

• When Kernels Multiply, Clusters Unify: Fusing Embeddings with the Kronecker Product (NeurIPS 2025 · similarity 0.85)
• When to Align, When to Predict: A Phase Diagram for Multimodal Learning (NeurIPS 2026 · similarity 0.84)
• Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.84)
• CrossOver: 3D Scene Cross-Modal Alignment (CVPR 2025 · similarity 0.84)
• WikiDO: A New Benchmark Evaluating Cross-Modal Retrieval for Vision-Language Models (NeurIPS 2024 · similarity 0.84)
• Beyond Modality Collapse: Representation Blending for Multimodal Dataset Distillation (NeurIPS 2025 · similarity 0.83)