Research map / Vision-language and multimodal / Cross-modal learning
When Kernels Multiply, Clusters Unify: Fusing Embeddings with the Kronecker Product
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KrossFuse, fusing embeddings by multiplying their kernels via the Kronecker product of feature maps, plus RP-KrossFuse, a random-projection approximation, to combine cross-modal (CLIP, BLIP) and unimodal experts (DINOv2, E5) training-free.
Problem
Cross-modal embeddings such as CLIP lag behind modality-specific experts on single-modality tasks, and fusing embeddings with different structures into one representation is not handled by simple concatenation.
Approach
Fuse embeddings by multiplying their kernel similarity functions, realized as the Kronecker product of feature maps (so the fused kernel encodes the union of parent clusters); use Hadamard product of random projections to approximate the high-dimensional Kronecker space, and symmetrized embeddings with constant maps to handle a missing modality.
Key results
- RP-KrossFuse reaches 84.1% linear-probe accuracy on ImageNet, above CLIP (73.2%) and DINOv2 (83.3%).
- On SentEval text benchmarks, RP-KrossFuse (ViT-B/32) averages 88.0% versus 81.8% for CLIP ViT-B/32 and 86.2% for S-RoBERTa.
- In 1-shot cross-modal few-shot classification, Ours (I+T) averages 53.5% versus 44.3% for CLIP (I+T).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fuse CLIP and DINOv2 with RP-KrossFuse | CLIP image embeddings · ImageNet linear probe | accuracy: 83.3 -> 84.1 | Helped |
| Fuse CLIP and DINOv2 with RP-KrossFuse | CLIP image embeddings · GTSRB linear probe | accuracy: 83.1 -> 82.7 | Hurt |
| Fuse CLIP and DINOv2 with RP-KrossFuse | CLIP image embeddings · SVHN linear probe | accuracy: 63.6 -> 66.9 | Helped |
| Fuse CLIP and DINOv2 with RP-KrossFuse | CLIP image embeddings · ImageNet-A linear probe | accuracy: 48.5 -> 47.6 | Hurt |
| Fuse CLIP and DINOv2 with RP-KrossFuse | CLIP image embeddings · ImageNet-R linear probe | accuracy: 68.8 -> 67.4 | Hurt |
| Fuse CLIP text and S-RoBERTa using RP-KrossFuse | CLIP ViT-B/32 text features · Average SentEval | accuracy: 81.8 -> 88.0 | Helped |
| Fuse CLIP text and S-RoBERTa using RP-KrossFuse | CLIP ViT-L/14 text features · Average SentEval | accuracy: 83.2 -> 88.1 | Helped |
| Add text-modality samples to RP-KrossFuse (I) | RP-KrossFuse cross-modal few-shot classification · 1-shot cross-modal few-shot average | accuracy: 46.9 -> 53.5 | Helped |
| Add text-modality samples to RP-KrossFuse (I) | RP-KrossFuse cross-modal few-shot classification · 2-shot cross-modal few-shot average | accuracy: 55.2 -> 59.3 | Helped |
| Add text-modality samples to RP-KrossFuse (I) | RP-KrossFuse cross-modal few-shot classification · 4-shot cross-modal few-shot average | accuracy: 62.4 -> 63.9 | Helped |
| Add text-modality samples to RP-KrossFuse (I) | RP-KrossFuse cross-modal few-shot classification · 8-shot cross-modal few-shot average | accuracy: 65.7 -> 67.1 | Helped |
| Use cosine or RBF kernel instead of linear kernel (ablation) | RP-KrossFuse · ImageNet classification and SentEval | accuracy | Helped |
Surprising findings
- Concatenation-style fusion baselines (GATE, ATTN, COMM) and KPoMRP sometimes fall below the single parent embedding on several benchmarks, e.g., KPoMRP scores 49.4% on SVHN versus 63.6% for CLIP.
Limitations
- Requires extra hyperparameters (scaling constant C and projection dimension l), with the fused embedding needing around 3000 dimensions.
- Evaluated mainly on image and text modalities; extension to other modalities and multi-way fusion left to future work.
- Kernel-product fusion behavior for kernels with negative values relies on orthogonality arguments rather than strict separation.
Experimental setup
Datasets: ImageNet GTSRB SVHN ImageNet-A ImageNet-R MR CR SUBJ MPQA SST2 TREC MRPC
Benchmarks: ImageNet GTSRB SVHN ImageNet-A ImageNet-R SentEval (MR, CR, SUBJ, MPQA, SST2, TREC, MRPC) Caltech101 Food101 DTD FGVC-Aircraft MSCOCO Flickr30k
Metrics: linear probe accuracy few-shot accuracy zero-shot retrieval recall cosine similarity distribution
Baselines: KPoMRP GATE ATTN COMM CLIP DINOv2
Builds on: random Fourier features Johnson-Lindenstrauss lemma CLIP
Closest papers by meaning
- Towards an Explainable Comparison and Alignment of Feature Embeddings (ICML 2025 · similarity 0.87)
- Learning Shared Representations from Unpaired Data (NeurIPS 2025 · similarity 0.86)
- AKRMap: Adaptive Kernel Regression for Trustworthy Visualization of Cross-Modal Embeddings (ICML 2025 · similarity 0.85)
- Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoders (NeurIPS 2026 · similarity 0.85)
- ITO: Multi-View Alignment and Training-Time Fusion for Image-Text Pretraining (NeurIPS 2026 · similarity 0.84)
- Latent Functional Maps: a spectral framework for representation alignment (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.