Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
From Bricks to Bridges: Product of Invariances to Enhance Latent Space Communication
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a product space of invariant relative-representation components (cosine, Euclidean, L1, L-infinity) that is aggregated into one latent space to improve zero-shot latent stitching across vision, text, and graph modalities.
Problem
Representations learned by different neural networks are related by transformation classes that vary with seed, architecture, dataset, and task, so no single predefined invariance (as in relative representations with cosine) suffices for latent space communication and zero-shot stitching.
Approach
Extends relative representations by using multiple distance functions (cosine, Euclidean, L1, L-infinity), each inducing a different known invariance, and aggregates the resulting relative subspaces into a single product space via concatenation, MLP+sum, or self-attention, without needing to know the right transformation a priori.
Key results
- On CIFAR-10 stitching with CViT-B/32 and RViT-B/16 encoders, the product-space aggregation reaches 0.88 and 0.95 accuracy, matching or beating the best single projection.
- On CORA GCN stitching, the product of four projections reaches 0.77 accuracy versus 0.53 for cosine alone, with a stitching index of 1.00.
- Cross-seed similarity analysis shows no single invariance (cosine, Euclidean, L1, L-infinity) consistently yields the highest latent similarity across architectures and datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine cosine, Euclidean, L1, and L∞ projections with MLP+Sum | ALBERT zero-shot stitching · DBpedia | accuracy: 0.52 \pm 0.01 -> 0.53 \pm 0.01 | Helped |
| Combine cosine, Euclidean, L1, and L∞ projections with MLP+Sum | ALBERT zero-shot stitching · TREC | accuracy: 0.65 \pm 0.02 -> 0.65 \pm 0.02 | No clear effect |
| Combine cosine, Euclidean, L1, and L∞ projections with MLP+Sum | GCN zero-shot stitching · CORA | accuracy: 0.53 \pm 0.06 -> 0.77 \pm 0.01 | Helped |
| Combine cosine, Euclidean, L1, and L∞ projections with MLP+Sum | CViT-B/32 zero-shot stitching · CIFAR-100 | accuracy: 0.53 \pm 0.04 -> 0.58 \pm 0.03 | Helped |
| Combine cosine, Euclidean, L1, and L∞ projections with MLP+Sum | CViT-B/32 zero-shot stitching · CIFAR-10 | accuracy: 0.87 \pm 0.02 -> 0.88 \pm 0.02 | Helped |
| Replace MLP+Sum with another aggregation function (ablation) | Zero-shot stitching classification · Vision, text, and graph | accuracy: 0.85 \pm 0.00 vs 0.81 \pm 0.00; 0.55 \pm 0.04 vs 0.54 \pm 0.03; 0.77 \pm 0.01 vs 0.75 \pm 0.02 | Hurt |
| Replace MLP+Sum with MLP+SelfAttention aggregation (ablation) | Zero-shot stitching classification · Vision, text, and graph | accuracy: 0.85 \pm 0.00 vs 0.84 \pm 0.01; 0.55 \pm 0.04 vs 0.51 \pm 0.03; 0.77 \pm 0.01 vs 0.63 \pm 0.13 | Hurt |
| Replace MLP+Sum with SelfAttention aggregation (ablation) | Zero-shot stitching classification · Vision, text, and graph | accuracy: 0.85 \pm 0.00 vs 0.76 \pm 0.03; 0.55 \pm 0.04 vs 0.36 \pm 0.22; 0.77 \pm 0.01 vs 0.76 \pm 0.02 | Hurt |
| Fine-tune QKV parameters for space selection and blending (ablation) | SelfAttention zero-shot stitching · CIFAR-100 | accuracy: 0.17 -> 0.75 | Helped |
| Fine-tune the MLP classification head (ablation) | SelfAttention zero-shot stitching · CIFAR-100 | accuracy: 0.17 -> 0.52 | Helped |
| Increase the number of anchors from 2 to 1280 (ablation) | Product of invariances (MLP+Sum) · CIFAR-100 | accuracy: 0.07 \pm 0.02 -> 0.83 \pm 0.04 | Helped |
| Use the product of four projections instead of cosine projection (ablation) | CViT-B/32 zero-shot stitching · CIFAR-100 with 1280 anchors | accuracy: 0.80 \pm 0.02 -> 0.83 \pm 0.04 | Helped |
Surprising findings
- The best-correlated projection changes across architectures (e.g., VAE vs linear VAE) and datasets, challenging the assumption that angle-preserving transformations are the main driver of cross-seed latent correlation.
- Fine-tuning only the QKV attention projections at stitching time gives much larger gains than fine-tuning the classifier head.
Limitations
- Similarity functions must have a closed form; learning similarity functions is left for future work.
- Selecting the optimal invariant spaces can match performance with fewer components, so space selection remains open.
- Stitching-time fine-tuning of attention weights is used only in one experiment and is outside the zero-shot setting.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 MNIST Fashion-MNIST DBpedia TREC News CORA
Benchmarks: CIFAR-10 CIFAR-100 MNIST Fashion-MNIST DBpedia TREC CORA
Metrics: accuracy Pearson correlation Spearman correlation linear CKA stitching index
Baselines: Cosine relative representation Euclidean relative projection L1 relative projection L-infinity relative projection Concat aggregation MLP+SelfAttention SelfAttention aggregation
Builds on: relative representations CKA DeepSets zero-shot model stitching
Closest papers by meaning
- Latent Functional Maps: a spectral framework for representation alignment (NeurIPS 2024 · similarity 0.88)
- Connecting Neural Models Latent Geometries with Relative Geodesic Representations (NeurIPS 2025 · similarity 0.87)
- Invariant Features in Language Models: Geometric Characterization and Model Attribution (NeurIPS 2026 · similarity 0.87)
- On Making $SE(2)$-Invariant Networks Optimal (NeurIPS 2026 · similarity 0.87)
- Interpreting Equivariant Representations (ICML 2024 · similarity 0.87)
- In-Context Symmetries: Self-Supervised Learning through Contextual World Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.