Research map / Vision-language and multimodal / Cross-modal learning

Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoders

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Contrastive vision-language alignment · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that cross-modal feature directions for the same concept differ in VLM joint embeddings, explaining SAE modality split, and proposes modality-specific SAEs with post-hoc Hungarian-based latent alignment.

Problem

Sparse autoencoders trained on joint image-text embeddings of VLMs exhibit modality split (same concept activates different latents per modality); prior methods assume corresponding features share directions and force latent alignment via auxiliary losses, which degrades reconstruction.

Approach

Cross-modal feature heterogeneity: same concept has different feature directions per modality. Train separate SAEs per modality to preserve each modality's geometry, then align latent coordinates post hoc via coactivation correlation and Hungarian matching, without changing decoder directions.

Key results

• Post-hoc alignment reaches image-to-text Recall@1 of 16.0 on MS-COCO vs 7.1 for the strongest baseline (group-sparse loss), and text-to-image Recall@1 of 11.4 vs 4.3.
• Reconstruction MSE of modality-specific SAE plus post-hoc alignment is 0.089 on MS-COCO, matching the best (0.089) and lower than group-sparse loss (0.105).
• Zero-shot ImageNet1K top-1 accuracy is 25.1 for the proposed method, versus 26.6 for group-sparse loss and 15.7 for shared SAE.

Surprising findings

• Modality-specific SAEs without alignment yield near-zero cross-modal retrieval (Recall@1 about 0.0), showing that the alignment step is essential.
• Cosine distances between coactivating image and text features remain centered around positive values even for high-correlation pairs (c>=0.8).

Limitations

• Theory relies on the linear representation hypothesis and simplified generative assumptions (Top-1, independent sparse latents).
• Alignment assumes correspondence can be inferred from coactivation statistics, which may be unreliable in noisy or weakly aligned settings.
• Experiments focus on CLIP-like contrastive VLMs; LLM-based autoregressive VLMs are not evaluated.

Experimental setup

Datasets: CC-3M MS-COCO ImageNet1K

Benchmarks: MS-COCO ImageNet1K

Metrics: MSE Recall@1 Recall@5 Recall@10 top-1 zero-shot accuracy mAP MRR monosemanticity score feature collapse rate feature recovery error

Baselines: Shared SAE Iso-Energy alignment loss Group-sparse loss

Builds on: sparse autoencoders linear representation hypothesis Top-K SAE modality split

Closest papers by meaning

• Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion (ICLR 2025 · similarity 0.89)
• Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment (ICML 2025 · similarity 0.88)
• Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.88)
• WikiDO: A New Benchmark Evaluating Cross-Modal Retrieval for Vision-Language Models (NeurIPS 2024 · similarity 0.88)
• Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs (NeurIPS 2026 · similarity 0.88)
• On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.88)