Research map / Vision-language and multimodal / Cross-modal learning
Learning Modality Knowledge Alignment for Cross-Modality Transfer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoNA, a meta-learning method that first learns a target embedder to align modality knowledge (P(Y|X)) with a frozen-source-quality objective, then fully finetunes a pretrained transformer for cross-modality transfer.
Problem
Cross-modality transfer from large pretrained models (vision/language) to scientific data modalities often fails because the knowledge in the source model is misaligned with the target modality; the effect of this modality gap on knowledge reuse is not well understood.
Approach
Formalizes modality knowledge discrepancy via conditional distributions P(Y|X) and learns a target embedder with bi-level meta-learning, whose outer loss measures source-data representation quality (alignment and uniformity) after a simulated inner-loop finetuning, before standard full finetuning.
Key results
- MoNA achieves top performance on nine of ten NAS-Bench-360 tasks, e.g., CIFAR-100 error 6.48% vs ORCA 6.53% and Spherical 27.13% vs 29.85%.
- On PDEBench, MoNA achieves state-of-the-art on four of eight tasks and outperforms ORCA on seven tasks.
- On AudioSet-20k, ESC50, and UCF101, MoNA reaches 0.523 mAP, 9.64% and 13.45% error respectively, beating ORCA on all three.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MoNA instead of ORCA | Cross-modal transfer on NAS-Bench-360 · NAS-Bench-360 | prediction error: CIFAR-100: 6.53 -> 6.48; Spherical: 29.85 -> 27.13; Darcy Flow: 7.28E-3 -> 6.80E-3; PSICOV: 1.91 -> 0.99; Cosmic: 0.152 | Helped |
| Use MoNA instead of ORCA | Cross-modal transfer on PDEBench · PDEBench | nRMSE: Advection 1D: 0.0098 -> 0.0088; Burgers 1D: 0.0120 -> 0.0114; Diffusion-Reaction 1D: 0.0030 -> 0.0028; Diffusion-Sorptio | Helped |
| Remove the inner-loop loss from the outer-loop objective (ablation) | MoNA · CIFAR-100, Spherical, NinaPro, FSD50K | prediction error: CIFAR-100: 6.48 -> 8.00; Spherical: 27.13 -> 28.76; NinaPro: 7.28 -> 7.74; FSD50K: 0.55 -> 0.58 | Hurt |
| Replace alignment and uniformity outer loss with contrastive loss (ablation) | MoNA · CIFAR-100, Spherical, NinaPro, FSD50K | prediction error: CIFAR-100: 6.48 -> 6.51; Spherical: 27.13 -> 27.90; NinaPro: 7.28 -> 7.44; FSD50K: 0.55 -> 0.55 | Hurt |
| Replace alignment and uniformity outer loss with clustering metric (ablation) | MoNA · CIFAR-100, Spherical, NinaPro, FSD50K | prediction error: CIFAR-100: 6.48 -> 7.09; Spherical: 27.13 -> 28.02; NinaPro: 7.28 -> 8.19; FSD50K: 0.55 -> 0.56 | Hurt |
| Increase inner-loop update steps (ablation) | MoNA · Spherical and NinaPro | accuracy | Helped |
| Set trade-off parameter lambda much higher or lower than the optimum (ablation) | MoNA · Spherical and NinaPro | accuracy | Hurt |
Surprising findings
- Finetuning on some target modalities (e.g., FSD50K) substantially degrades the source encoder's ability to discriminate CIFAR-10 images, while finetuning on CIFAR-100 preserves it.
- Training from scratch on some target tasks outperforms vanilla finetuning of the pretrained model (per ORCA observation).
Limitations
- Results depend on surrogate source datasets (CIFAR-10, CoNLL-2003) chosen following ORCA, which may influence transfer outcomes.
- Single-step inner loop MAML-style approximation is a simplification; multi-step inner loops give small gains with more cost and variance.
- The modality knowledge discrepancy is only approximated, using random permutation and subset search, and is not directly computable for real tasks.
Experimental setup
Datasets: NAS-Bench-360 PDEBench CIFAR-10 CoNLL-2003 AudioSet-20k ESC50 UCF101
Benchmarks: NAS-Bench-360 PDEBench AudioSet-20k ESC50 UCF101
Metrics: 0-1 error relative l2 MAE8 1-AUROC 1-mAP 1-F1 score nRMSE linear probing error
Baselines: ORCA FPT Finetuning Finetuning++ Frozen-encoder Emb Perceiver IO DASH PINN FNO U-Net
Builds on: ORCA MAML FPT Swin Transformer
Closest papers by meaning
- Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs (NeurIPS 2026 · similarity 0.91)
- C2KD: Bridging the Modality Gap for Cross-Modal Knowledge Distillation (CVPR 2024 · similarity 0.89)
- Guiding Cross-Modal Representations with MLLM Priors via Preference Alignment (NeurIPS 2025 · similarity 0.88)
- Quantifying Cross-Modality Memorization in Vision-Language Models (NeurIPS 2025 · similarity 0.88)
- DeepAlign: Mitigating Modality Conflict through Modality-Specific Alignment (CVPR 2026 · similarity 0.87)
- Vision-Language Models Create Cross-Modal Task Representations (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.