Research map / Vision-language and multimodal / Cross-modal learning
Enhancing Cross-Modal Fine-Tuning with Gradually Intermediate Modality Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PaRe, an end-to-end cross-modal fine-tuning method that uses a gate network to score patches and progressively replaces target patches with source patches to build easy-to-hard intermediate modalities.
Problem
Fine-tuning large pretrained models (text/vision) on specialized modalities with large modality discrepancy and scarce labeled data is unstable and suboptimal, and prior two-stage alignment (ORCA) only partly closes the gap.
Approach
Gate-network patch scoring selects top-k source patches to replace bottom-k target patches in the shared embedding space, with k decreasing linearly over training so intermediate modalities move from source-like to target-like; the loss is a weighted mix of source and target losses, trained end-to-end.
Key results
- On NAS-Bench-360, PaRe achieves best error on all 10 tasks, e.g., CIFAR-100 6.25% vs ORCA 6.53% and NinaPro 6.53% vs ORCA 7.54%.
- On PDEBench, PaRe outperforms ORCA on 6 of 8 tasks (e.g., Advection 1D 2.70E-03 vs 9.80E-03).
- On OpenML-CC18 (30 datasets), PaRe has the highest average AUROC 0.9030 vs AutoGluon 0.8947 and ORCA 0.8946, winning 15/30 datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace ORCA with PaRe | Cross-modal fine-tuning · NAS-Bench-360: CIFAR-100 | 0-1 error (%): 6.53 -> 6.25 | Helped |
| replace ORCA with PaRe | Cross-modal fine-tuning · NAS-Bench-360: PSICOV | MAE₈: 1.91 -> 0.99 | Helped |
| replace ORCA with PaRe | Cross-modal fine-tuning · PDEBench: Advection 1D | nRMSE: 9.80E-03 -> 2.70E-03 | Helped |
| replace ORCA with PaRe | Cross-modal fine-tuning · PDEBench: Navier-Stokes 1D | nRMSE: 6.20E-02 -> 6.62E-02 | Hurt |
| replace ORCA with PaRe | Cross-modal fine-tuning · OpenML-CC18 (30 datasets) | Avg. AUROC: 0.8946 -> 0.9030 | Helped |
| use gradual linear k schedule rather than non-gradual k (ablation) | PaRe · NAS-Bench-360: CIFAR-100 | 0-1 error (%): 6.59 -> 6.25 | Helped |
| use piecewise k schedule rather than non-gradual k (ablation) | PaRe · NAS-Bench-360: CIFAR-100 | 0-1 error (%): 6.59 -> 6.22 | Helped |
| use exponential k schedule rather than non-gradual k (ablation) | PaRe · NAS-Bench-360: CIFAR-100 | 0-1 error (%): 6.59 -> 6.38 | Helped |
| replace random patch selection with gate scoring (ablation) | PaRe · NAS-Bench-360: CIFAR-100 | 0-1 error (%): 6.52 -> 6.25 | Helped |
| replace random patch selection with gate scoring (ablation) | PaRe · NAS-Bench-360: Spherical | 0-1 error (%): 28.06 -> 25.55 | Helped |
| replace patch replacement with Mixup (ablation) | PaRe · NAS-Bench-360: Cosmic | 1-AUROC: 0.121 -> 0.500 | Hurt |
| use MLP plus sigmoid gate instead of FC plus sigmoid (ablation) | PaRe · NAS-Bench-360: CIFAR-100 | 0-1 error (%): 6.25 -> 6.40 | Hurt |
Surprising findings
- Mixup fails to train on the Cosmic dataset, producing random outcomes.
- Increasing gate network complexity (MLP, MLP with dropout) did not improve results over a single FC layer with sigmoid.
Limitations
- Choosing a suitable source proxy dataset for a target modality remains unresolved and effects vary across modalities.
- Target-only data augmentation helps some modalities but is not universal; a modality-agnostic augmentation is needed.
- Does not yet exploit unlabeled target-modality data.
Experimental setup
Datasets: NAS-Bench-360 PDEBench OpenML-CC18 CIFAR10 Tiny-ImageNet PASCAL VOC CoNLL-2003 miniDomainNet
Benchmarks: NAS-Bench-360 PDEBench OpenML-CC18
Metrics: 0-1 error nRMSE relative l2 MAE 1-AUROC 1-mAP 1-F1 AUROC performance profiles
Baselines: ORCA NAS-Bench-360 hand-designed models DASH Perceiver IO FPT NFT PINN FNO U-Net XGBoost CatBoost LightGBM
Builds on: ORCA Mixup CutMix Gumbel softmax optimal transport dataset distance
Closest papers by meaning
- Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs (NeurIPS 2026 · similarity 0.87)
- Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing (ICML 2025 · similarity 0.86)
- When to Align, When to Predict: A Phase Diagram for Multimodal Learning (NeurIPS 2026 · similarity 0.86)
- Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.85)
- Explore How to Inject Beneficial Noise in MLLMs (AAAI 2026 · similarity 0.85)
- Learning Modality Knowledge Alignment for Cross-Modality Transfer (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.