Research map / Vision-language and multimodal / CLIP and zero-shot
The Inter-Intra Modal Measure: A Predictive Lens on Fine-Tuning Outcomes in Vision-Language Models
Laura Niss, Kevin Vogt-Lowell, Theodoros Tsiligkaridis
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use IIMM instead of GBC to predict gain over zero-shot error (Kendall tau) | IIMM · CoCa, EVA-02, CLIP, SigLIP | Kendall tau: CoCa 0.78 -> 0.89, EVA-02 0.72 -> 0.83, CLIP 0.78 -> 0.78, SigLIP 0.83 -> 0.78 | Mixed |
| use IIMM instead of TransRate for gain over zero-shot error | IIMM · CoCa | Kendall tau: 0.67 -> 0.89 | Improved |
| use IIMM instead of NCTI | IIMM · EVA-02 | Kendall tau: -0.59 -> 0.83 | Improved |
| use IIMM instead of H-Score | IIMM · EVA-02 | Kendall tau: -0.78 -> 0.83 | Improved |
| use IIMM instead of GBC to predict fine-tuned accuracy | IIMM · CoCa | Kendall tau: 0.61 -> 0.72 | Improved |
| use IIMM instead of GBC to predict fine-tuned accuracy | IIMM · SigLIP | Kendall tau: 0.67 -> 0.61 | No clear effect |
| fit IIMM linearly instead of GBC against log gain | IIMM · CLIP | R^2: 0.59 -> 0.92 | Improved |
| use image-image similarity (intra-images) term alone instead of intra-texts (ablation) | IIMM component · CoCa | R^2: 0.37 -> 0.88 | Improved |
| use incorrect-alignment term alone instead of intra-texts term (ablation) | IIMM component · SigLIP | R^2: 0.17 -> 0.80 | Improved |
| combine inter and intra terms (alpha 0.5) rather than inter only (alpha 0.00) (ablation) | IIMM · CLIP | Pearson r: 0.79 -> 0.96 | Improved |
| combine inter and intra terms (alpha 0.5) rather than intra only (alpha 1.00) (ablation) | IIMM · CLIP | Pearson r: 0.90 -> 0.96 | Improved |
| apply IIMM to CLIP-Adapter instead of other PEFT methods | IIMM · CLIP-Adapter | Pearson r: 0.76 -> 0.96 for Attention-Weight Tuning at alpha 0.5 | Hurt |
Closest papers by meaning
- Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion (ICLR 2025 · similarity 0.89)
- Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Models (NeurIPS 2025 · similarity 0.89)
- CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models (CVPR 2026 · similarity 0.89)
- VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models (CVPR 2026 · similarity 0.88)
- Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs (NeurIPS 2025 · similarity 0.88)
- Jointly Modeling Inter- & Intra-Modality Dependencies for Multi-modal Learning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
