Research map / Vision-language and multimodal / CLIP and zero-shot
Geodesic Multi-Modal Mixup for Robust Fine-Tuning
Changdae Oh, Junhyuk So, Hoyoon Byun, YongTaek Lim, Minchul Shin, Jong-June Jeon, Kyungwoo Song
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use geodesic mixup instead of linear mixup with L2 normalization in m3-Mix (ablation) | m3-Mix · ImageNet variants mean accuracy | accuracy: 48.48 -> 49.80 (tau=0.05) | Improved |
| add m2-Mix to FT (ablation) | CLIP FT · ImageNet few-shot + shifts | avg accuracy: 46.70 -> 47.81 | Improved |
| add L-Mix to FT (ablation) | CLIP FT · Few-shot Pets/SVHN/CLEVR | avg accuracy: 62.62 -> 63.92 | Improved |
| add V-Mix to FT (ablation) | CLIP FT · Few-shot Pets/SVHN/CLEVR | avg accuracy: 62.62 -> 62.66 | No clear effect |
| combine m2-Mix, uni-Mix and VL-Mix into m3-Mix (ablation) | CLIP FT · Few-shot Pets/SVHN/CLEVR | avg accuracy: 62.62 -> 66.28 | Improved |
| increase temperature to 0.05 in m3-Mix (ablation) | m3-Mix · Few-shot Pets/SVHN/CLEVR | avg accuracy: 66.28 -> 68.45 | Improved |
| increase FT temperature to 0.10 | CLIP FT · Flickr30k i->t | R1: 81.2 -> 75.7 | Hurt |
| add m3-Mix to LP-FT | LP-FT · Few-shot Pets/SVHN/CLEVR | avg accuracy: 62.82 -> 70.16 | Improved |
| add m3-Mix to MaPLe | MaPLe · ImageNet few-shot + shifts | avg accuracy: 53.44 -> 53.46 | No clear effect |
| add m2-Mix to GMC | GMC · CMU-MOSEI visual only | accuracy: 64.7 -> 64.2 | Mixed |
| fine-tune with m3-Mix instead of FT | CLIP FT · Flickr30k calibration i->t | ECE: 2.26 -> 1.54 | Improved |
| add m2-Mix to contrastive+captioning loss | CoCa ViT-L/14 · MS COCO captioning | CIDEr: 124.9 -> 125.6 | Improved |
Closest papers by meaning
- Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion (ICLR 2025 · similarity 0.88)
- Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.87)
- The Double-Ellipsoid Geometry of CLIP (ICML 2025 · similarity 0.87)
- Accept the Modality Gap: An Exploration in the Hyperbolic Space (CVPR 2024 · similarity 0.87)
- Understanding Transferable Representation Learning and Zero-shot Transfer in CLIP (ICLR 2024 · similarity 0.86)
- Learning Shared Representations from Unpaired Data (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
