Research map / Vision-language and multimodal / CLIP and zero-shot
Unified Generative and Discriminative Training for Multi-modal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Sugar, which fine-tunes VILA with an added Dynamic Time Warping / global alignment kernel loss over MLLM hidden states plus a Triple Kernel of DINOv2, CLIP and BGE features, so one model handles both generative and discriminative tasks.
Problem
Generative MLLMs are weak at discriminative tasks such as interleaved and fine-grained retrieval and show position bias over long interleaved inputs, while CLIP-style discriminative models struggle with interleaved documents and fine-grained semantics.
Approach
Impose semantic similarity between interleaved input samples as a structural constraint on the MLLM's last-token hidden states, computed via a global alignment kernel (DTW-style) over slice embeddings, with a Triple Kernel that combines uni-modal and cross-modal features from discriminative encoders, jointly trained with the next-token loss.
Key results
- Sugar (Vicuna-7B) reaches 86.6 on POPE and 1550.8 on MME Perception, and 60.4 on VizWiz versus 57.8 for VILA-7B.
- On Winoground, Sugar scores 40.0 Text, 36.3 Image, 27.0 Group, outperforming all listed discriminative models.
- On VIST 5c+4i retrieval, Sugar reaches R@1 21.9 vs 18.2 for FROMAGe; on DEMON, Sugar leads 6 of 7 categories.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Sugar's structure-induced discriminative training | VILA-7B · VizWiz | accuracy | Helped |
| Add Sugar's structure-induced discriminative training | VILA-7B · POPE | accuracy | Helped |
| Add Sugar's structure-induced training | VILA-7B · DEMON TRQA | score: +36.1% | Helped |
| Add Sugar's structure-induced training | VILA-7B · DEMON MMC | score | Hurt |
| Add Sugar's retrieval method | VILA-7B · VIST 5c+4i | R@1: 18.2 -> 21.9 | Helped |
| Add Sugar's retrieval method | VILA-7B · MSCOCO text-to-image | R@1: 23.4 -> 22.0 | Hurt |
| Add retrieval-augmented generation | Sugar · VizWiz | accuracy: +2.5% | Helped |
| Add retrieval-augmented generation | Sugar · ScienceQA-IMG | accuracy: +3.6% | Helped |
| Remove generative-task data (ablation) | Sugar · KGQA | accuracy: 64.4 -> 62.5 | Hurt |
| Remove generative-task data (ablation) | Sugar · POPE | accuracy: 86.6 -> 86.4 | Hurt |
| Remove discriminative-task data (ablation) | Sugar · SQA-I | accuracy: 72.6 -> 72.8 | No clear effect |
| Remove the Global Alignment Kernel (GAK) (ablation) | Sugar · VIST 5c+4i | R@5: 46.7 -> 33.5 | Hurt |
Surprising findings
- Removing the triple kernel drops Winoground Image score from 36.3 to 20.5 (a 43.5% decrease), indicating the kernel drives fine-grained discrimination.
- Off-the-shelf RAG with CLIP or BLIP-2 retrieval degrades VILA on VizWiz, while Sugar's own retrieval improves both VizWiz and SQA-IMG.
Limitations
- Inherits hallucination and bias from base models such as Vicuna and CLIP.
- Training data may contain mismatched image-text pairs that hurt training.
- Image-text retrieval on MSCOCO is only comparable to FROMAGe (R@1 lower).
Experimental setup
Datasets: MMC4 COYO LLaVA-1.5 SFT WebQA MMVP-VLM DEMON MSCOCO VIST Winoground
Benchmarks: VQA-v2 GQA VizWiz ScienceQA-IMG TextVQA POPE MME MMBench LLaVA-Bench (In-the-Wild) MM-Vet DEMON MSCOCO
Metrics: accuracy Recall@k MME score
Baselines: VILA-7B LLaVA-1.5 BLIP-2 InstructBLIP VPG-C FROMAGe CLIP ViT-L/14 Winoground discriminative models
Builds on: VILA Dynamic Time Warping global alignment kernel FROMAGe LLaVA-1.5
Closest papers by meaning
- VladVA: Discriminative Fine-tuning of LVLMs (CVPR 2025 · similarity 0.92)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.92)
- VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks (ICLR 2025 · similarity 0.90)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.90)
- Towards General Continuous Memory for Vision-Language Models (NeurIPS 2025 · similarity 0.90)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.