Research map / Vision-language and multimodal / Multimodal LLMs
Enhancing Cognition and Explainability of Multimodal Foundation Models with Self-Synthesized Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a self-synthesis framework that uses information-bottleneck-based concept selection over LMM-generated image descriptions, plus reward-model-free rejection sampling over iterative LoRA fine-tuning of LLaVA-1.5 to improve accuracy and explanation quality on domain-specific visual classification.
Problem
LMMs struggle with fine-grained, domain-specific visual classification and give unjustifiable explanations, and annotating image-specific visual features for fine-tuning data is costly and hard to scale.
Approach
Self-synthesize image-specific interpretable answers: sample many LMM descriptions to approximate image content, select expert-defined class concepts via an InfoNCE-based information bottleneck criterion, rewrite them into answers, then iteratively fine-tune and select best answers by InfoNCE alignment with selected concepts (no reward model).
Key results
- On Stanford Dogs, the method reaches 85.29% accuracy after iteration 1 and 86.91% after iteration 4, versus 12.2% for base LLaVA-1.5 and 84.27% for naive label fine-tuning at iteration 4.
- On PLD, accuracy rises from 75.96% to 97.16% over four iterations, compared with 93.95% for naive label fine-tuning.
- Concept selection precision peaks at 72.89% with 25 descriptions, above GPT-4o (63.95%) and CLIP/LLaVA (about 55%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train using self-synthesized image-specific concepts and iterative rejection sampling instead of labels alone | LLaVA-1.5-7B · CUB-200 | accuracy (iteration 4): 82.21 -> 85.02 | Helped |
| Train using self-synthesized image-specific concepts and iterative rejection sampling instead of labels alone | LLaVA-1.5-7B · Stanford Dogs | accuracy (iteration 4): 84.27 -> 86.91 | Helped |
| Train using self-synthesized image-specific concepts and iterative rejection sampling instead of labels alone | LLaVA-1.5-7B · FGVC-Aircraft | accuracy (iteration 4): 87.73 -> 91.99 | Helped |
| Train using self-synthesized image-specific concepts and iterative rejection sampling instead of labels alone | LLaVA-1.5-7B · PLD | accuracy (iteration 4): 93.95 -> 97.16 | Helped |
| Train using self-synthesized image-specific concepts and iterative rejection sampling instead of labels alone | LLaVA-1.5-7B · HAM10000 | accuracy (iteration 4): 81.71 -> 85.06 | Helped |
| Train using self-synthesized image-specific concepts and iterative rejection sampling instead of labels alone | LLaVA-Med · Chest X-ray Pneumonia | accuracy (iteration 4): 89.58 -> 98.72 | Helped |
| Add reward model-free filtering of synthetic answers (ablation) | LLaVA-1.5-7B trained on synthetic data · CUB-200 | accuracy (iteration 4): 70.45 -> 85.02 | Helped |
| Add reward model-free filtering of synthetic answers (ablation) | LLaVA-1.5-7B trained on synthetic data · FGVC-Aircraft | accuracy (iteration 4): 76.78 -> 91.99 | Helped |
| Add reward model-free filtering of synthetic answers (ablation) | LLaVA-1.5-7B trained on synthetic data · Stanford Dogs | accuracy (iteration 4): 78.26 -> 86.91 | Helped |
| Add reward model-free filtering of synthetic answers (ablation) | LLaVA-1.5-7B trained on synthetic data · PLD | accuracy (iteration 4): 62.59 -> 97.16 | Helped |
| Add reward model-free filtering of synthetic answers (ablation) | LLaVA-1.5-7B trained on synthetic data · HAM10000 | accuracy (iteration 4): 72.29 -> 85.06 | Helped |
| Use E5 rather than BERT-Base for concept-selection embeddings (ablation) | Concept selection · Expert-annotated concepts | concept selection accuracy: 69.7 -> 72.9 | Helped |
Surprising findings
- Naive label-only fine-tuning already reaches high accuracy (e.g., 82.73% on Stanford Dogs), leaving the explanation-quality gain as the main differentiator.
- Training on general label-level explanations (L+GE) lowers accuracy on some datasets; on HAM10000 it reaches only 7.47-9.35% accuracy, below the base model's gain expectations.
Limitations
- Evaluated only on classification tasks; the authors note that more complex tasks remain future work.
- Relies on GPT-4o as judge for explanation existence and cognition scores, and on expert-defined label-level concepts generated by GPT-4o.
- Cross-dataset transfer from CUB-200 to Stanford Dogs is weak (16.6% vs 86.91% in-domain).
Experimental setup
Datasets: CUB-200-2011 Stanford Dogs FGVC-Aircraft HAM10000 Chest X-Ray Pneumonia PlantDoc-like Plant Disease Dataset (PLD) MMStar SEED-Bench-2 Plus MMBench MME
Benchmarks: CUB-200-2011 Stanford Dogs FGVC-Aircraft HAM10000 Chest X-Ray Pneumonia PLD MMMU MMStar SEED-Bench-2 Plus MMBench MME
Metrics: classification accuracy explanation existence cognition score fluency (perplexity) concept selection precision
Baselines: Base LLaVA-1.5 Naive Label Fine-tuning Label with General Explanations (L+GE) GPT-4o concept extraction CLIP concept selection
Builds on: LLaVA-1.5 rejection sampling / best-of-N InfoNCE information bottleneck
Closest papers by meaning
- MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods (NeurIPS 2026 · similarity 0.90)
- Latent Implicit Visual Reasoning (CVPR 2026 · similarity 0.90)
- A Concept-Based Explainability Framework for Large Multimodal Models (NeurIPS 2024 · similarity 0.90)
- Toward Cognitive Supersensing in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.89)
- Long Grounded Thoughts: Synthesizing Visual Problems and Reasoning Chains at Scale (ICML 2026 · similarity 0.89)
- Thinking Diffusion: Penalize and Guide Visual-Grounded Reasoning in Diffusion Multimodal Language Models (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.