Research map / Vision-language and multimodal / Multimodal LLMs
SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SymDPO, a DPO variant that replaces textual answers in in-context demonstrations with random symbols so LMMs must use visual context, improving Open-Flamingo and IDEFICS ICL performance.
Problem
Large multimodal models under-use visual information in multimodal in-context demonstrations (visual context overlook), relying instead on textual patterns, and standard DPO data is poorly suited to fixing this.
Approach
Build DPO preference data where demonstration answers are replaced by semantically meaningless symbols, with the chosen answer being the symbol matching the query's demonstration, forcing the model to link demonstration images to symbols rather than rely on text.
Key results
- OF-3B with SymDPO at 4-shot improves COCO Caption CIDEr from 82.7 to 87.4 (+4.7) and Flickr30K from 59.1 to 61.2.
- IDEFICS-9B with SymDPO at 16-shot improves COCO CIDEr from 99.7 to 107.9 (+8.2) and OK-VQA from 48.4 to 50.6.
- General DPO and Video DPO give little or negative gains compared to SymDPO across most settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace demonstration answers with symbols and train with SymDPO | Open-Flamingo 3B (I) · COCO Caption, 4-shot | CIDEr: 82.7 -> 87.4; +4.7 | Helped |
| Replace demonstration answers with symbols and train with SymDPO | Open-Flamingo 3B (I) · Flickr-30K, 4-shot | CIDEr: 59.1 -> 61.2; +2.1 | Helped |
| Replace demonstration answers with symbols and train with SymDPO | Open-Flamingo 3B (I) · VQAv2, 4-shot | accuracy: 45.7 -> 46.2; +0.5 | Helped |
| Replace demonstration answers with symbols and train with SymDPO | Open-Flamingo 3B (I) · OK-VQA, 4-shot | accuracy: 30.6 -> 31.6; +1.0 | Helped |
| Replace demonstration answers with symbols and train with SymDPO | Open-Flamingo 3B (I) · TextVQA, 4-shot | accuracy: 28.1 -> 28.3; +0.2 | Helped |
| Replace answers with symbols, but use standard supervised symbol fine-tuning rather than DPO (ablation) | Open-Flamingo 3B · COCO Caption | CIDEr | Hurt |
| Add RICES demonstration selection to SymDPO (ablation) | Open-Flamingo 3B (I) · COCO Caption, 4-shot | CIDEr: 82.7 -> 93.5; +10.8 | Helped |
| Add RICES demonstration selection to SymDPO (ablation) | Open-Flamingo 3B (I) · Flickr-30K, 4-shot | CIDEr: 59.1 -> 62.0; +2.9 | Helped |
| Add RICES demonstration selection to SymDPO (ablation) | Open-Flamingo 3B (I) · VQAv2, 4-shot | accuracy: 45.7 -> 46.6; +0.9 | Helped |
| Add RICES demonstration selection to SymDPO (ablation) | Open-Flamingo 3B (I) · OK-VQA, 4-shot | accuracy: 30.6 -> 33.4; +2.8 | Helped |
| Add RICES demonstration selection to SymDPO (ablation) | Open-Flamingo 3B (I) · TextVQA, 4-shot | accuracy: 28.1 -> 29.1; +1.0 | Helped |
| Use RICES without SymDPO (ablation) | Open-Flamingo 3B (I) · Flickr-30K, 4-shot | CIDEr: 59.1 -> 53.9; −5.2 | Hurt |
Surprising findings
- Open-Flamingo performs equal or better when demonstration images are replaced with blanks or removed entirely, indicating heavy reliance on text in demonstrations.
- Symbol tuning (SymTune) via standard fine-tuning fails to improve and even declines on captioning, whereas symbol DPO improves.
Limitations
- Gains on some benchmarks (e.g., TextVQA, VQAv2) are small, and some baseline settings show negative deltas.
- Exclusive reliance on symbolic data does not give optimal performance; mixing ratio is task-dependent.
Experimental setup
Datasets: VQAv2 GQA ImageNet
Benchmarks: COCO Caption Flickr30K VQAv2 OK-VQA TextVQA
Metrics: CIDEr accuracy
Baselines: General DPO Video DPO MIA-DPO Symbol Tuning (SymTune) RICES
Builds on: DPO Symbol Tuning Flamingo RICES
Closest papers by meaning
- AIM: Let Any Multimodal Large Language Models Embrace Efficient In-Context Learning (AAAI 2025 · similarity 0.91)
- VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning (ICLR 2025 · similarity 0.88)
- LIVE: Learnable In-Context Vector for Visual Question Answering (NeurIPS 2024 · similarity 0.88)
- Large Language Models are Demonstration Pre-Selectors for Themselves (ICML 2025 · similarity 0.88)
- Beyond task performance: evaluating and reducing the flaws of large multimodal models with in-context-learning (ICLR 2024 · similarity 0.87)
- In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.