Research map / Large language models / Alignment and preferences
DAMA: Data- and Model-aware Alignment of Multi-modal LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DAMA, a DPO variant for MLLM hallucination alignment that scales beta per instance by CLIP-based data hardness and by batch-level implicit reward gaps (model responsiveness), reporting large gains on Object HalBench, AMBER, MMHal-Bench, LLaVA-Bench and MM-Vet.
Problem
DPO for multimodal LLM alignment responds imbalanced to preference pairs of varying hardness: it overfits easy-to-distinguish pairs and underfits hard-to-distinguish ones under a static beta.
Approach
Dynamically scale the DPO temperature beta per instance by combining a data-aware factor (CLIP similarity of LLM-split sub-sentences, softmax-normalized, giving pair hardness) with a model-aware factor (outlier-filtered batch mean of implicit reward gaps, moving-averaged), multiplied element-wise.
Key results
- On Object HalBench, DAMA-7B reaches 90.9 response-level and 95.3 mentioned-level non-hallucination versus 78.3 and 89.5 for LLaVA-1.5-7B+DPO.
- On AMBER, DAMA-13B reaches 84.3 accuracy and 88.1 F1, above GPT-4V (83.4 and 87.4).
- Ablation on Object HalBench: DAMA (90.87 response) exceeds D2PO (82.54) and MDPO (88.00) alone over DPO (78.29).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace DPO with DAMA | LLaVA-1.5-7B · Object HalBench | response-level non-hallucination rate: 78.3 -> 90.9 | Helped |
| Replace DPO with DAMA | LLaVA-1.5-13B · AMBER | Accuracy: 78.5 -> 84.3 | Helped |
| Add data-aware optimization to DPO (D²PO) (ablation) | LLaVA-1.5-7B · Object HalBench | response-level non-hallucination rate: 78.29 -> 82.54 | Helped |
| Add data-aware optimization to DPO (D²PO) (ablation) | LLaVA-1.5-7B · Object HalBench | mentioned-level non-hallucination rate: 89.48 -> 90.64 | Helped |
| Add model-aware optimization to DPO (MDPO) (ablation) | LLaVA-1.5-7B · Object HalBench | response-level non-hallucination rate: 78.29 -> 88.00 | Helped |
| Add model-aware optimization to DPO (MDPO) (ablation) | LLaVA-1.5-7B · Object HalBench | mentioned-level non-hallucination rate: 89.48 -> 93.74 | Helped |
| Transform CLIP similarity scores into probabilities for hardness estimation (ablation) | D²PO · Object HalBench | response-level non-hallucination rate: 80.65 -> 82.54 | Helped |
| Transform CLIP similarity scores into probabilities for hardness estimation (ablation) | D²PO · Object HalBench | mentioned-level non-hallucination rate: 89.62 -> 90.64 | Helped |
| Filter only the largest-distance samples (Bottom) instead of filtering both extremes (ablation) | MDPO · Object HalBench | response-level non-hallucination rate: 86.66 -> 76.36 | Hurt |
| Filter only the smallest-distance samples (Top) instead of filtering both extremes (ablation) | MDPO · Object HalBench | response-level non-hallucination rate: 86.66 -> 86.34 | Hurt |
| Filter both largest- and smallest-distance samples (ablation) | MDPO · Object HalBench | response-level non-hallucination rate: 86.66 -> 88.00 | Helped |
| Combine data-aware and model-aware factors by multiplication rather than weighted sum (ablation) | DAMA · Object HalBench | response-level non-hallucination rate | Helped |
Surprising findings
- The weighted-sum combination of the two factors underperforms element-wise multiplication by a significant margin, per the authors.
- Data-aware optimization alone gives only modest gains over model-aware optimization, which the authors attribute to high preference data quality.
Limitations
- Evaluated only on image-text MLLMs; video and audio not explored.
- Does not test commercial multimodal LLMs.
- Data hardness relies on CLIP similarity of LLM-decomposed sub-sentences, which adds an LLM preprocessing step.
Experimental setup
Datasets: released preference dataset of 22k pairs (from prior work) Object HalBench AMBER MMHal-Bench LLaVA-Bench MM-Vet
Benchmarks: Object HalBench AMBER MMHal-Bench LLaVA-Bench MM-Vet
Metrics: response-level non-hallucination rate mentioned-level non-hallucination rate accuracy F1 MMHal score hallucination rate LLaVA-Bench score MM-Vet score
Baselines: LLaVA-1.5 DPO VCD Less-is-more OPERA CCA-LLaVA HA-DPO POVID RLHF-V RLAIF-V CSR TPO GPT-4V
Builds on: DPO RLHF-V CLIP
Closest papers by meaning
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.91)
- Panacea: Pareto Alignment via Preference Adaptation for LLMs (NeurIPS 2024 · similarity 0.90)
- Robust LLM Alignment via Distributionally Robust Direct Preference Optimization (NeurIPS 2025 · similarity 0.90)
- Direct Density Ratio Optimization: A Statistically Consistent Approach to Aligning Large Language Models (ICML 2025 · similarity 0.90)
- ComPO: Preference Alignment via Comparison Oracles (NeurIPS 2025 · similarity 0.89)
- TODO: Enhancing LLM Alignment with Ternary Preferences (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.