Research map / Large language models / Alignment and preferences

Aligning Visual Contrastive learning models via Preference Optimization

ICLR 2025 · Large language models › Alignment and preferences · approach: DPO design choices · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Applies DPO, IPO, and KTO preference optimization to CLIP-style contrastive vision-language models, framing contrastive classification as a one-step MDP, to improve robustness to typographic attacks and disentangle gender concepts, with a linear SVD-based head for post-hoc control.

Problem

Contrastive vision-language models like CLIP are vulnerable to typographic attacks and encode gender and task biases, and preference optimization has not been explored for non-generative contrastive models.

Approach

Reformulates contrastive learning as a one-step MDP with a softmax policy over image-caption similarities, derives DPO/IPO gradients for CLIP embeddings, adds a KL regularizer on clean data, and controls behavior post-training by scaling singular values of a learned linear projection (W_t = U Σ^t V^T).

Key results

• On typographic-attack variants of 9 datasets, Ours (KTO) averages 51.74% vs 31.31% for CLIP and best baseline (PAINT/Defense-Prefix) with up to 19.63-point gain on EuroSAT.
• On real-world typographic datasets, Ours (IPO) averages 67.54% vs 64.52% for Defense-Prefix and 46.82% for CLIP.
• Preference-optimized models show lower KL divergence to the reference than cross-entropy fine-tuning, and singular-value scaling trades off OCR vs object detection without retraining.

Measured effects reported in the paper

Surprising findings

• On the original (non-typographic) datasets, the proposed methods lose accuracy relative to the best baseline on several datasets (e.g., Flowers102 drops 5.63 points for KTO-vs-best-baseline difference).

Limitations

• Preference data is synthetic (generated typographic attacks) rather than human-curated.
• Accuracy on clean original datasets drops relative to some baselines on several benchmarks.
• Large positive or negative scaling of singular values requires normalization to remain stable.

Experimental setup

Datasets: ImageNet-100 Food101 VL-Bias Caltech101 OxfordPets StanfordCars Flowers102 FGVCAircraft DTD SUN397 EuroSAT Materzynska

Benchmarks: Caltech101 OxfordPets StanfordCars Flowers102 FGVCAircraft DTD SUN397 EuroSAT Materzynska PAINT RTA-100

Metrics: classification accuracy KL divergence to reference model percentage of typographic label predictions

Baselines: CLIP Materzynska+ PAINT Defense-Prefix

Builds on: DPO IPO KTO PAINT Defense-Prefix

Closest papers by meaning

• Multimodal Large Language Models Make Text-to-Image Generative Models Align Better (NeurIPS 2024 · similarity 0.90)
• Diffusion Model Alignment Using Direct Preference Optimization (CVPR 2024 · similarity 0.88)
• RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data (AAAI 2025 · similarity 0.88)
• Personalized Preference Fine-tuning of Diffusion Models (CVPR 2025 · similarity 0.88)
• The "Law'' of the Unconscious Contrastive Learner: Probabilistic Alignment of Unpaired Modalities (ICLR 2025 · similarity 0.88)
• DrPO: Drifting Preference Optimization for One-Step Generative Models (NeurIPS 2026 · similarity 0.87)