Research map / Image generation / Text-to-image diffusion

CustomContrast: A Multilevel Contrastive Perspective for Subject-Driven Text-to-Image Customization

AAAI 2025 · Image generation › Text-to-image diffusion · approach: Multi-subject personalization · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CustomContrast, a finetune-free subject-driven T2I customization method that uses multilevel contrastive learning (cross-modal semantic and multiscale appearance) with a multimodal Qformer-based feature injection encoder to disentangle subject intrinsic features from irrelevant attributes.

Problem

Subject-driven text-to-image customization suffers a trade-off between subject similarity and text controllability because self-reconstruction misconstrues irrelevant attributes (view, pose, background) as intrinsic subject attributes.

Approach

Replace self-reconstruction with a cross-differential view: contrastive learning enforces intra-consistency across views of the same subject and inter-distinctiveness across subjects, with CLIP-based appearance similarity scaling factors aligning learned distances to real subject distances.

Key results

• On SD-v1.5, CustomContrast reaches CLIP-T 0.325 and ImageReward 0.45 versus 0.313 and -0.24 for MOMA, with E-DI 0.591.
• On SDXL, CLIP-T 0.329 and E-DI 0.597, improving over MS-Diffusion (0.312) and EMU2 (0.583).
• Ablation: adding CSCL and MACL to the MFI-encoder raises E-CI from 0.754 to 0.788 and E-DI from 0.563 to 0.591 on SD-v1.5.

Measured effects reported in the paper

Limitations

• Relies on segmented subject images and Grounded SAM for similarity evaluation and positive-sample construction.
• Positive samples depend on frames of the same subject from multi-view datasets.

Experimental setup

Datasets: MVImageNet (filtered subset) OpenImage DreamBench

Benchmarks: DreamBench

Metrics: DINO-I CLIP-I CLIP-T ImageReward

Baselines: Custom Diffusion DreamMatcher BLIP-Diffusion ELITE IP-Adapter-plus SSR-Encoder KOSMOS-G MOMA λ-ECLIPSE IP-Adapter-XL EMU2 MS-Diffusion

Builds on: Stable Diffusion IP-Adapter ELITE MOMA CLIP

Closest papers by meaning

• Direct Consistency Optimization for Robust Customization of Text-to-Image Diffusion models (NeurIPS 2024 · similarity 0.86)
• Decoupled Textual Embeddings for Customized Image Generation (AAAI 2024 · similarity 0.86)
• Harmonizing Visual and Textual Embeddings for Zero-Shot Text-to-Image Customization (AAAI 2025 · similarity 0.86)
• MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance (ICLR 2025 · similarity 0.86)
• Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directions (CVPR 2025 · similarity 0.85)
• Selectively Informative Description can Reduce Undesired Embedding Entanglements in Text-to-Image Personalization (CVPR 2024 · similarity 0.85)