Research map / Image generation / Text-to-image diffusion
VerbDiff: Text-Only Diffusion Models with Enhanced Interaction Awareness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes Stable Diffusion v1.4 cross-attention layers on HICO-DET with triplet, alignment, and direction losses to better distinguish interaction verbs in text-only generation, without bounding boxes.
Problem
Text-to-image diffusion models struggle to depict human-object interactions accurately because they fail to distinguish semantically different interaction verbs, partly due to CLIP's object bias; prior fixes rely on extra bounding-box or LLM conditions.
Approach
Relation Disentanglement Guidance pushes generated image features away from frequency-based anchor-verb text and toward real-image features (with a triplet loss and class-balanced weighting), and Interaction Direction Guidance aligns feature differences within localized interaction regions extracted from cross-attention centroids, all without extra grounding conditions.
Key results
- VerbDiff reaches 0.733 CLIP T2T and 0.633 S-BERT T2T similarity vs. 0.725 and 0.251/0.620 for SD in Table 1 (S-BERT row as reported).
- On HICO-DET HOI accuracy with SOV-STG-Swin-L (Default Full), VerbDiff scores 27.05 vs. 21.69 for SD and 24.86 for InteractDiffusion.
- Ablation shows RDG and IDG each improve HOI accuracy over reconstruction loss alone; the full combination gives the best scores (27.05 Default Full).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VerbDiff instead of Stable Diffusion | Stable Diffusion v1.4 · HICO-DET | CLIP T2T similarity: 0.725 -> 0.733 | Helped |
| use VerbDiff instead of Stable Diffusion | Stable Diffusion v1.4 · HICO-DET | CLIP T2I similarity: 0.251 -> 0.242 | Hurt |
| use VerbDiff instead of Stable Diffusion | Stable Diffusion v1.4 · HICO-DET | S-BERT T2T similarity: 0.620 -> 0.633 | Helped |
| use VerbDiff instead of Stable Diffusion | Stable Diffusion v1.4 · HICO-DET | VQA-Score I2T alignment: 0.769 -> 0.771 | Helped |
| use VerbDiff instead of Stable Diffusion | Stable Diffusion v1.4 · HICO-DET | VQA score: 0.765 -> 0.766 | Helped |
| use VerbDiff instead of InteractDiffusion | InteractDiffusion · HICO-DET, SOV-STG-Swin-L Default Full | accuracy: 23.53 -> 27.05 | Helped |
| add Interaction Direction Guidance (IDG) (ablation) | VerbDiff · HICO-DET, full setting | S-BERT T2T similarity: 0.589 -> 0.633 | Helped |
| add Relation Disentanglement Guidance (RDG) (ablation) | VerbDiff · HICO-DET, full setting | S-BERT T2T similarity: 0.588 -> 0.633 | Helped |
| add image alignment loss Lalign (ablation) | VerbDiff with reconstruction, Ltriple, and IDG · HICO-DET, full setting | S-BERT T2T similarity: 0.610 -> 0.633 | Helped |
| add IDG and Lalign (ablation) | VerbDiff with reconstruction loss only · HICO-DET, full setting | S-BERT T2T similarity: 0.582 -> 0.633 | Helped |
| add Interaction Direction Guidance (IDG) (ablation) | VerbDiff · HICO-DET, full setting | SOV-STG-Swin-L Default Full accuracy: 20.32 -> 27.05 | Helped |
| add Relation Disentanglement Guidance (RDG) (ablation) | VerbDiff · HICO-DET, full setting | SOV-STG-Swin-L Default Full accuracy: 20.21 -> 27.05 | Helped |
Surprising findings
- Using only reconstruction plus IDG (without RDG triplet and alignment losses) yields a larger gain than RDG alone in the ablation table, and the paper notes that the model without the align loss scores well.
Limitations
- Evaluated only on HICO-DET with 501 interaction prompts after excluding multi-human/object classes containing 'and'.
- Text-to-image score is lower than vanilla SD, and VQA gains over SD are slight.
Experimental setup
Datasets: HICO-DET
Benchmarks: HICO-DET SOV-STG HOI accuracy
Metrics: CLIP score Sentence-BERT similarity VQAScore HOI classification accuracy
Baselines: Stable Diffusion GLIGEN InteractDiffusion
Builds on: Stable Diffusion CLIP InteractDiffusion class-balanced loss triplet loss
Closest papers by meaning
- InteractDiffusion: Interaction Control in Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.90)
- Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models (NeurIPS 2024 · similarity 0.89)
- De-Diffusion Makes Text a Strong Cross-Modal Interface (CVPR 2024 · similarity 0.88)
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.88)
- CONFORM: Contrast is All You Need for High-Fidelity Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.88)
- Diffusion PID: Interpreting Diffusion via Partial Information Decomposition (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.