Research map / Image generation / Text-to-image diffusion
Novel Object Synthesis via Adaptive Text-Image Harmony
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Adaptive Text-Image Harmony (ATIH), a training-free SDXL-Turbo-based inversion-and-editing method that balances object text and image inputs to synthesize novel hybrid objects.
Problem
Object synthesis that combines an object text and an object image into a new object image; diffusion models often over-weight either the text or the image, yielding a result dominated by one input.
Approach
Introduces a cross-attention scale factor alpha, an injection step i for self-attention, and a balance loss that optimizes the sampling noise; alpha and i are adapted via a similarity score (DINO image similarity, CLIP text similarity) with golden-section search to harmonize text and image integration.
Key results
- On the TIF dataset, ATIH reaches AES 6.124 and HPS 0.383 and the best F score (1.362) and lowest balance similarity (0.075), versus 1.174 F score for MagicMix and 0.768 for InstructPix2Pix.
- In user studies (570 votes from 95 participants), ATIH received 74.03% and 79.47% of votes across the two comparisons.
- ATIH runs in about ten seconds per image on SDXL-Turbo with four denoising steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ATIH instead of InfEdit | Object synthesis on TIF dataset · TIF dataset | AES: 6.080 -> 6.124 | Helped |
| use ATIH instead of MagicMix | Object synthesis on TIF dataset · TIF dataset | HPS: 0.373 -> 0.383 | Helped |
| use ATIH instead of MagicMix | Object synthesis on TIF dataset · TIF dataset | F-score: 1.174 -> 1.362 | Helped |
| use ATIH instead of MagicMix | Object synthesis on TIF dataset · TIF dataset | Bsim: 0.167 -> 0.075 | Helped |
| set lambda to 125 rather than 0 (ablation) | ATIH · Class fusion dataset | AES: 6.116 -> 6.153 | Helped |
| set lambda to 125 rather than 0 (ablation) | ATIH · Class fusion dataset | CLIP-T: 0.413 -> 0.417 | Helped |
| set lambda to 125 rather than 0 (ablation) | ATIH · Class fusion dataset | Dino-I: 0.927 -> 0.902 | Hurt |
| set lambda to 260 rather than 125 (ablation) | ATIH · Class fusion dataset | AES: 6.153 -> 6.012 | Hurt |
| set lambda to 260 rather than 125 (ablation) | ATIH · Class fusion dataset | CLIP-T: 0.417 -> 0.419 | Helped |
| set lambda to 260 rather than 125 (ablation) | ATIH · Class fusion dataset | Dino-I: 0.902 -> 0.760 | Hurt |
| add the balance loss for optimizing sampling noise (ablation) | ATIH object synthesis · Object synthesis examples | image fidelity | Helped |
| add adaptive injection of self-attention features (ablation) | ATIH object synthesis · Corgi-to-Fire Engine synthesis | fusion smoothness | Helped |
Surprising findings
- InfEdit and InstructPix2Pix achieve higher DINO-I and CLIP-T scores than ATIH, but worse aesthetic, HPS and balance scores, showing that single similarity metrics favor one input.
Limitations
- Weak semantic correlation between the two categories yields texture changes rather than deeper semantic transformations.
- Parameters (k, I_sim min/max, lambda) are set by observation and experience rather than learned.
- Evaluated on a small TIF set of 1,800 pairs built from 60 texts and 30 images.
Experimental setup
Datasets: PIE-bench ImageNet
Benchmarks: TIF
Metrics: DINO-I CLIP-T AES HPS F score B sim
Baselines: MasaCtrl InfEdit InstructPix2Pix MagicMix ConceptLab ControlNet PnPinv Kosmos-G
Builds on: Plug-and-Play PnP inversion MasaCtrl Prompt-to-Prompt ReNoise inversion
Closest papers by meaning
- Object-Aware Inversion and Reassembly for Image Editing (ICLR 2024 · similarity 0.89)
- Progressive Text-to-Image Diffusion with Soft Latent Direction (AAAI 2024 · similarity 0.89)
- Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing (CVPR 2024 · similarity 0.89)
- Multi-Region Text-Driven Manipulation of Diffusion Imagery (AAAI 2024 · similarity 0.88)
- RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Models (NeurIPS 2024 · similarity 0.88)
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.