Research map / Image generation / Editing and style transfer

ATA: Adaptive Transformation Agent for Text-Guided Subject-Position Variable Background Inpainting

CVPR 2025 · Image generation › Editing and style transfer · approach: Texture and sketch synthesis · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes subject-position variable background inpainting, where a Hunyuan-DiT-based Adaptive Transformation Agent (ATA) predicts subject displacement via reverse-ordered PosAgent blocks with SFT and a position switch embedding for fixed/variable modes.

Problem

Foreground-conditioned background inpainting usually pins the subject to its original position, which can conflict with the generated background and produce inconsistent subject-background layouts.

Approach

Adaptively predict subject displacement (scale and shift) at multiple feature levels using PosAgent blocks arranged in a reverse (deep-to-shallow) structure with SFT, plus a learnable position switch embedding that toggles between variable and fixed subject position, trained with a 1:1 hybrid dataset.

Key results

• ATA achieves the best FID of 79.66 versus 81.14 for SD3-ControlNet and 82.82 for FLUX-ControlNet in Table 1.
• ATA reduces extension ratio to 7.92% and multi-subject rate to 4.73%, versus best baseline values of 14.27% and 9.37%.
• ATA achieves GPT-4o position rationality of 92.8 and VQA score of 0.883, the highest rationality score among compared methods.

Measured effects reported in the paper

Surprising findings

• Removing the position switch and training only on variable-position data gave unstable performance and worse metrics than with the switch.
• Without text embedding in PosAgent, the multi-subject rate slightly improved while image quality and text alignment worsened.

Limitations

• Position rationality is judged by GPT-4o and user studies rather than a standardized metric.
• Variable-position training pairs are synthetic, created by centering and enlarging subjects from fixed-position data.

Experimental setup

Datasets: 79K collected subject images (following Pinco) datasets used for inpainting tasks (cited, rejected for quality)

Benchmarks: in-house text-guided subject-position variable background inpainting test set

Metrics: FID OER extension ratio (BiRefNet) VQA Score FlorenceV2 multi-subject rate GPT-4o position rationality user study preference

Baselines: PowerPaint BrushNet-SD1.5 BrushNet-SDXL LayerDiffusion SD3-ControlNet FLUX-ControlNet GLIGEN ELITE BLIP-Diffusion

Builds on: Hunyuan-DiT Pinco SFT BrushNet Self-IPA

Closest papers by meaning

• HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models (ICLR 2025 · similarity 0.84)
• Large-Scale Text-to-Image Model with Inpainting is a Zero-Shot Subject-Driven Image Generator (CVPR 2025 · similarity 0.84)
• MTADiffusion: Mask Text Alignment Diffusion Model for Object Inpainting (CVPR 2025 · similarity 0.84)
• TurboFill: Adapting Few-step Text-to-image Model for Fast Image Inpainting (CVPR 2025 · similarity 0.84)
• Brush2Prompt: Contextual Prompt Generator for Object Inpainting (CVPR 2024 · similarity 0.83)
• Keyframe-Guided Creative Video Inpainting (CVPR 2025 · similarity 0.82)