Research map / Image generation / Editing and style transfer
RelationAdapter: Learning and Transferring Visual Relation with Diffusion Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RelationAdapter, a lightweight DiT attention-injection module that extracts editing intent from source-target image pairs, plus the Relation252K dataset of 218 editing tasks, for visual prompt-based image editing built on FLUX.1-dev.
Problem
Single-reference visual prompt editing methods mainly handle style or appearance changes and struggle with non-rigid transformations, while concatenating exemplar pairs into the input is memory-hungry and degrades text prompt guidance.
Approach
A dual-branch SigLIP-encoded visual prompt (source-target pair) is injected into each DiT block through a decoupled key-value attention that shares the backbone query and is fused with weight alpha; the In-Context Editor adds positional encoding cloning and a noise-free clean source-condition paradigm, trained with LoRA.
Key results
- On the Relation252K benchmark, vs Edit Transfer MSE 0.020 vs 0.043 and CLIP-I 0.905 vs 0.827; vs VisualCloze MSE 0.025 vs 0.049 and CLIP-I 0.894 vs 0.802.
- Ablation: removing RelationAdapter raises seen-task MSE from 0.044 to 0.055 and lowers CLIP-I from 0.852 to 0.787.
- User study: 73.19% preference for edit accuracy over the in-context baseline on seen tasks, and 97.11% over Edit Transfer on edit accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RelationAdapter instead of EditTransfer | RelationAdapter · Common task | MSE: 0.043 -> 0.020 | Helped |
| use RelationAdapter instead of EditTransfer | RelationAdapter · Common task | CLIP-I: 0.827 -> 0.905 | Helped |
| use RelationAdapter instead of EditTransfer | RelationAdapter · Common task | GPT-C: 4.234 -> 4.437 | Helped |
| use RelationAdapter instead of EditTransfer | RelationAdapter · Common task | GPT-A: 3.508 -> 4.429 | Helped |
| use RelationAdapter instead of VisualCloze | RelationAdapter · Common task | MSE: 0.049 -> 0.025 | Helped |
| use RelationAdapter instead of VisualCloze | RelationAdapter · Common task | CLIP-I: 0.802 -> 0.894 | Helped |
| use RelationAdapter instead of VisualCloze | RelationAdapter · Common task | GPT-C: 3.594 -> 4.084 | Helped |
| use RelationAdapter instead of VisualCloze | RelationAdapter · Common task | GPT-A: 3.411 -> 3.918 | Helped |
| add RelationAdapter (ablation) | RelationAdapter · Seen tasks | MSE: 0.055 -> 0.044 | Helped |
| add RelationAdapter (ablation) | RelationAdapter · Seen tasks | CLIP-I: 0.787 -> 0.852 | Helped |
| add RelationAdapter (ablation) | RelationAdapter · Seen tasks | GPT-C: 3.909 -> 4.079 | Helped |
| add RelationAdapter (ablation) | RelationAdapter · Seen tasks | GPT-A: 3.597 -> 4.106 | Helped |
Surprising findings
- Against the in-context learning baseline on low-level tasks, the GPT-based scores slightly drop, and the reported depth and surface normal results are mixed; VisualCloze beats the method on edge detection precision and segmentation pixel accuracy.
Limitations
- Sometimes fails to render text details accurately.
- Performance varies on rare or unseen tasks, sensitive to task-specific nuances.
- Weaker fine-grained spatial alignment and low-resolution limits, as shown in failure cases.
Experimental setup
Datasets: Relation252K MidJourney-generated images GPT-4o-generated images public low-level benchmark datasets (BSDS500, COCO, NYUv2, BIPED)
Benchmarks: Relation252K benchmark subset (6,540 samples; seen and unseen tasks) BSDS500 COCO NYUv2 BIPED
Metrics: MSE CLIP-I GPT-C GPT-A user preference
Baselines: Edit Transfer VisualCloze in-context learning concatenation variant (w/o RA)
Builds on: FLUX.1-dev DiT FLUX.1 Redux LoRA VisualCloze Edit Transfer
Closest papers by meaning
- Textualize Visual Prompt for Image Editing via Diffusion Bridge (AAAI 2025 · similarity 0.91)
- An Item Is Worth a Prompt: Versatile Image Editing with Disentangled Control (AAAI 2025 · similarity 0.89)
- Localizing and Editing Knowledge In Text-to-Image Generative Models (ICLR 2024 · similarity 0.88)
- Concept Lancet: Image Editing with Compositional Representation Transplant (CVPR 2025 · similarity 0.88)
- Imagine Before You Draw: Visual Prompt Engineering for Image Generation (NeurIPS 2026 · similarity 0.88)
- ACE: All-round Creator and Editor Following Instructions via Diffusion Transformer (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.