Research map / Vision-language and multimodal / Human and embodied modeling

Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulation

CVPR 2025 · Vision-language and multimodal › Human and embodied modeling · approach: Scene graph generation · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes InstaManip, a multimodal autoregressive model (LLaMA2-13B backbone with LoRA) that performs few-shot text-plus-exemplar image manipulation via group self-attention separating learning and applying stages, plus a CLIP-based relation regularization.

Problem

Few-shot text-guided image manipulation with visual exemplars requires reasoning to separate transformation from content, which diffusion-based in-context editors handle poorly; in-context learning for this task with autoregressive models is understudied.

Approach

Splits the in-context prompt into two attention groups joined only by learnable manipulation tokens (learning stage on exemplars/instruction, applying stage on query image and generation tokens), and regularizes manipulation-token similarity to match CLIP text-instruction similarity.

Key results

• In-distribution, InstaManip reaches CLIP-Dir 19.81 and CLIP-Vis 32.39 vs. best prior few-shot baseline PromptDiffusion 17.13 and 27.69.
• Out-of-distribution, InstaManip achieves CLIP-Dir 18.27 and CLIP-Vis 28.23 vs. PromptDiffusion 15.41 and 25.49.
• User study shows InstaManip preferred over ImageBrush, VISII and PromptDiffusion by a margin of at least 19% in both settings.

Measured effects reported in the paper

Surprising findings

• Visual exemplars alone outperform textual instructions alone in the ablation, which the authors attribute to a text-image token domain gap in the MLLM feature space.
• CLIP-I is judged an unreliable metric for this task, since very high scores indicate trivial edits and InstaManip lags InstructPix2Pix on it.

Limitations

• Out-of-distribution performance drops notably relative to in-distribution.
• CLIP-based metrics are flawed for judging edit quality (CLIP-I), so evaluation relies partly on human study.

Experimental setup

Datasets: InstructPix2Pix dataset

Benchmarks: InstructPix2Pix dataset held-out test set (in-distribution and out-of-distribution splits)

Metrics: CLIP-Dir CLIP-Vis CLIP-T CLIP-I human preference rate

Baselines: InstructPix2Pix ImageBrush VISII PromptDiffusion

Builds on: InstructPix2Pix Emu SEED-X LLaMA2

Closest papers by meaning

• RelationAdapter: Learning and Transferring Visual Relation with Diffusion Transformers (NeurIPS 2025 · similarity 0.88)
• InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists (ICLR 2024 · similarity 0.88)
• Textualize Visual Prompt for Image Editing via Diffusion Bridge (AAAI 2025 · similarity 0.87)
• Learning to Manipulate Artistic Images (AAAI 2024 · similarity 0.87)
• Prompt Augmentation for Self-supervised Text-guided Image Manipulation (CVPR 2024 · similarity 0.87)
• Devil is in the Detail: Towards Injecting Fine Details of Image Prompt in Image Generation via Conflict-free Guidance and Stratified Attention (CVPR 2025 · similarity 0.86)