Research map / Vision-language and multimodal / CLIP and zero-shot
Cropper: Vision-Language Model for Image Cropping through In-Context Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Cropper adapts black-box VLMs (Gemini 1.5 Pro, GPT-4o) to image cropping via CLIP-based retrieval of in-context examples and feedback-driven iterative crop refinement, training-free, across free-form, subject-aware, and aspect-ratio-aware cropping.
Problem
Existing image cropping methods are trained on specific datasets and require retraining for new requirements or cropping constraints, while VLM in-context learning has not been explored for cropping.
Approach
Use CLIP-embedding-based retrieval of top-S similar images with their ground-truth crops as in-context examples, then iteratively refine VLM-proposed crops using feedback scores from an aesthetic scorer (VILA), CLIP content similarity, and area size, with no weight updates.
Key results
- On GAICD free-form cropping, Cropper reaches SRCC 0.904 and Acc5 84.3 versus 0.872 SRCC and 64.8 Acc5 for the best prior method (Chao et al.), without training.
- On SACD subject-aware cropping, Cropper achieves IoU 0.769 and Disp 0.0372, beating SAC-Net (0.767, 0.0491).
- On FCDB aspect-ratio-aware cropping, Cropper reaches IoU 0.756 versus 0.735 for Mars; user study shows 60.8-63.4% preference over baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Cropper instead of Chao et al. | Free-form cropping · GAICD test | Acc5: 70.0 -> 84.3 | Helped |
| Use Cropper instead of SAC-Net | Subject-aware cropping · SACD test | Disp: 0.0491 -> 0.0372 | Helped |
| Use Cropper instead of Mars | Aspect ratio-aware cropping · FCDB | IoU: 0.735 -> 0.756 | Helped |
| Use CLIP top-S retrieval instead of random retrieval (ablation) | Cropper · GAICD test free-form | IoU: 0.740 -> 0.748 | Helped |
| Add in-context examples and Cropper retrieval instead of zero-shot inference (ablation) | Gemini 1.5 Pro on Cropper · GAICD test free-form | IoU: 0.509 -> 0.748 | Helped |
| Add iterative refinement (ablation) | Cropper · GAICD test free-form | IoU: 0.722 -> 0.748 | Helped |
| Add iterative refinement (ablation) | Cropper · FCDB free-form | IoU: 0.642 -> 0.667 | Helped |
| Select the final-iteration output instead of the highest-scoring crop across iterations (ablation) | Cropper · GAICD test free-form | IoU: 0.714 -> 0.748 | Helped |
| Select the final-iteration output instead of the highest-scoring crop across iterations (ablation) | Cropper · SACD subject-aware | IoU: 0.760 -> 0.769 | Helped |
| Select the final-iteration output instead of the highest-scoring crop across iterations (ablation) | Cropper · FCDB aspect ratio-aware | IoU: 0.756 -> 0.714 | Hurt |
| Replace Gemini 1.5 Pro with Gemini 1.5 Flash (ablation) | Cropper · GAICD test free-form | IoU: 0.748 -> 0.781 | Helped |
| Replace Gemini 1.5 Pro with Mantis-8B-Idefics2 (ablation) | Cropper · GAICD test free-form | IoU: 0.748 -> 0.672 | Hurt |
Surprising findings
- Zero-shot Gemini 1.5 Pro achieves only 0.509 IoU on GAICD, far below Cropper's 0.748, showing that the raw VLM struggles with cropping without ICL and refinement.
- Gemini-1.5-flash gives higher Acc5 (87.2) than Gemini-1.5-pro (84.3) on GAICD, though with lower SRCC and PCC.
Limitations
- Inference speed is constrained by retrieving suitable in-context examples from the dataset.
- Relies on closed-source API VLMs whose latency is high (about 5.83 s per call for Gemini 1.5 Pro).
Experimental setup
Datasets: GAICD FCDB SACD
Benchmarks: GAICD FCDB SACD
Metrics: SRCC PCC Acc K/N IoU Disp user preference
Baselines: A2RL VPN VFN VEN GAIC CGS TransView Chao et al. SAC-Net LVRN Mars GPT-4o
Builds on: in-context learning visual in-context learning VILA CLIP
Closest papers by meaning
- Learning Subject-Aware Cropping by Outpainting Professional Photos (AAAI 2024 · similarity 0.87)
- From Local Details to Global Context: Advancing Vision-Language Models with Attention-Based Selection (ICML 2025 · similarity 0.87)
- Global Compression Commander: Plug-and-Play Inference Acceleration for High-Resolution Large Vision-Language Models (AAAI 2026 · similarity 0.86)
- FOCUS: Internal MLLM Representations for Efficient Fine-Grained Visual Question Answering (NeurIPS 2025 · similarity 0.86)
- Spatial-Semantic Collaborative Cropping for User Generated Content (AAAI 2024 · similarity 0.85)
- Unifying Image Processing as Visual Prompting Question Answering (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.