Research map / Vision-language and multimodal / Multimodal LLMs
Mini-Monkey: Alleviating the Semantic Sawtooth Effect for Lightweight MLLMs via Complementary Image Pyramid
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Complementary Image Pyramid (CIP), a plug-and-play multi-scale cropping scheme that mitigates the semantic sawtooth effect in high-res MLLMs, plus a parameter-free Scale Compression Mechanism (SCM) for token pruning, yielding the 2B Mini-Monkey.
Problem
Sliding-window or non-overlapping cropping of high-resolution images cuts objects and words, causing semantic discontinuity that hurts lightweight MLLMs on fine-grained and text-heavy understanding.
Approach
Build a complementary image pyramid from three aspect-ratio groups (detailed, adaptive, global) so that crop boundaries of the detailed tiles are covered by offset adaptive tiles and a global view, and compress redundant detailed tokens using attention from the LLM's early layers.
Key results
- Mini-Monkey (2B) reaches 806 on OCRBench, surpassing InternVL2-8B (794) by 12 points.
- CIP on InternVL2-2B training-free raises TextVQA from 73.4 to 75.2 and OCRBench from 784 to 800; with fine-tuning it reaches 76.0 and 806.
- SCM at 50% compression achieves TextVQA 74.7 and OCRBench 794 versus FastV at 73.4 and 781.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CIP to InternVL2-2B (ablation) | InternVL2-2B · OCRBench | score: 784 -> 806 | Helped |
| Add CIP to InternVL2-2B (ablation) | InternVL2-2B · TextVQA | VQA score: 73.4 -> 76.0 | Helped |
| Fine-tune InternVL2-2B without CIP (ablation) | InternVL2-2B · MME | score: 1876.8 -> 1858.3 | Hurt |
| Add CIP during fine-tuning (ablation) | InternVL2-2B · OCRBench | score: 784 -> 806 | Helped |
| Replace CIP with overlapping cropping (ablation) | Baseline · TextVQA | VQA score: 73.4 -> 70.6 | Hurt |
| Replace CIP with fixed-size high-resolution cropping (ablation) | Baseline · OCRBench | score: 806 -> 772 | Hurt |
| Add CIP to MiniCPM-V-2-2.8B (ablation) | MiniCPM-V-2-2.8B · OCRBench | score: 605 -> 627 | Helped |
| Add CIP to LLaVA-OV-0.5B (ablation) | LLaVA-OV-0.5B · OCRBench | score: 577 -> 600 | Helped |
| Add CIP to InternVL2-1B (ablation) | InternVL2-1B · TextVQA | VQA score: 70.5 -> 72.3 | Helped |
| Use SCM with 90% token compression (ablation) | Mini-Monkey · OCRBench | score: 792 -> 801 | Helped |
| Replace SCM with pooling at 50% compression (ablation) | Mini-Monkey · OCRBench | score: 794 -> 256 | Hurt |
| Increase baseline sub-image count from 24 to 48 (ablation) | Baseline · OCRBench | score: 783 -> 767 | Hurt |
Surprising findings
- Directly fine-tuning InternVL2-2B without CIP slightly degrades MME (-18.5) and TextVQA, while fine-tuning with CIP improves performance.
- Overlapping cropping degrades performance (e.g., TextVQA 70.6 vs 73.4 baseline), attributed to duplicate information.
Limitations
- Image-centric pyramid without trainable parameters; the authors propose a trainable FPN as future work.
- Overlapping cropping can introduce hallucinations and degrade performance.
- Evaluation of document accuracy metric depends on response containing ground truth, a relaxed criterion.
Experimental setup
Datasets: DocVQA ChartQA DVQA AI2D GeoQA+ LLaVA-150K (zh)
Benchmarks: MME RealWorldQA AI2D CCBench SEED Image HallusionBench POPE MathVista ChartQA DocVQA InfoVQA TextVQA
Metrics: accuracy ANLS VQA score relaxed accuracy
Baselines: InternVL2-2B InternVL2-8B MiniCPM-V-2 TextMonkey FastV LLaVA-OneVision-0.5B InternVL2-1B GLM4-V
Builds on: InternVL2 Monkey FastV TextMonkey
Closest papers by meaning
- Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · similarity 0.89)
- When MLLMs Meets Compression Distortion: A Coding Paradigm Tailored to MLLMs (ICLR 2026 · similarity 0.88)
- Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision (ICLR 2024 · similarity 0.88)
- ApET: Approximation-Error Guided Token Compression for Efficient VLMs (CVPR 2026 · similarity 0.88)
- CoLLM: A Large Language Model for Composed Image Retrieval (CVPR 2025 · similarity 0.87)
- Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMs (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.