Research map / Vision-language and multimodal / Multimodal LLMs

QLIP: A Dynamic Quadtree Vision Prior Enhances MLLM Performance Without Retraining

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes QLIP, a training-free-for-MLLM drop-in CLIP replacement that uses content-aware quadtree patchification plus a small coordinate MLP to interpolate positional embeddings, improving LLaVA-1.5 on V* and POPE without retraining the vision encoder or LLM.

Problem

CLIP vision encoders in MLLMs are limited to fixed input resolution and uniform grid patches, causing poor fine-grained VQA and hallucination; replacing the encoder usually requires retraining the entire MLLM.

Approach

Replace uniform grid patchification with a content-aware image quadtree (gradient-based pruning, downsampling leaves to patch size) and train a small coordinate-based MLP (Fourier features) to interpolate CLIP positional embeddings to arbitrary resolutions, so the CLIP encoder and LLM stay frozen.

Key results

• On V*, LLaVA-1.5-13B with QLIP reaches 58.6 vs 45.0 baseline (+13.6), and LLaVA-1.5-7B with QLIP reaches 53.4 vs 42.4 (+11.0).
• POPE F1 improves by +5.2 for 7B (74.4 to 79.6) and +1.2 for 13B (82.4 to 83.6).
• On MM-Bench, CV-Bench, ScienceQA, MME and RealWorld-QA changes are small, with a -2.8 drop on MM-Bench and -1.7 on RealWorld-QA for 7B.

Surprising findings

• Reducing image tokens via quadtree pruning can exceed baseline accuracy on V* while using fewer visual tokens.
• Random quadtree pruning yields large performance gaps relative to gradient-based selection, indicating selection content matters rather than token count alone.

Limitations

• Not designed for dense prediction tasks such as segmentation or detection; not validated on non-natural image domains.
• MLP hyperparameters were not fully swept due to compute limits; benchmark sweeps select best-of-sweep scores per benchmark.
• Does not transfer directly to newer encoders with integrated token reduction or relative position encodings (InternVL, Qwen-VL, SigLIP-style).

Experimental setup

Datasets: Imagenette

Benchmarks: V* MM-Bench POPE CV-Bench ScienceQA MME RealWorld-QA

Metrics: accuracy F1

Baselines: LLaVA-1.5-7B LLaVA-1.5-13B S^2 SEAL

Builds on: LLaVA-1.5 CLIP quadtree image representation Fourier features

Closest papers by meaning

• Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.89)
• Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.89)
• VladVA: Discriminative Fine-tuning of LVLMs (CVPR 2025 · similarity 0.89)
• FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs (CVPR 2025 · similarity 0.89)
• VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction-Editing Data and Long Captions (NeurIPS 2025 · similarity 0.89)
• Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (CVPR 2024 · similarity 0.88)