Research map / Vision-language and multimodal / Multimodal LLMs

SigLIP-HD by Fine-to-Coarse Supervision

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Fine-tunes a SigLIP 2 vision encoder at 512px to mimic fused 512+1024px multi-scale features from the frozen encoder, improving MLLM fine-grained and OCR performance at no extra inference cost.

Problem

Higher-resolution inputs for MLLMs improve fine-grained perception but add multiple forward passes and more visual tokens; the paper asks whether a standard-resolution encoder can reach fine-grained perception without larger images.

Approach

A fine-to-coarse supervision: the trainable encoder at 512px is trained with patch-level L1 loss to match the averaged (interpolated) features of the frozen encoder run on 512px and 1024px views, distilling high-resolution detail into the standard-resolution tokens with no architectural change.

Key results

• With LLaVA-NeXT data and frozen encoder, SigLIP-HD improves DocVQA from 56.0 to 59.6 and ChartQA from 61.6 to 65.2 over SigLIP 2 at the same 512px token budget.
• On HRBench, SigLIP-HD raises the score from 43.5 to 48.3 (+4.8) under the LLaVA-NeXT setting with unfrozen encoder.
• Under AnyRes, SigLIP-HD averages 66.3 vs 64.8 for SigLIP 2, with ChartQA improving from 63.9 to 67.4.

Surprising findings

• Overlapping sliding windows for high-resolution inference degraded performance, contrary to dense prediction practice.
• Increasing the weight of high-resolution features in the fusion degraded results, even on fine-grained benchmarks such as DocVQA.

Limitations

• Some fine visual details may be lost after down-sampling, so native-resolution practice is not fully bypassed.
• Fusion hyperparameters and scale choices were selected empirically and more sophisticated configurations were not explored.
• Training requires a high-resolution teacher pass (34 hours on 32 A100 GPUs), adding one-time training cost.

Experimental setup

Datasets: Cambrian-1 collected data (4.5M raw images) LLaVA-1.5 SFT data LLaVA-NeXT SFT data

Benchmarks: DocVQA ChartQA TextVQA InfoVQA TextCaps HRBench RealWorldQA GQA MME Perception POPE MMBench ScienceQA-IMG

Metrics: accuracy average score across benchmarks

Baselines: SigLIP 2 OpenAI-CLIP CLIP-HD multi-scale CLIP baseline

Builds on: SigLIP 2 LLaVA AnyRes AM-RADIO EVA

Closest papers by meaning

• Fine-R1: Make Multi-modal LLMs Excel in Fine-Grained Visual Recognition by Chain-of-Thought Reasoning (ICLR 2026 · similarity 0.85)
• FineRS: Fine-grained Reasoning and Segmentation of Small Objects with Reinforcement Learning (NeurIPS 2025 · similarity 0.84)
• Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.84)
• Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · similarity 0.84)
• Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (NeurIPS 2024 · similarity 0.83)
• Mini-Monkey: Alleviating the Semantic Sawtooth Effect for Lightweight MLLMs via Complementary Image Pyramid (ICLR 2025 · similarity 0.83)