Research map / Vision-language and multimodal / Multimodal LLMs

FRISM: Fine-Grained Reasoning Injection via Subspace-Level Model Merging for Vision–Language Models

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FRISM, which merges an LRM into a VLM at SVD-subspace granularity, learning per-subspace sigmoid gates with a label-free KL self-distillation plus spectral-magnitude injection loss, improving reasoning while largely preserving perception.

Problem

Merging VLMs with LRMs to add reasoning usually trades off visual perception, because existing merging methods act at coarse layer or model level.

Approach

Decompose the LRM task vector per layer via SVD, freeze the singular vectors and values, and learn a sigmoid gate per singular subspace, trained with a KL distillation loss on unlabeled VQA images (VizWiz) plus a term that maximizes the injected spectral magnitude.

Key results

• On Qwen2.5-VL-7B merged with DeepSeek-R1-Distill-Qwen-7B, FRISM reaches 49.4 average on VL reasoning benchmarks vs 47.4 for the base VLM (+2.0) and best baseline 47.8 (DARE).
• On Qwen2.5-VL-32B merged with QwQ-32B, FRISM averages 59.7 on reasoning vs 57.3 base, with MathVision improving by 4.4 points.
• On Qwen2.5-VL-3B merged with SmallThinker-3B, FRISM improves reasoning average by 1.8 points over the base and keeps perception at 79.7.

Surprising findings

• Layer-wise merging methods such as IP-Merging collapse on some settings (e.g., Qwen2.5-VL-7B with OpenThinker3-7B), attributed to exceptionally large merging coefficients.
• Removing the injection loss makes the merged model revert close to the base VLM, indicating the distillation loss alone drives gates toward no merging.

Limitations

• Gains on reasoning are modest (about 1-2 points average) relative to base VLM and remain below some post-trained reasoning VLMs in absolute terms.
• Theoretical analysis relies on an assumed decoupling of SVD subspaces under the visual Hessian (diagonal approximation), which is not verified.
• Injection loss is a norm-based proxy for reasoning injection rather than a direct reasoning signal; the sensitivity to alpha and normalization is described only briefly.

Experimental setup

Datasets: VizWiz

Benchmarks: MathVista MathVision MathVerse MMMU R1-OneVision MMStar TextVQA POPE SeedBench OCRBench MME ChartQA

Metrics: accuracy average normalized score (Avg-R, Avg-V)

Baselines: Base VLM Task Arithmetic Ties-Merging DARE IP-Merging DRIFT OpenVLThinker-7B VLAA-Thinker

Builds on: Task Arithmetic TIES-Merging DARE IP-Merging FRANK

Closest papers by meaning

• Multi-step Visual Reasoning with Visual Tokens Scaling and Verification (NeurIPS 2025 · similarity 0.88)
• From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models (ICML 2026 · similarity 0.87)
• DiVE-k: DIFFERENTIAL VISUAL REASONING FOR FINE-GRAINED IMAGE RECOGNITION (ICLR 2026 · similarity 0.87)
• Fine-R1: Make Multi-modal LLMs Excel in Fine-Grained Visual Recognition by Chain-of-Thought Reasoning (ICLR 2026 · similarity 0.87)
• SwiftVLM: Efficient Vision-Language Model Inference via Cross-Layer Token Bypass (NeurIPS 2026 · similarity 0.87)
• VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoning (ICLR 2026 · similarity 0.87)