Research map / Vision-language and multimodal / Multimodal LLMs

Beyond the Linear Separability Ceiling: Aligning Representations in VLMs

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces the Linear Separability Ceiling (LSC), a nearest-centroid probe on VLM visual embeddings, to diagnose an "alignment gap" on Bongard tasks, and adds a contrastive centroid loss to next-token prediction to surpass it.

Problem

Whether VLM failures on abstract Bongard-style visual reasoning come from poor bottom-up perception or faulty top-down reasoning; existing diagnostics lack a way to quantify this.

Approach

A non-parametric nearest-centroid linear probe on raw vision-encoder embeddings (the LSC) serves as a perception baseline; generative accuracy is tested against it with chi-squared and significance tests. Training combines next-token loss with an InfoNCE-style contrastive loss pulling query final embeddings toward their true positive/negative prototype.

Key results

• Phi with LoRA (combined loss) reaches 95.6% on OpenWorld versus an 84.4% LSC, and 79.2% on HOI versus a 71.9% LSC.
• Pixtral with LoRA (combined loss) reaches 95.0% on OpenWorld and 79.6% on HOI, with final-layer probe accuracy rising to 96.2% on OpenWorld.
• On Winoground text retrieval, Pixtral improves from 28.75% baseline to 54.75% with the combined loss.

Surprising findings

• Postfix tuning, which cannot alter visual representations, surpasses the LSC on OpenWorld (94.2% vs 84.0%), suggesting dormant non-linear reasoning in VLMs.
• Generative accuracy is inversely correlated with the linear separability of final embeddings for most configurations of several models.

Limitations

• Combined-loss models degrade under altered prompt formats and on multiple-choice VQA, suggesting a trade-off with general robustness.
• Gemma baselines show environment-dependent instability in evaluation.
• Prompt tuning is highly sensitive to architectural choices and overfits easily.

Experimental setup

Datasets: Bongard OpenWorld Bongard HOI Winoground POPE A-OKVQA ScienceQA

Benchmarks: Bongard OpenWorld Bongard HOI Winoground POPE A-OKVQA ScienceQA

Metrics: accuracy linear separability ceiling chi-squared test of dependence

Baselines: Direct prompting baseline Projector tuning Prompt tuning Postfix tuning CLIP ViT-L/14 linear probe

Builds on: LoRA prompt tuning CLIP InfoNCE

Closest papers by meaning

• Progressive Multi-granular Alignments for Grounded Reasoning in Large Vision-Language Models (AAAI 2025 · similarity 0.90)
• Not Only Text: Exploring Compositionality of Visual Representations in Vision-Language Models (CVPR 2025 · similarity 0.90)
• Fill the GAP: A Granular Alignment Paradigm for Visual Reasoning in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.90)
• Multi-step Visual Reasoning with Visual Tokens Scaling and Verification (NeurIPS 2025 · similarity 0.89)
• Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs? (ICML 2025 · similarity 0.89)
• VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs (CVPR 2026 · similarity 0.89)