Research map / Vision-language and multimodal / Multimodal LLMs

Words That Make Language Models Perceive

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that a sensory cue ("see"/"hear") in the prompt, combined with generative (autoregressive) hidden-state representations, steers a frozen text-only Qwen3 LLM toward the kernel geometry of vision (DINOv2) and audio (BEATs) encoders.

Problem

Whether text-only LLMs encode perceptually grounded structure, and whether representational alignment with vision and audio encoders can be elicited at inference time rather than being a fixed property of training.

Approach

Introduces "generative representations" that average hidden states over autoregressive continuations rather than single-pass embeddings, and shows that a sensory instruction ("see" or "hear") in the prompt steers these representations toward the matching sensory encoder's kernel geometry, measured by mutual-kNN alignment.

Key results

• Under SEE cue, Qwen3-32B generative representations increase alignment with DINOv2 while HEAR cue increases alignment with BEATs; single-pass embeddings do not show this steering.
• Alignment rises with generation length up to 256 tokens and can decline at 512 tokens.
• VQA without V on MME (non-OCR, n=2334) improves overall accuracy from 64.78 to 67.14 with a visual framing prompt.

Surprising findings

• Mismatched cues (e.g., SEE evaluated on audio) can still outperform the no-cue baseline at longer generation lengths.
• Without an explicit cue, smaller models default to a visual framing, even for audio captions.

Limitations

• Only lightweight cues ('see', 'hear') and a few instruction verbs were explored.
• Audio alignment is lower than vision alignment, attributed to low-level acoustic features of BEATs.
• Sensory prompts may induce hallucinated perceptual details.

Experimental setup

Datasets: WiT DCI AudioCaps Clotho v2 Visual Genome MME

Benchmarks: WiT DCI AudioCaps Clotho v2 MME

Metrics: mutual-kNN alignment accuracy Cohen's d AUROC

Baselines: single-pass embeddings (z_e) no cue prompt CLIP CLAP BEATs+

Builds on: Platonic Representation Hypothesis VQA without V

Closest papers by meaning

• Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach (CVPR 2025 · similarity 0.89)
• Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.87)
• Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.87)
• Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.87)
• Senses Wide Shut: A Representation-Action Gap in Omnimodal LLMs (NeurIPS 2026 · similarity 0.86)
• Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting (ICLR 2024 · similarity 0.86)