Research map / Vision-language and multimodal / Multimodal LLMs

Directed-Tokens: A Robust Multi-Modality Alignment Approach to Large Language-Vision Models

NeurIPS 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Direct-LLaVA, which adds image-patch and text-word order reconstruction tasks with a learnable directed token and an image-to-response attention loss on top of the LLaVA-1.5 recipe, reporting gains over LLaVA-1.5 on academic and instruction-following benchmarks.

Problem

Large multimodal models tend to rely on language priors and under-use visual inputs, so visual-textual alignment is weak; the paper asks whether models can reconstruct shuffled image and text order to improve this alignment.

Approach

Adds jigsaw-style image patch order reconstruction and text word order reconstruction as auxiliary tasks during LMM instruction tuning, uses a learnable directed token appended at the end of the sequence to predict the permutation index, and adds an Image-to-Response attention loss that pushes attention from visual tokens to response tokens.

Key results

• Direct-LLaVA with Vicuna-13B reaches 77.3 on SciQA-IMG and 65.4 on TextVQA versus 71.6 and 61.3 for LLaVA-1.5-13B.
• Direct-LLaVA-13B scores 1572.1 on MME and 65.6 on SEED-Bench (all) versus 1531.3 and 61.6 for LLaVA-1.5-13B.
• On LLaVA-OneVision data with Qwen2.5-0.5B, Direct-LLaVA with 100% pretext samples improves MME from 1238 to 1494 over LLaVA.

Measured effects reported in the paper

Surprising findings

• Replacing the image with a black image lowers LLaVA-v1.5-7B SciQA-IMG only from 66.8 to 64.1 and MMMU from 35.3 to 32.4, suggesting weak visual reliance.
• Shuffling images in pretraining with directed tokens gives gains comparable to shuffling text, with image shuffling slightly more effective.

Limitations

• Balance weights among the learning objectives were not explored.
• Experiments limited to 7B-13B LLMs and LLaVA v1.5 data scale due to compute.
• Image-to-Response loss alone is insufficient for robust multimodal reasoning according to the authors' ablation.

Experimental setup

Datasets: LLaVA v1.5 training data (558K pretrain, 665K finetune) LLaVA-OneVision data LLaVA v1 data (595K pretrain, 158K finetune)

Benchmarks: VQAv2 GQA VizWiz SciQA-IMG TextVQA POPE SEED-Bench MME LLaVA-Wild MM-Vet MMMU-Val AI2D

Metrics: accuracy F1 score

Baselines: LLaVA-1.5-7B LLaVA-1.5-13B Qwen-VL Qwen-VL-Chat InstructBLIP BLIP-2 IDEFICS-9B IDEFICS-80B Shikra LLaVA 7B

Builds on: LLaVA v1.5 jigsaw puzzle self-supervised learning visual instruction tuning

Closest papers by meaning

• Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.91)
• Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.90)
• TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.90)
• Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.90)
• Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment (AAAI 2026 · similarity 0.90)
• Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations (NeurIPS 2025 · similarity 0.90)