Research map / Vision-language and multimodal / Human and embodied modeling
AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AlignVLM, a connector that maps SigLIP visual features to probability distributions over LLM vocabulary and takes a weighted average of text embeddings, so visual inputs lie in the convex hull of the LLM text embedding space; evaluated on document understanding with Llama 3.x backbones.
Problem
Shallow-fusion VLM connectors (MLPs, Perceiver Resampler, Ovis visual embeddings) do not constrain projected visual features to the LLM's text embedding distribution, causing misalignment, noise sensitivity, and data inefficiency, especially for document understanding.
Approach
The connector maps each visual feature to a softmax distribution over the LLM vocabulary (with the second linear layer initialized from the LM head) and outputs the weighted sum of the LLM's text embeddings, so visual tokens lie in the convex hull of text embeddings.
Key results
- Llama-3.2-3B-AlignVLM reaches 58.81 average on document benchmarks versus 53.06 for the MLP connector and 54.72 for Ovis under the same high-resource training.
- In the low-resource LLaVA-NeXT setting, AlignVLM averages 59.32 on document tasks (DocVQA 71.43 vs 42.11 for MLP) and 56.88 overall versus 53.59 for MLP.
- Under Gaussian feature noise (sigma=3), AlignVLM drops only 1.67 points versus 25.54 for the MLP connector.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the MLP connector with AlignModule | Llama-3.2-3B VLM · General document benchmarks | Avg. Score: 53.06 -> 58.81 | Helped |
| replace the Perceiver Resampler with AlignModule | Llama-3.2-3B VLM · General document benchmarks | Avg. Score: 50.68 -> 58.81 | Helped |
| replace Ovis with AlignModule | Llama-3.2-3B VLM · General document benchmarks | Avg. Score: 54.72 -> 58.81 | Helped |
| replace the MLP connector with AlignModule | Llama-3.2-3B VLM · Low-resource document tasks | Avg.: 40.61 -> 59.32 | Helped |
| replace the best baseline, HoneyBee, with AlignModule | Llama-3.2-3B VLM · Low-resource general vision tasks | Avg.: 54.33 -> 56.88 | Helped |
| restrict the connector to 3.4K high-probability embeddings (ablation) | AlignModule · General document benchmarks | Avg. Score: 58.81 -> 58.69 | No clear effect |
| add Gaussian noise (σ = 3) to visual features | AlignModule · Robustness to Gaussian noise | Avg. Score: 58.81 -> 57.14; ↓ 1.67 | Hurt |
| add Gaussian noise (σ = 3) to visual features | MLP connector · Robustness to Gaussian noise | Avg. Score: 53.06 -> 27.52; ↓ 25.54 | Hurt |
| replace the same-data Llama3.2-11B baseline with AlignVLM-Llama-3.1-8B | multimodal document understanding model · General document benchmarks | Avg. Score: 60.26 -> 62.88; 2.62% improvement | Helped |
Surprising findings
- Restricting the connector to about 3.4K high-probability vocabulary embeddings causes negligible performance loss (58.81 vs 58.69 average), suggesting a compact subset is used.
- The per-patch vocabulary distribution is dense rather than sparse, with the highest single-token probability only about 0.0118.
Limitations
- Training on the Instruct-style datasets is limited; the paper compares against models trained on undisclosed data, so direct comparisons are not fully fair.
- Model fails on VCR examples where similar landmark names differ by one character, indicating a bias toward common vocabulary tokens.
- Due to limited compute, maximum tiles is 9 and the training data is restricted to document-focused datasets.
Experimental setup
Datasets: CC-12M BigDocs-7.5M DocDownstream LLaVA-558K LLaVA-NeXT-779K VCR
Benchmarks: DocVQA InfoVQA DeepForm KLC WTQ TabFact ChartQA TextVQA TableVQA MMMU SeedBench MMVet
Metrics: accuracy average score cosine distance
Baselines: MLP connector Perceiver Resampler Ovis H-Reducer HoneyBee Qwen2-VL-2B DocOwl1.5-8B Llama3.2-11B Qwen2.5-VL-7B GPT-4o
Builds on: LLaVA-NeXT Ovis Perceiver Resampler HoneyBee BigDocs-7.5M
Closest papers by meaning
- Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models (NeurIPS 2025 · similarity 0.91)
- Fill the GAP: A Granular Alignment Paradigm for Visual Reasoning in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.90)
- Text as Partial Constraint: Core–Residual Alignment for Robust Vision–Language Learning (NeurIPS 2026 · similarity 0.90)
- Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models (ICML 2024 · similarity 0.90)
- Deep Pre-Alignment for VLMs (ICML 2026 · similarity 0.90)
- Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.