Research map / Vision-language and multimodal / Human and embodied modeling
Pixel-Aligned Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PixelLLM, a vision-language model that regresses a pixel location for each output word from LLM features, trained on Localized Narratives, and adapted to referring localization, location-conditioned captioning, and dense object captioning.
Problem
Vision-language models generate text but cannot perform word-level or region-level localization (e.g., word grounding, referring localization, region captioning) within the LLM framework.
Approach
Adds a small MLP on the LLM's pre-vocabulary token features to regress a 2D trace location for each generated word, avoiding location tokens in the vocabulary; uses a learnable prompt feature extractor for location prompts and trains on dense human mouse-trace word alignments from Localized Narratives.
Key results
- Achieves 89.8 P@0.5 on RefCOCO val referring localization, beating the second-best by 2.3 points.
- Reaches 19.9 METEOR and 148.9 CIDEr on Visual Genome location-conditioned captioning, state of the art in Table 3.
- Obtains 17.02 mAP on Visual Genome dense object captioning, outperforming GRiT (15.48).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use per-token regression instead of raw-string or extra-token localization (ablation) | PixelLLM · RefCOCO | P@0.5: 84.4 -> 87.6; 86.1 -> 87.6 | Helped |
| Add Localized Narratives pretraining to GoldG (ablation) | PixelLLM with per-token regression · RefCOCO | P@0.5: 87.6 -> 89.8 | Helped |
| Increase language model size from T5 Small through T5 XL (ablation) | PixelLLM without LoRA · RefCOCO val | P@0.5: 67.0; 70.3; 73.6; 81.9 | Helped |
| Increase language model size from T5 Small through T5 XL (ablation) | PixelLLM with LoRA · RefCOCO val | P@0.5: 75.6; 80.8; 84.8; 89.8 | Helped |
| Add LoRA fine-tuning (ablation) | PixelLLM with T5 Small, T5 Base, T5 Large, or T5 XL · RefCOCO val | P@0.5: 67.0 -> 75.6; 70.3 -> 80.8; 73.6 -> 84.8; 81.9 -> 89.8 | Helped |
| Use PixelLLM rather than the strongest listed baseline | Referring localization · RefCOCO val | P@0.5: 87.5 -> 89.8 | Helped |
| Use PixelLLM rather than the strongest listed baseline | Referring localization · RefCOCO+ val | P@0.5: 81.6 -> 83.2 | Helped |
| Use PixelLLM rather than the strongest listed baseline | Referring localization · RefCOCOg val | P@0.5: 83.9 -> 84.6 | Helped |
| Use PixelLLM rather than the strongest listed baseline | Referring segmentation · RefCOCO val | cIoU: 74.9 -> 76.9 | Helped |
| Use PixelLLM rather than the strongest listed baseline | Referring segmentation · RefCOCO+ val | cIoU: 65.1 -> 69.2 | Helped |
| Use PixelLLM rather than the strongest listed baseline | Referring segmentation · RefCOCOg val | cIoU: 67.9 -> 70.7 | Helped |
| Use PixelLLM rather than GRiT | Dense object captioning · Visual Genome | mAP: 15.48 -> 17.02 | Helped |
Surprising findings
- Frozen T5 with LoRA-adapted training improves RefCOCO P@0.5 by 8-10 points over the frozen counterparts, suggesting a frozen LLM already holds localization ability.
- Regressing locations on token features outperforms raw-string and extra-token localization formulations even with only sparse data, and dense Localized Narratives supervision adds a further 2.3 points.
Limitations
- Trace-based localization supervision from mouse traces can be noisy and not all word-location pairs are visually accurate.
- Referring localization comparisons are system-level, since models differ in backbones and pretraining data.
Experimental setup
Datasets: WebLI Localized Narratives RefCOCO RefCOCO+ RefCOCOg GoldG Visual Genome
Benchmarks: RefCOCO RefCOCO+ RefCOCOg Visual Genome Localized Narratives
Metrics: P@0.5 cIoU mAP METEOR CIDEr LBM
Baselines: Shikra-7B Ferret-7B UniTAB MDETR LISA GRiT Kosmos-2 GPT4RoI MITR
Builds on: BLIP-2 SAM GRiT Localized Narratives Pix2Seq
Closest papers by meaning
- Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding (CVPR 2025 · similarity 0.90)
- Grounding Everything: Emerging Localization Properties in Vision-Language Transformers (CVPR 2024 · similarity 0.89)
- RegionGPT: Towards Region Understanding Vision Language Model (CVPR 2024 · similarity 0.87)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.87)
- LocCa: Visual Pretraining with Location-aware Captioners (NeurIPS 2024 · similarity 0.87)
- Locality Alignment Improves Vision-Language Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.