Research map / Vision-language and multimodal / Multimodal LLMs
PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a fully open, reproducible vision-language model (PLM 1B/3B/8B) trained without proprietary distillation, releases 2.8M human-annotated video QA and spatio-temporal caption labels, and introduces PLM-VideoBench.
Problem
Closed-source VLMs and distillation from black-box teachers obscure data and training recipes, making it hard to measure scientific progress and to know which training gaps (especially fine-grained and spatio-temporal video understanding) are real.
Approach
Train a VLM (Perception Encoder + Llama 3 decoder) using only open-access data and a white-box synthetic data engine built from open models, study scaling of synthetic data, identify that synthetic data fails on challenging video tasks, and close the gap with newly collected human-annotated fine-grained video QA (PLM-FGQA) and spatio-temporal captions (PLM-STC), plus a new benchmark PLM-VideoBench
Key results
- PLM-8B reaches 55.6 average on PLM-VideoBench versus 38.5 for InternVL2.5-8B and 51.6 for GPT-4o.
- PLM-8B scores 67.7 MBAcc on FGQA versus 61.2 for GPT-4o, and 59.1 meanR on RTLoc versus 33.1 for GPT-4o.
- Synthetic data power-law exponent for challenging video tasks (HardQA) is -0.03 versus -0.15 for general video QA, indicating synthetic data does not scale on hard tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add PLM-STC human-annotated data (ablation) | PLM with synthetic data and reduced SFT mix · PLM-VideoBench ST | 3-metric average: +27.4 points | Helped |
| Add PLM-STC human-annotated data (ablation) | PLM with synthetic data and reduced SFT mix · video captioning | Dream 1K score: +2.4 points | Helped |
| Add PLM-STC human-annotated data (ablation) | PLM with synthetic data and reduced SFT mix · spatial and temporal tasks | 4-benchmark average: +6.8 points | Helped |
| Add PLM-FGQA human-annotated data (ablation) | PLM with synthetic data and reduced SFT mix · PLM-VideoBench FGQA | MBAcc: +13.1 points | Helped |
| Add PLM-FGQA human-annotated data (ablation) | PLM with synthetic data and reduced SFT mix · PLM-VideoBench SGQA | accuracy: +7.3 points | Helped |
| Add PLM-FGQA human-annotated data (ablation) | PLM with synthetic data and reduced SFT mix · fine-grained video tasks | 5-benchmark average: +1.3 points | Helped |
| Add PLM-FGQA human-annotated data (ablation) | PLM with synthetic data and reduced SFT mix · video hallucination tasks | 2-benchmark average: +3.0 points | Helped |
| Add PLM-FGQA human-annotated data (ablation) | PLM with synthetic data and reduced SFT mix · spatial and temporal tasks | 4-benchmark average: +2.2 points | Helped |
| Use all human-annotated data (PLM-STC and PLM-FGQA) (ablation) | PLM with synthetic data and reduced SFT mix · selected benchmarks | total average: 57.9 -> 61.2 | Helped |
| Increase vision encoder size from 300M to 2B | PLM synthetic-data scaling setup · Video QA | power-law exponent: -0.15 to -0.19 | Helped |
| Train with up to 16 rather than 8 tiles/frames | PLM synthetic-data scaling setup · Video QA | metric error: 32.2 vs 30.2 | Helped |
| Train with higher input size (16 rather than 8 tiles/frames) | PLM synthetic-data scaling setup · Natural QA | scaling trend | No clear effect |
Surprising findings
- Challenging video tasks (HardQA) scale poorly with synthetic data, with a power-law exponent of -0.03 compared with -0.15 for general video QA.
- Open-access video models default to repeating the same caption across time intervals on spatio-temporal tasks such as RDCap and RCap.
Limitations
- Synthetic data does not improve challenging fine-grained and temporal video tasks, so reliance on costly human annotation remains.
- Smart-glasses QA and LLM-judge metrics depend on an open LLM judge (Llama 3.3 70B), which may introduce evaluation bias.
- PLM-8B underperforms InternVL2.5-8B on SGQA (46.2 vs 48.3) and several image benchmarks.
Experimental setup
Datasets: SA-1B PLM-FGQA PLM-STC PixmoCap DocVQA ArxivCap PDFAcc YT-1B Ego4D Spoken Moments Kinetics710 The Cauldron
Benchmarks: DocVQA ChartQA TextVQA InfoVQA AI2D OCRBench MMMU VQAv2 OK-VQA VizWiz SEED BLINK
Metrics: accuracy MBAcc F1 LLM-judge accuracy SODA score mIoU meanR CIDEr
Baselines: GPT-4o Gemini 1.5 Pro Gemini 2.0 Flash Molmo-7B-O LLaVA-OneVision-7B Qwen2.5-VL InternVL2.5
Builds on: Perception Encoder Llama 3 SAM 2 LLaVA-OneVision
Closest papers by meaning
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.92)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.91)
- Perception Encoder: The best visual embeddings are not at the output of the network (NeurIPS 2025 · similarity 0.91)
- VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understanding (NeurIPS 2025 · similarity 0.91)
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.90)
- Not Another Text Benchmark: Putting the “Visual" Back in Visual Question Answering for Large Video Models (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.