Research map / Vision-language and multimodal / Multimodal LLMs
PUMA: Empowering Unified MLLM with Multi-granular Visual Generation
Rongyao Fang, Chengqi Duan, Kun Wang, Hao Li, Linjiang Huang, Hao Tian, Xingyu Zeng, Rui Zhao, Jifeng Dai, Hongsheng Li, Xihui Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use f0 (256 tokens) instead of f4 (1 token) features for decoding (ablation) | PUMA · ImageNet val reconstruction | PSNR_r: 10.76 -> 18.16 | Improved |
| use f4 (1 token) instead of f0 features for decoding (ablation) | PUMA · ImageNet val decoding diversity | LPIPS_d: 0.1559 -> 0.5751 | Improved |
| use f3 scale instead of f4 scale (ablation) | PUMA · MSCOCO 30K text-to-image | CLIP-I: 0.699 -> 0.703 | Improved |
| use f3 scale instead of f4 scale (ablation) | PUMA · MSCOCO 30K text-to-image | LPIPS_d: 0.613 -> 0.558 | Hurt |
| use f0 scale instead of f2 for text-to-image (ablation) | PUMA · MSCOCO 30K text-to-image | CLIP-I: 0.703 -> 0.621 | Hurt |
| select best of 5 scale outputs (5-scale Max) (ablation) | PUMA · MSCOCO 30K text-to-image | CLIP-I: 0.703 -> 0.736 | Improved |
| use f0 instead of f1 scale features (ablation) | PUMA · Emu-Edit test | DINO: 0.679 -> 0.784 | Improved |
| use f0 instead of f1 scale features (ablation) | PUMA · Emu-Edit test | CLIP-I: 0.802 -> 0.840 | Improved |
| use f0 instead of f1 scale features (ablation) | PUMA · Emu-Edit test | CLIP-T: 0.258 -> 0.264 | No clear effect |
| use all 5 scales f4-f0 as understanding input instead of f4 only (ablation) | PUMA encoder · MMBench | score: 56.8 -> 65.1 | Improved |
| use f0 (256 tokens) instead of f2 (16 tokens) as understanding input (ablation) | PUMA encoder · GQA | score: 46.6 -> 58.8 | Improved |
| use f4-f0 (341 tokens) instead of f0 (256 tokens) as understanding input (ablation) | PUMA encoder · GQA | score: 58.8 -> 61.0 | Improved |
Closest papers by meaning
- Kosmos-G: Generating Images in Context with Multimodal Large Language Models (ICLR 2024 · similarity 0.88)
- MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model (NeurIPS 2024 · similarity 0.88)
- Granulon: Awakening Pixel-Level Visual Encoders with Adaptive Multi-Granularity Semantics for MLLM (CVPR 2026 · similarity 0.88)
- LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.88)
- Visual Perception by Large Language Model’s Weights (NeurIPS 2024 · similarity 0.88)
- Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
