Research map / Vision-language and multimodal / Multimodal LLMs

MM-SeR: Multimodal Self-Refinement for Lightweight Image Captioning

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Replaces LLaVA-1.5's LLaMA-7B with a 125M OPT model for lightweight captioning, then adds a multimodal self-refinement stage (SeR-Connector using prior caption and multi-layer ViT features) that improves captions and long-range VideoQA.

Problem

Image captioning with MLLMs is too computationally expensive for edge devices and streaming applications, and small captioners lag in reliability and detail.

Approach

Shows a LLaVA-style captioner with a 125M OPT language model is competitive with MLLMs on factual captioning, then adds a second-pass refinement conditioned on the model's own initial caption plus multi-layer vision features, trained on GPT-4o-mini-perturbed pseudo-initial captions.

Key results

• Lightweight OPT-125M captioner (450M total) reaches 129.6 CIDEr on MS COCO, 6.9 above SmallCap (also OPT-125M), roughly matching LLaVA-1.5 (133.7).
• MM-SeR raises CIDEr on MS COCO from 129.6 to 133.5 (+3.9) and CAPT on ShareGPT4V/DCI from 45.9 to 48.4 (+2.5).
• On long-range VideoQA with Qwen2.5-14B, MM-SeR reaches 50.8 accuracy versus 51.1 for LLaVA-1.5 (7.3B), with 5m10s vs ~29m runtime.

Surprising findings

• A 125M-parameter language model matches MLLMs with 7B LLMs on factual single-sentence captioning, suggesting captioning relies less on LLM reasoning.
• Iterative refinement gives no measurable gain for OPT-125M but helps OPT-1.3B, indicating small LMs cannot exploit extra refinement passes.

Limitations

• Small LM shows repetitive phrasing, reduced fluency, limited OCR and world knowledge.
• Trained on about 500K machine-generated ShareGPT4V captions, limiting data quality.
• Existing evaluation metrics correlate poorly with human judgment and some rankings are inconsistent.

Experimental setup

Datasets: MS COCO ShareGPT4V DCI CC558K GLaMM

Benchmarks: MS COCO ShareGPT4V DCI GLaMM benchmark LLoVi long-range VideoQA

Metrics: BLEU@4 METEOR CIDEr BERTScore CAPTURE CLAIR GPT-4o-mini judge score accuracy inference time

Baselines: LLaVA-1.5 Qwen-VL Cambrian SmallCap Tag2Text I-Tuning CapPa LocCa ViPCap GLaMM BLIP-2

Builds on: LLaVA-1.5 SmallCap DPO Self-Refine LLoVi

Closest papers by meaning

• CapsFusion: Rethinking Image-Text Data at Scale (CVPR 2024 · similarity 0.89)
• Top-Down Semantic Refinement for Image Captioning (AAAI 2026 · similarity 0.89)
• CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness (NeurIPS 2025 · similarity 0.89)
• Enhancing Descriptive Captions with Visual Attributes for Multimodal Perception (CVPR 2026 · similarity 0.88)
• Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.88)
• Patch Matters: Training-free Fine-grained Image Caption Enhancement via Local Perception (CVPR 2025 · similarity 0.88)