Research map / Vision-language and multimodal / Multimodal LLMs
Q-Instruct: Improving Low-level Visual Abilities for Multi-modality Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Collects 58K human natural-language low-level visual feedbacks (Q-Pathway) and converts them via GPT into 200K instruction-response pairs (Q-Instruct) to instruction-tune MLLMs for low-level perception, description, and quality assessment.
Problem
Multimodal LLMs answer low-level visual questions (quality, distortions, attributes) inaccurately, largely because training data focuses on high-level vision.
Approach
Collect pathway feedbacks (detailed low-level description followed by an overall quality conclusion) from trained human subjects, then use GPT to convert them into diverse VQA, yes/no, and extended conversation instruction data for low-level visual instruction tuning of existing MLLMs.
Key results
- Q-Instruct mix-strategy tuning raises LLaVA-v1.5-7B LLVisionQA-test overall accuracy from 60.07% to 69.30%.
- On never-seen IQA datasets, average SRCC gain of +0.243 is reported for the tuned models.
- Description sum score for LLaVA-v1.5-7B rises from 3.21 to 3.86 under the mix strategy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Tune MLLMs with Q-Instruct mixed with high-level data | LLaVA-v1.5-7B · LLVisionQA-test | Overall accuracy: 60.07% -> 69.30% | Helped |
| Tune MLLMs with Q-Instruct after high-level tuning | InternLM-XComposer-VL · LLVisionQA-test | Overall accuracy: 64.35% -> 70.37% | Helped |
| Tune with Q-Instruct | MLLMs · Low-level image description | relevance: +0.31 | Helped |
| Tune with Q-Instruct | MLLMs · Low-level image description | precision: +0.04 | Helped |
| Tune with Q-Instruct | MLLMs · Never-seen IQA datasets | SRCC: +0.243 | Helped |
| Use full Q-Instruct instead of only Q-Pathway (ablation) | LLaVA-v1.5-7B · Low-level image description (mix) | sum score: 3.74 -> 3.86 | Helped |
| Use full Q-Instruct instead of only the VQA subset (ablation) | LLaVA-v1.5-7B · LLVisionQA-test (mix) | Overall accuracy: 67.6% -> 69.3% | Helped |
| Replace high-level tuning data with Q-Instruct (ablation) | LLaVA-v1.5-7B · LLVisionQA-test | Overall accuracy: 69.3% -> 64.1% | Hurt |
| Increase the amount of Q-Instruct training data (ablation) | LLaVA-v1.5-7B · Low-level visual perception accuracy | accuracy | Helped |
| Mix training with high-level data rather than replacing it (ablation) | LLaVA-v1.5-7B · LLVisionQA-test | Overall accuracy: 64.1% -> 69.3% | Helped |
| Use the full Q-Instruct dataset rather than only Q-Pathway (ablation) | LLaVA-v1.5-7B · Low-level image description (after high-level tuning) | sum score: 3.69 -> 3.82 | Helped |
Surprising findings
- Quality scores are learned from text-only instruction tuning without any MOS values as supervision, yet generalize to unseen datasets.
- Replacing high-level tuning data with Q-Instruct is notably worse than mixing or sequential tuning, indicating high-level awareness matters.
Limitations
- Tuned models degrade on general-purpose and language-centric tasks.
- Accuracy on LLVisionQA-test (68-71%) remains below average human (~74%).
- Dataset is mainly in-the-wild natural images, limiting coverage of other visual content.
Experimental setup
Datasets: KonIQ-10k SPAQ LIVE-FB LIVE-itw AGIQA-3K ImageRewardDB COCO Q-Pathway
Benchmarks: LLVisionQA Q-Bench KonIQ-10k SPAQ LIVE-FB LIVE-itw AGIQA-3K CGIQA-6K KADID-10K KonViD-1k
Metrics: accuracy completeness precision relevance SRCC PLCC
Baselines: NIQE LLaVA-v1.5 (7B) LLaVA-v1.5 (13B) mPLUG-Owl-2 InternLM-XComposer-VL
Builds on: LLaVA-v1.5 Q-Bench A-OKVQA COCO-VQA
Closest papers by meaning
- Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision (ICLR 2024 · similarity 0.95)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.90)
- Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs (CVPR 2024 · similarity 0.90)
- VHELM: A Holistic Evaluation of Vision Language Models (NeurIPS 2024 · similarity 0.89)
- Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · similarity 0.89)
- Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.