Research map / Image generation / Autoregressive and latent generation
Boosting Image Quality Assessment through Efficient Transformer Adaptation with Local Feature Enhancement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoDa, which freezes a pretrained ViT-B and injects multi-scale local distortion features from a frozen pretrained ResNet-50 via cross-attention adapters, training only the extractor, injector and head, to reach SOTA on seven IQA datasets.
Problem
Image quality assessment (IQA) suffers from limited labeled data and the locality weakness of large pretrained transformers, and it is unclear whether scaling large pretrained high-level models benefits low-level IQA.
Approach
Keep a large pretrained ViT frozen and inject multi-scale local distortion features extracted from a pretrained CNN into its tokens via down-projected cross-attention with near-zero-initialized gates, so only small adapter modules are trained.
Key results
- LoDa with ViT-B and ResNet-50 reaches 0.931 SRCC / 0.936 PLCC on KADID-10k and 0.932 / 0.944 on KonIQ-10k, competitive with or above prior SOTA.
- Cross-dataset generalization: LoDa achieves the best SRCC in all four train/test settings (e.g., 0.805 on LIVEC when trained on FLIVE? and 0.811 when trained on KonIQ and tested on LIVEC).
- Using only ~9M trainable parameters, LoDa outperforms full fine-tuning of ViT (KADID-10k SRCC 0.931 vs 0.889) and Adapter-ViT, LoRA-ViT and VPT-ViT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add a local distortion extractor and injector to ViT (ablation) | LoDa · KADID-10k | SRCC: 0.889 -> 0.931 | Helped |
| add a local distortion extractor and injector to ViT (ablation) | LoDa · KonIQ-10k | SRCC: 0.874 -> 0.932 | Helped |
| add a local distortion injector (ablation) | ViT + Extractor · KADID-10k | SRCC: 0.915 -> 0.931 | Helped |
| add a local distortion injector (ablation) | ViT + Extractor · KonIQ-10k | SRCC: 0.925 -> 0.932 | Helped |
| use ViT-B rather than ViT-T (ablation) | LoDa · KADID-10k | SRCC: 0.892 -> 0.931 | Helped |
| use ViT-B rather than ViT-T (ablation) | LoDa · KonIQ-10k | SRCC: 0.914 -> 0.932 | Helped |
| use ImageNet-21K-pretrained weights rather than ImageNet-1K-pretrained weights (ablation) | LoDa · KADID-10k | SRCC: 0.912 -> 0.931 | Helped |
| use multi-modal pretrained weights rather than ImageNet-21K pretrained weights (ablation) | LoDa · KADID-10k | SRCC: 0.931 -> 0.897 | Hurt |
| use multi-modal pretrained weights rather than ImageNet-21K pretrained weights (ablation) | LoDa · KonIQ-10k | SRCC: 0.932 -> 0.909 | Hurt |
Surprising findings
- Multi-modal pretrained weights (CLIP-style) performed worse than ImageNet-21K weights for IQA, which the authors attribute to abstract-concept focus.
Limitations
- Performance with multi-modal pretrained weights is weaker than ImageNet-21K weights, which the method does not handle well.
- Relies on a frozen CNN and ViT backbone, so inference cost includes both networks.
Experimental setup
Datasets: LIVE TID2013 KADID-10k LIVEC KonIQ-10k SPAQ FLIVE
Benchmarks: LIVE TID2013 KADID-10k LIVEC KonIQ-10k SPAQ FLIVE
Metrics: SRCC PLCC
Baselines: ILNIQE BRISQUE WaDIQaM-NR DB-CNN TIQA MetaIQA P2P-BM HyperIQA MUSIQ TReS DEIQT LIQE
Builds on: Vision Transformer (ViT) Adapter ResNet DEIQT HyperIQA
Closest papers by meaning
- Transformer-Based No-Reference Image Quality Assessment via Supervised Contrastive Learning (AAAI 2024 · similarity 0.89)
- Adaptive Feature Selection for No-Reference Image Quality Assessment by Mitigating Semantic Noise Sensitivity (ICML 2024 · similarity 0.87)
- Distilling Spatially-Heterogeneous Distortion Perception for Blind Image Quality Assessment (CVPR 2025 · similarity 0.87)
- Toward Generalized Image Quality Assessment: Relaxing the Perfect Reference Quality Assumption (CVPR 2025 · similarity 0.87)
- Adaptive Image Quality Assessment via Teaching Large Multimodal Model to Compare (NeurIPS 2024 · similarity 0.86)
- Augmenting Perceptual Super-Resolution via Image Quality Predictors (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.