Research map / Architectures and efficiency / Pruning and efficient tuning
Charm: The Missing Piece in ViT Fine-Tuning for Image Aesthetic Assessment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Charm, a ViT tokenization that keeps some image patches at native resolution while downscaling others, preserving aspect ratio, composition, and multi-scale detail for image aesthetic and quality assessment without additional pre-training.
Problem
Standard ViT pipelines resize or crop images to small fixed sizes for batch processing and fixed position embeddings, losing aspect ratio, composition, and high-resolution detail that matter for image aesthetic assessment.
Approach
Tokenize the image at a coarse patch size, keep randomly (or saliency/frequency/gradient/entropy) selected patches at full resolution and downscale the rest to a fixed token budget, interpolate the frozen pre-trained position embeddings separately for high- and low-resolution tokens, and add learnable scale embeddings, with no changes to the ViT architecture.
Key results
- On AVA, Dinov2-small + Charm reaches PLCC 0.779, SRCC 0.777, ACC 0.826 versus 0.710/0.706/0.802 for the fine-tuned baseline (about 6.9% and 7.1% gains).
- Charm improves performance up to 7.5% PLCC, 8.1% SRCC and 14.8% accuracy across datasets (largest on TAD66k).
- For a 640x640 image, 2-scale Charm reduces Dinov2-small GMACs by 84%, runtime by 77%, GPU memory by 83%, and token count by 75%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace standard tokenization with Charm | Dinov2-small · AVA | PLCC: 0.710 -> 0.779 | Helped |
| Replace standard tokenization with Charm | Dinov2-small · AADB | PLCC: 0.695 -> 0.767 | Helped |
| Replace standard tokenization with Charm | Dinov2-small · TAD66k | PLCC: 0.429 -> 0.488 | Helped |
| Replace standard tokenization with Charm | Dinov2-small · PARA | PLCC: 0.904 -> 0.938 | Helped |
| Replace standard tokenization with Charm | Dinov2-small · BAID | PLCC: 0.428 -> 0.439 | Helped |
| Replace standard tokenization with Charm | Dinov2-small · SPAQ | PLCC: 0.911 -> 0.919 | Helped |
| Replace standard tokenization with Charm | Dinov2-small · KonIQ10k | PLCC: 0.896 -> 0.944 | Helped |
| Preserve aspect ratio and high-resolution details with multiscale inputs (ablation) | Dinov2-small + Charm · AVA | PLCC: 0.734 -> 0.779 | Helped |
| Use multiscale inputs after downscaling images to 256 x 256 (ablation) | Dinov2-small · AVA | PLCC: 0.734 -> 0.710 | Hurt |
| Use multiscale inputs while preserving high resolution but changing aspect ratio (ablation) | Dinov2-small · AVA | PLCC: 0.734 -> 0.763 | Helped |
| Select high-resolution patches randomly each epoch rather than by saliency (ablation) | Dinov2-small + Charm · AADB | PLCC: 0.751 -> 0.767 | Helped |
| Use a downscaling factor of 0.5 instead of larger factors (ablation) | Charm · not specified | performance | Helped |
Surprising findings
- Random patch selection each epoch outperformed saliency, frequency, gradient and entropy-based selection, acting as augmentation against overfitting.
- Initially downscaling to a fixed 256x256 before multi-scale tokenization (as in prior mixed-resolution work) decreased IAA performance relative to the standard baseline.
Limitations
- Selective downscaling can lose information in unselected regions.
- Charm is incompatible with Swin architectures per the authors' conclusion.
- Multi-scale inputs can hurt performance on very small images, and gains plateau beyond a resolution threshold.
Experimental setup
Datasets: AVA PARA TAD66k AADB BAID SPAQ KonIQ10k
Benchmarks: AVA PARA TAD66k AADB BAID SPAQ KonIQ10k
Metrics: PLCC SRCC ACC
Baselines: Ghosal et al. Chen et al. MUSIQ FlexiViT Swin Muller et al. high-resolution method NaViT-style padding baseline ELTA
Builds on: Vision Transformer MUSIQ mixed-resolution ViT mixed-scale tokenization SAM-HQ
Closest papers by meaning
- Vision Transformers with Self-Distilled Registers (NeurIPS 2025 · similarity 0.86)
- Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More (ICML 2025 · similarity 0.86)
- DA-VPT: Semantic-Guided Visual Prompt Tuning for Vision Transformers (CVPR 2025 · similarity 0.85)
- Rethinking Token Reduction with Parameter-Efficient Fine-Tuning in ViT for Pixel-Level Tasks (CVPR 2025 · similarity 0.85)
- Linearly Decomposing and Recomposing Vision Transformers for Diverse-Scale Models (NeurIPS 2024 · similarity 0.85)
- Vision Transformers Need Registers (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.