Research map / Image generation / Text-to-image diffusion
CLIPtone: Unsupervised Learning for Text-based Image Tone Adjustment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
CLIPtone performs text-based image tone adjustment trained unsupervised on unpaired images and color-name prompts, using a CLIP directional loss to modulate a pretrained 3D LUT enhancer via a hyper-network.
Problem
Text-based image tone adjustment normally requires expert-retouched paired or triplet datasets, which are costly and limit adjustments to styles seen in training; the paper seeks text-driven tone adjustment without paired data that generalizes to unseen descriptions.
Approach
Use CLIP's directional similarity as a perceptual criterion to train a hyper-network that modulates the AdaInt and weight-predictor parameters of a pretrained 3D LUT enhancer (Yang et al.) from text descriptions, with no paired or labeled data.
Key results
- CLIPtone reaches grayscale SSIM 0.858 and CLIP text-image directional similarity 0.083, versus T2ONet 0.936/0.021 and CLIPstyler 0.095 (highest directional score).
- On pseudo-ground-truth Lightroom filter comparison, CLIPtone achieves the highest CLIP directional similarity (0.467 vs 0.359 for T2ONet).
- CLIPtone runs in 10.3 ms per 512x512 image with 111.4M parameters, faster than IP2P (7177.4 ms) and T2ONet (45.1 ms).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CLIPtone instead of T2ONet | text-based tone adjustment · MIT-Adobe 5K, 500 images × 50 descriptions | CLIP text-image directional similarity: 0.021 -> 0.083 | Helped |
| use CLIPtone instead of CLIPstyler | text-based tone adjustment · MIT-Adobe 5K, 500 images × 50 descriptions | Grayscale SSIM: 0.484 -> 0.858 | Helped |
| use CLIPtone instead of IP2P | text-based tone adjustment · MIT-Adobe 5K, 500 images × 50 descriptions | CLIP image similarity: 0.932 -> 0.975 | Helped |
| use CLIPtone instead of CLIPstyler | text-based tone adjustment · MIT-Adobe 5K, 500 images × 50 descriptions | CLIP text-image directional similarity: 0.095 -> 0.083 | Hurt |
| use CLIPtone instead of IP2P | text-based tone adjustment · Adobe Lightroom pseudo-ground-truth test set | CLIP directional similarity: 0.310 -> 0.467 | Helped |
| use CLIPtone instead of T2ONet | text-based tone adjustment · Adobe Lightroom pseudo-ground-truth test set | CLIP image similarity: 0.956 -> 0.948 | Hurt |
| train without target keywords (ablation) | CLIPtone · MIT-Adobe 5K, target-keyword test set | CLIP text-image directional similarity: 0.089 -> 0.087 | No clear effect |
| remove the sampling interval loss (ablation) | CLIPtone · tone adjustment examples | aesthetic quality | Hurt |
| vary the modulation scaling factor s (ablation) | CLIPtone · “Sepia tone photo” adjustment | adjustment strength control | Helped |
Surprising findings
- T2ONet, a supervised method, scores highest on structure preservation but barely changes the image and performs poorly on text alignment.
- Training with target keywords excluded (CLIPtone-0) performs nearly as well as the full model, supporting zero-shot generalization.
Limitations
- Inherent CLIP biases can lead to adjustments misaligned with human perception.
- Only global tone adjustment; cannot perform local adjustments.
- Relies only on text directions, so cannot capture styles hard to express in text (no CLIP image encoder).
Experimental setup
Datasets: MIT-Adobe 5K Color Names Dataset
Benchmarks: MIT-Adobe 5K test subset with 50 manually collected tone descriptions pseudo-ground-truth images from Adobe Lightroom Classic filters
Metrics: grayscale SSIM CLIP image similarity CLIP text-image directional similarity user study score (1-5) parameter count inference time
Baselines: T2ONet IP2P CLIPstyler
Builds on: 3D LUT image enhancement (Yang et al.) StyleGAN-NADA directional CLIP loss CLIP-IQA relative similarity hyper-networks
Closest papers by meaning
- A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation (ICLR 2024 · similarity 0.86)
- Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Data (CVPR 2025 · similarity 0.86)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.86)
- UniPercept: Towards Unified Perceptual-Level Image Understanding across Aesthetics, Quality, Structure, and Texture (ICML 2026 · similarity 0.86)
- TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives (NeurIPS 2024 · similarity 0.85)
- Text and Image Are Mutually Beneficial: Enhancing Training-Free Few-Shot Classification with CLIP (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.