Research map / Vision-language and multimodal / CLIP and zero-shot
Unified Coding for Both Human Perception and Generalized Machine Analytics with CLIP Supervision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UG-ICM, a learned image compression framework that uses multi-scale CLIP supervision and a preference-conditional decoder to produce human- and machine-oriented reconstructions from one bitstream, trained self-supervised without downstream models.
Problem
Image coding for machines lacks generalization to unseen analytics tasks and typically needs separate bitstreams or task-specific supervision to serve both human perception and machine analytics.
Approach
Replaces downstream-task supervision with global, local, and instance-level CLIP cosine-similarity losses (MS-CLIP) for machine-oriented training, and adds a Preference Conditional Decoding Module that conditions the decoder on a human or machine preference flag so one bitstream yields two reconstructions.
Key results
- With the MLIC++ backbone, UG-ICM gives average bpp-analytics gains of 2.8% (classification), 3.9% (object detection), 6.0% (semantic segmentation), and 3.2% (instance segmentation).
- With the Bmshj2018 backbone, average gains are 1.7% top-1 accuracy, 1.7% mAP, 3.2% mIoU, and 1.2% mAP for classification, detection, semantic segmentation, and instance segmentation.
- The machine-preference decoding gives about 3% higher bpp-mAP than the human-preference decoding on object detection but is about 1 dB lower in bpp-PSNR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add UG-ICM to the Bmshj2018 backbone | Bmshj2018 · ImageNet-1k classification | Top-1 accuracy: +1.7% | Helped |
| Add UG-ICM to the Bmshj2018 backbone | Bmshj2018 · PASCAL VOC object detection | mAP: +1.7% | Helped |
| Add UG-ICM to the Bmshj2018 backbone | Bmshj2018 · PASCAL VOC semantic segmentation | mIoU: +3.2% | Helped |
| Add UG-ICM to the Bmshj2018 backbone | Bmshj2018 · COCO val2017 instance segmentation | mAP: +1.2% | Helped |
| Add UG-ICM to the MLIC++ backbone | MLIC++ · ImageNet-1k classification | bpp-analytics: +2.8% | Helped |
| Add UG-ICM to the MLIC++ backbone | MLIC++ · PASCAL VOC object detection | bpp-analytics: +3.9% | Helped |
| Add UG-ICM to the MLIC++ backbone | MLIC++ · PASCAL VOC semantic segmentation | bpp-analytics: +6.0% | Helped |
| Add UG-ICM to the MLIC++ backbone | MLIC++ · COCO val2017 instance segmentation | bpp-analytics: +3.2% | Helped |
| Decode with machine preference instead of human preference (ablation) | UG-ICM · PASCAL VOC object detection | bpp-mAP: +3% | Helped |
| Decode with machine preference instead of human preference (ablation) | UG-ICM · Kodak24 human perception | bpp-PSNR: -1dB | Hurt |
| Add local- and instance-wise CLIP supervision to global-wise CLIP loss (ablation) | UG-ICM · PASCAL VOC object detection and semantic segmentation | machine analytics performance | Helped |
Surprising findings
- Offline CLIP-guided pixel updates alone improved unseen detection and segmentation accuracy while keeping perceptual quality, motivating the approach.
Limitations
- Relies on an external instance segmentation model for the instance-level CLIP loss.
- Machine-preference decoding reduces perceptual quality (about 1 dB PSNR loss versus human-preference decoding).
Experimental setup
Datasets: COCO train2017 PASCAL VOC ImageNet-1k COCO val2017 Kodak24
Benchmarks: ImageNet-1k PASCAL VOC COCO val2017 Kodak24
Metrics: top-1 accuracy mAP@0.5 mAP@[0.5:0.95] mIoU PSNR SSIM LPIPS bpp
Baselines: Chamain's TransTIC SA-ICM MLIC++ Bmshj2018
Builds on: MLIC++ Bmshj2018 TransTIC SA-ICM
Closest papers by meaning
- Test-Time Fine-Tuning of Image Compression Models for Multi-Task Adaptability (CVPR 2025 · similarity 0.87)
- All-in-One Image Coding for Joint Human-Machine Vision with Multi-Path Aggregation (NeurIPS 2024 · similarity 0.87)
- Towards Unified Human Perception and Machine Understanding: Token Flow Guided Compression Framework (CVPR 2026 · similarity 0.87)
- When MLLMs Meets Compression Distortion: A Coding Paradigm Tailored to MLLMs (ICLR 2026 · similarity 0.87)
- Bridging Compressed Image Latents and Multimodal Large Language Models (ICLR 2025 · similarity 0.87)
- Benchmarking and Enhancing VLM for Compressed Image Understanding (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.