Research map / Vision-language and multimodal / Human and embodied modeling
All-in-One Image Coding for Joint Human-Machine Vision with Multi-Path Aggregation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Multi-Path Aggregation (MPA), an all-in-one learned image codec that routes shared latent features to task-specific MLP paths via a learned importance predictor, fine-tuning only ~1.89% of parameters for machine tasks.
Problem
Joint human-machine image coding: existing approaches use multiple task-specific encoder-decoder pairs (high parameter and bitrate overhead) or a unified representation whose multi-objective optimization degrades performance across tasks.
Approach
Insert parallel MLP paths (main and side) into the transform blocks of a learned image codec and use a lightweight importance-score predictor with Gumbel-Sigmoid to allocate latent features among paths; a two-stage training fixes the generalized model first, then optimizes only side paths and predictors for each machine task.
Key results
- MPA fine-tunes only 1.89% of parameters for machine vision and reaches classification accuracy comparable to fully fine-tuned models on ImageNet-1K.
- MPA's MSE path achieves 16.04% BD-rate against VTM on Kodak, with a smaller decoder (9.295M vs 13.38M for CRDR) and 7.27% partial fine-tuning.
- MPA's FID at low bitrate is lower than CRDR's, with performance comparable to state-of-the-art unified models for human vision.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MPA instead of LoRA fine-tuning | TinyLIC · Kodak | BD-Rate against VTM: 16.26% -> 16.04% | Helped |
| use MPA instead of LoRA fine-tuning | TinyLIC · CLIC | BD-Rate against VTM: 21.80% -> 21.43% | Helped |
| add the predictors (ablation) | MPA encoder · Kodak | BD-Rate against VTM: 16.25% -> 16.04% | Helped |
| add the high-quality path (ablation) | MPA encoder · Kodak | BD-Rate against VTM: 17.05% -> 16.04% | Helped |
| add the low-quality path (ablation) | MPA encoder · Kodak | BD-Rate against VTM: 17.18% -> 16.04% | Helped |
| use inverted bottleneck MLP instead of bottleneck MLP for the MSE path (ablation) | MPA · Kodak | BD-Rate: 19.51% -> 16.04% | Helped |
| use inverted bottleneck MLP instead of bottleneck MLP for the classification path (ablation) | MPA · ImageNet-1K | accuracy: 76.72% -> 77.16% | Helped |
| use inverted bottleneck MLP instead of bottleneck MLP for the segmentation path (ablation) | MPA · ADE20K | mIoU: 37.41% -> 37.76% | Helped |
| use the task-specific MSE path for MSE coding rather than another task's path (ablation) | MPA · Kodak | BD-Rate: 49.61% (perceptual path), 32.81% (classification path), 34.19% (segmentation path) vs. 16.04% (MSE path) | Helped |
| use the task-specific classification path rather than another task's path (ablation) | MPA · ImageNet-1K | accuracy: 76.66% (perceptual), 60.59% (MSE), 73.57% (segmentation) vs. 76.77% (classification) | Helped |
| use the task-specific segmentation path rather than another task's path (ablation) | MPA · ADE20K | mIoU: 36.17% (perceptual), 28.34% (MSE), 35.34% (classification) vs. 37.41% (segmentation) | Helped |
| combine perceptual loss and cross-entropy loss rather than use cross-entropy loss only (ablation) | MPA classification path · ImageNet-1K | Top-1 accuracy on ConvNeXt-T: 76.79% -> 77.01% | Helped |
Surprising findings
- Perceptually optimized (LPIPS) features serve as the best main path and can even yield higher PSNR than MSE-optimized features when fine-tuned with MSE at low bitrate.
- Machine-vision paths use bottleneck MLPs since their accuracy is less sensitive to path complexity, whereas human-vision paths benefit from inverted bottleneck MLPs.
Limitations
- Increased latency compared to the baseline due to flattening, selection, and reassembly operations.
- Each task requires separate training of its side path, rather than joint multi-task training.
- Fine-tuning and training steps differ from baselines, with comparisons limited to the authors' setup for baselines without public code.
Experimental setup
Datasets: Flickr2W DIV2K CLIC ImageNet-1K ADE20K Kodak ImageNet validation ADE20K validation MS-COCO 2017
Benchmarks: Kodak CLIC ImageNet-1K ADE20K MS-COCO 2017
Metrics: BD-Rate PSNR FID LPIPS top-1 accuracy mIoU mAP
Baselines: MRIC DIRAC CRDR HiFiC VTM17.1 SFT TransTIC LoRA
Builds on: TinyLIC HiFiC Gumbel-Sigmoid LPIPS
Closest papers by meaning
- Unified Coding for Both Human Perception and Generalized Machine Analytics with CLIP Supervision (AAAI 2025 · similarity 0.87)
- Test-Time Fine-Tuning of Image Compression Models for Multi-Task Adaptability (CVPR 2025 · similarity 0.85)
- When MLLMs Meets Compression Distortion: A Coding Paradigm Tailored to MLLMs (ICLR 2026 · similarity 0.83)
- OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence (NeurIPS 2026 · similarity 0.83)
- Perception Encoder: The best visual embeddings are not at the output of the network (NeurIPS 2025 · similarity 0.83)
- LSNet: See Large, Focus Small (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.