Research map / Vision-language and multimodal / Human and embodied modeling
Test-Time Fine-Tuning of Image Compression Models for Multi-Task Adaptability
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes instance-specific test-time fine-tuning of a pretrained learned image compressor (TIC) using SVD-LoRA, tuning only singular values of the decoder to keep bitstream overhead low, for closed-set and open-set machine vision tasks.
Problem
Adapting human-perception learned image compressors to closed-set and especially open-set (unseen at training) machine vision tasks while limiting bitstream overhead and avoiding task-specific pre-training.
Approach
Apply test-time fine-tuning to both encoder and decoder of a frozen LIC backbone via SVD-based LoRA, transmitting only the decoder-side singular values per image, so adaptation needs no pre-training and works for open-set tasks.
Key results
- Ours (w/ TTFT) achieves BD-Rate of -56.49% on instance segmentation and -54.96% on object detection under closed-set conditions, versus TransTIC's -47.45% and -46.64%.
- Under open-set settings, Ours (w/ TTFT) reaches BD-Rate -49.51% (instance segmentation) and -48.94% (object detection), beating TransTIC's closed-set results.
- Training all SVD-LoRA parameters in the decoder increases bitstream overhead by 15.3x compared with singular-value-only training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add instance-specific TTFT to SVD-LoRA | TIC adaptation · Instance Segmentation, closed-set | BD-mAP: 1.99 -> 2.68 | Helped |
| Add instance-specific TTFT to SVD-LoRA | TIC adaptation · Object Detection, closed-set | BD-mAP: 2.17 -> 2.98 | Helped |
| Add instance-specific TTFT to SVD-LoRA | TIC adaptation · Classification, closed-set | BD-accuracy: 0.08 -> 0.14 | Helped |
| Add instance-specific TTFT to SVD-LoRA | TIC adaptation · Instance Segmentation, open-set | BD-mAP: 1.97 -> 2.43 | Helped |
| Add instance-specific TTFT to SVD-LoRA | TIC adaptation · Object Detection, open-set | BD-mAP: 1.87 -> 2.69 | Helped |
| Increase TTFT iterations from 5 to 40 (ablation) | TIC on open-set object detection · MS-COCO validation | mAP: 35.28 -> 36.25 | Helped |
| Increase TTFT iterations from 5 to 40 (ablation) | TIC on open-set object detection · MS-COCO validation | TTFT time (sec): 1.18 -> 14.13 | Hurt |
| Tune both encoder and decoder rather than only the encoder or decoder (ablation) | TIC on open-set object detection · Object Detection, open-set | mAP | Helped |
| Tune only decoder singular values instead of all decoder SVD-LoRA parameters (ablation) | TIC on open-set object detection · Object Detection, open-set | Bitstream overhead: 15.3x | Helped |
Surprising findings
- Decoder-side test-time fine-tuning with only singular values adds under 1% bitstream overhead, while full SVD-LoRA decoder tuning adds about 15.3x more overhead.
Limitations
- TTFT adds per-instance encoding latency that grows linearly with iterations.
- Open-set initialization relies on a classification model to provide generalized features.
- Evaluated on only one backbone in the main text (TIC), with CNN-based LIC results only in supplementary material.
Experimental setup
Datasets: MS-COCO ImageNet-1K Flickr2W
Benchmarks: MS-COCO ImageNet-1K
Metrics: BD-Rate BD-mAP BD-accuracy top-1 accuracy mAP
Baselines: TIC Full fine tuning TransTIC UIC TIC+SFT TIC+channel selection TIC+prefilter
Builds on: LoRA SVD-LoRA TransTIC TIC
Closest papers by meaning
- Unified Coding for Both Human Perception and Generalized Machine Analytics with CLIP Supervision (AAAI 2025 · similarity 0.87)
- Benchmarking and Enhancing VLM for Compressed Image Understanding (ICML 2026 · similarity 0.86)
- VoCo-LLaMA: Towards Vision Compression with Large Language Models (CVPR 2025 · similarity 0.86)
- Bridging Compressed Image Latents and Multimodal Large Language Models (ICLR 2025 · similarity 0.86)
- Few-Shot Domain Adaptation for Learned Image Compression (AAAI 2025 · similarity 0.85)
- Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.