Research map / Probabilistic and scientific modeling / Variational and generative sampling
Latent Modulated Function for Computational Optimal Continuous Image Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Latent Modulated Function (LMF), which moves most HR-HD decoding of INR-based arbitrary-scale super-resolution into LR-HD latent space and renders with a tiny modulated LD MLP, plus a CMSR multi-scale rendering algorithm to cut compute.
Problem
Existing INR-based arbitrary-scale super-resolution (ASSR) methods decode every high-resolution pixel with a large MLP in high-dimensional space, so computation and runtime grow quadratically with the scale factor.
Approach
Decouple HR-HD decoding into a shared latent MLP in LR-HD space that outputs FiLM-style latent modulations (scale and shift) for each LR latent code, and a small low-dimensional render MLP in HR space modulated by these; add CMSR, which renders each latent code only at its minimum sufficient scale based on modulation intensity.
Key results
- LMF reduces decoder MACs by 90.4% to 99.9% and runtime by 2.4x to 56.9x (up to 57x) relative to the original INR-based ASSR methods, while saving 45.1% to 76.0% of parameters.
- LM-LIIF with 170K parameters matches or slightly exceeds LIIF (355K) PSNR on DIV2K100 in-scale (x2: 34.65 vs 34.67 with EDSR-b, roughly comparable), with LM-LIIF MACs of 125.32G vs 3.567T at x2.
- CMSR with MSE threshold 2e-5 keeps PSNR while cutting rendering MACs by 20.58% at x8 and 72.75% at x16.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Convert LIIF to LMF | LIIF with EDSR-b · DIV2K validation, ×2 to ×18 | MACs: reduces computational cost by 96.6% to 99.9% | Helped |
| Convert LTE to LMF | LTE with EDSR-b · DIV2K validation, ×2 to ×18 | MACs: 91.3% to 99.8% | Helped |
| Convert CiaoSR to LMF | CiaoSR with EDSR-b · DIV2K validation, ×2 to ×18 | PSNR (dB) | Hurt |
| Convert INR-based ASSR methods to LMF | LIIF, LTE, and CiaoSR · ASSR | GPU runtime: 2.4× to 56.9× | Helped |
| Shrink the LIIF decoder to a 9-layer MLP with 16 dimensions without modulation (ablation) | LIIF · DIV2K validation ×2 | PSNR (dB): 34.67 -> 33.61 | Hurt |
| Use coarse-to-fine decoding without latent modulation (ablation) | LIIF · DIV2K validation ×2 | PSNR (dB): 34.67 -> 34.11 | Hurt |
| Use scale modulation only instead of scale and shift (ablation) | LM-LIIF · DIV2K validation ×2 | PSNR (dB): 34.66 -> 34.61 | Hurt |
| Use shift modulation only instead of scale and shift (ablation) | LM-LIIF · DIV2K validation ×2 | PSNR (dB): 34.66 -> 34.59 | Hurt |
| Compress the latent code to 16 dimensions for the render MLP (ablation) | LM-LIIF · DIV2K validation ×16 | MACs: 32.79G -> 18.90G | Helped |
| Reduce latent MLP depth from 4 layers to 2 layers (ablation) | LM-LIIF · DIV2K validation ×2 | PSNR (dB): 34.64 -> 34.66 | No clear effect |
| Reduce latent MLP dimension from 192 to 128 (ablation) | LM-LIIF · DIV2K validation ×2 | PSNR (dB): 34.66 -> 34.63 | Hurt |
| Reduce render MLP dimension from 128 to 16 (ablation) | LM-LIIF · DIV2K validation ×2 | MACs: 232.18G -> 123.10G | Helped |
Surprising findings
- LM-CiaoSR shows slightly lower PSNR than CiaoSR, attributed to moving attention-in-attention modules into latent space.
- A coarse-to-fine variant of LIIF with the same LMF-sized structure fails to match LIIF PSNR, showing latent modulation is critical for shrinking the render MLP.
Limitations
- LM-CiaoSR shows slightly lower PSNR than CiaoSR in several settings.
- CMSR only reduces render MLP cost, so gains are limited at small scales.
- Better forms of latent modulation and more efficient decoders are left to future work.
Experimental setup
Datasets: DIV2K Set5 Set14 BSD100 Urban100
Benchmarks: DIV2K100 Set5 Set14 BSD100 Urban100
Metrics: PSNR MACs runtime parameters GPU memory
Baselines: LIIF LTE CiaoSR EDSR-b RDN SwinIR Bicubic
Builds on: LIIF LTE CiaoSR FiLM
Closest papers by meaning
- Continuity-Preserving Convolutional Autoencoders for Learning Continuous Latent Dynamical Models from Images (ICLR 2025 · similarity 0.80)
- MINIMA: Modality Invariant Image Matching (CVPR 2025 · similarity 0.80)
- LMD: Faster Image Reconstruction with Latent Masking Diffusion (AAAI 2024 · similarity 0.80)
- RaPD: Resolution-Agnostic Pixel Diffusion via Semantics-Enriched Implicit Representations (NeurIPS 2026 · similarity 0.80)
- SSC-VAE: Structured Sparse Coding Based Variational Autoencoder for Detail Preserved Image Reconstruction (AAAI 2025 · similarity 0.79)
- Dynamic View Synthesis as an Inverse Problem (NeurIPS 2025 · similarity 0.79)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.