Research map / Image generation / Fast diffusion sampling
ReFrame: Layer Caching for Accelerated Inference in Real-Time Rendering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends DeepCache-style layer caching to encoder-decoder rendering networks (ExtraNet, FBSR, Implicit Depth), using adaptive frame-delta and motion-vector cache refresh policies to get about 1.4x average inference speedup on an RTX 2080 Ti with negligible quality loss.
Problem
Neural network inference in real-time rendering (frame extrapolation, supersampling, image composition) is latency-sensitive, and the temporal coherence of frames is unexploited in the deeper layers of encoder-decoder networks.
Approach
Cache the deep encoder features that feed skip-connection concatenations and reuse them in subsequent frames, refreshing the cache adaptively when the input frame delta (SMAPE) or average motion vector exceeds a threshold, rather than on a fixed every-N schedule.
Key results
- Delta_L policy gives about 1.6x average speedup with 72% of inferences skipped and a 40% reduction in encoder-decoder FLOPs across workloads.
- Delta_H policy reduces FLOPs by 19% with negligible quality loss (FLIP mostly below 0.025 for FE workloads).
- On FBSR supersampling, 3x upscaling with cache (Delta_H) improves FLIP over the 4x baseline while saving 0.19s over 20 frames.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ReFrame caching with Delta_L instead of no caching | ExtraNet · Frame extrapolation, Sun Temple | speedup: 1.72 | Helped |
| Use Delta_L rather than Delta_H refresh sensitivity (ablation) | ReFrame on ExtraNet · Frame extrapolation, Asian Village | FLIP: 0.021 -> 0.035 | Hurt |
| Refresh every 5 frames rather than use Delta_L (ablation) | ReFrame on ExtraNet · Frame extrapolation, Asian Village | FLIP: 0.035 -> 0.040 | Hurt |
| Refresh every 2 frames rather than use Delta_L (ablation) | ReFrame on ExtraNet · Frame extrapolation, Asian Village | FLIP: 0.035 -> 0.025 | No clear effect |
| Use motion-vector-triggered refresh rather than Delta_L (ablation) | ReFrame on ExtraNet · Frame extrapolation, Asian Village | FLIP: 0.035 -> 0.030 | No clear effect |
| Use the non-linear N-5 refresh schedule rather than Delta_L (ablation) | ReFrame on ExtraNet · Frame extrapolation, Asian Village | FLIP: 0.035 -> 0.040 | Hurt |
| Cache at Level 3 rather than Level 1 (ablation) | ReFrame on ExtraNet · Frame extrapolation, Asian Village | FLOPs: 46.2% -> 71.8% | Helped |
| Cache at Level 3 rather than Level 1 (ablation) | ReFrame on ExtraNet · Frame extrapolation, Asian Village | FLIP: 0.035 -> 0.023 | Helped |
| Cache all additional U-Net++ blocks rather than only the last block (ablation) | ReFrame on Implicit Depth · Image composition | FLIP: 0.014 -> 0.013 | Helped |
| Increase the input from 4× to 3× upscaling while caching (ablation) | ReFrame Delta_H on FBSR · Supersampling, Sun Temple | FLIP: 0.399 -> 0.396 | Helped |
| Combine DeltaCNN with ReFrame rather than use ReFrame alone (ablation) | ExtraNet · Frame extrapolation, Asian Village | latency: 2.0 ms -> 1.9 ms | No clear effect |
| Replace cached features with arbitrary zero, random, or normally distributed values (ablation) | ReFrame on ExtraNet · Sun Temple | FLIP | Hurt |
Surprising findings
- Filling the cache with zero, random, or normal noise gives worse FLIP than not running the network at all (No Update).
- DeltaCNN gave actual FLOPs reductions but little realized speedup, since its scattered computation is poorly suited to commodity GPUs.
Limitations
- Only applies to U-Net-like encoder-decoder or concatenation-based networks, not transformers.
- Average latency is reduced but worst-case frame latency is not, since periodic cache refreshes remain full inference.
- Test set is built from free Unreal Engine scenes and the ExtraNet dataset is not freely available, so exact results are hard to reproduce.
Experimental setup
Datasets: Unreal Engine Marketplace scenes (Asian Village, Cyberpunk, Sun Temple test sequences) Implicit Depth sample scenes (Garden Chair)
Benchmarks: Sun Temple Cyberpunk Asian Village Garden Chair
Metrics: FLIP SSIM PSNR LPIPS MSE speedup Enc-Dec FLOPs skipped frames latency 95th percentile
Baselines: DeepCache every-N policy DeltaCNN Non-linear N-5 policy Motion vector policy No cache baseline
Builds on: DeepCache DeltaCNN U-Net U-Net++
Closest papers by meaning
- Cache Me if You Can: Accelerating Diffusion Models through Block Caching (CVPR 2024 · similarity 0.88)
- Accelerating Diffusion Transformers with Token-wise Feature Caching (ICLR 2025 · similarity 0.86)
- Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching (NeurIPS 2024 · similarity 0.85)
- DC-DiT: Adaptive Compute and Elastic Inference for Visual Generation via Dynamic Chunking (NeurIPS 2026 · similarity 0.85)
- Diffusion Renderer: Neural Inverse and Forward Rendering with Video Diffusion Models (CVPR 2025 · similarity 0.85)
- High-Quality Real-Time Rendering Using Subpixel Sampling Reconstruction (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.