Research map / Architectures and efficiency / Quantization
ESCA: Enabling Seamless Codec Avatar Execution through Algorithm and Hardware Co-Optimization for Virtual Reality
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ESCA, a full-stack framework combining 4-bit post-training quantization for Codec Avatar VAE decoders (ICAS, FFAS, UV-weighted Hessian PTQ) with a custom input-combining systolic accelerator for transposed convolutions.
Problem
Real-time photorealistic Codec Avatar decoding on resource-constrained VR headsets is too slow; transposed convolutions cause activation outliers that break low-bit quantization and create structured sparsity that underutilizes accelerators.
Approach
Adapt SmoothQuant-style channel smoothing to transposed convolutions (fused into preceding LeakyReLU layers), skip smoothing on facial-detail channels identified via UV/facial masks, weight GPTQ-style Hessian calibration by UV importance, and add an input-combining mechanism in a weight-stationary systolic array to skip zero activations from zero-inserted im2col.
Key results
- At W4A4, FFAS-UV reaches VDP 5.8541 (front) vs. 5.4980 for GPTQ, i.e. up to +0.39 VDP gain on side views.
- Input-combining accelerator reduces INT8 decoder latency from 42.04 ms to 12.51 ms (3.36x speedup).
- End-to-end pipeline estimates 10 ms effective per-frame latency, i.e. 100 FPS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add FFAS, UV-weighted PTQ, and ICAS rather than using GPTQ | Codec Avatar decoder · MultiFace, W4A4, front/left/right views | FovVideoVDP: +0.36/+0.39/+0.39 VDP | Helped |
| add FFAS-UV instead of GPTQ | Codec Avatar decoder · MultiFace spontaneous expressions, W4A4 | FovVideoVDP: +0.49 to +0.66 VDP | Helped |
| use ICAS smoothing without UV weighting or FFAS instead of FFAS-UV (ablation) | Codec Avatar decoder · MultiFace, W4A4, front/left/right views | FovVideoVDP: 5.5901 vs 5.8541; 4.7317 vs 4.9795; 4.7536 vs 4.9605 | Hurt |
| use UV-weighted quantization without smoothing or FFAS instead of FFAS-UV (ablation) | Codec Avatar decoder · MultiFace, W4A4, front/left/right views | FovVideoVDP: 5.7559 vs 5.8541; 4.8130 vs 4.9795; 4.8187 vs 4.9605 | Hurt |
| remove FFAS from the smoothed UV-weighted method (ablation) | Codec Avatar decoder · MultiFace, W4A4, front/left/right views | FovVideoVDP: 5.6438 vs 5.8541; 4.9145 vs 4.9795; 4.9057 vs 4.9605 | Hurt |
| add input-combining to the accelerator | Codec Avatar decoder · INT8 inference | latency: 42.04 ms -> 12.51 ms; 3.36× | Helped |
| overlap transmission and decoding and process multiple frames in a pipelined fashion | Codec Avatar end-to-end pipeline · VR headset end-to-end inference | frame rate: 100 FPS | Helped |
| apply ICAS-only smoothing at W8A8 instead of less aggressive alternatives (ablation) | Codec Avatar decoder · MultiFace, Front, W8A8 | FovVideoVDP: reduction of 0.9 | Hurt |
Surprising findings
- ICAS-only variants underperform other methods at 8-bit, suggesting excessive smoothing harms high-frequency detail.
- Standard rotation-based LLM quantization methods cannot be applied directly to the decoder because LeakyReLU non-linearities block offline weight rotation.
Limitations
- Relies on accurate facial UV priors.
- Only accelerates the decoding stage; the rest of the pipeline is unoptimized.
- End-to-end rate is estimated from per-stage latency references rather than fully measured on-device.
Experimental setup
Datasets: MultiFace
Benchmarks: MultiFace
Metrics: FovVideoVDP LPIPS PSNR SSIM latency end-to-end frame rate
Baselines: Full Codec Avatar (FP32) AdaRound+LSQ POCA 2DQuant GPTQ ICAS UV-W ICAS-UV
Builds on: SmoothQuant GPTQ QuaRot LSQ AdaRound
Closest papers by meaning
- LUCAS: Layered Universal Codec Avatars (CVPR 2025 · similarity 0.82)
- RAM-Avatar: Real-time Photo-Realistic Avatar from Monocular Videos with Full-body Control (CVPR 2024 · similarity 0.82)
- HERA: Hybrid Explicit Representation for Ultra-Realistic Head Avatars (CVPR 2025 · similarity 0.82)
- Stratified Avatar Generation from Sparse Observations (CVPR 2024 · similarity 0.82)
- D-VLA: A High-Concurrency Distributed Asynchronous RL Framework for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.81)
- Relightable Gaussian Codec Avatars (CVPR 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.