Research map / 3D vision / Depth and stereo
CLiFT: Compressive Light-Field Tokens for Compute Efficient and Adaptive Neural Rendering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLiFT, a reconstruction-free neural rendering representation that compresses multi-view transformer tokens into a small set of K-means-selected, condensed light-field tokens, with a renderer that adapts to the number of tokens used.
Problem
Reconstruction-free novel view synthesis from sparse posed images produces large token sets (e.g., LVSM's decoder inputs, splats in feed-forward 3DGS), so storage and rendering compute are high and cannot be traded off flexibly at inference.
Approach
Select cluster-centroid rays via latent-space K-means over encoder tokens, condense all tokens into those centroids with a lightweight cross-attention transformer, and render with a transformer decoder trained on a random number of tokens, so one model supports variable storage and render budgets.
Key results
- CLiFT reaches PSNR comparable to MVSplat and DepthSplat with about 5-7x less data size and to LVSM with about 1.8x less data size on RealEstate10K and DL3DV.
- Latent K-means with condenser achieves 25.21 PSNR at 256 tokens vs 22.64 for patch-wise grouping in the ablation.
- With 4096 storage CLiFTs, reducing render CLiFTs to 1024 lowers PSNR by 0.97 (26.72 to 25.75) while raising FPS from 54.3 to 80.77.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use latent K-means instead of patch-wise grouping with the condenser (ablation) | CLiFT · RealEstate10K, 256 tokens | PSNR: +2.24 PSNR | Helped |
| Add the condenser to K-means-selected tokens (ablation) | CLiFT · RealEstate10K, 256 tokens | PSNR: 24.46 -> 25.21 | Helped |
| Add the condenser to K-means-selected tokens (ablation) | CLiFT · RealEstate10K, 1024 tokens | PSNR: 28.17 -> 28.41 | Helped |
| Add the condenser to patch-wise grouped tokens (ablation) | CLiFT · RealEstate10K, 256 tokens | PSNR: 22.64 -> 22.97 | Helped |
| Add the condenser to patch-wise grouped tokens (ablation) | CLiFT · RealEstate10K, 1024 tokens | PSNR: 27.48 -> 27.55 | Helped |
| Reduce render CLiFTs from 4096 to 3072 (ablation) | CLiFT · RealEstate10K, Ns=4096 | PSNR: 26.72 -> 26.71 (-0.01) | No clear effect |
| Reduce render CLiFTs from 4096 to 2048 (ablation) | CLiFT · RealEstate10K, Ns=4096 | FPS: 54.3 -> 66.44 (+22%) | Helped |
| Reduce render CLiFTs from 4096 to 1024 (ablation) | CLiFT · RealEstate10K, Ns=4096 | PSNR: 26.72 -> 25.75 (-0.97) | Hurt |
| Reduce render CLiFTs from 4096 to 1024 (ablation) | CLiFT · RealEstate10K, Ns=4096 | FPS: 54.3 -> 80.77 (+49%) | Helped |
| Reduce render CLiFTs from 4096 to 512 (ablation) | CLiFT · RealEstate10K, Ns=4096 | PSNR: 26.72 -> 23.89 (-2.83) | Hurt |
| Reduce render CLiFTs from 4096 to 512 (ablation) | CLiFT · RealEstate10K, Ns=4096 | FPS: 54.3 -> 90.15 (+66%) | Helped |
Surprising findings
- Patch-wise + condenser gains only 0.33 PSNR at 256 tokens, while K-means + condenser gains 0.75 over K-means alone, suggesting the condenser benefits depend on the selection method.
Limitations
- Degrades when camera motion deviates from the training distribution (smooth translations in RealEstate10K).
- Blurry renderings when target views are not covered by input views in large DL3DV scenes.
Experimental setup
Datasets: RealEstate10K DL3DV
Benchmarks: RealEstate10K DL3DV
Metrics: PSNR SSIM LPIPS FPS FLOPs data size
Baselines: LVSM MVSplat DepthSplat
Builds on: LVSM Plücker ray embeddings faiss k-means
Closest papers by meaning
- CodecNeRF: Toward Fast Encoding and Decoding, Compact, and High-quality Novel-view Synthesis (AAAI 2025 · similarity 0.85)
- SCINeRF: Neural Radiance Fields from a Snapshot Compressive Image (CVPR 2024 · similarity 0.85)
- GoLF-NRT: Integrating Global Context and Local Geometry for Few-Shot View Synthesis (CVPR 2025 · similarity 0.85)
- TeTriRF: Temporal Tri-Plane Radiance Fields for Efficient Free-Viewpoint Video (CVPR 2024 · similarity 0.84)
- ColNeRF: Collaboration for Generalizable Sparse Input Neural Radiance Field (AAAI 2024 · similarity 0.84)
- Few-Shot Neural Radiance Fields under Unconstrained Illumination (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.