Research map / Video / Video generation
AKiRa: Augmentation Kit on Rays for Optical Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AKiRa, an augmentation kit that simulates zoom, lens distortion and bokeh on video frames paired with extended Plücker ray camera maps, and trains a camera adapter on a frozen video backbone for optical video generation.
Problem
Text-conditioned video diffusion models offer little or no control over camera motion and optical parameters (focal length/zoom, lens distortion, aperture/focus), and no training data pairs videos with such optical parameters.
Approach
Extend Plücker ray camera representation with a distortion-aware direction map and a per-pixel aperture map, and create training data by augmenting frames (cropping/resizing for zoom, radial distortion warping, depth-based virtual bokeh) jointly with the corresponding camera parameters, then train a camera adapter on a frozen backbone.
Key results
- On SVD, AKiRa reaches CD-FVD 398.3 vs 424.9 for CameraCtrl and FlowSim 80.11 vs 79.64.
- On AnimateDiff, AKiRa achieves FlowSim 70.97 vs 53.74 for CameraCtrl and lower RPE-t (1.480 vs 1.807).
- In a 25-person user study, AKiRa was ranked first by 44.2% of participants on average vs 29.2% for CameraCtrl.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add AKiRa camera and optics adapter trained with optical augmentations | AnimateDiff · WebVid | FVD: 384.5 -> 332.3 | Helped |
| Add AKiRa camera and optics adapter trained with optical augmentations | AnimateDiff · WebVid | FlowSim: 53.74 -> 70.97 | Helped |
| Add AKiRa camera and optics adapter trained with optical augmentations | SVD · WebVid | FVD: 173.8 -> 162.8 | Helped |
| Add AKiRa camera and optics adapter trained with optical augmentations | SVD · WebVid | FlowSim: 79.64 -> 80.11 | Helped |
| Add AKiRa optical control | AnimateDiff · WebVid, zoom | ZoomSim: 78.98 -> 86.82 | Helped |
| Add AKiRa optical control | AnimateDiff · WebVid, distortion | DistortSim: 62.13 -> 81.19 | Helped |
| Add focal-length augmentation (ablation) | AKiRa on AnimateDiff · WebVid, augmentation ablation | CD-FVD: 331.0 -> 328.7 | Helped |
| Add lens-distortion augmentation (ablation) | AKiRa on AnimateDiff · WebVid, augmentation ablation | CD-FVD: 332.7 -> 328.7 | Helped |
| Add aperture augmentation (ablation) | AKiRa on AnimateDiff · WebVid, augmentation ablation | CD-FVD: 359.4 -> 328.7 | Helped |
| Add focal-length and lens-distortion augmentation (ablation) | AKiRa on AnimateDiff · WebVid, augmentation ablation | CD-FVD: 359.4 -> 328.7 | Mixed |
| Add focal-length and aperture augmentation (ablation) | AKiRa on AnimateDiff · WebVid, augmentation ablation | CD-FVD: 332.7 -> 328.7 | Helped |
| Add lens-distortion and aperture augmentation (ablation) | AKiRa on AnimateDiff · WebVid, augmentation ablation | FlowSim: 71.97 -> 70.97 | No clear effect |
Surprising findings
- CameraCtrl was preferred for zoom in the user study (43.6% vs 35.6% for AKiRa), attributed to its overly fast zoom that lacks subject consistency.
- Distortion combined with bokeh augmentation performs strongly because distortion's crop-and-rescale implicitly acts as a focal length change.
Limitations
- Relies on monocular depth estimation for bokeh augmentation, so bokeh quality depends on depth estimator accuracy.
- Aperture fixed at omega=50 for quantitative evaluation to keep realism, limiting analysis of bokeh control.
- Evaluation of camera motion depends on ParticleSfM trajectory estimation, which is computationally expensive and noisy.
Experimental setup
Datasets: WebVid RealEstate10K
Benchmarks: WebVid
Metrics: FID FVD CD-FVD RPE-R RPE-t FlowSim VBench consistency VBench smoothness VBench flickering ZoomSim
Baselines: MotionCtrl CameraCtrl AnimateDiff baseline
Builds on: CameraCtrl MotionCtrl AnimateDiff Stable Video Diffusion Plücker coordinates
Closest papers by meaning
- Frame In-N-Out: Unbounded Controllable Image-to-Video Generation (NeurIPS 2025 · similarity 0.84)
- IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation (NeurIPS 2025 · similarity 0.84)
- FloVD: Optical Flow Meets Video Diffusion Model for Enhanced Camera-Controlled Video Synthesis (CVPR 2025 · similarity 0.83)
- CineTechBench: A Benchmark for Cinematographic Technique Understanding and Generation (NeurIPS 2025 · similarity 0.83)
- CRePE: Curved Ray Expectation Positional Encoding for Unified-Camera-Controlled Video Generation (NeurIPS 2026 · similarity 0.83)
- AnimateAnything: Consistent and Controllable Animation for Video Generation (CVPR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.