Research map / Vision-language and multimodal / Human and embodied modeling
DVI:A Derivative-based Vision Network for INR
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DVI, which fuses high-order derivative maps of an INR into pre-existing raster-based vision networks via progressive Swin-based feature extraction and fusion, improving image, volume, and video tasks.
Problem
Vision networks for INR data are either purely INR-based (limited operations, weak performance) or raster-based (lose structural information when converting INR to raster form).
Approach
Compute high-order derivative maps of the INR with an efficient recursive method and progressively extract and fuse these structural features into intermediate layers of any raster-based vision network via Swin-transformer extractors and cross-attention fusion modules.
Key results
- Image SR: DVI(SwinIR) reaches 31.96 PSNR on Set5 vs 30.00 for SwinIR and 19.37 for INSP.
- 3D volume segmentation: DVI(UNet3D) improves mean DSC from 68.46 to 80.49 on Synapse.
- Video optical flow: DVI(FlowFormer) lowers Sintel final EPE from 6.35 to 5.67 (all).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DVI with SwinIR instead of the raster-based SwinIR baseline | Image super-resolution · Set5 (×2) | PSNR: 30.00 -> 31.96 | Helped |
| use DVI with SwinIR instead of the raster-based SwinIR baseline | Image denoising · Kodak24 | PSNR: 34.53 -> 35.95 | Helped |
| use DVI with MedSegDiff-V2 instead of MedSegDiff-V2 | 3D volume segmentation · Synapse | DSC: 75.79 -> 85.46 | Helped |
| use DVI with VD-Diff instead of VD-Diff | Video deblurring · GoPro | PSNR: 28.23 -> 29.07 | Helped |
| use DVI with FlowDiffuser instead of FlowDiffuser | Video optical flow estimation · Sintel (final) | EPE (all): 4.94 -> 3.92 | Helped |
| reduce spatio-temporal resolution (ablation) | DVI(ViViT) · Something-Something V2 | Top1-accuracy: 57.0 -> 60.2 | Helped |
| reduce spatio-temporal resolution and increase derivative order (ablation) | DVI(ViViT) · Something-Something V2 | Top1-accuracy: 60.2 -> 64.9 | Helped |
| remove feature extraction and fusion modules (ablation) | DVI(DnCNN) · Kodak24 | PSNR | Hurt |
| replace the INR derivative map with derivatives computed numerically from the raw signal (ablation) | DVI(VNET) · Synapse | DSC: 83.60 -> 74.33 | Hurt |
| use ReLU positional encoding INR instead of SIREN INR (ablation) | DVI(VNET) · Synapse | DSC: 83.60 -> 80.29 | Hurt |
| use FFN INR instead of SIREN INR (ablation) | DVI(VNET) · Synapse | DSC: 83.60 -> 80.00 | Hurt |
Surprising findings
- Numerical derivatives computed from the raw signal are ineffective compared with derivatives from the INR (e.g., 74.33 vs 83.60 DSC).
- Performance drops notably when the highest derivative order reaches 5 across tasks.
Limitations
- DVI adds computational overhead from derivative computation and the feature extraction/fusion networks.
- Underperforms on tasks requiring coarse structural information such as video classification without resolution reduction and higher derivative order.
- Complex data augmentations (color jitter, noise, scaling) are not supported on-the-fly for INR pairs and were removed from experiments.
Experimental setup
Datasets: DIV2K Set5 Set14 BSD100 Urban100 Manga109 BSD500 WED CBSD68 Kodak24 McMaster Synapse
Benchmarks: Set5 Set14 BSD100 Urban100 Manga109 CBSD68 Kodak24 McMaster Synapse GoPro Sintel Something-Something V2
Metrics: PSNR SSIM DSC EPE Top-1 accuracy MAC parameter count
Baselines: INSP EDSR SwinIR StableSR DnCNN DiffBIR UNet3D VNET MedSegDiff-V2 PVDNet VDTR VD-Diff
Builds on: INSP SwinIR HOPE recursive high-order derivatives SIREN
Closest papers by meaning
- Implicit Representations via Operator Learning (ICML 2024 · similarity 0.86)
- VI^3NR: Variance Informed Initialization for Implicit Neural Representations (CVPR 2025 · similarity 0.85)
- HIIF: Hierarchical Encoding based Implicit Image Function for Continuous Super-resolution (CVPR 2025 · similarity 0.85)
- Grids Often Outperform Implicit Neural Representation at Compressing Dense Signals (NeurIPS 2025 · similarity 0.85)
- IINet: Implicit Intra-inter Information Fusion for Real-Time Stereo Matching (AAAI 2024 · similarity 0.84)
- Explaining the Implicit Neural Canvas: Connecting Pixels to Neurons by Tracing their Contributions (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.