Research map / Video / Video segmentation and tracking

Explaining the Implicit Neural Canvas: Connecting Pixels to Neurons by Tracing their Contributions

CVPR 2024 · Video › Video segmentation and tracking · approach: Neural video representations · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes XINC, a framework that computes per-neuron contribution maps linking neurons of MLP (Fourier feature network) and CNN (NeRV) implicit neural representations to output pixels, and uses them to analyze what INRs learn.

Problem

Implicit neural representations (INRs) used for video compression and view synthesis are opaque; existing interpretability tools like NetDissect and Grad-CAM do not transfer straightforwardly to coordinate-to-signal networks such as NeRV and MLP-based INRs.

Approach

Dissect INRs by mapping each neuron's activations (weights times inputs) to every output pixel, forming contribution maps that together form an "implicit neural canvas"; for NeRV this extends to convolution kernels, PixelShuffle rearrangement, and k×k aggregation across layers.

Key results

• Clustering by instances, RGB colors, and Gabor features yields much higher contribution variance than spatial gridcells, indicating INRs ignore spatial position in favor of object, color, and edge features.
• NeRV contributions focus heavily on edges, while FFN contributions correlate almost perfectly with image intensity but are more evenly distributed per pixel.
• Changes in NeRV neuron contributions over time are concentrated around regions of optical flow, indicating motion drives contribution change.

Surprising findings

• INRs lack high-level class semantics: contributions to instances of the same category are not grouped together.
• Nearly 50% of neurons in FFN Layer 2 have almost no contributions, indicating many dead neurons.

Limitations

• Analysis limited to small-resolution (128x256) videos and small models chosen for matched bits-per-pixel.
• Only one MLP-based (FFN) and one CNN-based (NeRV) architecture analyzed.

Experimental setup

Datasets: Cityscapes-VPS VIPSeg

Benchmarks: Cityscapes-VPS VIPSeg

Metrics: variance of normalized contribution differences across clust fraction of neurons contributing per pixel neuron contribution counts per pixel

Builds on: NeRV Fourier Feature Networks Network Dissection Grad-CAM

Closest papers by meaning

• Bias for Action: Video Implicit Neural Representations with Bias Modulation (CVPR 2025 · similarity 0.88)
• Grids Often Outperform Implicit Neural Representation at Compressing Dense Signals (NeurIPS 2025 · similarity 0.88)
• Implicit Representations via Operator Learning (ICML 2024 · similarity 0.87)
• Long-Range Feedback Spiking Network Captures Dynamic and Static Representations of the Visual Cortex under Movie Stimuli (NeurIPS 2024 · similarity 0.86)
• A sampling theory perspective on activations for implicit neural representations (ICML 2024 · similarity 0.86)
• VI^3NR: Variance Informed Initialization for Implicit Neural Representations (CVPR 2025 · similarity 0.85)