Research map / 3D vision / Face and shape
Implicit Zoo: A Large-Scale Dataset of Neural Implicit Functions for 2D Images and 3D Scenes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds Implicit-Zoo, a large dataset of over 1.5M trained implicit neural representations (SIREN for CIFAR-10, ImageNet-1K, Cityscapes; NeRF for OmniObject3D), and benchmarks learnable tokenization for ViT classification/segmentation and transformer-based NeRF pose regression.
Problem
Research on implicit neural representations is limited by the lack of large-scale datasets of INRs, which are expensive to produce, and by the lack of benchmarks for tasks such as classification, segmentation, and pose regression on INRs.
Approach
Fit INRs to large existing 2D and 3D datasets (CIFAR-10, ImageNet-1K, Cityscapes, OmniObject3D) at scale, quality-filter them to PSNR thresholds, and use the continuous coordinate access of INRs to make tokenization locations learnable within ViT-style models.
Key results
- On CIFAR-10-INRs, ViT with learnable centers (LC) reaches 81.33% accuracy and ViT with learnable pixels plus regularization (LP+Reg) reaches 81.57%, versus 80.82% for the ViT baseline.
- On CityScapes-INRs, MiT-B0 with LP+Reg improves fine mIOU from 39.95 to 40.61.
- On OmniObject3D-INRs pose regression with seen scenes, Ours+Pre+LP reduces translation error from 3.50 cm to 2.99 cm and rotation error from 15.29 to 14.17 degrees.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use learnable centers and learnable scaling instead of fixed patch locations (ablation) | ViT · CIFAR-10-INRs | accuracy: 80.82$\pm$0.86% -> 81.33$\pm$ 0.23% | Helped |
| use learnable scaling with uniform centers (ablation) | ViT · CIFAR-10-INRs | accuracy: 80.82$\pm$0.86% -> 80.24$\pm$ 0.47% | Hurt |
| learn all pixel coordinates without regularization (ablation) | ViT · CIFAR-10-INRs | accuracy: 80.82$\pm$0.86% -> 79.51$\pm$ 0.23% | Hurt |
| randomly initialize learnable pixel coordinates (ablation) | ViT · CIFAR-10-INRs | accuracy: 80.82$\pm$0.86% -> 59.43 $\pm$ 1.21 % | Hurt |
| add non-close regularization to learnable pixel coordinates (ablation) | ViT · CIFAR-10-INRs | accuracy: 79.51$\pm$ 0.23% -> 81.57$\pm$ 0.29% | Helped |
| use learnable centers and learnable scaling (ablation) | MiT-B0 · Cityscapes fine | mIOU (fine): 39.95 $\pm$0.9 -> 40.33$\pm$0.7 | Helped |
| use learn pixel coordinates with regularization (ablation) | MiT-B0 · Cityscapes fine | mIOU (fine): 39.95 $\pm$0.9 -> 40.61$\pm$0.6 | Helped |
| use learnable centers and learnable scaling (ablation) | ViT · ImageNet-100 | accuracy: 84.81$\pm$0.92 % -> 85.02$\pm$1.02 % | No clear effect |
| use learn pixel coordinates with regularization (ablation) | ViT · ImageNet-100 | accuracy: 84.81$\pm$0.92 % -> 84.92$\pm$0.93 % | No clear effect |
| pretrain the volume encoder (ablation) | pose regressor · OmniObject3D-INRs unseen scenes | RE: 21.90 \textdegree -> 20.24 \textdegree | Helped |
| use learnable volume centers and scale with a pretrained encoder (ablation) | pose regressor · OmniObject3D-INRs unseen scenes | RE: 20.24 \textdegree -> 20.09 \textdegree | Helped |
| learn volume points with a pretrained encoder (ablation) | pose regressor · OmniObject3D-INRs unseen scenes | RE: 20.24 \textdegree -> 20.02 \textdegree | Helped |
Surprising findings
- Random-initialized learnable pixels (LP+rand) drop CIFAR-10 accuracy to 59.43%, far below the baseline.
- Learnable tokenization improves pixel-level segmentation as well as classification, despite the misalignment between input pixels and labels.
Limitations
- Scalability restricted by INR querying, forcing small batch and model sizes in benchmarks.
- PSNR threshold of 30 may cause artifacts in repetitive backgrounds.
- Pose regression struggles with symmetric objects.
Experimental setup
Datasets: CIFAR-10 ImageNet-1K Cityscapes OmniObject3D ImageNet-100
Benchmarks: CIFAR-10-INRs ImageNet-100-INRs CityScapes-INRs OmniObject3D-INRs
Metrics: accuracy precision F1 mIOU translation error rotation error PSNR
Baselines: ViT SegFormer (MiT-B0) DWSNet HyperRepresentation ViT+LC ViT+LP+Reg
Builds on: SIREN NeRF Vision Transformer SegFormer DWSNet
Closest papers by meaning
- MagicPony: Learning Articulated 3D Animals in the Wild (CVPR 2023 · similarity 0.87)
- ImageNet3D: Towards General-Purpose Object-Level 3D Understanding (NeurIPS 2024 · similarity 0.87)
- LEAP: Liberate Sparse-View 3D Modeling from Camera Poses (ICLR 2024 · similarity 0.87)
- BEAST3D: animal behavioral analysis and neural encoding from multi-view video via Gaussian splatting (NeurIPS 2026 · similarity 0.86)
- DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Features (NeurIPS 2024 · similarity 0.86)
- Large Spatial Model: End-to-end Unposed Images to Semantic 3D (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.