Research map / Architectures and efficiency / Attention and transformers
Pre-training Vision Models with Mandelbulb Variations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes formula-driven supervised pre-training datasets (MandelbulbVAR-1k and MandelbulbVAR-Hybrid-21k) rendered from colored, shaded 3D Mandelbulb fractal variations, and benchmarks them for CNN/ViT pre-training and PatchCore anomaly detection.
Problem
ImageNet pre-training raises copyright, privacy, and bias concerns, and existing formula-driven supervised learning (FDSL) datasets are 2D or grayscale and have not been shown to work jointly for CNN and ViT pre-training and for frozen-feature cold-start anomaly detection.
Approach
Model 3D Mandelbulb Variations parametrized by n and a 9-bit boolean vector, color them with orbit traps, shade them under a simulated light, and render RGB projections from random virtual cameras, plus hybrid variations that compose two variations to increase shape diversity.
Key results
- ResNet-50 pre-trained on MandelbulbVAR-1k averages 84.5% top-1 across five targets, versus 83.5% for the second-best FDSL dataset (FractalDB-10k).
- PatchCore with WideResNet-50 pre-trained on MandelbulbVAR-1k reaches 97.2% average image-level AUROC on MVTec AD, versus 92.7% for VisualAtom-1k and 99.1% for ImageNet-1k.
- ViT-B pre-trained on MandelbulbVAR-Hybrid-21k reaches 82.2% top-1 on ImageNet-1k, above ImageNet-21k pre-training (81.8%) and 0.1 below VisualAtom-1k (82.3%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pre-train ResNet-50 on MandelbulbVAR-1k instead of VisualAtom-1k | ResNet-50 · ImageNet-1k val | top-1 accuracy: 73.2 -> 73.4 | Helped |
| Pre-train ResNet-50 on MandelbulbVAR-1k instead of ImageNet-1k | ResNet-50 · CIFAR-10 val | top-1 accuracy: 97.1 -> 93.9 | Hurt |
| Pre-train ResNet-50 on MandelbulbVAR-1k instead of ImageNet-1k | ResNet-50 · CIFAR-100 val | top-1 accuracy: 84.4 -> 76.2 | Hurt |
| Pre-train ResNet-50 on MandelbulbVAR-1k instead of ImageNet-1k | ResNet-50 · Flowers val | top-1 accuracy: 98.3 -> 96.6 | Hurt |
| Pre-train ResNet-50 on MandelbulbVAR-1k instead of VisualAtom-1k | ResNet-50 · ImageNet-100 val | top-1 accuracy: 81.4 -> 82.3 | Helped |
| Pre-train WideResNet-50 on MandelbulbVAR-1k instead of ImageNet-1k | PatchCore with WideResNet-50 · MVTec AD | average image-level AUROC: 99.1 -> 97.2 | Hurt |
| Pre-train ViT-Base on MandelbulbVAR-Hybrid-21k instead of ImageNet-21k | ViT-Base · ImageNet-1k val | top-1 accuracy: 81.8% -> 82.2% | Helped |
| Use MandelbulbVAR-Hybrid-21k instead of MandelbulbVAR-1k for ViT-Tiny pre-training | ViT-Tiny · CIFAR-10/100, Flowers, ImageNet-100 average | average top-1 accuracy: 91.2% vs. 91.9% | Helped |
| Use MandelbulbVAR-Hybrid-21k instead of MandelbulbVAR-1k for ViT-Base pre-training | ViT-Base · CIFAR-10/100, Flowers, ImageNet-100 average | average top-1 accuracy: 91.7% vs. 92.0% | Mixed |
Surprising findings
- ViT-B pre-trained on MandelbulbVAR-Hybrid-21k slightly exceeds ImageNet-21k pre-training on ImageNet-1k fine-tuning despite containing no natural images.
Limitations
- Datasets lack semantic content, so they may be poor for semantic representation learning.
- MandelbulbVAR-1k does not outperform contour-based datasets (RCDB, VisualAtom) for ViT pre-training on average.
- Gains are often within about 1 point and some comparisons use differing pre-training budgets against the baselines.
Experimental setup
Datasets: MandelbulbVAR-1k MandelbulbVAR-Hybrid-21k FractalDB-1k FractalDB-10k VisualAtom-1k RCDB-1k ExFractalDB-1k ImageNet-1k ImageNet-21k CIFAR-10 CIFAR-100 Flowers
Benchmarks: IN1k CIFAR-10 CIFAR-100 Flowers ImageNet-100 MVTec AD
Metrics: top-1 accuracy image-level AUROC pixel-wise AUROC PRO
Baselines: FractalDB-1k FractalDB-10k VisualAtom-1k RCDB-1k ExFractalDB-1k ImageNet-1k pre-training ImageNet-21k pre-training VisualAtom-21k RCDB-21k PatchCore
Builds on: FractalDB VisualAtom ExFractalDB PatchCore Mandelbulb Variations
Closest papers by meaning
- Should VLMs be Pre-trained with Image Data? (ICLR 2025 · similarity 0.83)
- Dataset Distillation for Pre-Trained Self-Supervised Vision Models (NeurIPS 2025 · similarity 0.83)
- Scaling Laws of Synthetic Images for Model Training ... for Now (CVPR 2024 · similarity 0.82)
- ADPretrain: Advancing Industrial Anomaly Detection via Anomaly Representation Pretraining (NeurIPS 2025 · similarity 0.82)
- Asymmetric Masked Distillation for Pre-Training Small Foundation Models (CVPR 2024 · similarity 0.82)
- ScaleKD: Strong Vision Transformers Could Be Excellent Teachers (NeurIPS 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.