Research map / Learning theory and algorithms / Prediction and decision losses
Task-Driven Wavelets using Constrained Empirical Risk Minimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Constrained Empirical Risk Minimization (CERM), which runs Riemannian SGD so CNN filters stay exactly wavelets, and applies it to task-driven wavelet networks for organ contour prediction on medical images.
Problem
Strictly enforcing function-space constraints (here, that convolutional filters remain wavelets in a multiresolution analysis) is hard in DNNs; soft loss penalties only satisfy constraints approximately.
Approach
Formulate constraints as a regular system F(θ)=0 so the feasible parameters form a Riemannian manifold; perform SGD intrinsically on it using graph charts from the implicit function theorem, so wavelet constraints hold exactly during training. Wavelet low-pass filter constraints (MRA conditions) are derived as F_M, and the wavelet filters are learned end-to-end in a U-Net-like network predicting w
Key results
- Best wavelet networks (order 7-8 for spleen, order 5-6 for prostate) outperform the 2d-nnUNet baseline on test Dice: spleen 0.921 vs 0.914 (order 7); prostate 0.935 vs 0.896 (order 5).
- Wavelet networks use 5M-10M parameters versus roughly 40M (spleen) and 18M (prostate) for nnUNet.
- Task-optimized wavelets differ significantly from random initial wavelets and appear simpler after training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the order 7 wavelet network instead of nnUNet | Wavelet network · Spleen test set | Dice score: 0.914 (1.74 · 10−1) -> 0.921 (6.91 · 10−2) | Helped |
| Use the order 4 wavelet network instead of nnUNet | Wavelet network · Prostate test set | Dice score: 0.896 (1.27 · 10−1) -> 0.935 (3.48 · 10−2) | Helped |
| Use wavelet order 3 (ablation) | Wavelet network · Spleen test set | Dice score: 0.921 (6.91 · 10−2) -> 0.911 (7.38 · 10−2) | Hurt |
| Use wavelet order 3 (ablation) | Wavelet network · Prostate test set | Dice score: 0.935 (3.48 · 10−2) -> 0.929 (4.66 · 10−2) | Hurt |
| Use wavelet order 4 (ablation) | Wavelet network · Spleen test set | Dice score: 0.921 (6.91 · 10−2) -> 0.911 (7.85 · 10−2) | Hurt |
| Use wavelet order 5 (ablation) | Wavelet network · Spleen test set | Dice score: 0.921 (6.91 · 10−2) -> 0.916 (6.94 · 10−2) | Hurt |
| Use wavelet order 5 (ablation) | Wavelet network · Prostate test set | Dice score: 0.935 (3.48 · 10−2) -> 0.935 (4.11 · 10−2) | No clear effect |
| Use wavelet order 6 (ablation) | Wavelet network · Spleen test set | Dice score: 0.921 (6.91 · 10−2) -> 0.917 (7.17 · 10−2) | Hurt |
| Use wavelet order 6 (ablation) | Wavelet network · Prostate test set | Dice score: 0.935 (3.48 · 10−2) -> 0.934 (4.14 · 10−2) | Hurt |
| Use wavelet order 7 (ablation) | Wavelet network · Prostate test set | Dice score: 0.935 (3.48 · 10−2) -> 0.928 (4.03 · 10−2) | Hurt |
| Use wavelet order 8 (ablation) | Wavelet network · Spleen test set | Dice score: 0.921 (6.91 · 10−2) -> 0.919 (6.64 · 10−2) | Hurt |
| Use wavelet order 8 (ablation) | Wavelet network · Prostate test set | Dice score: 0.935 (3.48 · 10−2) -> 0.934 (3.62 · 10−2) | Hurt |
Surprising findings
- Learned wavelets become simpler and 'cleaned up' relative to their random initialization.
Limitations
- Computational cost rises when many parameters are constrained relative to the number of constraints.
- Setting up well-posed constraints can be non-trivial due to redundancies; the constrained manifold may be disconnected, making initialization matter.
- Experiments limited to one-dimensional wavelets and two medical contour tasks.
Experimental setup
Datasets: Medical Segmentation Decathlon spleen CT Medical Segmentation Decathlon prostate T2 MRI
Benchmarks: Medical Segmentation Decathlon spleen Medical Segmentation Decathlon prostate
Metrics: dice score
Baselines: 2d-nnUNet
Builds on: nnUNet Riemannian SGD Medical Segmentation Decathlon
Closest papers by meaning
- An Empirical Evaluation of Current Convolutional Architectures' Ability to Manage Nuisance Location and Scale Variability (CVPR 2016 · similarity 0.85)
- Learning multi-scale local conditional probability models of images (ICLR 2023 · similarity 0.85)
- Sample-Specific Output Constraints for Neural Networks (AAAI 2021 · similarity 0.85)
- Band-limited Training and Inference for Convolutional Neural Networks (ICML 2019 · similarity 0.85)
- Improving Robustness of Deep-Learning-Based Image Reconstruction (ICML 2020 · similarity 0.84)
- Learning Across Scales---Multiscale Methods for Convolution Neural Networks (AAAI 2018 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.