Research map / Probabilistic and scientific modeling / Variational and generative sampling
Learning Set Functions with Implicit Differentiation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves convergence of the mean-field fixed-point iterations for learning set functions from optimal subset oracles and replaces loop-unrolled autodiff with implicit differentiation (iDiffMF), reducing memory and improving subset selection accuracy.
Problem
Learning set functions from optimal subset oracle data via energy-based models with mean-field variational inference; the fixed-point iterations may diverge and unrolled autodiff through them is memory- and time-prohibitive.
Approach
Establishes a contraction condition (multilinear relaxation bounded by 1/|V|) guaranteeing unique convergence of the fixed-point iterations, and computes gradients via implicit function theorem (linear system with Hessian of the multilinear relaxation) instead of unrolling iterations.
Key results
- iDiffMF_* achieves best test JC on BindingDB (77.48 ± 1.04) vs 73.22 for DiffMF and 73.59 for EquiVSet_ind.
- iDiffMF variants obtain best or second-best JC in most test cases, e.g., Amazon carseats 22.42 (iDiffMF_*) vs 20.03 (EquiVSet_copula).
- Implicit differentiation keeps GPU memory constant as fixed-point iterations increase, whereas autodiff memory grows linearly.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use iDiffMF variants instead of the strongest listed baseline | Set anomaly detection · CelebA, Gaussian, and Moons | Test JC | Helped |
| Use iDiffMF variants instead of EquiVSet copula | Product recommendation · Amazon carseats, furniture, and safety | Test JC | Helped |
| Use iDiffMF variants instead of EquiVSet copula | Product recommendation · Amazon apparel, diaper, feeding, gear, and toys | Test JC | Hurt |
| Use nuclear-norm scaling instead of Frobenius-norm scaling (ablation) | iDiffMF · Test JC across the reported datasets | Test JC | Mixed |
| Replace loop-unrolled automatic differentiation with implicit differentiation | Mean-field fixed-point training · GPU memory usage as fixed-point iterations increase | GPU memory usage | Helped |
Surprising findings
- EquiVSet_copula is faster yet is not consistently the second-best performer, and iDiffMF gives significantly better results on several Amazon datasets.
Limitations
- Convergence assumption (multilinear relaxation bounded by 1/|V|) is not verified exactly; Lipschitz constant of DNNs is NP-hard to compute, so heuristic scaling is used.
- Training time is comparable or higher than baselines on several datasets, and frameworks differ (JAX vs PyTorch).
Experimental setup
Datasets: CelebA Gaussian Moons Amazon (12 categories) BindingDB
Benchmarks: CelebA Gaussian Moons Amazon product recommendation (12 categories) BindingDB
Metrics: Jaccard coefficient training time (s) GPU memory usage
Baselines: DiffMF EquiVSet_ind EquiVSet_copula
Builds on: DiffMF EquiVSet deep equilibrium models JAXopt deep sets
Closest papers by meaning
- Enhancing Neural Subset Selection: Integrating Background Information into Set Representations (ICLR 2024 · similarity 0.85)
- A Unified Framework for Entropy Search and Expected Improvement in Bayesian Optimization (ICML 2025 · similarity 0.83)
- Bayesian Optimization for Unknown Cost-Varying Variable Subsets with No-Regret Costs (AAAI 2025 · similarity 0.83)
- Active Set Ordering (NeurIPS 2024 · similarity 0.83)
- Learning Augmented Exact Exponential Algorithms (NeurIPS 2026 · similarity 0.82)
- Two-timescale Derivative Free Optimization for Performative Prediction with Markovian Data (ICML 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.