Research map / Learning theory and algorithms / Combinatorial and search optimization

MAdam: Metric-Aware Multi-Objective Adam

NeurIPS 2026 · Learning theory and algorithms › Combinatorial and search optimization · approach: Multi-objective optimization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MAdam, a drop-in wrapper that preconditions the reconciled multi-objective solver direction by a preference-conditioned diagonal Fisher metric before Adam, so Adam's second moment becomes near-identity.

Problem

Adam, used as the default optimizer for MOO solvers (loss-balancing, gradient-balancing, Pareto-based), entangles time-varying preferences with gradient statistics (weighting mismatch) and imposes a non-Euclidean diagonal RMS metric that distorts the solver's geometry (geometric mismatch).

Approach

Precondition the solver's reconciled direction by the preference-conditioned diagonal Fisher of the scalarized gradient (built from cross-objective gradient second moments with stochastic pair sampling and rampup), so that Adam's second-moment denominator becomes approximately identity and the update follows the preference-conditioned metric; the solver and optimizer are left unchanged.

Key results

• On SARCOS, LS + MAdam lowers average error from 10.99 to 10.00 and CAGrad + MAdam from 10.98 to 10.38.
• On Cityscapes, IMTL + MAdam reduces relative depth error from 74.23 to 61.81.
• On SARCOS Pareto MTL, PaLoRA + MAdam raises hypervolume from 431.2 to 442.2 and PaMaL + MAdam from 378.3 to 394.6.

Surprising findings

• Adam's second-moment denominator can flip the sign of a task's per-objective loss change relative to the Euclidean direction, turning aligned objectives into apparent conflicts.
• On ISIC2018, MAdam performs on par with Adam under SwinTransformer rather than improving it.

Limitations

• Optimizer memory grows quadratically in number of objectives, storing C(C+1)/2 Fisher EMAs.
• Per-step runtime overhead of 12-19% on SARCOS.
• Gains are within one standard deviation where the base solver already balances tasks well.

Experimental setup

Datasets: MultiMNIST SARCOS UTKFace Cityscapes NYUv2 PINNacle ISIC2018 OASIS3

Benchmarks: MultiMNIST SARCOS UTKFace Cityscapes NYUv2 PINNacle ISIC2018 OASIS3

Metrics: top-1 accuracy MAE mIoU pixel accuracy absolute error relative error hypervolume L2RE Dice HD95

Baselines: Linear Scalarization Uncertainty Weighting Dynamic Weight Average IMTL CAGrad PaMaL PaLoRA LRA NTK RAR LAAF GAAF

Builds on: Adam PaLoRA PaMaL CAGrad PINNacle

Closest papers by meaning

• A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.86)
• Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold (NeurIPS 2025 · similarity 0.86)
• Scaling Exponents Across Parameterizations and Optimizers (ICML 2024 · similarity 0.84)
• Adam Exploits $\ell_\infty$-geometry of Loss Landscape via Coordinate-wise Adaptivity (ICLR 2025 · similarity 0.84)
• The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics (NeurIPS 2026 · similarity 0.84)
• On the Implicit Bias of Adam (ICML 2024 · similarity 0.84)