Research map / Optimization / Optimizers
Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add GradDrop gradient sign dropout layer | 9-layer CNN baseline · CelebA | error rate (%): 8.71 -> 8.52 | Helped |
| add GradDrop to mixed-batch transfer learning | DenseNet-100-BC mixed batch · ImageNet to CIFAR-100 | top-1 error (%): 29.8 -> 29.1 | Helped |
| add GradDrop on top of GradNorm | MB + GradNorm · ImageNet to CIFAR-100 | top-1 error (%): 29.4 -> 28.9 | Helped |
| add GradDrop | PointPillar · Waymo 3D detection | 3D APH (%): 53 -> 56.0 | Helped |
| add GradDrop on top of GradNorm | GradNorm · Waymo 3D detection | 3D AP (%): 51.0 -> 55.1 | Helped |
| add GradDrop on top of MGDA | MGDA · Waymo detection | 2D AP (%) / 3D AP (%): 2D AP 76.8 -> 73.7; 3D AP 20.0 -> 32.3 | Mixed |
| use sign-purity-based probability instead of random sign (Random GradDrop, k=0) (ablation) | GradDrop · CelebA | error rate (%): 8.60 -> 8.52 | Helped |
| use sign-purity-based probability instead of random sign (Random GradDrop, k=0) (ablation) | GradDrop · Waymo 3D detection | 3D APH (%): 50.5 -> 56.0 | Helped |
| use sign-purity-based probability instead of random sign (Random GradDrop, k=0) (ablation) | MB + GradNorm + GradDrop · ImageNet to CIFAR-100 | test loss: 1.04 -> 1.01 | Helped |
| set leak to 1.0 for source set (instead of transfer set) (ablation) | GradDrop · ImageNet to CIFAR-100 | top-1 error (%): 30.5 -> 28.9 | Helped |
| sum GradDrop signal over batch dimension (ablation) | GradDrop · CelebA | performance | Helped |
| replace PCGrad/MGDA with GradDrop for compute | multitask gradient methods · CelebA | speed vs baseline: 0.20 (PCGrad) -> 0.45 (GradDrop) | Helped |
Closest papers by meaning
- GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks (ICML 2018 · similarity 0.89)
- Gradient-Sign Masking for Task Vector Transport Across Pre-Trained Models (ICLR 2026 · similarity 0.88)
- DiSparse: Disentangled Sparsification for Multitask Model Compression (CVPR 2022 · similarity 0.86)
- On the Optimization and Generalization of Two-layer Transformers with Sign Gradient Descent (ICLR 2025 · similarity 0.85)
- DeepOBS: A Deep Learning Optimizer Benchmark Suite (ICLR 2019 · similarity 0.85)
- signSGD: Compressed Optimisation for Non-Convex Problems (ICML 2018 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.