Research map / Label-efficient and robust learning / Domain adaptation and generalization
TRAM: Bridging Trust Regions and Sharpness Aware Minimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TRAM, a sharpness-aware minimization variant whose ascent neighborhood is set by a trust-region divergence, improving out-of-distribution fine-tuning on vision, language modeling and cross-lingual NLI.
Problem
Sharpness-aware minimization targets parameter-space flatness but ignores representation curvature, which matters for transferring pre-trained structure during fine-tuning and out-of-distribution generalization.
Approach
TRAM replaces the fixed SAM/ASAM radius rho with a trust-region divergence (parameter-based or noise-based KL), so the adversarial parameter perturbation is bounded by how much the representation function changes, combining parameter sharpness and function curvature awareness.
Key results
- On XNLI, TRAM-x reaches 75.2 zero-shot average accuracy versus 74.0 for ASAM and 73.7 for SAM, with all TRAM variants significantly outperforming other methods (p<0.01).
- On M2D2 Wikipedia, TRAM-theta_{t-1} achieves 22.5 zero-shot average perplexity versus 23.2 for FSAM and 26.1 for SAM.
- On CIFAR-100 image adaptation, TRAM-x reaches 88.78% versus 87.99% for SAM; TRAM-theta_{t-1} reaches 97.07% on Flowers versus 95.05% for SAM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the fixed SAM neighborhood with a trust-region-aware neighborhood | ViT-base fine-tuning · Oxford Flowers | Top-1 accuracy: 95.26±0.03 -> 97.07±0.10 | Helped |
| Replace the fixed SAM neighborhood with a trust-region-aware neighborhood | GPT-2 Base fine-tuning · M2D2 Wikipedia zero-shot average | perplexity: 23.2 -> 22.5 | Helped |
| Replace the fixed SAM neighborhood with a trust-region-aware neighborhood | GPT-2 Base fine-tuning · M2D2 S2ORC zero-shot average | perplexity: 26.8 -> 26.2 | Helped |
| Replace the fixed SAM neighborhood with a trust-region-aware neighborhood | XLM-Roberta Base fine-tuning · XNLI zero-shot average | accuracy: 74.5 -> 75.2 | Helped |
| Estimate trust region using noisy inputs instead of the previous-step model (ablation) | TRAM · M2D2 Wikipedia zero-shot average | perplexity: 22.5 -> 23.2 | Hurt |
| Estimate trust region using noisy inputs instead of the previous-step model (ablation) | TRAM · M2D2 S2ORC zero-shot average | perplexity: 26.2 -> 26.6 | Hurt |
| Measure trust region against the pretrained model instead of the previous-step model (ablation) | TRAM · M2D2 Wikipedia zero-shot average | perplexity: 22.5 -> 23.2 | Hurt |
| Measure trust region against the pretrained model instead of the previous-step model (ablation) | TRAM · M2D2 S2ORC zero-shot average | perplexity: 26.2 -> 26.6 | Hurt |
| Use Fisher geometry for the trust-region neighborhood instead of previous-step divergence (ablation) | TRAM · M2D2 Wikipedia zero-shot average | perplexity: 22.5 -> 23.9 | Hurt |
| Use Fisher geometry for the trust-region neighborhood instead of previous-step divergence (ablation) | TRAM · M2D2 S2ORC zero-shot average | perplexity: 26.2 -> 26.9 | Hurt |
| Estimate trust region using noisy inputs instead of the previous-step model (ablation) | TRAM · XNLI zero-shot average | accuracy: 75.0 -> 75.2 | Mixed |
| Measure trust region against the pretrained model instead of the previous-step model (ablation) | TRAM · XNLI zero-shot average | accuracy: 75.0 -> 74.9 | Hurt |
Surprising findings
- Trust region methods (TRPO, R3F) perform no better than SAM-style methods on representation similarity to pre-trained models.
- TRAM-theta_{t-1} gives the worst perplexity on distant S2ORC domains (e.g., Art, Philosophy) despite best in-domain performance, showing a trade-off between easy and hard generalization.
Limitations
- Trust-region variants add one extra forward pass, increasing cost over SAM-style training.
- Gains are small on some tasks (e.g., CIFAR-100 and Cars) and TRAM does not fix degradation on distant domains.
- Sharpness-generalization correlation remains questioned, so the mechanism is not fully established.
Experimental setup
Datasets: ImageNet CIFAR-100 Stanford Cars Oxford Flowers M2D2 MultiNLI XNLI
Benchmarks: CIFAR-100 Stanford Cars Oxford Flowers M2D2 XNLI
Metrics: top-1 accuracy perplexity accuracy epsilon-sharpness CKA similarity
Baselines: Adam SGD SAM ASAM FSAM TRPO R3F MESA
Builds on: Adaptive SAM TRPO R3F Fisher SAM MESA
Closest papers by meaning
- Seeking Consistent Flat Minima for Better Domain Generalization via Refining Loss Landscapes (CVPR 2025 · similarity 0.88)
- Avoiding spurious sharpness minimization broadens applicability of SAM (ICML 2025 · similarity 0.87)
- Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late In Training (ICLR 2025 · similarity 0.86)
- Flatten Long-Range Loss Landscapes for Cross-Domain Few-Shot Learning (CVPR 2024 · similarity 0.86)
- Friendly Sharpness-Aware Minimization (CVPR 2024 · similarity 0.85)
- Unknown Domain Inconsistency Minimization for Domain Generalization (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.