Research map / Label-efficient and robust learning / Domain adaptation and generalization
FisherTune: Fisher-Guided Robust Tuning of Vision Foundation Models for Domain Generalized Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FisherTune, a selective fine-tuning method for vision foundation models in domain generalized semantic segmentation that picks domain-sensitive parameters via a variational Domain-Related Fisher Information Matrix (DR-FIM).
Problem
Fine-tuning vision foundation models for domain generalized semantic segmentation while preserving their cross-domain generalization; existing adapter-based and selective fine-tuning methods underuse VFM potential.
Approach
Extends the Fisher information matrix with a cross-domain term (Fisher change between seen and simulated unseen domain features) to score domain-sensitive parameters, and stabilizes its estimate by treating parameters as Gaussian variables with a pretrained prior under variational inference, then fine-tunes only the top-ranked parameters on a dynamic schedule.
Key results
- On GTAV→Cityscapes+BDD100K+Mapillary, FisherTune with DINOv2-Large reaches 66.5 average mIoU, versus 64.3 for Rein.
- With EVA02-Large, FisherTune averages 64.4 mIoU on GTAV→C+B+M, surpassing Rein's 63.4 and VQT's 59.5.
- On Cityscapes→ACDC with EVA02, DR-FIM selection gives 72.9 mIoU versus 69.5 for the Fisher-only variant.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| select 5% of parameters using DR-FIM rather than random selection (ablation) | EVA02 Large · Cityscapes → BDD100K | mIoU: 61.1 -> 65.8 | Helped |
| select 5% of parameters using DR-FIM rather than Fθ (ablation) | EVA02 Large · Cityscapes → BDD100K | mIoU: 63.8 -> 65.8 | Helped |
| select 5% of parameters using DR-FIM rather than ΔFθ (ablation) | EVA02 Large · Cityscapes → ACDC | mIoU: 71.3 -> 72.9 | Helped |
| select 5% of parameters using DR-FIM rather than Fθ (ablation) | DINOv2 Large · Cityscapes → ACDC | mIoU: 71.4 -> 77.5 | Helped |
| use stable variational estimation for DR-FIM (ablation) | FisherTune · DGSS generalization | mIoU: +2.6% mIoU | Helped |
| add feature perturbation (FP) (ablation) | FisherTune with EVA02 · GTAV → Avg. | mIoU: +0.1 | Helped |
| add feature perturbation (FP) (ablation) | FisherTune with DINOv2 · GTAV → Avg. | mIoU: +0.2 | Helped |
| use FisherTune rather than Rein | DINOv2 Large · Cityscapes → BDD100K | mIoU: 66.4 -> 67.7 | Helped |
| use FisherTune rather than Rein | EVA02 Large · Cityscapes → ACDC | mIoU: 70.9 -> 72.9 | Helped |
| use FisherTune rather than Rein | DINOv2 Large · Cityscapes → ACDC | mIoU: 77.6 -> 77.5 | Hurt |
Limitations
- Parameter selection relies on hand-set hyperparameters (gamma, tau, delta_min, delta_max) and a warm-up stage.
- Reported gains are relative to mostly adapter and selective baselines; comparisons with stronger VFM-based DG methods are limited to a few entries.
- Ablations are shown mainly on a subset of settings (e.g., 5% fine-tunable parameters).
Experimental setup
Datasets: GTAV Cityscapes BDD100K Mapillary ACDC
Benchmarks: Cityscapes BDD100K Mapillary ACDC Foggy Zurich Foggy Driving Dark Zurich Nighttime Driving
Metrics: mIoU
Baselines: Full fine-tuning Freeze LoRA VPT Rein VQT ChildTune AdaptFormer VLTSeg SDT CLOUDS
Builds on: Fishermask ChildTune Rein VQT
Closest papers by meaning
- Stronger Fewer & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation (CVPR 2024 · similarity 0.91)
- Sparse All-Layer Connector for Domain Generalised Semantic Segmentation (NeurIPS 2026 · similarity 0.89)
- A Simple Recipe for Language-guided Domain Generalized Segmentation (CVPR 2024 · similarity 0.88)
- Collaborating Foundation Models for Domain Generalized Semantic Segmentation (CVPR 2024 · similarity 0.88)
- PracticalDG: Perturbation Distillation on Vision-Language Models for Hybrid Domain Generalization (CVPR 2024 · similarity 0.88)
- Adapter Naturally Serves as Decoupler for Cross-Domain Few-Shot Semantic Segmentation (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.