Research map / Label-efficient and robust learning / Domain adaptation and generalization
Learning CNN on ViT: A Hybrid Model to Explicitly Class-specific Boundaries for Domain Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ECB, a hybrid ViT-CNN domain adaptation method where ViT finds class-specific boundaries via discrepancy maximization, the CNN clusters target features, and the two branches co-train with pseudo labels.
Problem
Domain adaptation methods typically use either CNN or ViT encoders alone, ignoring their complementary strengths (local vs global representations) and suffering from source-biased decision boundaries under UDA and SSDA settings.
Approach
Pairs a ViT encoder with a CNN encoder sharing two classifiers; ViT is used to find worst-case class-specific boundaries by maximizing classifier discrepancy, the CNN is trained to minimize discrepancy on target data, and both branches exchange pseudo labels via co-training (Finding-to-Conquering strategy).
Key results
- On Office-Home UDA (ResNet-50), ECB (CNN) reaches 81.2% mean accuracy, +5.4% over EIDCo (75.8%).
- On DomainNet SSDA 3-shot (ResNet-34), ECB (CNN) reaches 85.5% mean accuracy, versus 78.4% for G-ABC.
- Co-training ablation on DomainNet 3-shot: ECB branches reach 85.7% (ViT) and 85.5% (CNN), above one-direction teaching variants.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply ECB rather than EIDCo | Office-Home UDA · Office-Home UDA mean | accuracy: 75.8 -> 81.2 | Helped |
| apply ECB rather than G-ABC | DomainNet SSDA · DomainNet SSDA mean, 1-shot | accuracy: 77.2 -> 83.8 | Helped |
| apply ECB rather than G-ABC | DomainNet SSDA · DomainNet SSDA mean, 3-shot | accuracy: 78.4 -> 85.5 | Helped |
| add CNN-to-ViT pseudo-label teaching to ViT-to-CNN teaching (ablation) | ECB on DomainNet SSDA · DomainNet SSDA mean, 3-shot | accuracy: 79.4 -> 85.5 | Helped |
| add ViT-to-CNN pseudo-label teaching to CNN-to-ViT teaching (ablation) | ECB on DomainNet SSDA · DomainNet SSDA mean, 3-shot | accuracy: 64.1 -> 85.7 | Helped |
| replace one-direction teaching with bidirectional co-training (ablation) | ECB on DomainNet SSDA · DomainNet SSDA mean, 3-shot | accuracy: 76.1 -> 85.7 | Helped |
| replace one-direction teaching with bidirectional co-training (ablation) | ECB on DomainNet SSDA · DomainNet SSDA mean, 3-shot | accuracy: 79.4 -> 85.5 | Helped |
| use CNN-to-ViT teaching rather than bidirectional co-training (ablation) | ECB on DomainNet SSDA · DomainNet SSDA mean, 3-shot | accuracy: 85.7 -> 75.0 | Hurt |
| use CNN-to-ViT teaching rather than bidirectional co-training (ablation) | ECB on DomainNet SSDA · DomainNet SSDA mean, 3-shot | accuracy: 85.5 -> 64.1 | Hurt |
| add the Finding-to-Conquering strategy after supervised training (ablation) | ECB pseudo-label generation on DomainNet rel→clp, 3-shot · DomainNet rel→clp, 3-shot | total pseudo labels: around 8,000 -> 14,000 | Helped |
| add co-training after the Finding-to-Conquering strategy (ablation) | ECB pseudo-label generation on DomainNet rel→clp, 3-shot · DomainNet rel→clp, 3-shot | correct pseudo labels: 10,000 -> 16,000 | Helped |
| use the hybrid CNN and ViT architecture rather than CNN+CNN (ablation) | ECB on DomainNet SSDA · DomainNet SSDA mean, 3-shot | accuracy: around 77.0% -> about 85.5% | Helped |
Limitations
- Fixed confidence thresholds (tau_vit=0.6, tau_cnn=0.9) are time-consuming to tune.
- Requires training two backbones (ViT and CNN) jointly, increasing training cost.
- Evaluation only on Office-Home and DomainNet image benchmarks.
Experimental setup
Datasets: Office-Home DomainNet
Benchmarks: Office-Home DomainNet
Metrics: accuracy
Baselines: DANN MCD BNM MDD MCC GVB DCAN DALN FixBi ATDOC EIDCo ENT
Builds on: MCD FixBi DECOTA MVCL RandAugment
Closest papers by meaning
- Boosting Domain Adaptation by Discovering Latent Domains (CVPR 2018 · similarity 0.86)
- Vulcan: Crafting Compact Class-Specific Vision Transformers For Edge Intelligence (ICLR 2026 · similarity 0.86)
- DeiT-LT: Distillation Strikes Back for Vision Transformer Training on Long-Tailed Datasets (CVPR 2024 · similarity 0.84)
- Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation (NeurIPS 2024 · similarity 0.84)
- Channel Vision Transformers: An Image Is Worth 1 x 16 x 16 Words (ICLR 2024 · similarity 0.84)
- DA-VPT: Semantic-Guided Visual Prompt Tuning for Vision Transformers (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.