Research map / Label-efficient and robust learning / Knowledge distillation
Reinforced Cross-Domain Knowledge Distillation on Time Series Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RCD-KD, a cross-domain knowledge distillation framework for time series that uses a dueling DDQN to select target samples based on a student-capacity reward (uncertainty consistency and transferability) plus a DANN-style domain discriminator.
Problem
Unsupervised domain adaptation for time series relies on large models that cannot be deployed on resource-limited edge devices; existing KD-based domain adaptation aligns student and teacher outputs over all source and target samples, which ignores the student's capacity gap and causes negative tran
Approach
Formulates target sample selection as an MDP solved by a dueling DDQN whose reward reflects whether the compact student can match the teacher's MCD-estimated uncertainty and KL-based transferability, so only suitable target samples are used for distillation alongside an adversarial domain-invariant loss.
Key results
- On HAR, RCD-KD averages 94.68 macro F1 vs 83.62 for the best UDA baseline shown (HoMM 83.62 in Table 1 column) and 79.32 for UNI-KD in the KD-based comparison (Table 2).
- On FD, RCD-KD reaches 92.63 average macro F1 vs 90.06 for UNI-KD and a teacher score of 92.68.
- On SSC, RCD-KD reaches 68.26 average macro F1 vs 62.17 for UNI-KD, with the student-only baseline at 50.39.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RCD-KD instead of the strongest KD-based baseline, MLD-DA | compact 1D-CNN student · HAR | macro F1-score: 87.79 -> 94.68 | Helped |
| Use RCD-KD instead of the strongest KD-based baseline, MLD-DA | compact 1D-CNN student · HHAR | macro F1-score: 77.51 -> 82.37 | Helped |
| Use RCD-KD instead of the strongest KD-based baseline, UNI-KD | compact 1D-CNN student · FD | macro F1-score: 90.06 -> 92.63 | Helped |
| Use RCD-KD instead of the strongest KD-based baseline, UNI-KD | compact 1D-CNN student · SSC | macro F1-score: 62.17 -> 68.26 | Helped |
| Add knowledge distillation to domain-confusion training (ablation) | compact 1D-CNN student · HAR | macro F1-score: 85.51 -> 94.68 | Helped |
| Add knowledge distillation to domain-confusion training (ablation) | compact 1D-CNN student · HHAR | macro F1-score: 79.03 -> 82.37 | Helped |
| Add knowledge distillation to domain-confusion training (ablation) | compact 1D-CNN student · FD | macro F1-score: 77.95 -> 92.63 | Helped |
| Add knowledge distillation to domain-confusion training (ablation) | compact 1D-CNN student · SSC | macro F1-score: 62.39 -> 68.26 | Helped |
| Replace RL-selected target distillation with standard KD over target samples (ablation) | student trained with domain confusion · HHAR | macro F1-score: 82.37 -> 78.99 | Hurt |
| Replace RL-selected target distillation with standard KD over target samples (ablation) | student trained with domain confusion · SSC | macro F1-score: 68.26 -> 60.65 | Hurt |
| Add both uncertainty consistency and transferability to direct target-sample selection (ablation) | RCD-KD sample selection · HAR | macro F1-score: 89.32 -> 93.96 | Helped |
| Use RL to select samples with combined uncertainty consistency and transferability (ablation) | RCD-KD sample selection · HAR | macro F1-score: 93.96 -> 94.68 | Helped |
Surprising findings
- Coarse KD plus domain confusion (LKD + LDC) hurts performance on some datasets (e.g., HHAR and SSC) compared with LDC alone.
- Using student-side uncertainty alone for active-learning-style sample selection degrades performance relative to using all target samples.
Limitations
- Requires pre-training a cumbersome teacher with advanced UDA methods, adding training time.
- Sample transferability uses only logit distances and ignores feature-space information.
- Training is computationally heavier than other KD-based DA baselines because of the RL module and MCD inference.
Experimental setup
Datasets: HAR HHAR FD SSC PAMAP2
Benchmarks: HAR HHAR FD SSC PAMAP2
Metrics: macro F1-score
Baselines: Student-Only HoMM MDDA SASA DANN CoDATS AdvSKM KD-STDA KA-MCD MLD-DA REDA AAD
Builds on: knowledge distillation DANN dueling DDQN Monte Carlo dropout UNI-KD
Closest papers by meaning
- CauDiTS: Causal Disentangled Domain Adaptation of Multivariate Time Series (ICML 2024 · similarity 0.86)
- AuG-KD: Anchor-Based Mixup Generation for Out-of-Domain Knowledge Distillation (ICLR 2024 · similarity 0.86)
- Class-aware Domain Knowledge Fusion and Fission for Continual Test-Time Adaptation (NeurIPS 2025 · similarity 0.86)
- Enhancing Multivariate Time-Series Domain Adaptation via Contrastive Frequency Graph Discovery and Language-Guided Adversary Alignment (AAAI 2025 · similarity 0.86)
- Mix-CPT: A Domain Adaptation Framework via Decoupling Knowledge Learning and Format Alignment (ICLR 2025 · similarity 0.85)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.