Research map / Architectures and efficiency / Pruning and efficient tuning

CiTrus: Squeezing Extra Performance out of Low-data Bio-signal Transfer Learning

AAAI 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: Parameter-efficient transfer analysis · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CiTrus, a convolution-transformer hybrid with masked auto-encoding and frequency-based pre-training, plus a resampling-based fine-tuning method and a cross-validated evaluation for low-data bio-signal transfer learning from SleepEDF EEG.

Problem

Transfer learning for bio-signals with small labeled downstream datasets: which architectures and pre-training strategies transfer well from large EEG pre-training to other bio-signals, and how to fairly evaluate them given high test-fold variance.

Approach

A CNN encoder (channel-independent via batch concatenation) feeds a PatchTST transformer trained with masked auto-encoding; it adds a frequency (spectrogram) reconstruction pre-training task, multimodal EEG+EOG pre-training, and a resampling-based fine-tuning that aligns sampling rate to the pre-training data instead of interpolating length.

Key results

• CiTrus (s) achieves the best accuracy and ROC/PRC on HAR (80.71 ACC, 93.03 ROC) in the table, and CiTrus (fp) achieves the best ACC on ECG (82.44) at the lowest data regime.
• Frequency pre-training (CiTrus fp) gives the largest average gains on the lowest and highest data regimes, with pre-training improvements up to +12.9% on PPG.
• The proposed resampling-adaptive fine-tuning improves over temporal interpolation in almost all cases, with PPG gains reaching about 64% and overall improvements up to about 60%.

Measured effects reported in the paper

Surprising findings

• Variance across test folds is much larger than across random seeds, up to about 4609% higher variance.
• Pre-training is not necessary for EMG and FD-B; a convolutional model trained from scratch outperforms other models on FD-B.

Limitations

• Frequency pre-training performs poorly on some datasets (e.g., SleepEDF and Gesture high-data regimes, EMG and FD-B fp variants near chance)
• Pre-training not beneficial for EMG and FD-B and Epilepsy datasets
• SleepEDF evaluation averaged only over random seeds due to time constraints, not 10-fold

Experimental setup

Datasets: SleepEDF EMG Gesture Epilepsy FD-B PPG HAR ECG

Benchmarks: EMG ECG PPG HAR FDB Epilepsy Gesture SleepEDF

Metrics: accuracy ROC AUC PR AUC

Baselines: PatchTST bioFAME SimMTM TFC NLPatchTST

Builds on: PatchTST bioFAME SimMTM TFC

Closest papers by meaning

• Pre-Training Graph Contrastive Masked Autoencoders are Strong Distillers for EEG (ICML 2025 · similarity 0.85)
• EEGPT: Pretrained Transformer for Universal and Reliable Representation of EEG Signals (NeurIPS 2024 · similarity 0.84)
• Neuroformer: Multimodal and Multitask Generative Pretraining for Brain Data (ICLR 2024 · similarity 0.84)
• Standing on the Shoulders of Giants: Rethinking EEG Foundation Model Pretraining via Multi-Teacher Distillation (NeurIPS 2026 · similarity 0.83)
• Population Transformer: Learning Population-level Representations of Neural Activity (ICLR 2025 · similarity 0.83)
• Leveraging unlabelled data for generalizable neural population decoding (NeurIPS 2026 · similarity 0.83)