Research map / Large language models / Language, safety and interpretability
KD-MSLRT: Lightweight Sign Language Recognition Model Based on Mediapipe and 3D to 1D Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KD-MSLRT, a lightweight MediaPipe-landmark sign language recognizer distilled from the video-based CorrNet teacher (3D-to-1D distillation) with a self-supervised Transformer text-correction stage, and releases a Chinese long-sentence sign language dataset of 8,976 samples.
Problem
Video-based continuous sign language recognition and translation models are computationally expensive and hard to deploy on resource-constrained devices, and Chinese Sign Language lacks public large-scale datasets.
Approach
Distill knowledge from a video-based CorrNet teacher (3D) into a 1D CNN+BiLSTM student operating on 72 MediaPipe landmarks, combining convolutional-layer, BiLSTM and self-distillation losses with CTC, and add a dual-Transformer text correction network trained with synthetic noise on gloss outputs.
Key results
- KD-MSLRT reaches 17.4% / 17.8% WER (Dev/Test) on PHOENIX14 and 16.9% / 18.4% on PHOENIX14T, compared with CorrNet's 18.8/19.4 and 18.9/20.5.
- The TFLite quantized model is 12.93 MB, with mean CPU latency of 42.64 ms, but FLOPs of 1.9e9 versus CorrNet's 5e11 (as reported in the paper's table).
- Ablation shows combined KD (conv + BiLSTM + self) reduces base MSLR Dev WER from 43.8% to 28.7%; INT8 quantization changes WER by under 0.5%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add knowledge distillation to MSLR | MSLR · PHOENIX14 Dev WER | WER: 39.6 -> 28.7 | Helped |
| Add text correction to KD-MSLR (ablation) | KD-MSLR · PHOENIX14 Dev WER | WER: 28.7 -> 17.4 | Helped |
| Add data augmentation to the KD + TC model (ablation) | KD-MSLRT · PHOENIX14 Dev WER | WER: 18.2 -> 17.4 | Helped |
| Use text correction without knowledge distillation (ablation) | MSLR · PHOENIX14 Dev WER | WER: 43.8 -> 31.4 | Helped |
| Add a second Transformer (ablation) | KD-MSLRT text correction network · PHOENIX14 Dev WER | WER: 22.1 -> 18.3 | Helped |
| Preprocess the raw input text (ablation) | KD-MSLRT text correction network · PHOENIX14 Dev WER | WER: 18.3 -> 17.4 | Helped |
| Set 1D CNN kernel size to K=5 rather than K=3 (ablation) | MSLR · PHOENIX14 Dev WER | WER: 45.7 -> 43.8 | Helped |
| Increase kernel size from K=5 to K=7 (ablation) | MSLR · PHOENIX14 Dev WER | WER: 43.8 -> 47.9 | Hurt |
| Use convolutional-layer knowledge distillation (ablation) | MSLR · PHOENIX14 Dev WER | WER: 43.8 -> 34.6 | Helped |
| Use BiLSTM knowledge distillation (ablation) | MSLR · PHOENIX14 Dev WER | WER: 43.8 -> 30.2 | Helped |
| Use self-distillation only (ablation) | MSLR · PHOENIX14 Dev WER | WER: 43.8 -> 39.5 | Helped |
| Combine convolutional, BiLSTM, and self-distillation (ablation) | MSLR · PHOENIX14 Dev WER | WER: 43.8 -> 28.7 | Helped |
Surprising findings
- Text correction alone performs worse than KD combined with text correction, which the authors attribute to correction favoring the dominant (possibly incorrect) meaning when WER exceeds 40%.
- Self-distillation alone gives only a small improvement, while BiLSTM-level distillation from the 3D teacher gives the largest single gain.
Limitations
- Landmark-only input (72 keypoints) loses spatial and facial information, leaving a performance gap to video-based models.
- Text correction can amplify errors when the recognized sentence has high WER.
- Deployment evaluated only via simulated Intel CPU tests rather than on real mobile or edge hardware.
Experimental setup
Datasets: PHOENIX14 PHOENIX14T
Benchmarks: PHOENIX14 PHOENIX14T
Metrics: WER FLOPs mean CPU latency (ms) model size (MB) FPS
Baselines: CorrNet VAC SMKD TLP SEN STMC C2SLR DNF
Builds on: CorrNet Connectionist Temporal Classification MediaPipe Holistic C2SLR
Closest papers by meaning
- Scaling Sign Language Translation (NeurIPS 2024 · similarity 0.91)
- Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation (ICLR 2024 · similarity 0.90)
- Uni-Sign: Toward Unified Sign Language Understanding at Scale (ICLR 2025 · similarity 0.90)
- Conditional Variational Autoencoder for Sign Language Translation with Cross-Modal Alignment (AAAI 2024 · similarity 0.89)
- Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues (CVPR 2025 · similarity 0.89)
- LLMs are Good Sign Language Translators (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.