Research map / Large language models / Language, safety and interpretability
VSNet: Focusing on the Linguistic Characteristics of Sign Language
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VSNet, a skeleton-based isolated sign language recognition model that dynamically drops weak hand joints (SPD) and uses visual-symbol-grouped attention (VSformer) with a multi-grouping ensemble, reaching SOTA on WLASL, MSASL, NMFs-CSL and Slovo without pretraining.
Problem
Skeleton-based isolated sign language recognition models treat sign language as generic human motion, overlooking the linguistic structure of signs (joint relations and decomposition into visual symbols).
Approach
A Self-Pacing Dropping block that uses GCN-learned joint connection strengths to progressively discard the weakest joints, plus VSformer that groups joints into generalized visual symbols and applies spatial pairing and temporal window attention over them, ensembled across multiple groupings.
Key results
- On MSASL1000, the single-crop model reaches 70.71% top-1 (P-I), versus 65.22% for SSRL, a prior pre-trained skeleton method.
- On WLASL2000, the 4-crop ensemble reaches 55.98% top-1 versus 53.68% for DSTA, with no complex pretraining.
- On Slovo, the 4-crop model reaches 77.83% top-1 versus 64.09% for the MViTv2-small-32 baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use 4-crop inference | VSNet · MSASL1000 P-I | Top-1 accuracy: 70.71 -> 73.18 | Helped |
| use 4-crop inference | VSNet · WLASL2000 P-I | Top-1 accuracy: 53.54 -> 55.98 | Helped |
| use 4-crop inference | VSNet · NMFs-CSL | Top-1 accuracy: 75.3 -> 76.7 | Helped |
| use 4-crop inference | VSNet · Slovo | Top-1 accuracy: 76.53 -> 77.83 | Helped |
| use type-1 skeleton grouping alone instead of bone-vector data (ablation) | VSNet · WLASL300 | Top-1 accuracy: 74.70 -> 79.04 | Helped |
| combine the four grouping outputs (ablation) | VSNet · WLASL2000 | Top-1 accuracy: 53.54 -> 55.98 | Helped |
| replace manually assigned ensemble weights with accuracy-based weights (ablation) | VSNet multi-grouping ensemble · WLASL2000 | Top-1 accuracy: 55.98 -> 56.01 | No clear effect |
| replace SPD with three-layer Conv-2D (ablation) | VSNet · WLASL2000 | Top-1 accuracy: 46.32 -> 53.54 | Hurt |
| add weak-joint dropping to GCN (ablation) | VSNet skeleton block · WLASL2000 | Top-1 accuracy: 51.18 -> 53.54 | Helped |
| add TWA alongside SPA (ablation) | VSNet VSformer · WLASL300 | Top-1 accuracy: 77.69 -> 79.04 | Helped |
| add SPA alongside TWA (ablation) | VSNet VSformer · WLASL300 | Top-1 accuracy: 77.99 -> 79.04 | Helped |
Surprising findings
- Ensembling the four skeleton groupings with bone data gives about a 2% top-1 gain, and both weighting schemes tested (sum vs. accuracy-based) perform nearly identically.
- On MSASL100, SSRL slightly outperforms the single-crop VSNet, which the authors attribute to SPD needing more data to learn droppable nodes.
Limitations
- Slovo provides only 42 hand landmarks without arm data, which limits performance there.
- Visual symbol grouping is a fixed, simplified manual decomposition rather than learned; the authors note interpretability is weakened.
- Excludes facial expressions, which are important in sign language.
Experimental setup
Datasets: WLASL MSASL NMFs-CSL Slovo
Benchmarks: WLASL2000 WLASL300 WLASL100 MSASL1000 MSASL200 MSASL100 NMFs-CSL Slovo
Metrics: top-1 accuracy top-5 accuracy per-instance precision per-class precision
Baselines: SignBERT SignBERT+ MASA SSRL BEST ST-GCN DSTA SL-GCN NLA-SLR HMA BSL TCK
Builds on: decoupled graph convolution (DC-GCN / SL-GCN) SkateFormer SAM-SLR HRNet
Closest papers by meaning
- Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment (NeurIPS 2026 · similarity 0.88)
- SignGraph: A Sign Sequence is Worth Graphs of Nodes (CVPR 2024 · similarity 0.88)
- Sign-IDD: Iconicity Disentangled Diffusion for Sign Language Production (AAAI 2025 · similarity 0.87)
- TCNet: Continuous Sign Language Recognition from Trajectories and Correlated Regions (AAAI 2024 · similarity 0.87)
- Geo-Sign: Hyperbolic Contrastive Regularisation for Geometrically Aware Sign Language Translation (NeurIPS 2025 · similarity 0.87)
- Uni-Sign: Toward Unified Sign Language Understanding at Scale (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.