Research map / Large language models / Language, safety and interpretability
MM-WLAuslan: Multi-View Multi-Modal Word-Level Australian Sign Language Recognition Dataset
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MM-WLAuslan, a studio-recorded multi-view, multi-modal (RGB-D) word-level Australian Sign Language dataset with 282K+ videos over 3,215 glosses, and benchmarks existing ISLR models across single-view, multi-view, cross-camera and cross-view settings.
Problem
Auslan lacks a large-scale word-level dataset for isolated sign language recognition, and existing ISLR corpora have small vocabularies, single views, or no depth.
Approach
Record each gloss 22 times with four RGB-D cameras (three Kinect-V2 hemispherically placed plus one RealSense) by 73 signers, with four test subsets simulating backgrounds and temporal disturbances, and benchmark multi-modal and multi-view ISLR settings.
Key results
- Best single-view RGB result: NLA-SLR reaches 81.90 average Top-1 across the four test subsets.
- Multi-view RGB NLA-SLR reaches 89.63 average Top-1, while cross-camera transfer (train RealSense, test Kinect+) drops UMDR to 6.36 average Top-1.
- Cross-view transfer drops sharply, e.g., KVNet-V trained on front view and tested on left+right view gets 14.53 average Top-1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add depth to single-view RGB input for UMDR | UMDR · MM-WLAuslan (AVG.) | Top-1 accuracy: 47.78 -> 84.07 | Helped |
| Add depth to single-view RGB input for S3D | S3D · MM-WLAuslan (AVG.) | Top-1 accuracy: 52.94 -> 71.79 | Helped |
| Add RGB-D input in multi-view setting for UMDR | UMDR · MM-WLAuslan (AVG.) | Top-1 accuracy: 57.15 -> 86.20 | Helped |
| Add RGB-D input instead of RGB-only for NLA-SLR | NLA-SLR · MM-WLAuslan (AVG.) | Top-1 accuracy: 89.63 -> 89.51 | Hurt |
| Train on Kinect and test on RealSense instead of same-camera Kinect | KVNet-V · MM-WLAuslan cross-camera K→RS vs K→K | Top-1 accuracy: 62.82 -> 37.65 | Hurt |
| Train on Kinect and test on RealSense instead of same-camera Kinect | UMDR · MM-WLAuslan cross-camera K→RS vs K→K | Top-1 accuracy: 84.07 -> 48.80 | Hurt |
| Test on unseen side views instead of the trained front view | KVNet-V · MM-WLAuslan cross-view F→L+R vs F→F | Top-1 accuracy: 62.82 -> 14.53 | Hurt |
| Test on unseen side views instead of the trained front view | UMDR · MM-WLAuslan cross-view F→L+R vs F→F | Top-1 accuracy: 84.07 -> 22.82 | Hurt |
Surprising findings
- Adding multi-view RGB-D data lowered NLA-SLR accuracy compared to training on RGB alone, attributed to information redundancy.
- Cross-camera and cross-view generalization collapses for all evaluated models despite high same-setting accuracy.
Limitations
- Underrepresentation of African signers and limited demographic diversity.
- Test variations (ITW, SYN, TED) only approximate real-world scenarios; no multi-person or complex backgrounds.
- Benchmarked models are not designed specifically for sign language, leaving multi-view and multi-modal fusion underexplored.
Experimental setup
Datasets: MM-WLAuslan
Benchmarks: MM-WLAuslan
Metrics: Top-1 accuracy Top-5 accuracy
Baselines: ResNet2+1D TSN I3D S3D SlowFast Timesformer UMDR KVNet-V TGCN SL-GCN SPOTER DSTA-SLR
Builds on: ST-GCN SAM-SLR NLA-SLR KVNet UMDR
Closest papers by meaning
- YouTube-SL-25: A Large-Scale, Open-Domain Multilingual Sign Language Parallel Corpus (ICLR 2025 · similarity 0.87)
- KD-MSLRT: Lightweight Sign Language Recognition Model Based on Mediapipe and 3D to 1D Knowledge Distillation (AAAI 2025 · similarity 0.87)
- Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment (NeurIPS 2026 · similarity 0.86)
- FSboard: Over 3 Million Characters of ASL Fingerspelling Collected via Smartphones (CVPR 2025 · similarity 0.86)
- Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues (CVPR 2025 · similarity 0.85)
- SCOPE: Sign Language Contextual Processing with Embedding from LLMs (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.