Research map / Large language models / AI and society
LERMO: A Novel Web Game for AI-Enhanced Sign Language Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LERMO, a browser-based Wordle-style LIBRAS fingerspelling game that recognizes hand landmarks from a webcam with a compact MLP, with a new landmark dataset and a user survey.
Problem
Sign language learning tools that use AI for fingerspelling practice are underdeveloped, and existing SLR systems often need specialized hardware or heavy compute unsuitable for in-browser gameplay.
Approach
Combines a Wordle-inspired word-guessing game with keypoint-based LIBRAS fingerspelling recognition: MediaPipe hand landmarks (x,y only) feed a small MLP exported to ONNX and run client-side in the browser, with predictions aggregated by mode over 100 frames for stability.
Key results
- MLP trained on the merged dataset reached 0.8988 accuracy (normal) and 0.9017 (augmented) on held-out Actor 5, outperforming SVM, KNN, Random Forest and Naive Bayes.
- A survey of 41 respondents found LERMO user-friendly and innovative; 95% (reported as 94.9% in conclusions) believed it could enhance fingerspelling skills.
- Only 10.8% of users reported slow sign recognition speed; the MLP file is 100KB with 27,326 parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the MLP classifier instead of the strongest baseline, KNN | LERMO sign classifier, trained on Merged normal data · Actor 5 test | accuracy: 0.8698 -> 0.8988 | Helped |
| Use the MLP classifier instead of the strongest baseline, KNN | LERMO sign classifier, trained on Merged augmented data · Actor 5 test | accuracy: 0.8701 -> 0.9017 | Helped |
| Train the MLP on augmented rather than normal Merged data | LERMO sign classifier · Actor 5 test, Merged training data | accuracy: 0.8988 -> 0.9017 | Helped |
| Train the MLP on augmented rather than normal Merged data | LERMO sign classifier · Actor 5 test, Merged training data | F1 Score: 0.8787 -> 0.8846 | Helped |
| Train the MLP on augmented rather than normal Merged data | LERMO sign classifier · Actor 5 test, Merged training data | Precision: 0.8965 -> 0.9125 | Helped |
| Train SVM on augmented rather than normal Merged data | LERMO sign classifier · Actor 5 test, Merged training data | accuracy: 0.8643 -> 0.8566 | Hurt |
| Train Random Forest on augmented rather than normal Merged data | LERMO sign classifier · Actor 5 test, Merged training data | accuracy: 0.8650 -> 0.8445 | Hurt |
| Train Naive Bayes on augmented rather than normal Merged data | LERMO sign classifier · Actor 5 test, Merged training data | accuracy: 0.7143 -> 0.6694 | Hurt |
Surprising findings
- Naive Bayes and Random Forest performed notably worse than the MLP, and augmentation did not consistently improve accuracy across classifiers.
Limitations
- Not usable on mobile devices due to UI and hardware constraints.
- Frequent misclassification among similar letters (R, U, E, S, T, F) due to landmark noise and depth ambiguity.
- Only letter-level fingerspelling recognition; word-level recognition is future work.
Experimental setup
Datasets: LERMO landmark fingerspelling dataset (LIBRAS, 5 actors)
Benchmarks: LERMO landmark dataset (Actor 5 held-out test)
Metrics: accuracy F1 score precision user survey Likert/questionnaire responses
Baselines: SVM KNN Random Forest Naive Bayes
Builds on: MediaPipe Wordle Adam
Closest papers by meaning
- FSboard: Over 3 Million Characters of ASL Fingerspelling Collected via Smartphones (CVPR 2025 · similarity 0.85)
- Uni-Sign: Toward Unified Sign Language Understanding at Scale (ICLR 2025 · similarity 0.84)
- KD-MSLRT: Lightweight Sign Language Recognition Model Based on Mediapipe and 3D to 1D Knowledge Distillation (AAAI 2025 · similarity 0.84)
- Neural Sign Actors: A Diffusion Model for 3D Sign Language Production from Text (CVPR 2024 · similarity 0.83)
- A Wireframe-Based Approach for Classifying and Acquiring Proficiency in the American Sign Language (Student Abstract) (AAAI 2024 · similarity 0.82)
- YouTube-SL-25: A Large-Scale, Open-Domain Multilingual Sign Language Parallel Corpus (ICLR 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.