Research map / Vision-language and multimodal / Multimodal LLMs
STOLA: Self-Adaptive Touch-Language Framework for Tactile Commonsense Reasoning in Open-Ended Scenarios
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SToLa, a Mixture-of-Experts touch-language model built on Vicuna-7B with a two-stage training strategy, and introduces TactileBench, an open-ended tactile commonsense reasoning benchmark.
Problem
Touch-language models treat touch as a sub-modality of language and are trained/evaluated on narrow, templated tactile QA data, limiting open-ended tactile commonsense reasoning.
Approach
Insert MoE layers (router plus FFN experts replicated from stage I) into the LLM blocks with a shared self-attention layer so tactile and text tokens are routed to token-level experts, combined with a two-stage training pipeline (tactile token adaptation, then LoRA + MoE instruction tuning) and a free-form tactile QA benchmark.
Key results
- On PhysiCLeAR, SToLa (7B) reaches CIDEr 195.03 vs 141.20 for Octopi-13B, the best baseline.
- On TactileBench METEOR, SToLa scores 30.27 vs 28.83 for Octopi-13B.
- Removing MoE drops PhysiCLeAR CIDEr from 195.03 to 176.79.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add MoE layers in LLM blocks (replacing FFN) (ablation) | SToLa LLM · PhysiCLeAR and TactileBench | CIDEr, METEOR, GPT-4, DeepSeek-R1: CIDEr 176.79 to 195.03; TactileBench METEOR 28.71 to 30.27 | Helped |
| add LoRA fine-tuning of self-attention layers (ablation) | SToLa · PhysiCLeAR | CIDEr: 166.71 to 195.03 | Helped |
| add LoRA fine-tuning of self-attention layers (ablation) | SToLa · TactileBench | METEOR: 29.32 to 30.27 | Helped |
| add Stage I tactile token adaptation (ablation) | SToLa · PhysiCLeAR and TactileBench | CIDEr, METEOR: CIDEr 172.52 to 195.03; TactileBench METEOR 29.27 to 30.27 | Helped |
| SToLa (7B) vs Octopi-13B | SToLa · PhysiCLeAR overall | CIDEr: 195.03 vs 141.20 | Helped |
| SToLa (7B) vs Octopi-13B on property comparison subtask | SToLa · PhysiCLeAR subtasks | accuracy (Property Comparison): 62.28 vs 55.06 | Helped |
| SToLa (7B) vs Octopi-13B on Combined object property description | SToLa · PhysiCLeAR subtasks | accuracy (OPD combined): 48.72 vs 55.26 | Hurt |
| SToLa (7B) vs Octopi-13B on commonsense-driven reasoning | SToLa · TactileBench CDR | GPT-4 score: 7.61 vs 8.00 | Hurt |
Surprising findings
- SToLa-7B outperforms the 13B Octopi on overall metrics despite having fewer parameters.
Limitations
- Could not scale to a 13B LLM due to compute constraints, leaving it weaker than Octopi-13B on some subtasks.
- MoE experts are assigned by modality only, not by task.
- Reproduced baseline numbers differ from originally reported values due to unknown random seed in split.
Experimental setup
Datasets: Touch100k PhysiCLeAR Touch and Go
Benchmarks: PhysiCLeAR TactileBench
Metrics: CIDEr BLEU-4 METEOR accuracy GPT-4 score DeepSeek-R1 score
Baselines: Touch-LLM Octopi-7B Octopi-13B
Builds on: Octopi TLV-Link ViFi-CLIP LoRA
Closest papers by meaning
- Touch-R1: Reinforcing Touch Reasoning in MLLMs (NeurIPS 2026 · similarity 0.92)
- A Touch, Vision, and Language Dataset for Multimodal Alignment (ICML 2024 · similarity 0.87)
- Universal Visuo-Tactile Video Understanding for Embodied Interaction (NeurIPS 2025 · similarity 0.87)
- AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors (ICLR 2025 · similarity 0.86)
- Binding Touch to Everything: Learning Unified Multimodal Tactile Representations (CVPR 2024 · similarity 0.86)
- TextToucher: Fine-Grained Text-to-Touch Generation (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.