Research map / 3D vision / 3D generation

Tactile MNIST: Benchmarking Active Tactile Perception

NeurIPS 2026 · 3D vision › 3D generation · approach: 2D-to-3D pretrained representations · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Active Perception Gym (ap_gym) and the Tactile MNIST Benchmark Suite (TMBS), Gymnasium-compatible benchmarks for active tactile perception, plus a 13,580-model 3D MNIST dataset and 153,600 real GelSight Mini touches, and a CycleGAN tactile renderer.

Problem

Active tactile perception lacks standardized benchmarks, datasets, and evaluation protocols, making it hard to compare exploration policies and tactile methods fairly.

Approach

Frame active tactile perception as episodic POMDPs in a Gymnasium-compatible framework with simulated GelSight Mini tasks (classification, volume, pose, counting), a real-world touch dataset on 3D-printed digits, and a CycleGAN for sim-to-real tactile rendering, benchmarked with average and final accuracy metrics.

Key results

• Off-policy TAP variants (TAP-SAC, TAP-CrossQ) consistently outperform on-policy baselines HAM and TAP-PPO across evaluated environments.
• Across ap_gym and TMBS tasks, TAP requires millions of environment steps (up to 10M) and does not reach perfect accuracy, so the tasks are not solved.
• TAP-CrossQ was found to be much more robust to unseen environments than TAP-SAC in hyperparameter search.

Surprising findings

• HAM could not be tuned to meaningful performance despite extensive hyperparameter search.
• TAP-PPO was unstable when combined with a vision encoder.

Limitations

• No physics engine, so contact-rich interactions like grasping and in-hand pose estimation are out of scope.
• No online real-world evaluation; evaluation is on a static real dataset.
• CycleGAN rendering mode is only available for the TactileMNIST environment.

Experimental setup

Datasets: MNIST 3D Real Tactile MNIST Dataset Starstruck MNIST CIFAR-10 Tiny ImageNet CircleSquare

Benchmarks: CircleSquare TactileMNIST Starstruck Toolbox

Metrics: average accuracy final accuracy linear error angular error relative volume error mean-squared error

Baselines: TAP-SAC TAP-CrossQ TAP-PPO HAM random baseline (TAP-RND)

Builds on: TAP HAM Taxim CycleGAN Gymnasium

Closest papers by meaning

• AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors (ICLR 2025 · similarity 0.90)
• Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper (NeurIPS 2025 · similarity 0.89)
• Sensor-Invariant Tactile Representation (ICLR 2025 · similarity 0.88)
• Taccel: Scaling Up Vision-based Tactile Robotics via High-performance GPU Simulation (NeurIPS 2025 · similarity 0.88)
• EgoTac: In-the-wild Tactile Prediction from Egocentric Vision (NeurIPS 2026 · similarity 0.88)
• Touch-R1: Reinforcing Touch Reasoning in MLLMs (NeurIPS 2026 · similarity 0.87)