Research map / Architectures and efficiency / RNNs and normalization
PixelRNN: In-pixel Recurrent Neural Networks for End-to-end-optimized Perception with Neural Sensors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PixelRNN, a binary in-pixel convolutional recurrent encoder that runs on the SCAMP-5 sensor-processor to cut sensor readout bandwidth by up to 256x while keeping accuracy on hand gesture recognition and lip reading.
Problem
Digitizing and transmitting high-resolution video from edge-device image sensors is power-hungry and slow, and existing on-sensor feature extractors are either spatial-only CNNs or simple temporal-only encoders, with limited precision and memory on sensor-processors.
Approach
A lightweight convolutional GRU/minimal-gated-unit style recurrent encoder with binary {-1,1} weights and sign-based forget gating, placed in the pixel array so only a 64x64 output is read out every 16 frames, trained end to end with a downstream linear decoder and noise-aware quantization.
Key results
- On SCAMP-5 hardware, the noise-trained PixelRNN reaches 84.4% on hand gesture recognition (vs 68.9% without noise training; 85.6% in simulation) and 80.0% on lip reading (matching simulation).
- PixelRNN reduces readout bandwidth by up to 256x versus raw sensor data by outputting a 64x64 result every 16 frames.
- On EgoGesture with an 8x compressed readout, PixelRNN as a pre-processor retains 88.9% accuracy vs 74.5% for naive temporal downsampling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with simulated noise to account for analog noise (ablation) | PixelRNN on SCAMP-5 · Hand Gesture Recognition test set | accuracy: 68.9% -> 84.4% | Helped |
| train with simulated noise to account for analog noise (ablation) | PixelRNN on SCAMP-5 · Lip Reading test set | accuracy: 60.0% -> 80.0% | Helped |
| use PixelRNN encoder instead of naive temporal downsampling at the same bandwidth | EgoGesture decoder pipeline · EgoGesture | accuracy: 74.5% -> 88.9% | Helped |
| reduce PixelRNN readout bandwidth to about 1,000 values every 16 frames (ablation) | PixelRNN · Hand Gesture Recognition and Lip Reading | accuracy | No clear effect |
| reduce PixelRNN readout bandwidth beyond about 1,000 values every 16 frames (ablation) | PixelRNN · Hand Gesture Recognition and Lip Reading | accuracy | Hurt |
| use binary weights rather than full-precision weights (ablation) | PixelRNN compared with other RNN architectures · Hand Gesture Recognition and Lip Reading | accuracy | Helped |
| replace PixelRNN with larger LSTM architectures under binary weights (ablation) | in-pixel RNN feature encoder · Hand Gesture Recognition and Lip Reading | accuracy | Hurt |
| use the proposed CNN feature encoder instead of the listed CNN baselines | in-pixel CNN feature encoder · CIFAR-10 | accuracy: 39.8% -> 43.1% | Helped |
| use the proposed CNN feature encoder instead of the listed CNN baselines | in-pixel CNN feature encoder · Cambridge hand gesture images | accuracy: 60.2% -> 68.1% | Helped |
Surprising findings
- Larger RNNs such as LSTMs perform poorly under binary weight constraints, while leaner SRNN and PixelRNN train more reliably.
- Training with added Gaussian noise was needed to recover SCAMP-5 hardware accuracy (68.9% to 84.4% on hand gesture).
Limitations
- SCAMP-5 is not stacked, so limited in-pixel digital memory forces binary precision and small encoders.
- Evaluated on small datasets (Cambridge hand gesture, Tulips1); generalization to other tasks is only partly shown via EgoGesture with floating-point encoder.
- Hardware prototype uses a single-layer binary CNN + PixelRNN encoder; the source code is only promised for the project website.
Experimental setup
Datasets: Cambridge Hand Gesture Recognition dataset Tulips1 MNIST CIFAR-10 EgoGesture
Benchmarks: Cambridge Hand Gesture Recognition Tulips1 lip reading EgoGesture MNIST CIFAR-10
Metrics: classification accuracy memory bytes per pixel readout bandwidth (values)
Baselines: RAW camera difference camera convolutional LSTM GRU MGU simple RNN Bose et al. CNN Liu et al. CNN temporal downsampling
Builds on: GRU minimal gated unit Bose et al. in-pixel CNN difference camera / event camera
Closest papers by meaning
- A Low Power, Fully Event-Based Gesture Recognition System (CVPR 2017 · similarity 0.85)
- AsCAN: Asymmetric Convolution-Attention Networks for Efficient Recognition and Generation (NeurIPS 2024 · similarity 0.81)
- LION: Implicit Vision Prompt Tuning (AAAI 2024 · similarity 0.81)
- SpGesture: Source-Free Domain-adaptive sEMG-based Gesture Recognition with Jaccard Attentive Spiking Neural Network (NeurIPS 2024 · similarity 0.81)
- An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels (ICLR 2025 · similarity 0.81)
- Task-Optimized Convolutional Recurrent Networks Align with Tactile Processing in the Rodent Brain (NeurIPS 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.