Research map / Architectures and efficiency / RNNs and normalization
RTify: Aligning Deep Neural Networks with Human Behavioral Decisions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RTify, a trainable evidence-accumulator module that uses a first-order gradient approximation to align an RNN's decision time steps with human reaction times, and extends the Wong-Wang model into a differentiable multi-class module stackable on CNNs.
Problem
Existing image-computable vision models predict choice accuracy but ignore human reaction times, while decision-making models (drift-diffusion, Wong-Wang) are limited to simple artificial stimuli; the RT is an integer-valued, non-differentiable function of the network, which blocks gradient-based al
Approach
Relax the non-differentiable first-passage time into a continuous quantity and approximate its gradients with respect to the evidence-readout weights and threshold via a first-order Taylor expansion (1/(Phi_t - Phi_{t-1})), so a frozen task-optimized RNN can be trained to match human RT distributions directly, or to trade speed for accuracy via a self-penalty term; also a differentiable multi-popu
Key results
- On the RDM task, the supervised RTify model fits the full RT distribution and beats entropy-thresholding in MSE (Wilcoxon p<.05) across coherences.
- On object recognition, RTify model RTs correlate with human RTs at r = .51 (p<.001), and the self-penalized ideal observer gives r = .40 (p<.001) versus uncertainty proxy failing once high-uncertainty trials are removed (r = .05, p = 0.32).
- The RTified VGG with multi-class Wong-Wang module achieves r = .49 (p<.001) with human RTs; the base RNN reaches 75.2% on a held-out test set for the object task.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fit RTify to human RTs instead of entropy-thresholding | RTify supervised · RDM task, all coherence levels | RT-distribution MSE | Helped |
| Train RTify with time self-penalty instead of using an uncertainty proxy | RTify self-penalized · RDM task, all coherence levels | RT-distribution MSE | Helped |
| Add a RTified Wong-Wang module | C3D · RDM task, all coherence levels | RT-distribution MSE | Helped |
| Fit RTify to human RTs instead of entropy-thresholding | RTify supervised · Object recognition test set | correlation with human RT: 99.9% | Helped |
| Train RTify with time self-penalty instead of using an uncertainty proxy | RTify self-penalized · Object recognition test set | correlation with human RT: 87.1% | Helped |
| Exclude high-uncertainty trials | Uncertainty proxy · Object recognition test set, high-uncertainty trials excluded | correlation with human RT: .36 -> .05 | Hurt |
| Exclude high-uncertainty trials | RTify self-penalized · Object recognition test set, high-uncertainty trials excluded | correlation with human RT: .40 -> .41 | No clear effect |
| Add a multi-class RTified Wong-Wang module | VGG-19 · Object recognition test set | correlation with human RT | Helped |
| Use only the network output at the decision time point instead of combining outputs up to decision time (ablation) | RTify supervised and self-penalized · RDM task, all coherence levels | RT-distribution fit | Helped |
Surprising findings
- Fitting only the positive (correct-trial) RT part of the distribution led the model to reach human-comparable accuracy without accuracy supervision.
- Uncertainty correlated with human RTs only in high-uncertainty cases; after excluding them the correlation disappeared, while the proposed RT method remained significant.
Limitations
- Human datasets are small-scale and mostly synthetic stimuli; natural-image RT benchmarks are lacking.
- Only RT distributions are modeled, not the full broader set of behaviors; the approach is limited to predicting RT distributions.
- The object task used 10-way classification instead of the original binary pairwise task because individual binary pair data were not saved.
Experimental setup
Datasets: random dot motion RDM dataset (Kiani-style psychophysics, 21 participants) object recognition RT dataset (Kar et al., 88 participants) ImageNet COCO
Benchmarks: RDM reaction time distributions across coherence levels object categorization human RT correlation
Metrics: mean squared error between RT histograms Pearson correlation with human RTs classification accuracy two-sided Wilcoxon signed-rank test bootstrapping
Baselines: entropy-thresholding (Spoerer-style recurrent CNN) uncertainty proxy (evidential deep learning, C-RBP model)
Builds on: Wong-Wang model evidential deep learning CORnet-S backpropagation through time
Closest papers by meaning
- Computing a human-like reaction time metric from stable recurrent vision models (NeurIPS 2023 · similarity 0.91)
- Recurrent neural network dynamical systems for biological vision (NeurIPS 2024 · similarity 0.89)
- Learning dynamics in linear recurrent neural networks (ICML 2025 · similarity 0.86)
- Model–Behavior Alignment under Flexible Evaluation: When the Best-Fitting Model Isn’t the Right One (NeurIPS 2025 · similarity 0.86)
- NEvo: Neural-Guided Evolutionary Video Synthesis for Dynamic Visual Selectivity (NeurIPS 2026 · similarity 0.85)
- Flexible Context-Driven Sensory Processing in Dynamical Vision Models (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.