Research map / Architectures and efficiency / CNN design
Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistency
Robert Geirhos, Kristof Meding, Felix A. Wichmann
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CNNs instead of human observers as the compared decision makers | ImageNet-trained CNNs · Cue conflict, edge, and silhouette stimuli | error consistency (κ) | Improved |
| Compare ImageNet-trained CNNs with human observers | CNNs and human observers · Cue conflict and edge stimuli | error consistency (κ) | Hurt |
| Increase ImageNet validation accuracy | CNNs compared with humans · Cue conflict | error consistency (κ): F(1, 158) = 0.086, p=.769, R^2=0.001 | No clear effect |
| Increase ImageNet validation accuracy | CNNs compared with humans · Line drawing stimuli | error consistency (κ): F(1, 158) = 0.478, p=.491, R^2=0.003 | No clear effect |
| Increase ImageNet accuracy | CNNs compared with humans · Silhouette stimuli | error consistency (κ): F(1, 158) = 53.530, p=1.21\cdot 10^{-11}, R^2=0.253 | Improved |
| Increase top-5 ImageNet accuracy | CNNs compared with humans · ImageNet images | error consistency (κ): F(1, 30) = 8.162, p=.008, R^2=0.214 | Hurt |
| Use a recurrent CORnet-S instead of a feedforward ResNet-50 | CNN-human comparison · Object recognition behavioural stimuli | error consistency (κ): .068: ResNet-50; .066: CORnet-S | No clear effect |
| Use CORnet-S rather than ResNet-50 | CNN-CNN comparison · Object recognition behavioural stimuli | error consistency (κ): .711 | Improved |
| Increase model shape bias through stylized training data | CNNs compared with humans · Cue conflict stimuli | error consistency (κ): .066, .068, .098, .195 | Improved |
| Compare CNNs with humans on silhouette stimuli rather than cue conflict or line-drawing stimuli | CNN-human error consistency · Silhouette stimuli | error consistency (κ) | Improved |
Closest papers by meaning
- Quantifying Uncertainty in Error Consistency: Towards Reliable Behavioral Comparison of Classifiers (NeurIPS 2025 · similarity 0.93)
- Evaluating Multiview Object Consistency in Humans and Image Models (NeurIPS 2024 · similarity 0.89)
- Directional Confusions Reveal Divergent Inductive Biases Through Rate-Distortion Geometry in Human and Machine Vision (NeurIPS 2026 · similarity 0.89)
- Human and AI Perceptual Differences in Image Classification Errors (AAAI 2025 · similarity 0.88)
- Synthesizing Images on Perceptual Boundaries of ANNs for Uncovering and Manipulating Human Perceptual Variability (ICML 2025 · similarity 0.88)
- Model–Behavior Alignment under Flexible Evaluation: When the Best-Fitting Model Isn’t the Right One (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
