Research map / Label-efficient and robust learning / Test-time adaptation
Cold Case: The Lost MNIST Digits
Chhavi Yadav, Leon Bottou
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate classifiers on the reconstructed lost test examples rather than the official MNIST test set | KNN, SVM, MLP, and CNN classifiers · QMNIST50K versus MNIST test | misclassification rate: 2.95% -> 3.19%; 1.47% -> 1.8%; 1.61% -> 2.02%; 0.37% -> 0.53% | Hurt |
| Train LeNet-5 on reconstructed QMNIST instead of MNIST | LeNet-5 · MNIST test | misclassification rate: $0.82\%$ -> $0.81\% | No clear effect |
| Train LeNet-5 on reconstructed QMNIST instead of MNIST | LeNet-5 · QMNIST10K | misclassification rate: $0.81\%$ -> $0.80\% | No clear effect |
| Train KNN on QMNIST instead of MNIST | KNN (k=3) · QMNIST50K | misclassification rate: $3.19\%$ -> $3.19\% | No clear effect |
| Train SVM on QMNIST instead of MNIST | RBF SVM (C=10, g=0.02) · QMNIST50K | misclassification rate: $1.8\%$ -> $1.8\% | No clear effect |
| Train MLP on QMNIST instead of MNIST | MLP (800 hidden units) · QMNIST50K | misclassification rate: $2.02\%$ -> $2\% | No clear effect |
| Select a model using MNIST test performance | Classifiers evaluated on MNIST and QMNIST50K · MNIST and QMNIST50K | classifier ranking | Improved |
| Use paired comparisons on the same test examples instead of comparing absolute error-rate confidence intervals | Classifier comparisons · MNIST test | misclassification comparison sensitivity: four additional misclassifications out of 10,000 | Improved |
| Choose 1100 rather than 700 hidden units for the MLP | MLP · QMNIST50K | misclassification rate | Mixed |
Closest papers by meaning
- Scaling Down Deep Learning with MNIST-1D (ICML 2024 · similarity 0.80)
- $\texttt{dattri}$: A Library for Efficient Data Attribution (NeurIPS 2024 · similarity 0.79)
- On the Generalization of Handwritten Text Recognition Models (CVPR 2025 · similarity 0.78)
- The BatchNorm Illusion: Diagnosing Normalization Artifacts in Machine Unlearning Evaluation (NeurIPS 2026 · similarity 0.78)
- Learning the Signature of Memorization in Autoregressive Language Models (NeurIPS 2026 · similarity 0.77)
- Data Mixture Inference Attack: BPE Tokenizers Reveal Training Data Compositions (NeurIPS 2024 · similarity 0.77)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
