Research map / Label-efficient and robust learning / Recognition and long-tail
Knowledge Matters: Importance of Prior Information for Optimization
Çağlar Gülçehre, Yoshua Bengio
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add intermediate-level hints (two-phase training with patch-wise targets) | SMLP · Pentomino (80k train) | test error (%): 12.4 -> 0.01 | Improved |
| add intermediate-level hints | SMLP · Pentomino (40k train) | test error (%): 36.0 -> 3.1 | Improved |
| add intermediate-level hints | SMLP · Pentomino (20k train) | test error (%): 48.6 -> 30.7 | Improved |
| add standardization layer at intermediate level (online SGD) (ablation) | SMLP-nohints · Pentomino online | test error (%): 50.1 -> 27.5 | Improved |
| increase training set size 20k -> 80k | SMLP-nohints · Pentomino | test error (%): 48.6 -> 12.4 | Improved |
| increase training set size 40k to 320k | MLP with 3 hidden layers · Pentomino | test error (%) | No clear effect |
| pretrain with hints for 1 epoch, then continue without hints | SMLP · Pentomino (40k train) | test error (%): 39 -> 0 | Improved |
| add unsupervised pretraining (CAE/DAE) before supervised finetuning | SMLP-nohints · Pentomino (80k train) | test error (%): 12.4 -> 49.7 | Hurt |
| add unsupervised RBM pretraining of first layer | SMLP-nohints · Pentomino | test error (%) | No clear effect |
| use LBFGS optimizer | MLP with 2 hidden layers · Pentomino (40k train) | test error (%) | No clear effect |
| use larger hidden layer (2050 hidden units in P1NN) with extensive hyperparameter search | SMLP-nohints · Pentomino (80k train) | test error (%): 12.4 -> 6.7 | Improved |
| use maxout MLP instead of tanh MLP | MLP · Pentomino Exp3 one-hot 80 inputs | test error (%): 22.475 -> 0.0 | Improved |
Closest papers by meaning
- On the creation of narrow AI: hierarchy and nonlocality of neural network skills (NeurIPS 2025 · similarity 0.88)
- Towards Understanding Hierarchical Learning: Benefits of Neural Representations (NeurIPS 2020 · similarity 0.86)
- Sensitivity and Generalization in Neural Networks: an Empirical Study (ICLR 2018 · similarity 0.86)
- Evaluating alignment between humans and neural network representations in image-based learning tasks (NeurIPS 2024 · similarity 0.86)
- What Variables Affect Out-of-Distribution Generalization in Pretrained Models? (NeurIPS 2024 · similarity 0.86)
- Do Neural Networks Need Gradient Descent to Generalize? A Theoretical Study (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
