Research map / Learning theory and algorithms / Kernels and regression theory
Shortcut Features as Top Eigenfunctions of NTK: A Linear Neural Network Case and More
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes shortcut learning via NTK eigenfunctions in linear networks on clustered Gaussian mixtures, showing shortcut features have larger eigenvalues and influence, persisting under SD margin control, and proposes predictability/availability metrics validated on ResNet-18 and real datasets.
Problem
Neural networks trained on biased datasets, where a shortcut feature dominates most samples, prefer shortcut features over generalizable core features; the mechanism behind this is unclear, and the max-margin bias is often blamed.
Approach
Define features as NTK eigenfunctions; for linear networks on clustered Gaussian mixtures, show shortcut clusters yield larger eigenvalues (faster learning) and larger post-convergence weights tied to data variance; show margin control (SD) does not remove this bias, and introduce predictability and availability metrics for real data.
Key results
- Availability of shortcut labels exceeds that of ground-truth labels across Waterbirds, CelebA, Colored-MNIST, and Patched-MNIST, consistently across 5 runs (e.g., CE on Waterbirds 0.0153 difference at 1000 SGD updates, with std 0.045 reported).
- Test accuracy of shortcut labels on Waterbirds (~0.83) and CelebA (~0.953) exceeds ground-truth accuracy (~0.82, ~0.951) with ResNet-18.
- Under strong SD regularization (lambda to infinity), the weight ratios converge to those of the MSE solution, so the decision boundary still reflects shortcut features.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with CE, MSE, or SD; shortcut-label availability exceeds ground-truth-label availability | Two-layer ReLU FC and pretrained ResNet-18 · Waterbirds, CelebA, Colored-MNIST, Patched-MNIST | availability: 0.0153 ± 0.045; 0.0203 ± 0.0162; 0.0977 ± 0.0129; 0.0160 ± 0.0061 (CE); 0.0182 ± 0.0079; 0.0258 ± 0.0086; 0.0832 ± 0.004 | Helped |
| Use stronger shortcut correlation (ablation) | ResNet-18 availability measurement · Dataset with varied shortcut-label correlation | availability | Helped |
| Increase the patch size and shortcut strength (ablation) | Two-layer ReLU FC · Patched-MNIST | availability | Helped |
| Use pretrained rather than un-pretrained ResNet-18 (ablation) | ResNet-18 · Dogs vs Cats | availability | Mixed |
| Apply SD or use MSE rather than CE (ablation) | Two-layer ReLU FC and ResNet-18 · Colored-MNIST, Patched-MNIST, Waterbirds, CelebA | availability of shortcut label relative to ground-truth | No clear effect |
| Add SD regularization to the CE loss (ablation) | Linear neural network · Gaussian mixture toy classification | decision-boundary shortcut influence | No clear effect |
| Use very small rather than large within-cluster variance (ablation) | Two-layer ReLU fully-connected network · Four-cluster toy classification | decision-boundary shift with cluster-weight changes | Hurt |
| Reduce bird size and use an un-pretrained ResNet-18 (ablation) | ResNet-18 · Modified Waterbirds | availability as an indicator of learned shortcut | Mixed |
| Use shortcut labels instead of ground-truth labels | Two-layer ReLU FC · Colored-MNIST and Patched-MNIST | test accuracy: ≈ 0.92; ≈ 0.94; > 0.98 | Mixed |
| Use shortcut labels instead of ground-truth labels | Pretrained ResNet-18 · Waterbirds and CelebA | test accuracy: ≈ 0.83 for Waterbirds; ≈ 0.953 for CelebA; ≈ 0.82 for Waterbirds; ≈ 0.951 for CelebA | Helped |
Surprising findings
- Availability of shortcut labels was lower than ground-truth labels for a pretrained ResNet-18 on Dogs vs Cats, indicating pretraining reduces shortcut dominance.
- Availability can fail to detect a shortcut learned later when multiple shortcuts exist (un-pretrained Waterbirds experiment).
Limitations
- Theory limited to linear networks on Gaussian mixtures with comparable mean norms and infinite-width NTK assumption.
- Availability may not detect shortcuts learned later when multiple shortcuts exist.
- Shortcut labels were manually identified from model predictions rather than automatically.
Experimental setup
Datasets: Patched-MNIST Colored-MNIST Waterbirds CelebA Dogs and Cats
Benchmarks: Patched-MNIST Colored-MNIST Waterbirds CelebA Dogs vs Cats
Metrics: availability predictability test accuracy eigenvalue
Baselines: SD Marg-Ctrl
Builds on: Neural Tangent Kernel Tsilivis et al. feature decomposition Puli et al. SD (spurious debiasing) Hermann et al. predictability and availability
Closest papers by meaning
- Optimized Random Features for the Neural Tangent Kernel (Student Abstract) (AAAI 2025 · similarity 0.87)
- Understanding the Evolution of the Neural Tangent Kernel at the Edge of Stability (NeurIPS 2025 · similarity 0.87)
- Neural Tangent Kernels Motivate Cross-Covariance Graphs in Neural Networks (ICML 2024 · similarity 0.87)
- Better NTK Conditioning: A Free Lunch from (ReLU) Nonlinear Activation in Wide Neural Networks (NeurIPS 2025 · similarity 0.87)
- The Surprising Effectiveness of Infinite-Width NTKs for Characterizing and Improving Model Training (AAAI 2025 · similarity 0.86)
- Divergence of Neural Tangent Kernel in Classification Problems (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.