Research map / Vision-language and multimodal / Human and embodied modeling
CleverBirds: A Multiple-Choice Benchmark for Fine-grained Human Knowledge Tracing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CleverBirds, a large-scale knowledge tracing benchmark built from the eBird bird-species quiz (40k+ participants, 17.9M multiple-choice interactions, ~10.8k species), and benchmarks KT, MLP, tree, and LM baselines on it.
Problem
Modeling how human learners acquire fine-grained visual recognition expertise (e.g., bird species identification) is under-served by existing knowledge tracing datasets, which are concentrated in math, programming, and language with few concepts.
Approach
A large citizen-science quiz dataset with a very large concept space (~10,779 species) and multiple-choice interactions, paired with fixed image embeddings (DINOv2, ResNet-50) and user/species context features to evaluate multiple KT and classification baselines.
Key results
- Random forest with user and species context reaches 0.848 AP on predicting learner mistakes (binary task) on the full test set.
- MLP with user, species, and ResNet-50 image context reaches 76.0% multiple-choice accuracy and 24.5% on incorrectly answered questions, versus 20% random.
- Tested KT baselines (AKT, ATKT, simpleKT, etc.) cluster tightly, with AP around 0.35 and macro accuracy around 54-56%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add user and species context rather than species context alone | Random forest binary classifier · CleverBirds binary full | AP: 0.462 $\pm$ 0.000 -> 0.848 $\pm$ 0.000 | Helped |
| Add user context rather than species context alone | Random forest binary classifier · CleverBirds binary full | AP: 0.462 $\pm$ 0.000 -> 0.824 $\pm$ 0.002 | Helped |
| Add ResNet-50 image features to user and species context | MLP U+S · CleverBirds multiple choice incorrect | Accuracy (%): 6.40 $\pm$ 0.80 -> 24.50 $\pm$ 2.40 | Helped |
| Add ResNet-50 image features to user and species context | MLP U+S · CleverBirds multiple choice full | Accuracy (%): 72.70 $\pm$ 1.00 -> 76.00 $\pm$ 1.20 | Helped |
| Increase user-context length from Q1 to Q5 | RF U+S · CleverBirds test set, context-length quintiles | Binary AP: 0.62 -> 0.96 | Helped |
| Use DINOv2 rather than ResNet-50 image features | Image-only MLP species classifier · Bird species classification multiple choice | Accuracy (%): 74.90 $\pm$ 0.40 -> 88.80 $\pm$ 0.20 | Helped |
Surprising findings
- Knowledge tracing baselines perform similarly to each other (AP ~0.35) and underperform simple feature-engineered classifiers with user and species context.
- A confusion-prior classifier that never predicts the correct species achieves about 23.5% on incorrect-answer questions, close to the trained MLP's 24.5%.
Limitations
- Restricted to bird species identification, limiting immediate transfer to other domains such as medical imaging.
- Geographic and selection bias: participants skew toward the Global North and eBird users.
- Multiple-choice format and correct-label-only feedback limit ecological validity of the learning signal.
Experimental setup
Datasets: CleverBirds (eBird quiz interactions)
Benchmarks: CleverBirds
Metrics: multiple choice accuracy multiple choice incorrect-subset accuracy binary macro accuracy binary average precision (mistakes) AUC
Baselines: simpleKT KQN DKT DKT+ ATKT SAKT AKT DKVMN LM Seq2seq LM MCC Random Forest Logistic Regression
Builds on: DINOv2 simpleKT AKT DKT DKVMN
Closest papers by meaning
- RATS! Patches Talk Through Registers: Emergent Parts in Register Attention Transformers (NeurIPS 2026 · similarity 0.86)
- Seeing as Experts Do: A Knowledge-Augmented Agent for Open-Set Fine-Grained Visual Understanding (CVPR 2026 · similarity 0.85)
- VLM4Bio: A Benchmark Dataset to Evaluate Pretrained Vision-Language Models for Trait Discovery from Biological Images (NeurIPS 2024 · similarity 0.85)
- HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenes (ICLR 2026 · similarity 0.85)
- RealBirdID: Benchmarking Bird Species Identification in the Era of MLLMs (CVPR 2026 · similarity 0.85)
- UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.