Research map / Label-efficient and robust learning / Self-supervised and contrastive
Animal behavioral analysis and neural encoding with transformer-based self-supervised pretraining
Yanchen Wang, Han Yu, Ari Blau, Yizi Zhang, International Brain Laboratory, Liam Paninski, Cole Lincoln Hurwitz, Matthew R Whiteway
This paper tried a common change: Add a contrastive loss
add temporal contrastive loss to MAE pretraining (zero-shot) · IBL neural encoding (TCN) · BPS: 0.331 -> 0.292 (IBL); 0.311 -> 0.309 (IBL-whisker) Mixed
What 1,317 accepted papers report for this change
- 1,253 improved
- 22 no clear effect
- 16 mixed
- 26 hurt
- Beyond Looking Up, Try Looking Around: Harmonizing Global Structure and Local Consistency in Optimal Transport for Short Text Clustering ICML 2026 · add contrastive learning · SearchSnippets · ACC: 56.65 -> 80.12Improved
- SleepLM: Natural-Language Intelligence for Human Sleep ICML 2026 · add contrastive objective to captioning (CoCa vs Cap) · sleep event IoU · IoU: 29.2 -> 27.4Hurt
- LIMSSR: LLM-Driven Sequence-to-Score Reasoning under Training-Time Incomplete Multimodal Observations ICML 2026 · add contrastive loss L_con · FS1000 average · Spearman correlation: 0.759 -> 0.789Improved
See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add domain-specific MAE pretraining on IBL sessions to ImageNet MAE (ablation) | ViT-M · IBL neural encoding (TCN) | BPS: 0.321 -> 0.331 | Improved |
| add temporal contrastive loss to MAE pretraining (zero-shot) (ablation) | BEAST (IN+PT) · IBL neural encoding (TCN) | BPS: 0.331 -> 0.292 (IBL); 0.311 -> 0.309 (IBL-whisker) | Mixed |
| add session-specific fine-tuning to BEAST (ablation) | BEAST (IN+PT) · IBL neural encoding (TCN) | BPS: 0.292 -> 0.347 | Improved |
| combine MAE with contrastive loss rather than contrastive only (ablation) | BEAST · IBL-whisker neural encoding (TCN) | BPS: 0.283 -> 0.309 | Improved |
| use BEAST temporal-window positive sampling instead of ViC-MAE sampling (ablation) | BEAST pretraining · IBL-whisker neural encoding (TCN) | BPS: 0.289 -> 0.317 | Improved |
| use motion-filtered k-means frame selection instead of random (ablation) | BEAST pretraining · IBL neural encoding (TCN) | BPS: 0.319 -> 0.337 (IBL); 0.319 -> 0.317 (IBL-whisker) | Mixed |
| use extended augmentation (rotation, color jitter) instead of default (ablation) | BEAST pretraining · IBL neural encoding (TCN) | BPS: 0.337 -> 0.328 | No clear effect |
| use CLS token pooling instead of mean or attention pooling for contrastive loss (ablation) | BEAST pretraining · IBL neural encoding (TCN) | BPS: 0.321 -> 0.337 (mean); 0.323 -> 0.337 (attention) | Improved |
| use patch embeddings with attention pooling instead of CLS token (ablation) | BEAST (IN+PT) · CalMS21 action segmentation | F1: 0.63 -> 0.81 | Improved |
| add domain-specific pretraining with contrastive loss to ImageNet MAE backbone (ablation) | ViT-M patch · CalMS21 action segmentation | F1: 0.74 -> 0.81 | Improved |
| add domain-specific pretraining with contrastive loss to ImageNet MAE backbone (ablation) | ViT-M patch · IBL action segmentation | F1: 0.84 -> 0.87 | Improved |
| pretrain with contrastive loss only for pose estimation (ablation) | ViT-C · pose estimation (100 labels) | pixel error | Hurt |
Closest papers by meaning
- BehaVERT: A Transformer-Based Motion Language Model for Decoding Behavioral Semantics in Mice (IJCV 2026 · similarity 0.89)
- BEAST3D: animal behavioral analysis and neural encoding from multi-view video via Gaussian splatting (NeurIPS 2026 · similarity 0.89)
- Inpainting the Neural Picture: Inferring Unrecorded Brain Area Dynamics from Multi-Animal Datasets (NeurIPS 2025 · similarity 0.86)
- Neural Encoding and Decoding at Scale (ICML 2025 · similarity 0.86)
- UniAP: Towards Universal Animal Perception in Vision via Few-Shot Learning (AAAI 2024 · similarity 0.85)
- Pose Splatter: A 3D Gaussian Splatting Model for Quantifying Animal Pose and Appearance (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
