Research map / Vision-language and multimodal / Human and embodied modeling
Neural Concept Binder
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Neural Concept Binder, which combines SysBinder soft binding with HDBSCAN-based retrieval hard binding to produce discrete, inspectable and revisable object-factor concept-slot encodings from unlabeled images, and a new CLEVR-Sudoku benchmark.
Problem
Unsupervised visual concept learning yields concepts that are either continuous and hard to inspect or discrete and hard to learn; concepts must be both descriptive/distinct and inspectable and revisable by humans.
Approach
Stack a SysBinder block-slot encoder (soft binding) with a per-block HDBSCAN clustering whose prototypes/exemplars form a retrieval corpus that maps continuous block encodings to discrete concept symbols (hard binding), enabling symbolic inspection and revision by editing the corpus.
Key results
- On CLEVR-Easy with N=2000 encodings, NCB reaches 99.02 ± 1.00 property accuracy, nearly matching continuous SysBinder (99.83) and far above discrete SysBinder (92.49 ± 5.45 for the non-hard variant listed).
- With only 20 training encodings on CLEVR-Easy, NCB keeps 94.22 ± 4.11 accuracy versus 64.88 ± 10.89 for SysBinder (cont.).
- On CLEVR-Sudoku with 10 examples, NCB encodings solve roughly 50% of puzzles versus about 61% for supervised SA; SysBinder fails across variants.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use discrete NCB concept encodings instead of continuous SysBinder encodings for object-property classification with N=20 training examples | Object-property classifier on CLEVR-Easy · CLEVR-Easy, N=20 | object-property accuracy: 64.88 ± 10.89 -> 94.22 ± 4.11 | Helped |
| Use discrete NCB concept encodings instead of continuous SysBinder encodings for object-property classification with N=20 training examples | Object-property classifier on CLEVR · CLEVR, N=20 | object-property accuracy: 79.11 ± 8.75 -> 88.57 ± 4.68 | Helped |
| Use discrete NCB concept encodings instead of continuous SysBinder encodings for object-property classification with N=2000 training examples | Object-property classifier on CLEVR-Easy · CLEVR-Easy, N=2000 | object-property accuracy: 99.83 ± 0.24 -> 99.02 ± 1.00 | Hurt |
| Use NCB concept encodings rather than SysBinder discrete encodings | Object-property classifier · CLEVR-Easy and CLEVR | object-property accuracy | Helped |
| Use NCB concept encodings rather than SysBinder encodings in the Sudoku solver | CLEVR-Sudoku solver · Sudoku CLEVR-Easy and Sudoku CLEVR, 10-example configurations | percentage of solved puzzles: approximately 50% vs. approximately 61% | Helped |
| Use SysBinder discrete encodings instead of NCB concept encodings in the Sudoku solver | CLEVR-Sudoku solver · CLEVR-Sudoku | percentage of solved puzzles | Hurt |
| Revise NCB concepts using GPT-4 feedback | Neural Concept Binder · CLEVR-Sudoku with few examples | percentage of solved puzzles | Helped |
| Revise NCB concepts using GPT-4 feedback | Neural Concept Binder · CLEVR-Sudoku with more digit examples | percentage of solved puzzles | Hurt |
| Revise NCB concepts using human feedback | Neural Concept Binder · CLEVR-Sudoku, particularly configurations with fewer candidate examples | percentage of solved puzzles | Helped |
| Use a set-transformer classifier with unsupervised NCB concepts rather than supervised slot-attention encodings | CLEVR-Hans3 classifier · CLEVR-Hans3 | classification accuracy | No clear effect |
| Apply XIL feedback to the neural classifier's explanations | NCB + NN classifier · CLEVR-Hans3 non-confounded test set | classification accuracy | Helped |
| Zero activations of the undesired gray concept | NCB + NN classifier · CLEVR-Hans3 non-confounded test set | classification accuracy | Helped |
Surprising findings
- GPT-4-driven concept revision improves performance with few examples but hurts when more digit examples are present, due to inconsistent object descriptions.
- Zeroing out the activation of an undesired concept (gray) in the hard binder mitigated confounding more effectively than standard loss-based XIL on the neural classifier.
Limitations
- Performance depends heavily on quality of initial block-slot encodings from SysBinder.
- Unsupervised concepts may be misaligned with human knowledge and require alignment; clustering can overparameterize factors.
- Revisions rely on accurate human feedback and can be manipulated maliciously; tested mainly on simple CLEVR-style images, not video.
Experimental setup
Datasets: CLEVR CLEVR-Easy CLEVR-Hans3 CLEVR-Sudoku
Benchmarks: CLEVR-Sudoku CLEVR-Easy CLEVR CLEVR-Hans3
Metrics: object-property classification accuracy percentage of solved Sudokus test accuracy on CLEVR-Hans3
Baselines: SysBinder (cont.) SysBinder SysBinder (hard) SysBinder (step) NLOTM Supervised slot attention (SA) CBM NeSyCL GlanceNets VQ-VAE
Builds on: SysBinder slot attention HDBSCAN NLOTM concept bottleneck models
Closest papers by meaning
- Does Object Binding Naturally Emerge in Large Pretrained Vision Transformers? (NeurIPS 2025 · similarity 0.86)
- Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem (NeurIPS 2024 · similarity 0.85)
- NeuronEye: Query-Guided Visual Concept Activation for Vision-Language Reasoning (NeurIPS 2026 · similarity 0.85)
- Concept-RuleNet: Grounded Multi-Agent Neurosymbolic Reasoning in Vision Language Models (AAAI 2026 · similarity 0.85)
- Slot Abstractors: Toward Scalable Abstract Visual Reasoning (ICML 2024 · similarity 0.83)
- Slot State Space Models (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.