Research map / Architectures and efficiency / Attention and transformers
Names Don’t Matter: Symbol-Invariant Transformer for Open-Vocabulary Learning
İlker Işık, Wenchao Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use per-stream cross-attention instead of aggregated cross-attention (ablation) | Proposed · Propositional logic assignment (PropRandom35) (heatmap test set) | accuracy: 28.51% -> 95.05% | Improved |
| use per-stream cross-attention instead of aggregated cross-attention (ablation) | Proposed · LTL witness generation (LTLRandom35) (heatmap test set) | accuracy: 20.93% -> 90.47% | Improved |
| add aggregated cross-attention on top of per-stream (ablation) | Proposed · Propositional logic assignment (PropRandom35) (heatmap test set) | accuracy: 95.05% -> 92.66% | Hurt |
| add decoder per-stream attention (ablation) | Proposed · Propositional logic assignment (PropRandom35) (heatmap test set) | accuracy: 46.55% -> 95.05% | Improved |
| add decoder aggregated attention (ablation) | Proposed · Propositional logic assignment (PropRandom35) (heatmap test set) | accuracy: 84.48% -> 95.05% | Improved |
| add encoder aggregated attention (ablation) | Proposed · Propositional logic assignment (PropRandom35) (heatmap test set) | accuracy: 92.47% -> 95.05% | Improved |
| add decoder aggregated attention (ablation) | Proposed · LTL witness generation (LTLRandom35) (heatmap test set) | accuracy: 90.47% -> 89.47% | Hurt |
| add encoder per-stream attention (ablation) | Proposed · LTL witness generation (LTLRandom35) (heatmap test set) | accuracy: 84.13% -> 90.47% | Improved |
| use symbol-invariant proposed model instead of random embedding on renamed data | Proposed · Propositional logic assignment (PropRandom35) (Renamed training data) | correct predictions: 93.25% -> 98.03% | Improved |
| use symbol-invariant proposed model on reduced training data (80K) instead of baseline | Proposed · Propositional logic assignment (PropRandom35) (Reduced training data) | correct predictions: 63.26% -> 70.43% | Improved |
| use proposed model instead of baseline on renamed LTL data | Proposed · LTL witness generation (LTLRandom35) (Renamed training data) | correct predictions: 34.13% -> 98.24% | Improved |
| fine-tune converted pre-trained model for 1 epoch instead of training from scratch | Converted model · LTL witness generation (LTLRandom35) (heatmap test set) | heatmap accuracy: 84.13% -> 85.91% | Improved |
Closest papers by meaning
- Interchangeable Token Embeddings for Extendable Vocabulary and Alpha-Equivalence (ICML 2025 · similarity 0.88)
- When Symbol Names Should Not Matter: A Logistic Theory of Fresh-Symbol Classification (NeurIPS 2026 · similarity 0.88)
- Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling (ICML 2025 · similarity 0.86)
- Towards Universality: Studying Mechanistic Similarity Across Language Model Architectures (ICLR 2025 · similarity 0.85)
- The Foundations of Tokenization: Statistical and Computational Concerns (ICLR 2025 · similarity 0.85)
- Transformer Block Coupling and its Correlation with Generalization in LLMs (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
