Research map / Architectures and efficiency / Attention and transformers
Caracal: Causal Architecture via Spectral Mixing
BINGZHENG GAN, Tianyi Zhang, LI YUSU, Jing Huang, Wei Shi, Yangkai Ding, Tao Yu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add sliding window attention layers to MHF stack (ablation) | Caracal Large · Avg accuracy | avg accuracy (%): 48.22 -> 49.01 | Improved |
| add sliding window attention layers (ablation) | Caracal 345M, ctx 4096 · Avg accuracy (Table 3) | avg accuracy (%): 44.92 -> 46.62 | Improved |
| add pre-convolution (causal conv) in MHF (ablation) | Caracal Large w/o SWA · Avg accuracy | avg accuracy (%): 47.82 -> 48.22 | Improved |
| add explicit positional encodings (ablation) | Caracal Large · Avg accuracy | avg accuracy (%): 49.01 -> 48.94 | No clear effect |
| use two-stage gate (Linear+SiLU+group conv) instead of single linear projection (ablation) | Caracal Large · Avg accuracy | avg accuracy (%): 48.05 -> 49.01 | Improved |
| raise SWA density from 5:1 to 2:1 MHF:SWA (ablation) | Caracal Small · Avg accuracy | avg accuracy (%): 43.04 -> 43.35 | Improved |
| raise SWA density from 2:1 to 1:1 (ablation) | Caracal Small · Avg accuracy | avg accuracy (%): 43.35 -> 43.26 | Hurt |
| replace attention with MHF (Caracal) vs Llama on retrieval | Large models · SWDE | accuracy (%): 16.38 -> 8.82 | Hurt |
| replace attention with MHF (Caracal) vs Llama on LAMBADA perplexity | Large models · LAMBADA | perplexity: 24.92 -> 29.39 | Hurt |
Closest papers by meaning
- Flash Inference: Near Linear Time Inference for Long Convolution Sequence Models and Beyond (ICLR 2025 · similarity 0.86)
- The Hyperfitting Phenomenon: Sharpening and Stabilizing LLMs for Open-Ended Text Generation (ICLR 2025 · similarity 0.85)
- InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory (NeurIPS 2024 · similarity 0.85)
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads (ICML 2024 · similarity 0.85)
- Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.85)
- FFN Fusion: Rethinking Sequential Computation in Large Language Models (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
