Research map / Architectures and efficiency / Attention and transformers

FTerViT: Fully Ternary Vision Transformer

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Hybrid and interpretable ViT designs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Fully ternarizes a Vision Transformer (weights, patch embedding, LayerNorm affines, classifier head) via same-architecture KL distillation plus a low-LR recovery phase, reaching 79.64% ImageNet top-1 at 5.81 MB and running on an ESP32-S3 MCU.

Problem

Ternary ViTs keep patch embeddings, LayerNorms and classifier heads in full precision, and these components dominate the memory footprint of compact ViTs, blocking deployment on microcontroller-class hardware.

Approach

Ternarize every weight matrix and LayerNorm affine (TernaryBitConv2d with per-channel scales for patch embedding, TernaryLayerNorm) and train the ternary student from its FP32 teacher with KL-only distillation followed by a short low-LR recovery phase.

Key results

• FTerViT DeiT-III-S at 224 reaches 79.64% ImageNet-1K top-1 at 5.81 MB (-3.44 pp vs FP32), and at 384 reaches 82.43% at 6.09 MB.
• FTerViT DeiT-S reaches 77.47% top-1 at 5.81 MB, versus 74.2% for TerViT-DeiT-S at 6.0 MB.
• The ternary model runs on an ESP32-S3-EYE MCU with 2.83 MB peak PSRAM, with 21.06 s end-to-end latency per inference.

Surprising findings

• Patch embedding and LayerNorm, despite negligible parameter counts, account for a large share of importance under Taylor and Hessian estimators, yet can be ternarized with KD.
• Ternarized LayerNorm scale parameters converge to +1, effectively reducing normalization to identity scaling.

Limitations

• Only DeiT-Tiny and DeiT-Small (and DeiT-III-S) evaluated; larger models not explored.
• The C inference kernel uses basic unoptimized bit-unpacking, and on-device latency is about 21 s per inference.
• Two-stage training pipeline could potentially be unified.

Experimental setup

Datasets: ImageNet-1K CIFAR-10 CIFAR-100

Benchmarks: ImageNet-1K CIFAR-10 CIFAR-100

Metrics: top-1 accuracy model size (MB) compression ratio latency power/energy

Baselines: TerViT ViT-1.58b Q-ViT OFQ LSQ PTQ4ViT RepQ-ViT Bi-ViT BiViT MCUFormer TinyFormer LMaNet-Elite

Builds on: BitNet b1.58 TerViT TWN DeiT-III HAWQ

Closest papers by meaning

• GradQ-ViT: Robust and Efficient Gradient Quantization for Vision Transformers (AAAI 2025 · similarity 0.85)
• Outlier-aware Slicing for Post-Training Quantization in Vision Transformer (ICML 2024 · similarity 0.84)
• RPFQ-ViT: Rotated Phase-Frame Quantization for Extremely Low-Bit Weights in Vision Transformers (NeurIPS 2026 · similarity 0.84)
• AIQViT: Architecture-Informed Post-Training Quantization for Vision Transformers (AAAI 2025 · similarity 0.84)
• DeepCompress-ViT: Rethinking Model Compression to Enhance Efficiency of Vision Transformers at the Edge (CVPR 2025 · similarity 0.84)
• Bi-ViT: Pushing the Limit of Vision Transformer Quantization (AAAI 2024 · similarity 0.84)