Research map / Architectures and efficiency / Quantization

Analog Foundation Models

NeurIPS 2025 · Architectures and efficiency › Quantization · approach: Low-precision training formats · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Adapts pretrained LLMs (Phi-3-mini-4k-instruct, Llama-3.2-1B-Instruct) into analog foundation models via synthetic-data knowledge distillation and hardware-aware training with noise injection, static 8-bit I/O quantization and weight clipping, so they stay near 4-bit weight/8-bit activation baselines under analog noise.

Problem

Off-the-shelf LLMs degrade substantially when deployed on analog in-memory computing (AIMC) hardware, which imposes weight noise and static input/output quantization; no generic method existed to robustly adapt LLMs pre-trained on trillions of tokens to this setting.

Approach

Generate synthetic data by sampling from the teacher LLM itself, then train the model with pure knowledge distillation under hardware-aware training (learnable static input ranges, globally static output quantization, per-channel additive Gaussian weight noise, and iterative weight clipping), yielding analog-robust models using under 1% of pre-training tokens.

Key results

• Phi-3-mini-4k-instruct analog FM under hardware-realistic noise averages 66.22 vs 62.02 for the noisy off-the-shelf model and 60.58 for noisy LLM-QAT (W4).
• Llama-3.2-1B-Instruct analog FM under noise averages 40.91 vs 35.68 for noisy off-the-shelf model and 37.92 for noisy LLM-QAT.
• On MATH-500 test-time scaling, the noisy analog FM narrows the gap to the off-the-shelf model by 0.4% (Phi-3) and 3.58% (Llama-3.2-1B) as n grows, and outperforms the 4-bit quantized baselines.

Measured effects reported in the paper

Surprising findings

• Post-training SpinQuant models with static input ranges are less robust to hardware noise than the off-the-shelf model, sometimes collapsing (e.g., GSM8K 2.85% under noise for Phi-3).
• Weight clipping contributes more robustness than noise injection for LLMs, and the multiplicative noise component provided no robustness gain.

Limitations

• Training billion-parameter models is resource intensive and may be infeasible for many researchers.
• Substantial accuracy gap remains to FP16 baselines, especially on reasoning tasks such as GSM8K and MATH-500.
• Noise injection and output quantization slow training convergence and roughly triple linear kernel latency.

Experimental setup

Datasets: synthetic data generated by Phi-3-mini-4k-instruct and Llama-3.2-1B-Instruct FineWeb MATH-500 BookCorpus English Wikipedia CC-News OpenWebText GLUE

Benchmarks: MMLU GSM8K BoolQ Hellaswag MedQA AGIEval Arc-C Arc-E ANLI IFEval XSTest MATH-500

Metrics: accuracy average accuracy prompt-level accuracy instruction-level accuracy IPRR VPRR

Baselines: LLM-QAT SpinQuant Off-the-shelf FP16 models

Builds on: LLM-QAT AIHWKIT-Lightning SpinQuant GPTQ

Closest papers by meaning

• Analog In-memory Training on General Non-ideal Resistive Elements: The Impact of Response Functions (NeurIPS 2025 · similarity 0.90)
• Towards Exact Gradient-based Training on Analog In-memory Computing (NeurIPS 2024 · similarity 0.88)
• ScaleBITS: Scalable Bitwidth Search for Hardware-Aligned Mixed-Precision LLMs (NeurIPS 2026 · similarity 0.87)
• A2Q+: Improving Accumulator-Aware Weight Quantization (ICML 2024 · similarity 0.87)
• Reshape and Adapt for Output Quantization (RAOQ): Quantization-aware Training for In-memory Computing Systems (ICML 2024 · similarity 0.87)
• Scaling Laws for Floating–Point Quantization Training (ICML 2025 · similarity 0.86)