Research map / Architectures and efficiency / Attention and transformers

MiniPIC: Flexible Position-Independent Caching in <100LOC

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: KV cache eviction and compression · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MiniPIC, a minimal position-independent caching design for vLLM that stores unrotated keys, applies RoPE in the attention kernel, and exposes padding plus SSep/PDep special tokens to control block hashing, with interleaved span scheduling.

Problem

Prefix caching in vLLM cannot reuse KV entries for recurring spans (documents, code files) appearing at different positions or after different prefixes, while existing PIC implementations in inference servers require substantial engine changes or external KV storage.

Approach

Store position-free (unrotated) keys in the KV cache and apply RoPE on-the-fly inside the attention kernel using per-request logical positions, so shared blocks need no repositioning; keep PIC policy in the user layer via three token-level primitives (block-aligned padding, SSep, PDep) that modify block hashing.

Key results

• MiniPIC with ISPS (max_num_seqs=1024) reaches 48.01 samples/s vs 32.21 for baseline vLLM, a 49% improvement on 2WikiMultihopQA prefill throughput.
• MiniPIC with PIC disabled incurs 5.7% worst-case overhead (30.37 vs 32.21 samples/s).
• MiniPIC batched achieves 44.07 samples/s vs 36.14 for SPNL (CIDRA), and cached-span TTFT is reduced by up to two orders of magnitude versus vanilla vLLM.

Surprising findings

• Naive batched scheduling with a barrier between span prefill and final prompts underperforms interleaved scheduling, but the gain from interleaving is only about 9%.
• The position-free design adds only a small 5.7% overhead even when PIC is disabled.

Limitations

• Targets RoPE-based transformers only; ALiBi or NoPE need different handling.
• Incorrect padding or marker placement can cause false cache hits, placing burden on user.
• Span techniques approximate full causal attention and can reduce accuracy; no online traces evaluated and span eviction under high concurrency is possible.

Experimental setup

Datasets: 2WikiMultihopQA

Benchmarks: 2WikiMultihopQA

Metrics: prefill throughput (samples/s) speedup time-to-first-token preload time

Baselines: vanilla vLLM prefix caching SPNL (CIDRA) MEPIC

Builds on: vLLM Block-Attention EPIC Prompt Cache SPNL

Closest papers by meaning

• CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences (ICLR 2025 · similarity 0.88)
• SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.88)
• RazorAttention: Efficient KV Cache Compression Through Retrieval Heads (ICLR 2025 · similarity 0.88)
• KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs (NeurIPS 2026 · similarity 0.87)
• Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference (ICML 2024 · similarity 0.87)
• SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs (ICML 2025 · similarity 0.87)