AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Towards Optimal Caching and Model Selection for Large Model Inference

Banghua Zhu, Ying Sheng, Lianmin Zheng, Clark Barrett, Michael Jordan, Jiantao Jiao

NeurIPS 2023 · Architectures and efficiency › RNNs and normalization · approach: Neural scaling laws · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace LFU eviction with LEC (cost-aware) evictionlarge-model-only caching · Lambada offline, alpha=0.8cumulative cost (FLOPs, x10^3): 10.81 -> 10.36Improved
add learned model selector to LEC cachingLEC+large · Lambada offline, alpha=0.8, acc 100%cumulative cost (FLOPs, x10^3): 10.36 -> 2.50Improved
combine LEC with selector vs baseline LFU+largeLEC+selector · Lambada offline, alpha=0.8, acc 100%cumulative cost (FLOPs, x10^3): 10.81 -> 2.50Improved
raise selector accuracy from 80% to 100%LEC+selector · Lambada offline, alpha=0.8cumulative cost (FLOPs, x10^3): 4.76 -> 2.50Improved
use selector instead of cascade at 80% selector accuracyLEC caching · Lambada offline, alpha=0.2cumulative cost (FLOPs, x10^3): 1.50 -> 2.00Hurt
use selector instead of cascade at 100% selector accuracyLEC caching · Lambada offline, alpha=0.2cumulative cost (FLOPs, x10^3): 1.50 -> 0.99Improved
use cascade instead of always-large modelLFU caching · Lambada offline, alpha=0.8cumulative cost (FLOPs, x10^3): 10.81 -> 11.80Hurt
use cascade instead of always-large modelLEC caching · Lambada offline, alpha=0.8cumulative cost (FLOPs, x10^3): 10.36 -> 4.11Improved
replace LFU eviction with LEC evictionselector-based caching · OpenAssistant online, alpha=0.8cumulative cost (latency, x10^3): 21.14 -> 15.49Improved
combine LEC with selector vs baseline LFU+largeLEC+selector · OpenAssistant online, alpha=0.8cumulative cost (latency, x10^3): 28.24 -> 15.49Improved
use cascade instead of always-large modelLFU caching · OpenAssistant online, alpha=0.2cumulative cost (latency, x10^3): 9.31 -> 13.88Hurt
use cascade instead of always-large modelLEC caching · OpenAssistant onlinecumulative cost (latency, x10^3): 8.74 -> 8.82 (alpha=0.2); 26.07 -> 20.31 (alpha=0.8)Mixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy