RedKnot · Head-aware KV reuse for long-context LLM serving

Not every attention head needs the whole prompt.

Head-aware KV cache reuse for long-context LLM serving.

The idea in one picture

computed for this request reused from cache

Rows are attention heads, columns are prompt tokens.

SPLIT

$$\mathcal{H}=\mathcal{L}\cup\mathcal{G},\quad \tfrac{|\mathcal{G}|}{|\mathcal{H}|}\approx 3\text{–}17\%$$

A few global heads read everything. Most heads are local.

REUSE

$$W(i)=S_{\text{sink}}\cup[\,i-w,\;i\,]$$

A local head only looks at a few first tokens and a short recent window.

MERGE

$$y=W_b\,(z_{\text{off}}+z_{\text{on}})$$

Cached local part + fresh global part, combined in one step.

1.6–5.2×

faster first token

7B to 397B models, 16K–128K tokens

4.7–7.8×

more sessions per GPU

32K and 64K context

67–79%

less prefill compute

same model, no retraining

All numbers compare against recomputing the full prompt. Details and caveats are in the blog and the papers.

Partners