Explainer

RedKnot, explained: most attention heads can reuse their cache

Long prompts are slow because every attention head re-reads every token. Most heads only look nearby, so most of that work can be reused.

The RedKnot team · October 2026 · about 10 minutes · Based on RedKnot and RedKnot-MLA

TL;DR

Language models have many attention heads. We found that only a small share of them (about 3–17%) need to read the whole prompt. The rest look at the start and at the last few hundred tokens.

RedKnot recomputes the few heads that need the whole prompt and reuses cached work for all the others. The first token arrives 1.6–5.2× sooner, one GPU holds 4.7–7.8× more sessions, and answers stay close to full recomputation (on some long-context tests they get better). No retraining is needed.

In this post
  1. Why long prompts are slow
  2. Why fixing tokens one by one falls short
  3. Insight 1: heads have different jobs
  4. Insight 2: attention is not the only cost
  5. Insight 3: savings must show up in memory
  6. The same idea on DeepSeek-V4
  7. Results
  8. Cite us and join us

Why long prompts are slow

Today's LLM apps send very long prompts. A RAG system pastes in dozens of retrieved documents. A coding agent includes files, tool outputs, and earlier steps. Prompts of 10K to 100K+ tokens are common.

Before the model can write its first word, it has to read the whole prompt. This step is called prefill, and the wait it causes is the time to first token (TTFT). For example, prefill of a 64K-token prompt on Llama-3.3-70B with 4 GPUs takes about a minute.

While it reads, the model writes notes about every token, called the KV cache. A natural fix is to save these notes and reuse them. Today's engines do this with prefix caching, but it only works when a new prompt starts with exactly the same text as an old one.

Reused documents rarely sit at the same place in the prompt

Two RAG requests that share Doc B. A prefix cache can only reuse the part before the first difference.

Request 1SystemDoc ADoc BQuestion 1
Request 2System ✓Doc CDoc B ✗Question 2
Filled red = reused by a prefix cache. Doc B was already processed, but it now comes after Doc C, so a prefix cache processes it again. In a real RAG request stream (MuSiQue, 48,315 document uses), on average 95% of a document's reuses happened at a different position, where a prefix cache can't help.

So there is a lot of work to save, if we can reuse a document's cache wherever it appears. This is called position-independent caching. The catch is that each document's cache was computed alone, without seeing the text that now comes before it. The cached notes are slightly wrong, and something has to repair them.

Why fixing tokens one by one falls short

Earlier systems (for example CacheBlend and ProphetKV) repair the cache by picking a small set of tokens and recomputing them. The idea makes sense: if only a few tokens matter, fix those.

The problem is that a model has many attention heads, and each head cares about different tokens. A token-level fix recomputes a token for all heads at once, so it has to cover every token that any head cares about. Try it:

Each head cares about different tokens

Each row is one attention head and each column is one token. Red cells are the tokens that head needs fixed.

In real models the effect is strong. In shallow layers, the union of tokens that matter to some head covers nearly every token. A token-level method then has two bad options. It can recompute almost everything and lose the speedup, or recompute too little and leave errors in some heads. Those errors also spread: a token recomputed on top of stale neighbors is still slightly wrong.

Insight 1: heads have different jobs

So we stopped asking "which tokens?" and asked "which heads?" When we tested heads across five very different models, we found two kinds:

Most attention heads are local

Share of KV heads in each model that are local (reusable) vs. global (must be recomputed). Found with needle-in-a-haystack tests.

Each bar adds up to 100% of the model's KV heads. 83–97% of heads are local in every model we tested, from a 7B dense model to a 397B mixture-of-experts model.
Insight Whether a head is global or local depends on the model, not on the prompt. Head 5 in layer 30 behaves the same way on every request. We test each model once, offline, and the result costs nothing at serving time.

That gives RedKnot's main rule. For each reused document:

  1. Global heads are recomputed in full. They are few, so this is cheap, and they are exactly the heads that the new context changes.
  2. Local heads reuse the cache. We only refresh what they can actually see: the sink tokens plus a short window before each token.
$$W(i) \;=\; \underbrace{S_{\text{sink}}}_{\text{first few tokens}} \;\cup\; \underbrace{[\,i-w,\; i\,]}_{\text{recent window}}$$

For example, with a window of w = 256, token 1000 of a local head only needs tokens 744–1000 and the sink. Everything else is reused.

One more detail: a reused document has moved to a new position. Modern LLMs encode position as a rotation (RoPE), so moving a document just means rotating its cached keys by the difference. This step is exact and cheap:

$$K(p_{\text{new}}) \;=\; R(p_{\text{new}})\,R(p_{\text{old}})^{-1}\,K(p_{\text{old}})$$

RedKnot is also careful about where it saves work. Shallow layers are sensitive, and errors there spread to every later layer, so they stay close to dense. Deep layers are more selective, so that is where most of the reuse happens.

Insight 2: attention is not the only cost

Each transformer layer has two parts: attention, where tokens look at each other, and the feed-forward network (FFN), which processes each token on its own. Caching only helps attention. But how much of prefill time does attention take?

The FFN takes most of the prefill time

Share of prefill time spent in the FFN, at a short and a long prompt.

How to read: 61% means 61% of the time before the first token goes to FFN work that no attention cache can skip. Agent workloads often use chunks of 2K–8K tokens, where the FFN takes 57–62% of the time.

So even a perfect attention cache leaves more than half the time on the table. RedKnot uses the attention it just computed as a signal for which tokens matter. In deep layers, only those tokens go through the FFN. The others skip it and pass through the layer's residual connection unchanged.

Insight Skipping low-value tokens does more than save compute. In very long prompts, most tokens are noise for the current question, and leaving them out of later layers keeps that noise out of the answer. We return to this in the results.

Insight 3: savings must show up in memory

The last problem is less obvious. Standard engines (PagedAttention) store the KV cache in blocks of tokens, with all heads packed together. To tell a local head to skip most tokens, you have to add a mask. On GPUs, a mask turns off the fastest attention kernel (FlashAttention), and the masked path is 4.9–7.6× slower. The math is saved on paper, but the kernel still runs on the slow path.

RedKnot's SegPagedAttention stores the cache per head. A global head keeps pages for the whole prompt. A local head keeps pages only for its sink and window. No mask is needed, and every head stays on the fast path. It also means local heads use far less GPU memory, so more requests fit at once.

Per-head storage keeps attention fast as prompts grow

Time for attention prefill over 64 layers (shaped like Qwen3-32B). Lower is better.

How to read: "Dense + mask" is the usual layout with a mask that hides tokens. Both versions compute the same result (cosine similarity > 0.99998). At 128K tokens, SegPagedAttention takes 0.23 s instead of 5.42 s, which is 23× faster. The gap grows with length because the masked version still pays for the full prompt on every head.

The same idea on DeepSeek-V4

DeepSeek models use multi-head latent attention (MLA). Instead of one cache per head, all heads share one compressed cache, so the per-head cache that RedKnot splits does not exist. Splitting it by force would undo MLA's memory savings.

RedKnot-MLA splits the heads logically instead. DeepSeek-V4 groups its heads in sets of 8. In each group, 1 head stays global and is recomputed for the request, and 7 heads are local and reused. Each document is processed once offline, starting at position 0. At request time we rotate the query to the document's position instead of rewriting the cache. The two parts are then added together before the layer's shared output projection:

$$z \;=\; \underbrace{z_{\text{off}}}_{\text{local heads, cached}} \;+\; \underbrace{z_{\text{on}}}_{\text{global heads, fresh}}, \qquad y \;=\; W_b\, z$$

When the cached part is exactly what full recomputation would give, this sum is exact. Any error comes only from local heads now sitting in a new context. Global heads and a small set of protected tokens (the query, the edges of each document, and tokens DeepSeek's own indexer marks as important) correct it.

How much work can this skip? On DeepSeek-V4-Flash, 37 of 43 layers are reusable and 56 of 64 heads per layer are local:

$$\frac{37}{43}\times\frac{56}{64} \;\approx\; 75\%\ \text{of attention-head work can be reused}$$

Results

Every number below compares RedKnot with recomputing the whole prompt on the same model, the same GPUs, and the same input tokens. Speedup means time with full recomputation ÷ time with RedKnot, so 1× means no gain and 3× means the first token arrives in a third of the time.

Faster first token than token-level methods

TTFT speedup on three dense models

Higher is better. Each group is one model, dataset, and prompt length. The dashed line at 1× is full recomputation.

RedKnot is fastest in 3 of the 4 settings, and its lead grows with prompt length (3.5× at 64K). CacheBlend and ProphetKV level off around 2.0–2.4×. At 16K CacheBlend is slightly faster, because it recomputes only a small fixed share of tokens. The next chart shows what that costs in accuracy.

…while staying much closer to the original answer

Does the model still predict the same next token?

How often each method agrees with full recomputation on the first generated token. Higher is better; 1.0 = always the same.

Top-1: the single most likely next token is the same as with full recomputation. Top-10: overlap between the 10 most likely next tokens. RedKnot agrees 93% / 87% of the time. The token-level methods agree 40–50% of the time. On Llama-3.3-70B HotpotQA, RedKnot's exact-match score is even higher than full recomputation (0.80 vs. 0.60 at both 32K and 64K).

DeepSeek-V4-Flash: long contexts, up to 440K tokens

RedKnot-MLA TTFT speedup on DeepSeek-V4-Flash (8× H200)

Higher is better. Cached documents are already built ("hot"), which is the normal case when documents are reused. The dashed line at 1× is full recomputation.

The first token arrives 2.0–3.8× sooner. Each bar is a separate test with its own fixed settings, so the bars do not form a trend, and 256K being lower than 128K is not a scaling result. You can reproduce all four points with one script in the repo.

Answer quality at 256K tokens on DeepSeek-V4-Flash

Change in F1 score (answer word overlap) vs. full recomputation, in percentage points. Right of zero = RedKnot-MLA is better.

On average F1 goes up by 3.2 points (0.572 → 0.605) and exact match by 4.2 points, while about 79% of the main computation is skipped. To be fair, one dataset (MultiFieldQA) drops by 2.8 points, so we don't claim quality never drops.

More users per GPU

Concurrent sessions that fit in one GPU's memory

Higher is better. Standard dense KV storage vs. RedKnot with per-head SegPagedAttention.

Local heads store only their short window, so each session needs much less memory. At 32K tokens one GPU holds 31 sessions instead of 4 (7.8×). At 64K it holds 14 instead of 3 (4.7×).

Why it can beat full recomputation

Some results above beat the baseline. That sounds odd for a method that skips work, but there is a reason. As prompts grow, attention focuses on fewer tokens, and the rest acts like noise. Full recomputation lets that noise into every layer. RedKnot's sparse steps keep the tokens and heads that carry the signal. On Qwen3.5-397B and DeepSeek-V4-Flash, full recomputation loses accuracy as context grows, while RedKnot stays steady and overtakes it at long context.

A note on the numbers. All figures come from the two papers. Each was measured on a fixed model, prompt set, and GPU setup, and the papers list the exact settings. Results from the open-source code take priority over the paper: if you run the code and get different numbers, the code results win, and please tell us.

Like the idea? Cite us and build with us.

If RedKnot is useful for your research or product, please cite our papers. It helps the project and the people who build it.

RedKnot is fully open source (Apache-2.0) and belongs to its contributors. We would love your help:

  • Try it: one command reproduces the DeepSeek-V4-Flash results.
  • Open an issue: bug reports, questions, and requests for the next model to support are all welcome.
  • Send a pull request: there is open work on hybrid models (Qwen3.5 → Qwen4, GLM-5.3), DeepSeek-V4-Pro, the vLLM port, and the Ascend NPU port.

Meaningful contributions earn a bigger voice in the community.

★ Star on GitHub Open an issue Read the paper 
@article{liu2026redknot,
  title   = {RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention},
  author  = {Liu, Yang and Luo, Zhaokai and Jin, Huayi and Wang, Zhiyong and He, Ruozhou and Wang, Boyu and Chen, Guanjie and Liu, Yifei and Xie, Tao and Hu, Junhao},
  journal = {arXiv preprint arXiv:2606.06256},
  year    = {2026}
}

@article{liu2026redknotmla,
  title   = {RedKnot-MLA: Multi-Head Offline-Online Reuse for DeepSeek-V4 Long-Context Serving},
  author  = {Liu, Yang and Luo, Zhaokai and Jin, Huayi and He, Ruozhou and Hong, Chenchen and Ma, Mingxiao and Zhang, Biao and Wang, Zhiyong and Wang, Boyu and Chen, Guanjie and Liu, Yifei and Xie, Tao and Hu, Junhao},
  journal = {arXiv preprint arXiv:2609.07008},
  year    = {2026}
}