Prefill–Decode Disaggregation

Lecture 4 ended on an uncomfortable note. You learned a beautiful trick, chunked prefill, which lets a long prompt share a step with many decodes by capping the chunk size \(c\). But the trick has a knob, and the knob is cruel: whatever you give to prefill, you take from decode, and vice versa. You tuned and tuned, and the best you could do was pick one point on a trade-off curve where both of your SLOs were a little bit unhappy.

This lecture is about the moment you stop tuning the knob and remove it. The idea is as physical as it gets: prefill is a compute-hungry GEMM machine, decode is a bandwidth-hungry memory machine. So give each its own pool of GPUs, tuned its own way, on its own hardware if you like. A request is born in the prefill pool; its KV cache is then shipped to the decode pool, where it lives out its days generating tokens. The price of admission is one moving truck: the KV transfer. Most of this lecture is about quantifying that truck, hiding it, and sizing the two pools so the whole arrangement is worth the fuel.

Why One Pool Cannot Satisfy Two SLOs: The Zero-Sum, Formalized

Recall the continuous-batching step from Lecture 4: each iteration runs \(B\) decode tokens (one per active request) plus one prefill chunk of \(c\) tokens. For a dense model with \(P\) parameters in BF16 (weight bytes \(= 2P\)), one iteration costs \[ \text{FLOPs}_{iter} = 2P\,(c + B), \qquad \text{bytes}_{iter} = 2P + B\bar{L}\,kv + c\,kv, \] where \(\bar{L}\) is the mean resident context of the decode batch (every decode step re-reads the whole KV cache of every active request, Lecture 1) and \(c\,kv\) is the chunk's freshly written KV. The roofline then bounds the step time: \[ t_{iter}(c) \ge \max\!\left(\frac{2P\,(c+B)}{\pi_{eff}},\, \frac{2P + B\bar{L}\,kv + c\,kv}{\beta}\right). \]

Now here is the trap, in one line. A prompt of length \(L\) needs \(L/c\) such iterations to prefill, and each of those iterations is also a decode step for all \(B\) requests. Both SLOs are functions of the same knob: \[ \mathrm{TTFT}(c) \approx \frac{L}{c}\, t_{iter}(c), \qquad \mathrm{TPOT}(c) \approx t_{iter}(c). \] This is the zero-sum: raising \(c\) shortens TTFT but stretches every decode step; lowering it protects TPOT but stretches TTFT. One pool, two SLOs, one knob: you can only slide along the curve.

Disaggregation simply deletes the shared knob. Two pools, two instances of the same bound, no cross-terms: the prefill pool runs whole prompts (\(B = 0,\; c = L\)), the decode pool runs pure decode (\(c = 0\)): \[ \mathrm{TTFT} \approx \underbrace{\frac{2PL}{\pi_{eff}^{(P)}}}_{\text{prefill compute}} \;+\; t_{queue}^{(P)} \;+\; t_{transfer}, \qquad \mathrm{TPOT} \approx \underbrace{\frac{2P + B\bar{L}\,kv}{\beta_{agg}^{(D)}}}_{\text{decode step}}. \] Each SLO now depends only on its own pool's capacity and queue. The trade-off curve has collapsed into two independent points. That is the whole trick. Everything else in this lecture is the price tag.

Two Phases, Two Optima, and Two Independent Autoscalers

Why does physical separation beat a well-tuned \(c\)? Because the two phases disagree about everything, not just the token budget:

The decode intensity formula from Lecture 1, \(I \approx 2PB/(2P + B\bar{L}kv)\), makes the batching row precise: decode only nears the ridge when \(B\) is large, and large \(B\) needs a large KV pool, which is exactly what a bandwidth/capacity-oriented configuration gives. Prefill sits at \(I \approx L\) (hundreds to thousands) and is compute-bound on any realistic hardware; extra batching buys it almost nothing. In one pool, both phases share one parallel config and one hardware SKU; disaggregated, each takes its own optimum.

And because the SLOs decoupled, the scalability signals decouple too:

• Input-traffic surge (a batch of long-document jobs)? Scale out only the prefill pool.
• More concurrent sessions (more simultaneous conversations)? Scale out only the decode pool.

A single-pool deployment mixes the two pressures and scales the whole pool for either one. It pays for compute when it needs memory, and vice versa.

The Moving Truck: KV Transfer and the Feasible Region

Disaggregation is not free: every request's KV must physically move from the prefill pool to the decode pool. Time to put numbers on the truck. From Lecture 1, one request carries \[ V_{KV} = L \cdot kv, \qquad kv = 2 \cdot n_{layers} \cdot n_{kv} \cdot d_h \cdot b_{dtype} \;\; \text{bytes/token}, \] and over a link of bandwidth \(BW_{link}\) the serialized transfer takes \[ T_{transfer} = \frac{L \cdot kv}{BW_{link}}. \]

Take the canonical inter-node link (400 Gbps InfiniBand ≈ 50 GB/s per GPU, about \(18\times\) below the 900 GB/s NVLink domain) and our two canonical models:

Look at the GQA row. Against a typical 2 s TTFT SLO, that truck alone would eat 40%+ of the budget. The same 40 GiB crosses an NVLink domain at 900 GB/s in ≈ 48 ms. So the conclusion: cross-node PD disaggregation of large dense GQA models is basically infeasible at long context. It needs in-NVLink-domain transfer or KV compression. This is a system-level benefit of everything Lecture 2 did to \(kv\): MLA and quantization enlarge the feasible region of PD disaggregation directly.

Let us make "feasible" precise. Declare a transfer budget \(t_{budget}\), the fraction of the TTFT SLO you are willing to spend on the wire. Transfer is feasible iff \[ \frac{L \cdot kv}{BW_{link}} \le t_{budget} \qquad\Longleftrightarrow\qquad kv \le kv_{max} = \frac{t_{budget} \cdot BW_{link}}{L}. \] Note the \(1/L\): every 4× of context growth shrinks the admissible bytes-per-token 4×.

One more reading of feasibility, and this one no latency-hiding trick will ever fix. The fleet's aggregate transfer demand is \[ \lambda \cdot L \cdot kv \;\; \text{bytes/s}, \] and it must fit inside the deployed link capacity. At 128K context, one GQA-70B request per second already demands ≈ 43 GB/s: essentially a whole 400 Gbps link per request per second. Latency can be hidden; bandwidth cannot. Keep this sentence; we will need it again in a moment.

Hiding the Truck: Layer-Wise Pipelining and NIXL

Prefill produces KV layer by layer. So why wait for the whole model to finish before shipping? Once layer \(i\) is computed, its KV can start moving immediately, overlapping the computation of layer \(i+1\). The transfer then wraps up shortly after prefill ends. The only exposed remainder is the last layer's transfer.

The derivation is two lines. Layer \(i\) computes all \(L\) tokens in \(t_{comp} = 2PL/(n_{layers}\,\pi_{eff})\) and produces \(L \cdot kv/n_{layers}\) bytes, moving in \(t_{xfer} = L\,kv/(n_{layers}\,BW_{link})\). Full overlap of every layer but the last needs \(t_{comp} \ge t_{xfer}\): \[ \frac{2PL}{n_{layers}\,\pi_{eff}} \ge \frac{L\,kv}{n_{layers}\,BW_{link}} \quad\Longleftrightarrow\quad \frac{2P}{\pi_{eff}} \ge \frac{kv}{BW_{link}}, \] and both \(L\) and \(n_{layers}\) cancel. Overlap feasibility is a pure model/link property, independent of context length. When it holds, the exposed latency after a fully pipelined prefill is just \[ T_{exposed} \approx \frac{T_{transfer}}{n_{layers}} = \frac{L\,kv}{n_{layers}\,BW_{link}}. \]

Pipelining only works if transfers don't steal compute. The enabling mechanism is one-sided RDMA: the transfer occupies neither GPU's compute stream. Within a pool pair this is concretely NIXL (NVIDIA Inference Xfer Library), as used by Dynamo: KV is written directly from the prefill engine's VRAM into the decode engine's VRAM while forward passes at both ends keep serving other requests. Its operation, in the form relevant here:

1. Registration. At init, every worker registers memory descriptors of all its KV blocks with a metadata service (etcd). Afterwards, cross-node addressing needs only a block id; no addresses ride the hot path.
2. One-sided transfer. RDMA read/write: the prefill worker writes directly into (or the decode worker reads from) pre-allocated remote KV blocks, with no remote CPU or GPU involvement.
3. Automatic backend selection. NVLink inside a domain; IB/RoCE (via UCX) across nodes; plus NVMe and object-storage backends for tiered storage (Lecture 2). One interface covers both PD transfer and KV offload; the transfer layer is shared infrastructure.

Layout rearrangement, the underestimated part. The two pools usually run different parallel configs (say TP=4 on the prefill side and a large EP+TP mesh on the decode side), so KV is sharded differently at the two ends: which head of which layer lives on which card changes across the transfer. The transfer must rearrange KV mid-flight, decomposing into many non-contiguous, multi-target scatter writes (one source block → several destination cards). This is exactly why NIXL emphasizes non-contiguous transfer semantics. Ignore it and it quietly eats the gains.

Sizing the Two Pools: The xPyD Ratio

Disaggregation hands you a brand-new design variable: \(x\) prefill instances paired with \(y\) decode instances. The ratio is determined by the workload, not by hardware, and we can build it straight from the Lecture-1 spine.

Prefill pool: compute demand. Requests arrive at rate \(\lambda\) with mean input length \(L_{in}\); a fraction \(h\) of the prefix is already cached (Lecture 2), so only \((1-h)L_{in}\) tokens per request need real compute. With FLOPs/token ≈ \(2P\) plus the attention correction \(1 + L/L^*\) approaching the crossover \(L^* = 2P/(4\,d_{model}\,n_{layers})\) (Lecture 3): \[ \text{prefill demand} = \lambda\,(1-h)\,L_{in} \;\text{tok/s}, \qquad \text{per-GPU supply} \approx \frac{\pi_{eff}}{2P\,(1 + L_{in}/L^*)} \;\text{tok/s}, \] \[ x = \left\lceil \frac{\lambda\,(1-h)\,L_{in}}{G_P \cdot \text{supply}_{GPU}} \right\rceil \;\; \text{instances of } G_P \text{ GPUs}. \] Note the hit-rate discount sitting in the numerator: the prefix hit rate directly shrinks the prefill pool. Caching is capacity, not just latency.

Decode pool: capacity and bandwidth. By Little's law, mean concurrency is \(C = \lambda\,T_{sess}\) with \(T_{sess} \approx \mathrm{TTFT} + L_{out}\cdot\mathrm{TPOT}\). A decode instance of \(G_D\) GPUs (aggregate bandwidth \(\beta_{agg} = G_D\,\beta\), weight footprint \(W\)) supports a batch bounded two ways: \[ B_{cap} = \min\Big( \underbrace{\frac{G_D \cdot \mathrm{HBM} - W}{kv\,(L_{in} + L_{out}/2)}}_{\text{KV capacity}},\; \underbrace{\frac{(\mathrm{TPOT}_{SLO} - W/\beta_{agg})\,\beta_{agg}}{kv\,(L_{in} + L_{out}/2)}}_{\text{TPOT bandwidth}} \Big), \qquad y = \lceil C / B_{cap} \rceil, \] the first bound from the KV-pool size divided by per-session KV, the second from the step-time budget \((W + B \cdot kv \cdot L_{ctx})/\beta_{agg} \le \mathrm{TPOT}_{SLO}\).

Every quantity in these formulas is a moving target: the input-vs-output mix and the hit rate drift over hours and days. A fixed \(x{:}y\) sized for the morning mix is wrong by the evening. Production systems therefore implement dynamic P/D role conversion: instances switch roles (and reload/re-shard accordingly) instead of the partition being static; Mooncake and Dynamo both do this, and the capacity model above is what tells the controller which direction to convert.

The Short-Request Exception: Conditional Disaggregation

Hold on. The remote path carries fixed costs: extra queueing, two rounds of scheduling (decode-side allocation, then prefill execution), control traffic, the transfer itself. For a short prompt, local prefill is tens of milliseconds, and the remote path can be slower end-to-end. So... when is remote actually worth it? Let us derive the threshold instead of guessing.

Let \(O_{fixed}\) be the fixed remote overhead (queue + scheduling + control) and \(k_P\) the number of GPUs that would serve a local prefill. Local vs remote: \[ t_{local}(L) = \frac{2PL}{k_P\,\pi_{eff}}, \qquad t_{remote}(L) = O_{fixed} + \frac{L\,kv}{BW_{link}} + \underbrace{\frac{2PL}{x \cdot \text{supply}}}_{\approx\,0}, \] where the remote compute term vanishes because the prefill pool is provisioned so that queueing, not compute, dominates. Remote wins when \(t_{remote} < t_{local}\), i.e. beyond \[ L_{thresh} = \frac{O_{fixed}}{\dfrac{2P}{k_P\,\pi_{eff}} - \dfrac{kv}{BW_{link}}}. \]

Dynamo implements exactly this: it dynamically decides local vs remote prefill based on input length (and current load), with a configurable length threshold. Production PD is therefore hybrid: "long prefill disaggregated, short prefill in place."

The Global Queue and the Mooncake Form

With multiple prefill instances, requests must be load-balanced across them. The naive flow (send a request to some prefill instance, then later negotiate the KV handoff) turns out to be backwards. The production pattern (Dynamo, built on NATS) inverts it:

1. The decode-side worker allocates the KV blocks for the incoming request first.
2. It pushes a remote-prefill request, carrying the destination block ids, into a global prefill queue.
3. Prefill workers pull tasks from the queue, execute, and on completion write KV directly into the pre-allocated blocks via NIXL one-sided writes.

Pre-allocation is what makes the transfer fire-and-forget: the prefill worker never negotiates with the decode worker. Cache-aware routing (Lecture 2) then layers on top: prefer the prefill instance that already holds a matching prefix. The \((1-h)\) discount applies per routing decision, not just at capacity-planning time.

Mooncake carries the idea to its endpoint. Put a distributed, tiered (HBM / DRAM / SSD) KV pool at the center of the architecture, and let both GPU pools flow through it: a prefill instance first fetches the hit prefix KV, computes only the missed incremental part, and writes new KV back; a decode instance fetches the full KV from the pool (or straight from prefill) and starts generating. In this form, Lecture 2's cache infrastructure and this lecture's PD transfer merge into a single data path, and the transfer layer (an RDMA transfer engine / NIXL) is the common foundation. PD disaggregation, prefix caching, and tiered KV storage stop being three features. They become one system organized around KV movement.

So When Is Disaggregation Actually Worth It?

Worth it:

• Interactive workloads with dual SLOs on TTFT and TPOT. This is the setting where the single-pool trade-off curve actually hurts.
• Long inputs or highly fluctuating input/output ratios. The two pressures scale differently, so independent scaling saves real capacity.
• Existing cross-instance KV infrastructure. The transfer layer and metadata service are already paid for.

Not worth it:

• Offline, throughput-oriented workloads. No SLO conflict exists; disaggregation only adds transfer overhead. One saturated pool with large chunks wins.
• Very-short-input workloads. Conditional disaggregation degenerates to almost entirely local prefill; the decode-side second pool is idle weight.
• No high-speed interconnect. When \(T_{transfer}\) eats the gains, the feasible region is empty.

And how does this sit with Lecture 4's chunked prefill? The two are not mutually exclusive. Under disaggregation, the prefill pool can still chunk internally (to bound activation memory or mix priorities), and the occasional short local prefill inside the decode pool is mixed in as chunks:

Research Thinking

How to: read the starting point; read the question and think, for a minute, a day, a week... and only then open the answers. You are not supposed to reinvent several months of someone's research; it's the habit of thinking that counts.

Have Fun! The Moving Truck Calculator

Enough reading. Now you drive the truck. Pick a model, a context length, and a link, and see whether your KV fits inside a 200 ms transfer budget. Can you find a setting where pipelining does not rescue the GQA model? (And once you can't, try saying why in one sentence.)

Seminar & Homework

Take Lab 5: PD Disaggregation from the course's hands-on pack. On two GPUs you will:

• Deploy both architectures: a colocated baseline (one vLLM instance, TP=2) versus a 1P1D disaggregated pair: two single-GPU instances launched as KV producer/consumer with --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":...}' behind the vLLM disaggregated-prefill proxy.
• Measure goodput under the dual SLO (TTFT < 2 s, TPOT < 50 ms) on long inputs (\(L \in \{8\text{K}, 32\text{K}\}\), 128 outputs, --ignore-eos), with an open-loop rate sweep 0.25–4 req/s.
• Microbenchmark the truck: time KV movement for \(L\) = 8K/32K/64K, fit time vs \(L\), extract the effective bandwidth, and check with nvidia-smi nvlink / dcgmi whether you are actually on NVLink or PCIe (predicted: 4 GiB at 32K takes 4.8 ms NVLink, 67 ms PCIe, 86 ms IB). Measure, don't assume.
• Verify pipelining: at 32K, per-layer compute ≈ 33 ms ≫ per-layer PCIe transfer ≈ 2.1 ms, so exposed transfer should be ≈ \(67/32 \approx 2\) ms. Compare pair TTFT against baseline + predicted serialized transfer; the difference is what pipelining hides.

Analysis questions to answer in your report: Why does goodput under a dual SLO separate the two architectures more sharply than raw throughput? Where does the proxy/router itself appear in your TTFT decomposition? And at what \(\lambda\) does \(\lambda L\,kv\) saturate your link? Watch the pitfalls: attributing proxy/queue delay to transfer (decompose TTFT!), mismatched configs between baseline and pair, and single-point rate tests that miss the goodput knee.

Summary

• One pool ⇒ one knob: \(\mathrm{TTFT}(c) \approx \frac{L}{c}\,t_{iter}\), \(\mathrm{TPOT}(c) \approx t_{iter}\). This is a zero-sum whose whole curve lies above the two private optima (181 ms TPOT and 1.45 s TTFT, vs 93.1 ms and 1.16 s privately, in the 70B example).
• PD disaggregation gives each phase its own pool: TTFT and TPOT decouple, configs and hardware specialize per side, and the pools autoscale on different signals.
• The price is the KV truck: \(T_{transfer} = L\,kv/BW_{link}\); feasibility means \(kv \le t_{budget} BW_{link}/L\). GQA-70B crosses the (50 GB/s, 200 ms) boundary between 8K and 32K; MLA survives to 128K; at 1M even MLA needs the NVLink domain. K3 moves 1M more cheaply than GQA-70B moves 128K.
• Layer-wise pipelining hides per-request latency down to \(T_{transfer}/n_{layers}\) (overlap condition is \(L\)-independent: \(2P/\pi_{eff} \ge kv/BW_{link}\)), but aggregate demand \(\lambda L kv\) is a throughput fact that no scheduling will hide.
• Size pools from the workload: prefill from \(\lambda(1-h)L_{in}\) (caching is capacity: 45 vs 444 GPUs), decode from concurrency under both KV-capacity and TPOT-bandwidth bounds. The \(x{:}y\) ratio is a control output; convert roles as the mix drifts.
• Short prompts stay local: \(L_{thresh} = O_{fixed}/\left(\frac{2P}{k_P\pi_{eff}} - \frac{kv}{BW_{link}}\right)\) ≈ a few hundred tokens, set by fixed overhead, not the wire.
• Production form: decode-side pre-allocation → global prefill queue → fire-and-forget one-sided writes; in the Mooncake form this merges with tiered prefix caching into one system organized around KV movement.