Parallelism Strategies: DP, TP, PP, EP, CP and Large-Scale MoE Serving

Your 70B model arrives at the datacenter as 140 GB of BF16 weights. Your GPU holds 80 GB. Before a single token of KV cache, before a single request, the model simply does not fit. And even when a smaller model does fit, Lecture 1 handed you a second disappointment: one H100's \(\beta \approx 3.35\) TB/s of memory bandwidth caps a 70B decode at \(2P/\beta \approx 42\) ms per token. That is about 24 tok/s: drip, drip, drip. Two different walls, one direction out: more GPUs. This lecture is the map of how to cut, and the price tag of every cut.

Why More Than One GPU?

There are exactly two reasons, and you should be able to name both in your sleep:

1. Capacity. Weights plus KV cache do not fit in one GPU's HBM. A 70B model in BF16 is \(2P = 140\) GB, twice an H100's 80 GB, before a single token of KV. DeepSeek-V3's 671B parameters in FP8 (≈ 671 GB) overflow even a full 8-GPU node: \(671 > 640\) GB just to be resident.
2. Performance. Decode is bandwidth-bound (Lecture 1): every step streams \(2P + B \cdot kv\) bytes through HBM. Split those bytes across \(T\) GPUs and bandwidth aggregates to \(T\beta\): ideally TPOT divides by \(T\), and the \(2P/\beta\) latency floor from Lecture 1 becomes \(2P/(T\beta)\).

Inference borrows its whole parallel toolbox from training (tensor, pipeline, data, expert, and context parallelism all originated there), but it plays the game under different constraints:

• No gradient synchronization. Training's dominant collective (a gradient all-reduce every step) disappears. At inference, communication carries only activations and KV state, placed wherever the sharding scheme puts them.
• Decode has almost no compute per step. One token per request means there is no compute bulk to hide communication behind (prefill has it; training has it; decode does not). Inference is therefore far more sensitive to communication latency than training. This single fact drives nearly every design decision in this lecture.

The 18× Wall: Where Collectives May Live

Each strategy binds to exactly one communication primitive, and the primitive's cost is decided by the machine's topology. Meet the cast first:

Now the hardware those primitives run on. Inside one node, 8 GPUs form a fully connected NVLink clique at ≈ 900 GB/s (bidirectional). Between nodes, IB/RoCE RDMA delivers ≈ 400 Gbps, about 50 GB/s per GPU. Look at the ratio: \[ \frac{900}{50} = 18. \] The 18× gap is the single most important number in distributed serving. GB200 NVL72 stretches the NVLink domain from 8 to 72 GPUs. But notice: the hierarchy does not disappear, only the location of the boundary moves. Everything in this lecture phrased as "inside vs. outside the NVLink domain" applies verbatim with 72 substituted for 8.

Two Phases, Two Cost Structures

Same scheme, same primitive, and yet prefill and decode feel completely different pains, because message sizes differ by orders of magnitude. Take the TP all-reduce message:

• Prefill handles thousands of tokens per step: the message is \(T_{chunk} \cdot d_{model} \cdot b_{dtype}\). At \(T_{chunk} = 4096\), \(d_{model} = 8192\), BF16: \(4096 \times 8192 \times 2 = 64\) MB. Even on 50 GB/s RDMA that's \(64\,\text{MB}/50\,\text{GB/s} \approx 1.3\) ms: bandwidth-dominated.
• Decode handles one token per request: the same all-reduce carries only \(B \cdot d_{model} \cdot b_{dtype}\). At \(B = 32\): \(32 \times 8192 \times 2 = 512\) KB, whose transmission even on NVLink is ≈ 1.2 µs, dwarfed by the ∼ 10 µs fixed latency of the collective. Latency-dominated.

Formally, a collective is latency-dominated when \(m/\beta_{net} \ll t_{lat}\) and bandwidth-dominated when \(m/\beta_{net} \gg t_{lat}\): decode lives in the first regime, prefill in the second. File this away: it is why the two pools of a PD-disaggregated deployment (Lecture 5) may legitimately choose different parallelism degrees. The prefill pool optimizes a bandwidth problem; the decode pool, a latency problem.

Tensor Parallelism (TP): Buy TPOT with All-Reduces

The easiest thing you can do with several GPUs: cut every weight matrix into \(T\) pieces and give each GPU one piece. Megatron-style TP does it so that communication stays minimal: the MLP's up-projection \(W_1 \in \mathbb{R}^{d \times d_{ff}}\) is split column-wise (\(W_1 = [W_1^{(1)}\,|\,\cdots\,|\,W_1^{(T)}]\)), so each GPU computes a \(1/T\) slice of the intermediate activations with no traffic; the down-projection \(W_2\) is split row-wise conformally, so each GPU's partial output is already a partial sum of the full product; one all-reduce completes the layer. Attention is split by head: with \(n_h\) query heads and \(T\) GPUs, each GPU owns \(n_h/T\) heads (queries, keys, values, its own slice of the KV cache) and a column-slice of \(W_O\); one all-reduce after \(W_O\) merges the partial sums.

What does TP buy? Weights and KV divide evenly by \(T\): per GPU, the bytes read per decode step become \[ \frac{2P}{T} + \frac{B \cdot kv}{T} \qquad (3.6.2) \] Aggregate HBM bandwidth scales to \(T\beta\), memory capacity aggregates to \(T \times 80\) GB, and TPOT ideally decreases linearly in \(T\). TP is the most direct lever on single-request latency.

The TPOT model under TP

Assumptions, honestly stated: (i) decode is memory-bound, so step time is bytes-read over bandwidth; (ii) sharding is perfectly even; (iii) each all-reduce costs a fixed \(t_{lat} \approx\) 10 µs, because decode messages are small (see the previous section); (iv) two all-reduces per layer, not overlapped with compute: they sit on the critical path, since the next GEMM needs the reduced activation. Then the memory term is (3.6.2) divided by \(\beta\), plus the communication term: \[ \mathrm{TPOT}(T) \approx \underbrace{\frac{2P/T + B \cdot kv / T}{\beta}}_{\color{#5d8120}\text{HBM streaming}} + \underbrace{2\, n_{layers} \cdot t_{lat}(T)}_{\color{#4a6fa5}\text{all-reduce latency}} \qquad (3.6.3) \] Sanity checks before trusting it: as \(T \to \infty\) the first term vanishes but the second does not, so the curve is linear-then-flat, floored at \(2 n_{layers} t_{lat}\). And at \(T = 1\) there are no peers, so the true comm term is zero: (3.6.3) overstates the \(T=1\) row by \(2 n_{layers} t_{lat}\). Keep that in mind when reading the table below.

The three limits of TP

1. Latency overhead grows with degree, and cannot cross nodes. Two calls per layer × 60–80 layers × ∼ 10 µs is the fixed overhead above. Cross-node, the same 512 KB message pays \(512\,\text{KB}/50\,\text{GB/s} \approx 10\) µs of transmission alone, plus inter-node RTT and NCCL ring hops. Verdict: cross-node TP all-reduce latency is unusable for interactive decode. Hence the rule: TP degree ≤ NVLink domain size (8 on H100 HGX, 72 on GB200 NVL72).
2. KV-head count bounds KV sharding under plain TP. Attention is sharded by head, so the number of KV heads caps KV sharding. Llama-3.3-70B has only \(n_{kv} = 8\): at \(T = 8\) each GPU holds exactly one KV head. At \(T = 16\), KV heads must be replicated (two GPUs per head): the per-GPU KV read stops decreasing: each GPU still reads one eighth of the logical cache, rather than one sixteenth. Weight reads keep shrinking. DCP below lets those two GPUs store different token positions for their shared KV head.
3. MLA leaves its latent cache replicated under plain TP. An MLA latent cache (Lecture 2) is a single shared latent vector per token (576 elements per layer for DeepSeek-V3), not per-head tensors: there is nothing to shard by head. TP-sharding attention just replicates the full latent cache on every GPU: aggregate cache storage ×\(T\), with no reduction in per-GPU KV bytes. Attention DP and TP combined with DCP offer two ways to address this, with different request placement and communication costs.

Pipeline Parallelism (PP): Capacity That Crosses Nodes

You probably can guess why TP alone cannot save a 405B model: even at \(T = 8\), you need \(\ge 810/8 \approx 101\) GB of weights per GPU, more than any H100 has. The next cut is coarser: instead of slicing every matrix, give each GPU group whole layers. PP splits the model into \(s\) stages; each stage holds only its own layers' weights and those layers' KV cache. Between stages flows only the hidden state \(B \times d_{model} \times b_{dtype}\) (the same 512 KB at \(B = 32\)) as one-way P2P traffic. And here is the key difference from TP: this traffic is hideable. There is no partial sum on a critical path; a 512 KB hop costs ≈ 10 µs of transmission plus latency over RDMA, and it can be overlapped with the next microbatch's compute. PP is the only cross-node-friendly model sharding.

But you probably can guess PP's flaw too: a pipeline only earns its keep when several microbatches are in flight. With \(s\) stages and \(m\) microbatches per flush, the fill/drain phases idle \(s - 1\) stage-times against \(m + s - 1\) total: \[ \text{bubble fraction} \approx \frac{s - 1}{m + s - 1} \qquad (3.6.4) \] Check the limits with me: \(m \to \infty\) gives 0 (the throughput regime); \(m = 1\) gives \((s-1)/s\) (a single request fills only one stage at a time: utilization \(1/s\)). Numbers: \(s{=}4, m{=}8 \Rightarrow\) 27%; \(s{=}8, m{=}32 \Rightarrow\) 18%; \(s{=}8, m{=}4 \Rightarrow\) 64%. And here is the killer for interactive serving: \(m\) is not yours to choose. It fluctuates with the arrival process. At low concurrency few requests are available to fill the pipe, stages idle, and the bubble becomes the dominant cost exactly when latency SLOs matter most.

Two more fixed taxes. Every token sequentially traverses all \(s\) stages, so TTFT and TPOT each absorb \(s - 1\) inter-stage hops plus per-stage scheduling overhead: \(7 \times (\sim 20\,\mu s) \approx 0.14\) ms/token at \(s = 8\), before any bubbles. And, crucially: each stage still streams its own layers at full size every step, so PP does not reduce total weight bytes read per token. No TPOT bandwidth benefit, only capacity.

Expert Parallelism (EP): Turning Sparsity into Bandwidth

Now switch models. DeepSeek-V3, the reference MoE of this course, has 256 routed experts + 1 shared expert per layer, top-8 activated per token, 61 layers (58 MoE; the first 3 are dense), \(d_{model} = 7168\). Each V3 routed expert is \(3 \times 7168 \times 2048 \approx 44.0\)M params ≈ 44 MB in FP8.

EP vs. TP for MoE: who converts sparsity into bandwidth?

Compare the two ways to shard the same MoE layer across 8 GPUs:

• TP-8 cuts every expert matrix into 8 pieces. Every GPU holds a shard of all 256 experts, so each decode step it streams \(1/8\) of all expert weights: \(256/8 = 32\) experts' worth ≈ \(32 \times 44\,\text{MB} = 1.41\) GB. The fact that each token uses only 8 experts never reduces the bytes read: sparsity is not converted into bandwidth savings.
• EP-8 gives each GPU 32 whole experts. At realistic batch sizes it still streams ≈ all 32 resident experts, so EP-8 ≈ TP-8 in bytes... but EP's reads scale as (resident experts per GPU) \(= 256/E\), which keeps falling as \(E\) grows, while TP's per-GPU read is stuck at \(1/T\) of everything.

Why wide EP: the EP144 logic

DeepSeek's published V3/R1 deployment: prefill unit EP32 (4 nodes, 32 GPUs), decode unit EP144 (18 nodes, 144 GPUs). Why so wide on the decode side? Three chained reasons:

1. Minimize resident weights per GPU. Under EP144 each GPU holds \(\lceil 256/144 \rceil \approx 2\) routed experts per layer (≈ 88 MB FP8). Per-GPU weight bytes read per decode step (the denominator term of Lecture 1's intensity formula) is minimized, and the freed HBM all goes to KV cache, enlarging the batch ceiling.
2. Expert-granularity batch aggregation. An expert's GEMM processes only the tokens routed to it, so its arithmetic intensity is \[ I_{expert} \approx \frac{2 \cdot P_{expert} \cdot t_e}{P_{expert} \cdot b_{fp8}} \approx 2\, t_e \ \text{FLOP/byte} \qquad (3.6.6) \] where \(t_e\) = tokens hitting that expert per step. Against the H100 ridge \(\pi/\beta \approx 295\), an expert GEMM becomes compute-bound only at \(t_e \gtrsim 148\). A small single-GPU pool with \(B_{tot} = 8\) gets \(t_e = 8 \times 8/256 = 0.25\): hopeless GEMM shapes, pure weight streaming. A 144-GPU pool gathers all concurrent tokens and regroups by expert: at \(B_{tot} = 1024\), \(t_e = 32\) (\(I \approx 64\)); at a large decode batch \(B_{tot} = 18{,}432\) (128 per GPU), \(t_e = 576\), \(I \approx 1152\), firmly compute-bound. This is Lecture 1/4's batching logic replayed along the expert dimension, and it requires a wide pool: the aggregation ceiling is the pool's total concurrent tokens.
3. Prefill needs no extreme EP. Prefill is compute-bound (its tokens already saturate the GEMMs), so EP32 suffices; sizing the two pools independently is exactly the freedom PD disaggregation (Lecture 5) provides.

Load balancing: EPLB. Routing is data-dependent: some experts are hot, and the GPUs holding them become the bottleneck of both the all-to-all (everyone waits for the slowest destination) and the compute (straggler GEMM). The expert-parallelism load balancer periodically rearranges placement from routing statistics, with two tools. Redundant experts: a hot expert is replicated across GPUs and the router balances among replicas (at \(B_{tot} = 18{,}432\), an expert running 4× hot serves \(4 \times 576 = 2304\) tokens, \(I \approx 4600\), a straggler GEMM 4× the average cost; splitting it into 4 replicas restores \(t_e = 576\)). Topology-aware placement: experts frequently co-activated by the same token sit on the same node where possible, shrinking cross-node all-to-all traffic. EPLB is a control-loop cost (statistics + occasional weight movement), amortized against a permanent straggler tax on every decode step.

LatentMoE: Kimi K3's 2× all-to-all reduction

Kimi K3 (Moonshot AI, weights 2026-07-27, the canonical 2026 MoE of this course) scales the DeepSeek-V3 recipe to 896 routed + 2 shared experts, top-16, at the same \(d_{model} = 7168\). Substitute into (3.6.5) and the news looks bad: top-16 would double the per-token all-to-all volume to \(16 \times 7168 \times 3\,\text{B} = 344\) KB. LatentMoE cancels exactly that factor of two by changing what crosses the network. Define the latent dispatch width \(d_{lat} = 3584\): the routed path down-projects each token \(7168 \to d_{lat}\) before dispatch (fused with the router GEMM, so no extra kernel is launched), and up-projects \(d_{lat} \to 7168\) after combine. Equation (3.6.5) then holds with \(d_{lat}\) in place of \(d_{model}\) on the routed path, and the per-(token, expert) payload is exactly halved: \(3584 \times 3 = 10{,}752\) B vs. \(7168 \times 3 = 21{,}504\) B. The 2 shared experts run full-width locally and never enter the all-to-all: zero network bytes.

The Standard Form: Attention DP + MoE EP

If experts live pool-wide under EP, what happens to attention in these deployments? Surprisingly, the simplest thing of all: data parallelism. Each GPU maintains an independent request batch (local batch, local KV, local attention) and pool-wide communication happens only at the MoE-layer boundary (the two all-to-alls). Why this beats large TP for attention:

1. No 2× all-reduce per layer. The 1.6 ms fixed overhead of (3.6.3) disappears entirely from the attention blocks.
2. MLA's shared latent cache is not replicated. TP-Limit-3 dissolves: with DP, each GPU caches only its own requests' latents, 34 KB/token for V3, sharded by request rather than replicated by degree.

The cost is lockstep: all DP ranks' batches must advance together, because the whole pool enters each all-to-all every step. A rank with fewer tokens this step must pad to the maximum batch or wait. For example, eight ranks at local batches \([120, 118, 126, 90, 122, 124, 119, 121]\) padded to 126 waste ≈ 7% of expert-GEMM capacity. The remedy is scheduler-side cross-rank request balancing, an extra scheduler duty (Lecture 4) that only exists in multi-GPU settings. The standard form of large-scale MoE serving is, then: \[ \text{attention DP}\ (+ \text{small TP})\ \times\ \text{MoE EP}, \qquad \text{two pool-wide all-to-alls per layer} \qquad (3.6.7) \] and whether the whole construction works is decided by the quality of the all-to-all implementation. DeepEP and Mega-MoE kernels are Lecture 7's subject.

Context Parallelism (CP): Sharding a Single Sequence

Four strategies so far shard the model (weights, experts, layers) or the request set (DP). But look at what's left untouched: one very long sequence can itself be too big. At 128K tokens, a GQA-70B request carries \(320\,\text{KB} \times 131{,}072 \approx 43\) GB of KV. That cache competes with weights for HBM and takes time to read at every step. CP cuts along \(L\). Lecture 3 developed the algorithm (ring attention); here we analyze it as a parallel strategy.

Prefill CP: O(L) communication hidden by O(L²) compute

With \(N\) GPUs each holding an \(L/N\) query block, ring attention passes KV blocks around the ring; per GPU per layer the received volume is \[ V_{CP} \approx \frac{N-1}{N} \cdot L \cdot kv_{layer}, \qquad kv_{layer} = 2\, n_{kv}\, d_h\, b_{dtype} \qquad (3.6.8) \] which is linear in \(L\) (4 KB/token/layer for Llama-3.3-70B). Attention compute per GPU per layer is Lecture 1's spine formula divided by \(N\): \[ F_{CP} \approx \frac{4 L^2 d_{model}}{N} \qquad (3.6.9) \] which is quadratic in \(L\). Communication hides inside compute when \(V_{CP}/\beta_{net} \le F_{CP}/\pi\), i.e. \[ L \ge L^*_{CP} = \frac{(N-1)\, kv_{layer}\, \pi}{4\, d_{model}\, \beta_{net}} \qquad (3.6.10) \] The common \(1/N\) factors cancel, but \(N-1\) remains: the threshold depends on group size, model shape, and the network. Substituting Llama-3.3-70B constants (\(kv_{layer} = 4096\) B, \(d_{model} = 8192\), \(\pi = 990\) TFLOPS) with \(N = 8\): cross-node RDMA (50 GB/s) gives \(L^*_{CP} \approx\) 17.3K tokens; intra-node NVLink (450 GB/s/dir) gives ≈ 1.9K. Direct check at \(L = 128\)K, \(N = 8\): per GPU per layer, comm \(= (7/8) \times 131072 \times 4096\,\text{B} \approx 470\) MB (9.4 ms at 50 GB/s) vs. compute \(= 4 \times 131072^2 \times 8192 / 8 \approx 70\) TFLOP (71 ms at 990 TFLOPS). Compute exceeds communication by 7.6×, and the ratio grows linearly in \(L\). CP is one of the few high-communication parallelisms suited to cross-node: for long sequences even an RDMA P2P ring hides inside the quadratic compute.

Decode Context Parallelism: Share the History, Keep the GPUs

You have already chosen eight GPUs to serve a model, but its attention has only four KV heads. Plain TP gives two GPUs a copy of each head's entire history. Doubling the GPUs from four to eight shrank the weight shards; the KV cache per GPU stayed the same. What could you put in those duplicate copies instead? Different token positions from the same history. This is the motivation for vLLM's decode context parallelism (DCP). The deployment guide uses Qwen3-235B-A22B, with four KV heads, as this example.

For the examples below, keep prefill context parallelism at its default, PCP=1. In this configuration, vLLM forms DCP groups inside the existing TP group: TP=8, DCP=2 still uses eight GPUs. Linear layers retain their TP layout. DCP changes how attention uses those GPUs. The current CLI reference also describes layouts that combine PCP and DCP; their group arrangement differs.

Where does each token's cache go?

Zoom in on one of the four KV heads. Call its two GPUs A and B. With DCP=1, both store tokens 0, 1, 2, 3, … . With DCP=2, you can give A tokens 0, 2, 4, … and B tokens 1, 3, 5, … . The next token goes to its owner, so the cache stays balanced as generation continues. This is a token-level illustration of interleaved storage; the engine can interleave larger groups of tokens. The same growth problem and a round-robin solution appear in Helix Parallelism, §2.3.

Let \(M_{KV}\) be one request's logical cache size across all layers, before replication; \(H\) its KV-head count; \(T\) the TP degree; and \(D\) the DCP degree. For equal-sized heads, even head partitioning, and a supported DCP layout, your cache accounting is \[ M_{KV,\mathrm{GPU}} \approx \frac{M_{KV}}{\min(T,H)D}. \] This ignores block rounding and metadata. For a hypothetical 16 GiB cache with \(H=4\), TP=8 alone stores \(16/4=4\) GiB per GPU: 32 GiB across the group. Add DCP=2 and each GPU holds \(16/(4\cdot2)=2\) GiB: 16 GiB across the same eight GPUs. You removed duplication; the request still has all its tokens.

MLA makes the opportunity especially clear: its shared latent cache acts like \(H=1\) for this accounting. Plain TP=8 stores eight copies; DCP=8 can distribute one copy across those eight GPUs. Conversely, GQA with \(H=8\), TP=8 already has no duplicate KV heads to reclaim. For the TP-only layouts in vLLM's DCP walkthrough, when \(T\ge H\), choose \(D\) as a divisor of \(T/H\). Thus \(H=4,T=8\) allows DCP=1 or 2; MLA at TP=8 allows 1, 2, 4, or 8. With \(T<H\), this layout uses DCP=1. DCP is not an extra factor to multiply blindly into an already sharded cache.

One decode step: move the query, then merge the answers

The diagram above gives you the route. Follow one query head:

1. Share the query. Each TP rank initially owns a subset of query heads. An all-gather within its DCP group makes the group's query heads available on every KV shard.
2. Read local KV. Every rank computes attention for those queries against its own token positions. It produces a locally normalized output \(o_i\) and a log-sum-exp \(\ell_i\) for each query head. The long cached history stays on its owning GPUs.
3. Restore the global softmax. In the all-gather/reduce-scatter implementation, ranks all-gather the LSE values, reweight their local outputs, then reduce-scatter the weighted outputs back to the TP head layout. See vLLM's merge implementation.

Formally, for one query \(q\), let \(S_i\) be the token positions on rank \(i\), and let \(s_j=q^\top k_j/\sqrt{d_h}\) be a token's scaled attention score. Store \(\ell_i=\log\sum_{j\in S_i}\exp(s_j)\). Then the correct merge is \[ a=\max_i\ell_i,\qquad w_i=\frac{\exp(\ell_i-a)}{\sum_r\exp(\ell_r-a)},\qquad o=\sum_i w_i o_i. \] In your example, \(\ell_A=\log3\), \(\ell_B=0\), so subtracting \(a=\log3\) gives unnormalized weights 1 and 1/3, or 3/4 and 1/4 after normalization. Subtracting the maximum keeps the exponentials numerically stable. You recover full attention mathematically, with ordinary floating-point differences from changing reduction order. This is the same split-and-merge principle used by Flash-Decoding within one GPU; DCP distributes the cache and merge across GPUs.

Note: the collective sequence depends on the implementation. vLLM also exposes --dcp-comm-backend a2a to exchange partial outputs and LSE together, and supported MLA paths can replicate the query projection to skip query all-gather. Check the CLI reference for your version. These choices change the communication cost, while preserving the same attention merge.

Does half the cache mean half the token latency?

You can predict the memory saving before running the server. Latency needs more care. At fixed TP degree, batch, and context length, an illustrative model for a bandwidth-bound decode step is \[ t_{\mathrm{step}}(D) \approx t_{\mathrm{other}}(T) + \frac{t_{KV}(T,1)}{D} + t_{\mathrm{DCP}}(T,D). \qquad (3.6.11) \] Here \(t_{KV}(T,1)\) is the baseline KV-read time after TP sharding; \(t_{\mathrm{other}}\) includes the remaining work and existing TP communication; and \(t_{\mathrm{DCP}}\) is the added exposed communication and merge cost, zero at DCP=1. The \(1/D\) term assumes balanced shards and comparable achieved HBM bandwidth.

Try made-up timings: 6 ms of other work plus 8 ms of KV reads gives 14 ms. With DCP=4 and 0.6 ms of added overhead, you predict \(6+8/4+0.6=8.6\) ms, about 1.63× faster overall. If KV reads took only 0.4 ms, the same arithmetic gives 6.7 ms instead of 6.4 ms: a slowdown. The useful condition is \(t_{KV}(T,1)(1-1/D)>t_{\mathrm{DCP}}(T,D)\). This is a reasoning model, not a benchmark: kernels, query-head replication, and compute limits can change the attention timing.

There is evidence for both sides of this tradeoff. The August 2026 vLLM report compares TP and DCP on eight B200s with Kimi K2.6 in NVFP4 and an agentic trace. Its largest gains come from supporting concurrency beyond the replicated-cache baseline's capacity. In contrast, the million-token context parallelism paper, §4.3 shows cases where smaller local attention kernels lose their savings to query and output communication. Those are different implementations and workloads; together they explain why you should measure capacity, throughput, and token latency separately.

Try it in vLLM

Here is a two-GPU MLA example from the vLLM walkthrough. Restart the server with DCP=1 for your baseline, then DCP=2 for the comparison:

vllm serve deepseek-ai/DeepSeek-V2-Lite \
  --tensor-parallel-size 2 \
  --decode-context-parallel-size 2

For the four-KV-head GQA example, an eight-GPU configuration is:

vllm serve Qwen/Qwen3-235B-A22B \
  --tensor-parallel-size 8 \
  --decode-context-parallel-size 2

-tp and -dcp are the short forms. In the Python API, the corresponding arguments are tensor_parallel_size and decode_context_parallel_size. Choose hardware and dtypes that fit the model plus working memory. Verify DCP support for your model, GPU, KV dtype, and attention backend in the attention backend support matrix. Speculative decoding, sparse attention, hybrid layers, and P/D transfer connectors need compatible implementations too; support for DCP alone does not establish that every combination works.

1. Hold the comparison steady. Fix the GPUs, TP, model and KV dtypes, request lengths, prefix-cache policy, and scheduler limits. Compare DCP=1 against supported degrees at the same concurrency first.
2. Then test the capacity benefit. Sweep concurrency and, where needed, raise the scheduler's sequence limit to use the freed cache. Report KV usage, preemptions, output tokens/s/GPU, and p50/p95/p99 inter-token latency. Include TTFT when prefill shares the server.
3. Keep your latency target visible. More throughput is useful only while requests meet your SLO. Test both short and long contexts, and record whether each DCP group stays inside NVLink or crosses the network.

Choosing: What Each Cut Buys and Pays

Let us remember our main idea: every parallelism strategy is a choice of which tensor lives where plus which communication primitive pays for it, and feasibility is decided by the NVLink/RDMA boundary. Every strategy we have studied edits specific terms of Lecture 1's decode intensity \(I \approx 2PB/(2P + B \cdot kv)\) and latency model. The whole lecture fits in one table, and every cell is a claim we derived above:

Read each row as: which bytes shrink, which collective pays, and may it leave the NVLink domain?

Decision procedure. Fix three inputs, and usually only one or two combinations remain feasible:

1. Model structure. Dense or MoE? KV-head count \(n_{kv}\)? KV bytes per token (GQA vs. MLA latent)?
2. Load type & SLO (Lecture 1's metrics): interactive (TPOT/TTFT-driven) vs. throughput (goodput-driven); context length distribution.
3. Interconnect topology: NVLink domain size (8 on H100 HGX, 72 on GB200 NVL72), RDMA bandwidth.

The standing guidelines:

• Medium/large dense models: intra-node TP with \(T \le 8\), aligned with \(n_{kv}\), as far as interactive loads go. Example: Llama-3.3-70B, GQA-8 → TP=8 exactly.
• Dense models exceeding single-node capacity: add the shallowest PP that fits: TP × PP = (intra-node) × (cross-node).
• Long-context decode with replicated KV: compare TP alone with TP + DCP on the same GPUs. MLA and GQA deployments with TP beyond the KV-head count have cache capacity to reclaim; choose DCP degree using measured latency and goodput.
• Large MoE models: attention DP (MLA-style) or small TP (GQA-style) × MoE EP; under PD disaggregation the two pools choose EP sizes independently (EP32 prefill vs. EP144 decode in the DeepSeek deployment); long-context prefill additionally stacks CP.
• Frontier hybrid MoE, 2026 row (Kimi K3: KDA + Gated-MLA + LatentMoE): attention DP over the KDA layers (their fixed per-request state rides the DP partition; no per-token KV exists to shard along \(L\)); DP plus DCP for the 24 Gated-MLA layers whose 27 KiB/token cache dominates at long context; LatentMoE on EP64+ supernodes, the halved latent-width all-to-all kept inside the rack-scale NVLink domain.

Common Pitfalls

Research Thinking

How to: read the starting point; read the question and think, for a minute, a day, a week... and only then open the answers. You are not supposed to redesign EP144 from the starting point; each deployment took its authors months of work. It's the habit of thinking that counts: you have ideas, you try; if they don't work, you think again.

Have Fun! The TPOT Playground

You have the model; now feel it. Pick a model, a TP degree, and a batch, and watch the two terms of (3.6.3) fight. Missions: (i) find the \(T\) where the blue comm term first eats more than 40% of TPOT for the lab's 8B model, then check yourself against Lab 6's table below; (ii) notice what the 70B row does when you push \(T\) past \(n_{kv} = 8\); (iii) try to make TPOT worse by adding GPUs. Can you?

Seminar & Homework

Take Lab 6: Parallelism, the TP Sweep from the hands-on pack. Sweep --tensor-parallel-size over \(T \in \{1, 2, 4, 8\}\) on one node with vllm serve, under a deliberately latency-clean workload: fixed \(L = 1024\) prompts, 256 outputs, --max-concurrency 32 to hold \(B \approx 32\) steady, arrival rate low enough to avoid queueing. For each \(T\): warm up, run 400 requests, record TPOT P50/P99 and per-GPU resident memory from nvidia-smi.

Then do the fun part: regress measured TPOT against \(1/T\). The slope should match \((2P + B L\, kv)/\beta\); the intercept estimates \(2 n_{layers} t_{lat}\). Compare your fitted \(t_{lat}\) with the lecture's ≈ 10 µs (and ask yourself why ring hops growing as \(T-1\) might push it up). For reference, the pack's predictions at \(t_{lat} = 10\) µs (comm term \(2 \times 32 \times 10\,\mu s = 0.64\) ms):

The curve should halve, then bend: by \(T = 8\) the fixed comm term is ~46% of TPOT. Watch the GQA limit: \(n_{kv} = 8\) means TP=8 leaves exactly one KV head per GPU.

Optional, multi-node: repeat TP=8 across two 4-GPU nodes over IB and compare the intercept. Feel the 18× wall personally. Debrief questions the pack asks you to answer: why is the 256 KB decode payload latency- rather than bandwidth-dominated? At what \(T\) does adding GPUs stop paying for single-request latency, given your intercept? And why can't TP=8 serve more aggregate throughput than TP=1 even though TPOT falls? Pitfalls to dodge: reading TPOT under queueing (TP sweeps must stay latency-clean); forgetting KV duplication at small \(T\); comparing across different --max-num-seqs; and remembering that (3.6.3) overstates the \(T = 1\) row; note it in your report.

Summary

• Two reasons for many GPUs: capacity (140 GB of 70B weights vs 80 GB of HBM; V3's 671 GB vs a 640 GB node) and aggregate bandwidth (\(2P/\beta \to 2P/(T\beta)\)).
• Inference reuses training's toolbox without gradient sync and with almost no compute to hide behind, so communication latency rules decode. The hardware boundary is 900/50 = 18× (NVLink vs. per-GPU RDMA); NVL72 moves the boundary, 8 → 72.
• TP: two 512-KB all-reduces per layer; \(\mathrm{TPOT}(T) = (2P + B{\cdot}kv)/(T\beta) + 2 n_{layers} t_{lat}\) is linear-then-flat, floored at 1.6 ms for 70B, optimum at the NVLink-domain size in this simplified example. Plain TP's KV sharding is capped by \(n_{kv}\) for GQA; MLA's latent cache stays replicated unless another strategy such as DCP distributes it.
• PP: the only cross-node-friendly sharding, but pure capacity: bubble \((s-1)/(m+s-1)\), no per-token bandwidth benefit, \(s-1\) hops in TTFT/TPOT. Interactive loads: keep it shallow.
• EP: the only sharding converting MoE sparsity into decode bandwidth. EP144 reads ≈ 0.96 GB of experts per step where EP8/TP8 read ≈ 85 GB/layer-equivalent; wide pools also fix expert-GEMM shapes (\(I_{expert} \approx 2 t_e\)); priced at a pool-wide all-to-all + EPLB. LatentMoE (K3) halves wire bytes per (token, expert): top-16 at top-8 cost.
• Standard form for large MoE serving: attention DP (± small TP) × MoE EP, two all-to-alls per layer, lockstep padding as the residual cost.
• Prefill CP: shard the sequence; ring-attention traffic can hide in quadratic compute past \(L^*_{CP}\) (17.3K on RDMA for 70B-GQA, 2.8K for V3-MLA, at \(N=8\)).
• DCP: shard the cached history and merge partial outputs using their LSE weights. With PCP=1, vLLM reuses TP ranks: TP=8, DCP=2 still uses eight GPUs. Reclaimed KV capacity can raise concurrency; latency improves only when reduced KV-read time outweighs added overhead.
• Selection: fix model structure, SLO, topology, and the feasible set collapses to one or two combinations. When someone proposes cross-node TP, the review is over.