Framework Landscape, Evolution, and New Workloads

Imagine tomorrow is your first day owning a production LLM deployment. You open the course notes: paged KV, sparse attention, continuous batching, disaggregation, parallelism, kernels, speculative decoding. You can derive each one. Then you open GitHub and meet the names: vLLM, SGLang, TensorRT-LLM, Dynamo, Mooncake. Search "vLLM vs SGLang 2026" and drown in benchmark blog posts. And just as you settle on an engine, the RL team walks over: "we need to swap the model's weights into your serving fleet every ten minutes, and please don't crash." Nothing in your mental model of serving quite covers that.

This final lecture closes exactly those three gaps: who ships it (the framework landscape and how to choose), how we got here (the 2022–2026 evolution and its four main lines), and what breaks next (RL rollout, a workload that is inference but serves nobody). Everything stays anchored to the one ratio you have carried since Lecture 1.

The Spine Revisited: One Formula, Eight Modules

Let us remember our main idea, one last time:

Three structural remarks, because they are the difference between knowing the mechanisms and wielding them:

1. The interventions compose multiplicatively. Paging enlarges the feasible region of \(B\), scheduling fills it, sparsity and MLA shrink the cost of filling it, parallelism splits what remains, kernels make the hardware deliver its peak, and speculation changes what "one step" produces.
2. They are ordered by leverage. At low \(B\), the \(2P\) weight-read term dominates, and only batching, parallelism, and speculation help. At high \(B\) and long \(L\), the \(B \cdot kv\) term dominates, and KV compression and sparsity become the binding constraint. That is exactly why Modules 2 and 3 sit at the center of this course.
3. The model itself is a knob. MLA and DSA act on \(kv\) directly, and MTP acts on the speculative \(\tau\). The architecture is a dial on the same formula. Keep that thought; it is the seed of the co-design argument below.

The Framework Landscape: Four Systems on a Converging Checklist

Fine, the mechanisms are yours. Now somebody has to press the deploy button. The mechanisms do not exist in the abstract; they live inside a small number of production engines, and the honest way to picture them is not a league table but a map: each system has a hallmark, a thing it does more natively than anyone else.

Walking the map slowly, system by system:

• vLLM, the general-purpose engine. Hallmark: breadth. It originated PagedAttention (Module 2) and remains the default reference engine. After the v1 rewrite it is a multi-process EngineCore with asynchronous scheduling: scheduler, executor, and API server run as separate processes, so CPU-side scheduling overlaps GPU execution (Module 4's overlap theme). Its distinguishing commitment: the largest hardware matrix (NVIDIA, AMD, TPU, Trainium, ...), a day-0 convention for new models, and pluggable attention backends (FlashAttention, FlashInfer, FlashMLA, vendor kernels) swappable per model and platform. If your constraint is "many models, many platforms, one engine", it is natively satisfied here.
• SGLang. Hallmark: depth in scheduling and frontier-model support. It originated RadixAttention (Module 2's prefix reuse) and invests deepest where vLLM is broadest: a zero-overhead overlap scheduler, structured output compiled into the serving path, and the deepest DeepSeek support: large EP plus PD disaggregation reproducing DeepSeek's official deployment form, and day-0 support for V3.2's sparse attention including the dual-page-size KV work of Module 3. Frontier MoE models at scale tend to appear here first in open source.
• TensorRT-LLM, NVIDIA's official engine. Hallmark: kernel and compilation depth. First-day optimizations for new NVIDIA hardware appear here first because the engine is co-developed with the GPU roadmap; it sits in a closed loop with NIXL (KV transfer), Dynamo (orchestration), and Triton (kernel authoring). Brand-new, NVIDIA-endorsed silicon → shortest path to peak.
• Dynamo, which is not an engine at all but an orchestration layer above engines. It productizes everything in this course that lives beyond a single instance: global PD scheduling (a global prefill queue; conditional, per-request disaggregation), KV-aware routing (send each request to the worker holding the most overlapping prefix), KVBM (tiered KV blocks across HBM/DRAM/NVMe/remote), and NIXL as the transfer substrate. Workers can be vLLM, SGLang, or TensorRT-LLM; it is deliberately engine-agnostic.

Here is the layer-by-layer checklist, with the course mechanism behind each cell. Read the last two rows carefully, because they are the punchline:

Everything on the checklist exists everywhere. Speculation and sparse attention are no longer differentiators in kind, only in maturity and defaults.

This leads to the lecture's two observations. Honestly, they reframe what "choosing an engine" even means:

Which Framework When: Four Axes, Not a Checklist

Because the checklists converge, selection reduces to four axes: workload type, model family, hardware, team constraints. The decision guide compresses to a table you can interrogate:

The Evolution Timeline: Four Main Lines

Now the second gap: how did we get here? Line up 2022–2026 and, surprisingly, a plot appears. The field keeps attacking whichever term of the spine formula was binding at the time:

Each line deserves one paragraph of "what changed / why it matters / what comes next":

RL Rollout: The Third Workload

Now for the twist that breaks the frame. Since Lecture 1, we have classified traffic into two workload classes: latency-sensitive online serving (SLO-bound, goodput-defined) and throughput-bound offline batch. RL post-training adds a third. In frameworks like verl, the training loop alternates optimizer steps with rollout: the current policy generates samples (responses or full agentic trajectories) which the trainer then learns from. Rollout is inference (mainstream stacks embed vLLM or SGLang as the rollout engine), but it obeys neither class:

• No SLO, pure throughput. Nobody waits on a token stream; only total samples per training step matters. Goodput's latency constraints simply vanish.
• But with constraints serving never has: weight sync, partial rollout, numerical consistency, violent elasticity. Treating rollout as "offline serving with a big batch" is the classic pitfall.

Weight Sync: A Time-Budget Model

After every training step (a cadence of minutes or shorter) the updated weights must reach the rollout engine, in seconds. How much bandwidth does that need? Let's count. Let the trainer hold \(P\) parameters at \(b_{dtype}\) bytes/element, sharded across \(G\) training GPUs each with inter-node egress \(\beta_{net}\) (≈ 50 GB/s per GPU on 400 Gbps IB/RoCE). If the transfer must finish within a fraction \(\epsilon\) of step time \(T\): \[ \underbrace{\frac{P \cdot b_{dtype}}{G \cdot \beta_{net}}}_{\text{broadcast time (tree/pipelined)}} \;\le\; \epsilon\, T \qquad\Longrightarrow\qquad G\cdot\beta_{net} \;\ge\; \frac{P\cdot b_{dtype}}{\epsilon\, T}. \] The model assumes trainer shards stream in parallel (each sends its own \(P/G\) shard) with a pipelined broadcast, per-GPU egress being the bottleneck: the standard NCCL/RDMA weight-broadcast form.

Colocate vs a dedicated rollout fleet mirrors Module 5's disaggregation logic, one level up. Each earns its keep at a different scale:

As scale grows, the disaggregated form wins on utilization and isolation, and the weight broadcast becomes a first-class distributed-systems problem: tree topology, pipelining, FP8 quantization of the update, overlapping sync with the tail of the previous rollout.

Partial Rollout and the δ̄ Watchdog

Agentic trajectories reach hundreds of thousands of tokens, far longer than one training step's rollout window. If every trajectory must finish on one weight version, two failure modes follow: the long-tail stall (the slowest trajectory gates the whole step; utilization collapses while 99% of GPUs wait), and unbounded off-policy drift (the policy that wrote a long trajectory's early tokens is many updates old by the end). Partial rollout answers both: at a sync boundary, interrupt in-flight trajectories, keep their KV, swap weights, resume. Three requirements land squarely on Modules 2 and 4:

1. An interruptible request lifecycle. A suspend state distinct from running and from preempted-to-recompute: generation stops at a token boundary, but the request's KV blocks are pinned, not freed. Stronger than Module 4's preemption, which trades KV away for memory.
2. KV lifecycle across weight versions. Keep or recompute? An algorithmic decision exposed to the trainer.
3. Fairness under suspension. Suspended trajectories resume ahead of fresh requests if tail completion matters, behind them if throughput does. It is a scheduling knob with direct consequences for sample efficiency.

Now the subtlest constraint, and my candidate for "easiest way to silently break an RL stack." On-policy corrections assume generation probabilities are known exactly. The importance-sampling ratio for a token (multiplied, a trajectory): \[ \rho = \frac{\pi_{train}(a_t \mid s_t)}{\pi_{rollout}(a_t \mid s_t)}, \qquad \log \rho_{traj} = \sum_{t=1}^{N}\left(\log \pi_{train} - \log \pi_{rollout}\right)_t. \] The catch: rollout logprobs come from inference kernels (fused, reassociated, possibly FP8/TF32, speculative-verified) while training logprobs come from training kernels, with different reduction orders and different dtypes. Floating-point addition is non-associative, so the same sequence scores differently by small per-token gaps \(\delta_t = (\log\pi_{train} - \log\pi_{rollout})_t\) even at identical weights, and the ratio picks up the factor \(\exp\bigl(\sum_t \delta_t\bigr)\). How bad that factor gets depends entirely on the structure of \(\delta_t\):

Same magnitude, three fates. The operative question is never whether \(\delta_t\) is small (it always is) but whether it is biased. Mitigations fall straight out of the algebra: align reduction orders and dtypes in the logprob path (the sampling path can stay fast); keep per-token ratios so clipping absorbs local spikes; and measure \(\bar\delta\) on held-out sequences after every kernel or engine upgrade. It is a one-line regression test with \(10^4\)-token compounding consequences.

The epilogue is the familiar pattern of the timeline: the features RL rollout forced into existence (hot-update interfaces, interruptible lifecycles, coexistence with training frameworks) are now official feature areas of both vLLM and SGLang, not forks. A workload pressure appears, one system answers it, the answer becomes checklist. Whatever workload class follows agentic RL will arrive the same way: as new constraints on these same layers.

Common Pitfalls

Research Thinking

How to: read the starting point; read the question and think, for a minute, a day, a week... and only then open the answers. You are not supposed to reinvent several months of someone's research; it's the habit of thinking that counts.

Have Fun! The Weight-Sync Budget Calculator

Enough reading. Now you sign off on the sync budget. Set the checkpoint, the trainer size, and the step cadence, and watch the budget bind. Can you find the smallest sender count that keeps a 2-minute step inside 1%? And at what step cadence does any BF16 broadcast fail no matter how many GPUs you throw at it... until you flip to FP8?

Seminar & Homework

Lab 9: Framework Comparison and RL-Rollout Reading closes the hands-on pack with two halves, mirroring the lecture:

• Engine comparison under matched configs. On two GPUs (or one, sequentially), run one benchmark harness across vLLM and SGLang: same model revision, BF16 both, --max-num-batched-tokens matched against SGLang's --chunked-prefill-size, speculation off both, prefix caching off both for the engine comparison, then a second, clearly labeled pass with it on. Same seed. Optional: replay the Mooncake open trace at production-like rates. Deliverable: a one-page benchmark report in the course's required format (Module 1's report checklist: percentiles, input/output split, goodput under your chosen SLO, not a single blended number).
• Rollout config reading. Take a verl-style colocate rollout config and identify, in the actual settings you've learned this lecture to expect: the weight-sync mechanism (which broadcast, over which fabric, against which cadence) and the partial-rollout mechanism (where trajectories suspend, and what happens to their pinned KV).

Summary

• One spine, eight modules. Every mechanism is an intervention on \(I \approx 2PB/(2P + B\,kv)\): paging and scheduling act on the feasible region and fill of \(B\); MLA/DSA on \(kv\); parallelism splits \(2P + B\,kv\); kernels deliver \(\pi,\beta\); speculation amortizes the \(2P\) read over \(\tau\) tokens. Diagnosing a deployment = finding the dominating term at \((B,L)\).
• Frameworks: vLLM = breadth (hardware × models, async EngineCore); SGLang = depth (radix reuse, frontier-MoE form, day-0 sparse attention); TensorRT-LLM = kernel/compilation depth on new NVIDIA silicon; Dynamo = orchestration above engines. Single-engine checklists have converged; decide on the four axes, tuned configs, your traffic.
• Evolution: four lines: scheduling granularity (request → step → token budget → elastic/interruptible), memory as a managed resource (detail → paged → reusable → tiered & routable), deployment decoupling (colocated → PD → orchestrated pools), and model–system co-design (the non-convergent differentiator: 320 KiB/token → ≈11.7 KiB/token, ≈27×, decided at training time).
• RL rollout is a third workload class: no SLO, pure throughput, but with weight sync (\(G\beta_{net} \ge P b_{dtype}/\epsilon T\); 1.34 TB in 12 s needs only 3 senders, over PCIe it would take 21 s), partial rollout (suspend, pin KV, resume; accept mixed-version KV or re-prefill), numerical consistency (\(\bar\delta = 10^{-3}\) → 22,000× trajectory distortion; zero-mean noise → 1.11×; measure the bias), and ×10 elasticity.
• The pattern that closes the course: every workload pressure becomes an engine feature becomes a checklist item. You now know how to read the next one.