LLM-serving intuitions fail one by one on video generation — batching, acceleration, quality and streaming.
LLM serving has built up a fairly mature set of intuitions: decode is bound by memory bandwidth, so batching adds throughput almost for free; speculative decoding speeds things up without changing the output distribution; the KV cache is the main consumer of memory; TTFT and TBT describe the user experience. Video generation inference looks like a similar problem — the same Transformers, the same questions of scheduling, parallelism and memory — yet these intuitions fail almost one by one.
This post records where and why they fail. Public material mostly covers the parts of papers that worked or vendors’ final designs; the ideas that look reasonable but do not work are rarely written down, and this post starts from exactly such a case: a batching attempt that gained nothing. The numbers here are of three kinds: estimates derived from public model configs, measurements on a single A100 (marked “Measured” in the figures), and a few personal observations. Each states its conditions, and none is a complete benchmark.
The original idea was to bring over LLM-style continuous batching: even though most of a DiT is compute-bound, its memory-bound operations might still gain from batching, so requests would be co-batched at denoising-step boundaries. With a 5B model and long chunks, the profile showed compute was still the bottleneck, and co-batching did not bring the expected gain.
In hindsight, this follows directly from arithmetic intensity. For a linear layer over M tokens with hidden size d, the compute is 2Md² FLOPs and the bf16 traffic is about 2(2Md + d²) bytes:
| Workload | Tokens per forward pass M | Intensity I |
|---|---|---|
| LLM decode, batch 1 | 1 | ≈ 1 |
| LLM decode, batch 32 | 32 | ≈ 31 |
| Wan 1.3B, 480p, 81 frames, batch 1 | 32,760 | ≈ 750 |
| Same, batch 4 | 131,040 | ≈ 763 |
The LLM rows use d = 4096 and the Wan rows d = 1536; the VAE compresses 81 frames into 21 latent frames of 1,560 tokens each. An H100 SXM has about 989 TFLOPS of dense bf16 compute and about 3.35 TB/s of HBM bandwidth, so its ridge point is about 295 FLOP/byte. Only linear layers are counted, not attention or elementwise ops.
In LLM decode, going from batch 1 to 32 raises intensity about 30×, still below the ridge point, so batching pays off almost linearly. A single video DiT request is already far above the ridge point; a larger batch only pushes I toward its ceiling of d/2 = 768. From batch 1 to 4, intensity rises by only about 2%.
A larger model does not change this. For Wan2.2 TI2V 5B at 720p and 121 frames (27,280 tokens per forward pass), batch 2 still gives 1.00× the DiT throughput of batch 1; from batch 3 on, the job no longer fits on a 40 GB A100 even with the text encoder moved off the GPU. Batching does not raise throughput, and it runs into the memory limit first.
Solving I < 295 for M gives M < 295d / (d − 590): about 480 tokens at d = 1536 and about 330 at d = 5120. Linear layers return to the memory-bound regime only when a forward pass has a few hundred tokens, as in low-resolution text-to-image or very short clips.
So why doesn’t “exploiting the memory-bound operations” work either? Elementwise ops such as norms, activations and RoPE are indeed bandwidth-bound, but their intensity does not depend on batch size: batching scales their reads, writes and compute by the same factor, so the ratio stays the same. This overhead is better handled by kernel fusion than by batching.
For this kind of workload, then, scheduling is less about filling batches and more about when a request starts, which machine it runs on and how many GPUs it gets. SlackServe adjusts GPU resources, step count and resolution together under SLO constraints
N is the step count and G the number of GPUs; tstep includes communication that is not hidden, and Tother collects encoding, decoding and other overhead. C is the GPU-seconds a request occupies on G GPUs.
Under this decomposition, the common techniques rank roughly as step distillation ≫ sparse attention ≈ quantization > diffusion caching > VAE optimization. The ranking rests on which term each technique changes and where its ceiling lies, not on a particular measurement; it can shift with model, resolution and length. Kernel fusion does not fit neatly into this ranking: its gain depends mainly on how much the baseline already fuses.
Step distillation changes N. Methods such as DMD2 and rCM compress 20–50 steps to about 4
Sparse attention changes only the attention part of tstep
Estimated by FLOPs, p is about 0.7 for Wan 1.3B at 480p and 81 frames, so even if attention vanished entirely, a step would get at most about 3.4× faster. Attention compute grows with the square of the token count, so higher resolution and longer videos raise p, and the ceiling with it.
That p is a FLOPs estimate; measured by kernel time it is lower. The figure below was measured on an A100 (bf16, the FlashAttention kernel behind PyTorch SDPA, the default diffusers implementation): attention takes about 52%, which caps the per-step speedup at about 2.1×. Another 29% goes to elementwise ops such as norms, modulation and type conversions, whose FLOPs are negligible but whose time is not.
The same measurement on Wan2.2 TI2V 5B (720p, 121 frames, d = 3072) puts attention at only about 39%, a per-step ceiling of about 1.6×: with a larger hidden size and fewer tokens, the linear layers take a larger share.
Quantization also acts on tstep. FP8 has twice the peak compute of bf16, which is the theoretical ceiling for linear layers; methods like SageAttention extend quantization to attention
Diffusion caching skips part of the computation in similar adjacent steps
Kernel fusion targets the elementwise ops inside each step. Their FLOPs are negligible, yet in the profile above they take about 29% of kernel time; fusing norms, modulation and type conversions into neighboring kernels directly shortens tstep. That share comes from the default diffusers implementation without torch.compile, though; in an inference framework that already fuses these ops, much less is left.
VAE optimization handles the overhead outside the steps. It is barely visible in a 50-step model, but the smaller N gets, the larger Tother’s share: after distillation, encoding, decoding and data movement can go from negligible to significant. A faster step does not mean an equally faster request, and lower latency does not mean fewer GPU-hours.
In the A100 measurement below, VAE decoding takes only about 3% of end-to-end time at 50 steps with CFG; running the same checkpoint for just 4 steps without CFG raises that to about 43% (float32 VAE, no tiling).
The VAE’s cost is not only time: activations in the decoder grow with resolution and frame count. At the 480p, 81-frame setting below, the float32 VAE decode already peaks about 6.7 GB above DiT denoising; with tiling it drops to about the DiT’s level, at the cost of decode time (not measured here).
There is one more multi-GPU reason the speedups do not multiply: sparse attention reduces compute
An analytical estimate for seeing trends; the defaults are not calibrated against measurements.
In LLM inference, one of the most influential speedups is lossless: speculative decoding guarantees the output distribution does not change. In video generation, the speedups with large gains — distillation, sparsity, quantization, caching — are almost all lossy. Quality evaluation also lacks a cheap, reliable metric like accuracy, so every speedup carries an extra evaluation cost.
Parallelism can change the result too. Different sequence-parallel splits, reduction orders and attention backends give different floating-point results; denoising can amplify those initial differences, so multi-GPU and single-GPU outputs may not look the same. Where reproducibility or A/B comparison matters, check this beforehand.
Length is limited by the training distribution. A bidirectional model’s usable length is usually close to its training length; beyond that, repetition or structural collapse is common. Simply extending the sequence means extrapolating positional encodings
Changing the inference pipeline is riskier than it looks. Injecting the previous segment’s K/V into a bidirectional model, or adding an extra fusion module, both seem to carry information across segments; in my own attempts, though, the first thing such changes brought was unstable image quality. Possible causes include misaligned positional encodings and a train–inference mismatch, which I have not yet isolated one by one. Getting information in does not mean the model uses it as intended.
By contrast, video models are very sensitive to text conditioning. Without retraining, fuller and more consistent text descriptions may help cross-segment consistency more than changing the inference pipeline. This is the choice I made after experimenting so far, not a general conclusion.
Causal generators such as CausVid and Self Forcing let video be produced segment by segment
SlackServe organizes resource allocation around exactly this remaining playable time
Autoregression also brings the KV cache back to video generation. History KV directly consumes the memory budget that streams share; for bf16 standard multi-head attention:
Most current AR video models attend only within a local window; LongLive also keeps the first frame as a sink
Since 2026, finer-grained KV compression has appeared on the generation side: Quant VideoGen quantizes the KV to 2 bits
The two problems are not the same. On the understanding side, compression saves memory; the KV is only read by the question, and tokens can be selected based on it. On the generation side, the window already bounds memory, so compression is more about remembering a longer history within the same budget; and every compression error is reused as a condition for all later chunks and all denoising steps, where it may accumulate into drift segment after segment. Importance-based selection heuristics from the understanding side may therefore not carry over directly.
History length, the number of streams that fit and generation quality need to be weighed as one serving problem. This is one of the directions I consider most worth investing in.
The following is based on the open-source code as of late October 2026. These projects move quickly, and details may have changed.
Given Section 01, the value of batching and step-level scheduling in these two serving frameworks may lie mostly outside video throughput: image and low-resolution requests are still in the range where batching works; step-level execution gives a hook for preemption, interleaving and resource adjustment; and a unified request interface makes it easier to share infrastructure with LLM serving. For high-resolution video requests, batching itself is most likely not where the gains come from.
Not every direction leaves room for infra, but the following ones change the problems an inference system has to solve:
Looking back, video generation inference shares many components with LLM inference, but the bottleneck sits elsewhere: in compute, not bandwidth; in resource allocation, not batching; in quality constraints, not lossless speedups. Before carrying an LLM serving method over, first check that the assumption it relies on still holds.
October 2026. Personal observations are limited to the stated settings; quality comparisons are still to come. Schematics, intensities and capacities are estimates from public configs; figures marked "Measured" come from a single A100, with the setup in each caption.