A batch is a group of requests executed in one pass through a model. Batching raises throughput when each pass carries a fixed cost that does not scale with the number of inputs, so spreading that cost over more requests lowers the cost per request. The price is queue delay, the time a request waits before its batch starts. Service time is the execution time of a batch. Request latency is queue delay plus service time plus any surrounding stages. p99 latency is the value that 99% of requests meet or beat.
The gain depends on that fixed cost. In the Clipper paper, a 2 ms batching delay gave a 3.3x throughput gain for one model with high fixed per-batch cost, and no gain for a second model that already handled small batches efficiently. These are 2017-era results on specific models. The qualitative dependence is the reliable finding. The multiplier is not a general constant.
How a dynamic batcher decides when to dispatch
According to NVIDIA's Triton Inference Server batcher documentation, the dynamic batcher sends a batch on one of two triggers. Either a preferred or maximum batch size is reached, or the oldest queued request has waited for the configured maximum queue delay. The delay setting bounds time spent in the batcher queue. It does not bound end-to-end latency, which also includes preprocessing, data transfer and execution.
Tuning batch size and delay against a latency budget
NVIDIA's guide gives a procedure rather than values. Measure latency and throughput first with its Performance Analyzer. Then raise the maximum batch size or add queue delay only while measured latency stays within budget, and stop once the budget is exceeded.
Two points are the author's reading, not findings in the sources. First, the latency budget has to be split across queue delay, execution at the chosen batch size, and stages outside the model server. A queue delay that consumes most of the budget leaves little room for execution. Second, check the target percentile rather than the mean, because queueing effects concentrate in the tail.
A worked example the sources do not provide
None of the sources reviewed maps a p99 target to a concrete delay and batch size for a given arrival rate. Building one requires execution time measured at each batch size on your own hardware, your real arrival process, and your percentile target. The queueing model discussed below can estimate waiting time, but only under its stated assumptions.
Where batching stops helping
AlpaServe, a preprint on serving large models, reports that for large models a small batch can already saturate the GPU, so larger batches add little. It also argues that when execution time grows linearly with batch size, batching is a poor choice under tight service level objectives (SLOs). These results come from its specific experimental setting.
A queueing analysis preprint models dynamic batching with Poisson request arrivals and batch service time linear in batch size. It derives waiting-time behaviour under those assumptions. Whether a given workload matches either assumption has to be measured, not assumed.
One vendor result points the other way. An AWS Machine Learning blog post reports close to 2x throughput from dynamic batching on SageMaker with Triton, without a noticeable latency increase. The post is older and describes a single test configuration, so it should not be read as typical.
Continuous batching and head-of-line blocking
For autoregressive LLM serving, continuous (iteration-level) batching lets new requests join the running batch at each decode step instead of waiting for the whole batch to finish. Anyscale's benchmark of vLLM reports up to 23x throughput and an improved p50 latency. Those figures are from Anyscale's own benchmark. The sources reviewed do not establish that latency improves at every percentile. Cost per token is a separate question, covered in what inference cost per token depends on.
Continuous batching does not remove head-of-line blocking. An early-stage preprint reports that long generations in the active batch can delay shorter jobs, and its authors present this as motivation for lightweight scheduling changes.
Prefill stalls and chunked prefill
An LLM request has two phases with different compute profiles. Prefill processes the prompt. Decode generates output tokens one step at a time. Sarathi-Serve reports a throughput-latency trade-off in current systems, and states that increasing load can sharply raise tail latency in vLLM. Its remedy, chunked prefill, splits long prompts into pieces so that decode work for other requests can interleave between them.
A December 2025 analytic preprint addresses prefill-heavy clusters. There, a prefill instance processes a batch as a locked, non-preemptive unit, so immediate dispatch makes requests wait inside the engine. Under uniform-arrival assumptions, its model gives an expected wait that is lower under staggered dispatch, with the reduction growing with cluster size. The paper calls this large-cluster gain a theoretical result, and it has not been measured.
Overload and admission control
When more requests are admitted than capacity allows, queueing delay grows, and misses can cascade across requests. A common response is admission control: rejecting or deferring requests whose estimated completion would miss their SLO, so that the remaining requests still meet theirs. The evidence available for this is mostly preprints on text-to-image serving and LLM scheduling. Its applicability to classical tabular or computer-vision serving is not established.
What is established, contested and open
Established: the dispatch triggers documented for Triton, and the dependence of batching gain on per-batch fixed cost.
Contested: throughput multipliers. The 2x, 3.3x and 23x figures come from specific hardware, models and traces. ZeroEntropy's explainer gives 5 to 20x, which conflicts with Anyscale's figure, and no primary source reconciles them. Whether continuous batching improves latency at every percentile is also unresolved.
Open: a worked mapping from SLO to delay and batch size; how batch-wait time interacts with autoscaling, cold starts and multi-tenant priority queues; and current default settings in vLLM, Triton and TensorRT-LLM, which were not verified for this article. Evidence dates here are approximate.
If you are tuning a service now, measure execution time at each batch size on your hardware, split the p99 budget across queue, execution and external stages, and raise queue delay in small steps. Stop at the first step that breaks the percentile you care about.