How GPUs Avoid Waiting for the CPU During LLM Inference

A language model writes its answer one token at a time, and every token is one step of work for the GPU. Between steps, the CPU has work to do: it decides which requests run next, prepares their inputs, and reads the tokens that the step produced. If the GPU waits while the CPU does this, it sits idle between steps.

The cost of that idle time depends on the length of a step. Take Qwen3-8B, a small model, on one NVIDIA Blackwell B300 GPU. A decode step takes 3.4 ms (milliseconds) on the GPU, and the CPU needs 0.6 ms per step. If the GPU waits for the CPU after every step, the server loses 10 to 12% of its throughput with 1 to 32 concurrent requests. For GLM-5.3 on 4 B300 GPUs, where a step with one request takes 6.1 ms, waiting costs 9% with one request and 6% with 32.

SGLang avoids the wait with its overlap scheduler: the CPU prepares the next step while the GPU is still running the current one. The CPU can be only one step ahead, though, and this limit explains most of the design. If the CPU’s work for a step takes longer than the step takes on the GPU, the GPU waits even with overlap. The rest of the design exists so that the CPU doesn’t wait for a result from the GPU in the middle of that work.

For plain decoding, this works without much trouble, but with speculative decoding, it gets harder. There, a small drafter proposes a few draft tokens, the model checks all of them in one pass, and the request grows by 1 to 4 tokens per step in my setup. Only the GPU knows how many there were, and only once it has checked them. But the next step needs that number, which determines where in memory the step writes and what the drafter starts from.

On GLM-5.3 with one request, speculative decoding makes the server about twice as fast (325 instead of 156 tokens per second, at 2.8 tokens per step), and the speculative server is 8% faster with overlap than without. More than half of that comes from removing waits inside a speculative step rather than between steps. With 8 or more concurrent requests, a step is longer, and the gain is 4 to 5%.

There is a second reason to care, and it has nothing to do with speed. Once the CPU runs ahead, it changes tables on the GPU that the running step is still reading. If nothing orders the two, the server doesn’t crash. Instead, a step reads table rows that were written for a later step or for another request, and the answers get worse, which we notice only in an accuracy test. The pull request that fixed this was opened because such a test had become flaky.

Does the GPU have to wait between steps, and how can the next step be queued when the outcome of the current one is known only on the GPU? To answer this, I will ask the same question of every piece of state: what does the next step need from this one, and does that have to reach the CPU first?

I measured two models at SGLang commit f1065423, which is also the code I describe, on a machine I share with other people. The running example is GLM-5.3 on 4 GPUs (the NVFP4 checkpoint, with and without speculative decoding), and the second, with short steps, is Qwen3-8B on one GPU. Both get prompts from ShareGPT, a public set of chat logs, sent by SGLang’s bench_serving tool with 512 output tokens each at temperature 0, and throughput is the mean of 3 runs. The server’s log can’t show a gap of 0.3 ms. My timing patch records a CPU timestamp around each part of the scheduler’s loop and a GPU timestamp at both ends of each pass, that is, one run of a model over the batch. A GPU timestamp is a CUDA event, which the GPU stamps when it reaches that point of its queue. Per-step times are medians of 300 consecutive steps. A gap runs from the last GPU timestamp of one step to the first of the next, and the scheduler’s own small GPU operations fall inside it. With and without the patch, Qwen3-8B’s throughput differs by less than 1%.

1. The Serving Loop#

What does the CPU do between steps, and what does the GPU do in the meantime?

In the simplest serving loop, the CPU does four things for every step:

1. It prepares the step: it picks the requests that run next, reserves memory in the cache for what the step is going to write, and builds the tensors that describe the batch.
2. It launches the step’s GPU work.
3. It waits until the GPU is done and copies the result over.
4. It processes the result: it appends the new token to each request, checks the stop conditions, hands the tokens to the process that streams them to the client, and frees the memory of finished requests.

Then, it starts over, as shown in the top half of Figure 1.

While the CPU processes one result and prepares the next step, the GPU has nothing to do: the dashed outline between the steps at the top of Figure 1.

To see why the bottom half is possible, we need one fact about how a CPU drives a GPU. When the CPU calls a GPU operation, the call only puts the operation into a queue and returns, and the GPU works through this queue, called a stream, in order. The CPU has to wait only when it wants a result. To find out how long the GPU is idle in practice, I turned the overlap scheduler off and looked at the timestamps. Figure 2 shows three consecutive steps of Qwen3-8B with one request.

At the top of Figure 2, a step takes 3.59 ms between its two timestamps, and then the GPU waits 0.28 ms for the CPU, so a step comes every 3.87 ms. The step itself is also 0.17 ms longer than with overlap. For GLM-5.3 with plain decoding, the gap is 0.4 ms, and the step is 0.2 ms longer. I attribute the longer steps to this: without overlap, the CPU issues the operations of a step one by one while the GPU has nothing in its queue, so the GPU sometimes waits for the next operation. That is the time we want back.

2. Preparing One Step Ahead#

Launching a step only fills a queue, so the CPU is free again long before the GPU has finished. It can use this time to prepare the next step.

2.1 The Overlap Loop#

The overlap scheduler does this (the loop is here). While the GPU runs step N, the CPU prepares step N+1 and puts it into the queue behind step N. Only then does it wait for the result of step N, and it processes that result while the GPU is running step N+1. At the bottom of Figure 1, the CPU prepares step 2 before it reads result 1, and from then on, it stays one step ahead. A simplified round of the loop looks like this:

while True:
    batch = self.get_next_batch_to_run()    # prepare step N+1
    result = self.run_batch(batch)          # launch it; this only fills the queue
    self.result_queue.append((batch, result))
    pop_and_process()                       # wait for step N, then process it

For plain decoding, the CPU waits for the GPU only on the last line, and there it waits for the step before the one that it has just launched.

2.2 The Token Stays on the GPU#

What does step N+1 need from step N, and does that have to reach the CPU first? For plain decoding, the answer is short: it needs the token that step N produced, since that token is the input to step N+1.

SGLang doesn’t wait for this token to reach the CPU. The GPU keeps a buffer with one entry per request. Suppose a request has entry 1, and step N produces the token “The.” Step N writes “The” into entry 1, and step N+1, which the CPU queued with nothing more than “read entry 1,” takes its input from there. The CPU doesn’t need to know the token to queue the step. It needs to know only which requests take part, which it decides itself.

The bottom of Figure 2 shows the result for Qwen3-8B: the gap between steps shrinks to almost nothing, and a step comes every 3.43 ms instead of every 3.87 ms. In throughput, which includes the time for prompts, that’s 288 instead of 256 tokens per second with one request, and 6,949 instead of 6,130 with 32.

2.3 Only One Step Ahead#

Figure 1 also shows the limit of this scheme. The CPU can’t run more than one step ahead, because it builds the batch of step N+2 from the result of step N (which requests ended and each request’s new length). If preparing, launching, and processing a step take longer on the CPU than the step takes on the GPU, the queue is empty when the GPU finishes a step, and the GPU waits again. Figure 3 shows how far the two servers are from this limit.

For Qwen3-8B at the top of Figure 3, the CPU needs 0.6 ms for a 3.4 ms step (0.19 ms to prepare it, 0.37 ms to launch it, and 0.08 ms to process its result). That keeps it busy almost a fifth of the time. For GLM-5.3, it needs 0.9 ms of 6.1 ms, and with speculative decoding, where the CPU has more operations to issue per step, 1.8 ms of 8.2 ms. None is at the limit, but Figure 3 fits the numbers from the introduction. Without overlap, the GPU waits for part of that CPU work, between steps and inside a step, and the shorter the step, the more this costs.

Plain decoding is easy because the next step needs a single token.

3. What Changes with Speculative Decoding#

Let’s go over what a speculative step does, since the rest of the article uses its terms.

3.1 One Speculative Step#

A speculative step works on a short run of positions per request, which I will call the window. The first position holds the last token that the request has produced, and the others hold draft tokens, which a small drafter has proposed. The model that we serve is the target. It processes the whole window in one pass, the verify pass, which gives us, at every position, the token that the target would have written next. Figure 4 shows one such step.

The window in Figure 4 reads “The cat sat down.” The target would also have written “cat” after “The” and “sat” after “cat,” so the first 2 draft tokens are accepted. After “sat,” however, it would have written “on,” so “down” is rejected, and we take the target’s “on” in its place. The step gives us 3 tokens for one pass of the target: the 2 accepted draft tokens and the target’s own token, the bonus token, which becomes the first position of the next window. This count, which always includes the bonus token, is the accept length.

3.2 MTP as the Drafter#

In my running example, the drafter ships inside the GLM-5.3 checkpoint. It is a single extra layer, called MTP (for multi-token prediction), which continues from the target’s last hidden state, that is, from the vector that the target computes for a position right before turning it into a token. With 3 draft tokens, a step has four GPU passes: two short passes of the drafter, the verify pass, and one more pass of the drafter called draft extend. Draft extend recomputes the drafter’s state from what the target has just verified and yields the first draft token of the next step, which is why two draft passes are enough for 3 draft tokens. With one request, the step takes 8.2 ms, 7.4 ms of it in the verify pass, and the request gains 2.8 tokens on average. Plain decoding takes 6.1 ms for one token.

3.3 What the Next Step Needs#

Now, let’s ask our question again for the request in Figure 4. Suppose it has 100 tokens in its cache before the step. (“The” isn’t among them, since a token enters the cache only in the step that takes it as input.) What does the next step need?

First, it needs the request’s new length, 103, because its window covers positions 103 to 106. Second, it needs to know where in the cache to write the entries for these 4 positions. Third, it needs the bonus token “on” as the first position of its window. Fourth, for an MTP drafter, it needs the first draft token and the hidden state that goes with it, both produced by draft extend.

All four depend on the accept length, which is computed on the GPU right after the verify pass, because SGLang runs the comparison from Figure 4 there. When the CPU prepares the next step, it hasn’t seen any of this.

3.4 The Other Drafters#

SGLang supports several drafters, and to the scheduler, they look almost the same. Figure 5 lists what each of them hands to the next step.

In the top two rows of Figure 5, the GPU drafters differ only in the last two boxes. DFlash and DSpark, which fill the whole window in a single pass of a small model, hand over the new length and the bonus token by default. MTP and EAGLE add the first draft token and a hidden state. The bottom row is NGRAM, which has no model: it looks up its draft tokens on the CPU in an index (a trie) of text that the server has produced before. For this lookup, it needs the tokens that the previous step accepted.

We start with the first item of section 3.3: how the next step gets the length 103 without the CPU.

4. The Relay: Results That Stay on the GPU#

SGLang extends the buffer for the output token from section 2.2 to a set of buffers that hold everything the next step reads, with one entry per request. The code calls it FutureMap, and I will call it the relay.

4.1 Two Writes per Step#

A step writes into the relay at two points, as illustrated in Figure 6.

Request 1 in Figure 6 is the one from Figure 4. Right after the verify pass, the step writes the new length, 103. After draft extend, it writes the bonus token “on,” the first draft token “the,” and the drafter’s hidden state. In code, the two writes look like this (simplified from SGLang’s implementation):

def publish(self, reqs, new_seq_lens):        # right after verify
    self.new_seq_lens_buf[reqs] = new_seq_lens
    self.publish_ready.record()               # an event, see section 4.4

def stash(self, reqs, payload):               # at the end of the step
    self.output_tokens_buf[reqs] = payload.bonus_tokens
    self.topk_index_buf[reqs] = payload.topk_index        # first draft token
    self.hidden_states_buf[reqs] = payload.hidden_states

4.2 One Entry per Request#

When the scheduler first processes a request’s prompt, it hands the request an entry number from a free list. The request keeps this number while it runs, unless the scheduler takes the request out of the batch (section 5.3). The CPU hands step N+1 only these numbers, as it does for plain decoding, and a queued step carries no token. The step begins by reading its entries: it takes each request’s length, computes the positions of the window from it, and feeds the bonus token and the drafter’s inputs into the first pass.

What happens to an entry when its request ends? The entry number goes back to the scheduler, which gives it to the next request that arrives. The old length and the old bonus token are still in the buffers, but nothing reads them. The new request’s first pass, the one over its prompt, writes the prompt’s length and the first token after it into the entry before any speculative step looks at it. A new request needs no special handling. Neither does a client that disconnects: SGLang flags the request, lets it take part in one more step, and ends it normally.

4.3 Padding and Large Entries#

SGLang launches the passes of a step as CUDA graphs (recorded sequences of GPU operations that are replayed with a single call, at a fixed batch size). A batch of 5 requests is therefore padded to the next recorded size. The scheduler has one entry per request that it may run, plus one: entry 0 belongs to no request, so that padded rows read and write something harmless. An entry can be large: for GLM-5.3, the drafter’s hidden state is 6,144 numbers per request, and it never reaches the CPU.

4.4 Why the Lengths Are Written First#

The lengths are written before the rest because they are ready earlier, and in some configurations, a copy of them is needed on the CPU (see section 8.3). The line publish_ready.record() puts a marker into the stream. Such a marker is an event, and another stream, or the CPU, can wait for it without waiting for anything that comes later. This way, a copy of the lengths can start while draft extend is still running.

A step leaves in the relay everything that the next step reads, and the CPU hands over only entry numbers.

That settles three of the four items from section 3.3. The second one is still open: where the step writes its cache entries.

5. Reserving Slots Before the Length Is Known#

A server gives every token of a request a numbered slot in the KV cache (the memory in which a model keeps what it has computed for earlier tokens), and a slot holds that token’s cache entries for all layers. The drafter keeps a cache as well, and in SGLang, it uses the same slot numbers as the target.

A speculative step writes one slot per position of its window. Which slots these are depends on the request’s length, which the CPU doesn’t know when it prepares the step.

5.1 The Slot Table#

SGLang has the CPU reserve slots and the GPU pick among them. The scheduler keeps a table on the GPU with one row per request, and it writes each request’s reserved slot numbers into its row, in token order. At the start of a step, the GPU goes to the request’s row, starts at the request’s current length, which it has just read from the relay, and takes the next 4 slot numbers. In the code, this is a single indexing operation on the table, of the form req_to_token[request, length : length + 4].

5.2 Two Windows#

The CPU then has to reserve enough slots, whatever the length turns out to be. The CPU updates its own count of a request’s tokens only when it processes a result. When it prepares step N+1, its count doesn’t include what the request gains in step N, and the GPU’s length can be up to one window ahead of it. Step N+1 writes one more window. That’s why the CPU keeps each row filled up to two windows (8 slots in our example) beyond its count. Figure 7 traces this for one request over four steps.

Let’s go through Figure 7 from top to bottom. Before step 1, the CPU counts 100 tokens and fills the row up to position 107. Step 1 writes positions 100 to 103, and the request gains 3 tokens. When the CPU prepares step 2, step 1 is still running, so the CPU counts 100 and reserves nothing new. The GPU, however, starts step 2 at 103 and writes 103 to 106. Position 103 still holds what step 1 wrote for the rejected draft token, and step 2 overwrites it, so nothing has to be undone after a rejection. The CPU prepares step 3 after it has processed step 1, so it counts 103 and reserves up to 110. In step 2, the request gains all 4 tokens of the window, so step 3 writes 107 to 110. This is the worst case: the CPU’s count is a full window behind, and the window ends exactly on the last reserved position. In step 4, the CPU counts 107, and the GPU is at 108, so there is slack again.

The numbers in Figure 7 are positions in the row, and the slot numbers stored there are arbitrary. For GLM-5.3, SGLang hands out slots in pages of 64 and rounds the reservation up to a whole page. A request then has between 8 and 71 slots beyond the CPU’s count, and most steps reserve nothing.

5.3 When the Cache Is Full#

What if the cache is full and the scheduler can’t reserve two windows for every request? Before each step, it checks that every running request can be topped up. If that fails, it first drops cached prompts that no request uses (SGLang keeps the slots of a finished prompt, so that a later request with the same prefix can reuse them). Then, it takes requests out of the batch, starting with those that have produced the fewest tokens. It frees their slots immediately, keeps their output, and queues them again, so the server has to recompute their cache later.

With overlap, this can happen while a step with these requests is still running. The CPU needs nothing special here, since it drops these requests’ part of that step’s result, as it does for a finished request (section 6). On the GPU, that step still reads the rows of these requests and writes into slots that the scheduler has already given back. The ordering rule of section 7 holds back the scheduler’s later writes to those rows.

5.4 What It Costs#

The price of this scheme is the second window. It exists because of the lag under overlap, but SGLang reserves it either way. For 64 requests, that’s 256 slots before rounding and up to 4,300 after it (67 per request). The server has 2.5 million slots, so this is too little to matter. When a request ends, the slots that were reserved but never used are freed, and only the slots of counted tokens are kept.

5.5 A Shorter Verify Length#

The same reservation covers a case that looks like it should need more. DSpark estimates how likely each draft token is to be accepted, and SGLang can use these confidences (off by default) to verify fewer positions for requests whose draft will probably be rejected. The total that a batch verifies is the verify budget. Because the slots are reserved for the full window before this decision is applied and the relay carries one new length per request, a request with a shorter verify length looks to the scheduler like one that gained fewer tokens.

The GPU can now run a step without waiting for the CPU. The CPU, however, still has bookkeeping to do.

6. Working with Results That Are One Step Old#

The CPU needs two things from a step for its own bookkeeping: whether the step ended a request and how far each request has grown. Since the CPU is one step ahead, both arrive one step later than they would without overlap. For most decisions, this makes no difference, but a finished request costs one extra step.

Suppose the request from Figure 7 produces its end-of-sequence token in step 2, at a length of 107. The CPU finds out when it processes the result of step 2, and by then, step 3 is already in the queue with this request in its batch. The GPU still writes positions 107 to 110 for it, and the CPU throws these 4 positions away. Figure 8 shows the sequence.

How much does the magenta box in Figure 8 waste? For a request with 512 output tokens and an accept length of 2.8, it’s one window of one request in about 180 steps, or 0.6% of the work for that request. The same applies to every other stop condition, such as a stop string or the token limit, since the CPU checks all of these when it processes a result.

The client of a running request doesn’t see a delay. The CPU sends out the tokens of step N as soon as step N has finished, with or without overlap.

Two more parts of the server work with old results. The first is the repetition penalty. At this commit, SGLang computes it from the tokens that the CPU has seen, so it misses the newest token, which reportedly changes the output (an issue that I didn’t test). The second is the verify budget from section 5.5, which the CPU computes from DSpark’s confidences. These sit in the relay too. SGLang copies them to the CPU in the background without waiting for the copy, so each step sizes its budget with the confidences from two steps earlier. The GPU then uses the current step’s confidences to choose which positions fill the budget.

With the relay, the reserved slots, and one-step-old results, the CPU can queue the next step before the GPU has verified this one.

7. The Fence: Keeping the Scheduler’s Writes Behind the Step’s Reads#

The problem from the introduction remains: the scheduler now writes into GPU tables that a running step may still be reading.

7.1 Streams#

The scheduler and the passes of a step each get their own stream. The scheduler’s stream carries its small GPU operations, such as writing slot numbers into the table, and the forward stream carries the passes. A third stream copies results to the CPU. This way, the scheduler can queue its operations without waiting for a step. But the GPU works through the streams independently, so we no longer know which of two operations on different streams runs first.

7.2 What Can Go Wrong#

This matters for the slot table. At the beginning of each pass, the attention code reads the request’s row of the slot table to find out where the cache entries of the earlier tokens are. Take again the request that ended in step 2 (section 6). The CPU learns this while step 3 is in the queue, frees the row, and may immediately give it to a new request, whose slot numbers it writes into the same row. Two orders are possible, and Figure 9 shows both. If the passes of step 3 read the row first, they find the 107 slots of the old request, and everything is fine, since those slots still hold its entries. If the write runs first, step 3 looks up its slots in a row that now belongs to another request. The tokens that it computes are thrown away, but the pass also writes its 4 cache entries into whatever slots it finds there. If the new request has just been given these slots, its own first pass overwrites them later, and no harm is done. If they are slots that it shares with other requests through a cached prompt, they stay wrong for all of them. No check fails in either case, because every slot number in the row is valid, and which order we get depends on how busy the GPU is.

The comment above this write states the rule: “it must not overlap with the previous batch’s forward, which also reads req_to_token.” The example in Figure 9 follows this comment and the code. The observed case, the flaky test from the introduction, concerned a second table of the same kind, which some models keep for their most recent tokens.

7.3 Waiting for an Event#

SGLang adds one ordering rule between the two streams. After the scheduler has launched a step, it makes its own stream wait for an event that the step records as soon as it has finished reading the tables that it shares with the scheduler. Everything that the scheduler queues from then on runs behind that event. These are the slot numbers that it writes into the table, the row that it rewrites when it keeps a finished request’s prompt for later reuse, and its writes to the second table on the models that have one. The code calls this the WAR barrier, for write after read, and I will call it the fence. In the loop from section 2, it is one call right after the launch of a step (simplified):

def _apply_war_barrier(self):
    ev = runner.shared_read_done_event       # recorded by the step, if it declares one
    if ev is not None:
        self.schedule_stream.wait_event(ev)  # open when the reads are done
    else:
        self.schedule_stream.wait_stream(self.forward_stream)   # open when the step is done

Neither call makes the CPU wait. Each only tells the GPU not to start the scheduler’s later operations before the event, or before the end of everything that is queued on the forward stream so far.

7.4 Where the Event Sits#

On NVIDIA GPUs, a plain decoding step records the event inside its graph. When SGLang recorded the graph, it placed the event right after the attention code reads the tables, so every replay records it again, and the scheduler’s operations can run early in the step. For an MTP step, the last pass that reads the tables is draft extend, and at this commit, none of SGLang’s attention implementations declares where its reads end in that pass. The scheduler’s stream then waits for the end of the whole step. Figure 10 shows the streams for this case.

The top row of Figure 10 shows the CPU. It prepares step N+1 while step N runs and queues that step’s slot writes on the scheduler’s stream. There, they are held behind the fence until step N has ended and then run in the short stretch before step N+1. Step N+1 waits for them in turn, since every step begins by telling the forward stream to wait for whatever the scheduler has queued so far. At the bottom, the lengths are copied to the CPU right after the verify pass in the configurations that need them, and the result of the step is copied when the step is done.

The fence holds back only the scheduler’s own GPU operations, which are tiny, so step N+1 starts almost as soon as step N ends. The CPU isn’t held back as long as it only queues operations and never waits for them.

7.5 No Blocking Calls#

Any change to the scheduler must preserve this last condition. A stream runs its operations in order, so a blocking call on the scheduler’s stream returns only once everything queued before it has run. That includes the wait at the fence, which for an MTP step lasts for the whole step. A comment in the code describes what this means for a single line: “a blocking .to() syncs the schedule stream, which the WAR barrier has chained to the prev forward -> host stalls a full forward.” For our MTP step, one such call per step would make the CPU wait for up to 8.2 ms, and the idle time that overlap removed would return.

This has already happened once. According to the pull request that introduced the fence, it alone lowered the share of time in which the GPU runs a pass by 4%. Two existing copies on the scheduler’s stream were blocking: a copy of lengths to the GPU and the copy of the new lengths back to the CPU. The first one became non-blocking. The second one was moved to a separate fourth stream that waits only for the event from section 4.4, so the fence doesn’t hold it back. A line that waits for the GPU looks like any other line of PyTorch code, so it is easy to add one without noticing.

7.6 Summary#

To summarize, the scheduler’s stream waits for an event that the running step records once it has read the shared tables, so the scheduler’s writes always run behind those reads.

8. What Overlap Gains, and Where the GPU Still Waits#

How much idle time do these mechanisms recover?

8.1 One Measured Step#

Figure 11 shows three MTP steps of GLM-5.3, recorded like those in Figure 2.

At the top of Figure 11, the GPU waits about 0.3 ms between steps, and it waits inside each step. An MTP step is three graph replays (the two draft passes share one) with a few dozen small operations in between. Without overlap, the CPU issues them while the GPU is working, so the short passes of the drafter wait for the CPU. The step is about 0.4 ms longer than with overlap, and I attribute the difference to these waits. With overlap, the whole step is in the queue before it starts. The gap between steps is 0.01 ms, and a step comes every 8.3 ms instead of every 9.0 ms. The CPU’s 1.8 ms of work per step, most of it for issuing the step, now fits into the previous step.

8.2 Throughput#

Figure 12 shows the throughput gain.

For Qwen3-8B, the gain is 11 to 13% from 1 to 32 requests (the introduction counted the same runs as a loss of 10 to 12%). For GLM-5.3 with plain decoding, the gain is 10% with one request, 8% with 8, and 6% with 32 and 64. By the timestamps, a step grows from 6 ms with one request to 21 ms with 64, and the gap without overlap grows only from 0.4 ms to 0.7 ms. For GLM-5.3 with MTP, it is 8% with one request and 4 to 5% with 8, 32, and 64 requests, and at those loads, the timestamps show a step 3 to 5% shorter with overlap. The accept length was 2.8 with and without overlap.

Overlap shouldn’t change what a client receives, and I checked this in two ways. First, I compared the outputs with and without overlap, token for token, for 8 prompts with up to 256 output tokens each at temperature 0. All 8 were identical, for plain decoding and for MTP on GLM-5.3, and for plain decoding on Qwen3-8B. Second, I ran GSM8K, a set of 1,319 math questions and the kind of accuracy test that exposed the problem from the introduction, with SGLang’s own test script, twice per server. GLM-5.3 with MTP scored 0.912 and 0.907 with overlap, and 0.914 and 0.920 without, so the two servers are about as far apart as two runs of one server. I haven’t checked sampling, and with a repetition penalty, the outputs may differ for the reason given in section 6.

Overlap may have one cost that throughput doesn’t show. For GLM-5.3 with MTP, the median time to the first token is, with one request, 39 ms with overlap and 34 ms without, and with 64 requests, 104 ms and 95 ms. One possible cause is that a new request can’t join a step that is already in the queue, so its prompt waits behind that step. I haven’t verified this, and plain decoding, where the same holds, shows no consistent difference.

To see what overlap is worth on your own server, run it both ways. It is on by default, --disable-overlap-schedule turns it off, and the accept length in the server log tells you whether the two runs are comparable. I started the running example like this:

python -m sglang.launch_server \
  --model-path nvidia/GLM-5.3-NVFP4 \
  --tp-size 4 --quantization modelopt_fp4 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --chunked-prefill-size 8192 \
  --mem-fraction-static 0.85 \
  --max-running-requests 64

MTP runs through SGLang’s EAGLE code path, which is why the flags say EAGLE.

8.3 The Waits That Are Left#

Figure 11 shows a step that doesn’t wait for the CPU.

A step waits only when the CPU needs a fresh result before it can launch the next one. The code has more such points than I cover here. I go through three of them: two where the GPU waits and one where only the CPU waits. I measured the first, and I describe the other two from the code. Where SGLang gives up overlap, it does so for a single batch, inside the same loop.

The first case is NGRAM. Its lookup runs on the CPU and needs the tokens that the previous step accepted. These are written to the relay at the end of the step, so the CPU has to wait for the whole step, copy them over, do the lookup, and only then launch the next step. Figure 13 shows the sequence.

The dashed box in Figure 13 is the wait: the GPU is idle from the end of one verify pass until the CPU has copied the tokens and finished the lookup. I measured it on Qwen3-8B with one more timestamp there. With one request, that’s at least 0.30 ms per step. A step comes every 4.85 ms, so the wait is 6% of the time. The idle time grows with the number of requests, since the CPU goes through the requests one by one: it is 0.54 ms with 8 requests and 1.34 ms with 32, over 10% of a 12.7 ms step. Overlap can’t remove this wait, because the wait is the CPU’s work itself. In an earlier pair of NGRAM runs, with and without overlap, the throughput differed by less than the spread between runs (3,210 and 3,178 tokens per second with 32 requests, at an accept length of 1.9).

The second case is an attention implementation that prepares part of a pass on the CPU and needs each request’s length for it. With speculative decoding, the lengths exist only after the verify pass, so the CPU has to wait until then, and the next step can be queued only while draft extend runs. If that takes the CPU longer than draft extend takes the GPU, the GPU waits for the rest. The early write of the lengths and the fourth stream exist for this case. The sparse-attention code that GLM-5.3 uses keeps the lengths on the GPU, as do SGLang’s Triton, TensorRT-LLM, and FlashAttention implementations, so the running example doesn’t have this wait. The FlashInfer implementations still ask for the lengths on the CPU, so whether a server has this wait depends on which one it uses. NGRAM needs the lengths on the CPU with any of them.

The third case is constrained output. When a request has to follow a grammar (JSON output, for example), the server masks out the tokens that the grammar forbids, and the mask depends on every token accepted so far. For the GPU drafters, SGLang handles this inside step N+1. It queues the draft passes and the verify pass first, then waits for the result of step N, advances the grammar, and builds the mask, which is needed only after the verify pass. The CPU waits, but the GPU stays busy.

So far, every GPU has processed the same batch. The recipes that SGLang’s documentation recommends for high throughput give each GPU requests of its own.

9. With DP Attention#

With tensor parallelism, every GPU holds a slice of every layer, and all of them process the same batch. SGLang runs one scheduler process per GPU even then, but all 4 receive the same requests and make identical decisions, which is why we could treat them as one. The high-throughput recipe for GLM-5.3 adds a second mode on top, called DP attention, for data parallelism. There, every GPU gets its own requests and its own KV cache and computes the attention layers alone, while the MoE layers (the large feed-forward part of the model) are still computed by all 4 GPUs together. One consequence matters here: because the GPUs meet in every MoE layer, they have to run the same pass at the same time, even though each has a batch of its own.

9.1 One Scheduler per GPU#

Each GPU already had its own scheduler process, its own relay, its own slot table, and its own fence. Now, their contents differ, and none of them knows about the others. A request’s entry in the relay and its row of the slot table live on the one GPU that serves it, and two windows are reserved per request as before.

9.2 Agreeing on a Step#

What does the next step need here that it didn’t need before? It needs to know what the other 3 GPUs are about to run, and this never has to come from a GPU. Before every step, each scheduler tells all the others how many tokens it has, which kind of pass it wants, and whether it can use its recorded graphs. Then, it waits for their answers, so that all 4 GPUs agree on one kind of step. With speculative decoding, a step can require two such exchanges: the first asks whether any GPU has a prompt to process, and only if none has one do the schedulers build their decoding batches and exchange again (code). The exchange blocks the CPU process but not the scheduler’s stream, so it isn’t the kind of call that section 7.5 warns about. With overlap on, it goes over a CPU channel and never touches the GPU, so it runs while the previous step is still running. Every scheduler waits for the slowest of the 4. With overlap, the GPUs notice this only if the slowest one doesn’t finish inside the running step, and without overlap, they wait. At this commit, SGLang combines DP attention with MTP, EAGLE, and DSpark, and on NVIDIA GPUs, it refuses NGRAM, DFlash, UNO, and a standalone draft model.

9.3 Empty Batches and Pauses#

A speculative step changes in three ways, and Figure 14 shows the first two.

First, a GPU without requests can’t skip a step. It runs an empty batch through the draft passes, the verify pass, and draft extend, so that the shared MoE layers line up, and it writes nothing into its relay (GPU 4 in Figure 14).

Second, with speculative decoding, SGLang doesn’t mix prompts and decoding within one step. When one GPU has a new prompt to process, the others pause their requests for that step and run empty batches as well (GPUs 1 and 3 in the second step of Figure 14). Their relays keep the entries of the step before, so the pause needs no handling. When decoding resumes in the third step of Figure 14, GPU 1 reads the same length, 103, and the same bonus token that it would have read one step earlier. With plain decoding, the GPUs keep decoding in such a step, and SGLang can present each of their requests as a prompt of one token, so that all GPUs replay the same recorded graph. With speculative decoding, SGLang doesn’t allow this conversion.

Third, all 4 GPUs have to agree on whether to use the recorded graphs. If even one GPU can’t replay the drafter’s graph for its batch, all 4 run the draft passes without it, and the CPU then issues their operations one by one.

9.4 What Overlap Still Covers#

With DP attention, the CPU has more to do per step, and some of it depends on other processes. To see how much of it overlap still covers, I ran SGLang’s documented high-throughput recipe for GLM-5.3 on 4 GPUs (DP attention across them, MTP with 2 draft tokens, up to 256 running requests) with and without overlap.

With one request, a step now comes every 15.6 ms with overlap and every 16.4 ms without. The CPU’s preparation of a step, which includes the exchange, takes 1.2 ms where it took 0.2 ms before. Without overlap, the GPUs wait 0.85 ms between steps where they waited 0.3 ms, and with 64 requests, 16 per GPU, they wait 1.3 ms. With overlap, the gap is 0.01 to 0.02 ms at every load, as it was without DP attention.

In throughput, overlap is worth 9% with one request, 8% with 8, 6% with 32, 5% with 64, 3% with 128, and 2% with 256 (7,532 instead of 7,378 tokens per second), at an accept length of 2.4. The ranges of my runs with and without overlap are apart up to 64 requests and touch at 128 and 256. The median time to the first token rises here as well, from 56 to 71 ms with one request and from 109 to 176 ms with 64, and it is about the same with 256, at 533 and 521 ms.

The recipe itself is made for many requests. Up to 64, it is slower than the tensor-parallel server of section 8.2 (3,675 against 4,437 tokens per second with 64 requests, with 2 draft tokens against 3), which I ran with at most 64. The answer for the recipe is the same as for the running example: the relay, the reserved slots, and the fence work per GPU as before, and the agreement between the schedulers, which is new, fits into the running step.

10. What I Left Out#

To keep this article at a manageable length, I skipped a few things:

• How the drafters work and when each pays off.
• Speculative decoding with pipeline parallelism, where the layers are spread over several GPUs in sequence.
• Servers that process prompts and decode on separate machines, which changes the loop.
• What overlap does to tail latencies. I measured only throughput and the median time to the first token.

11. Summary#

Figure 15 collects the answers to our question for every piece of state that we met.

The thing to notice in Figure 15 is how little crosses from the left side to the right. Everything that depends on the accept length stays on the GPU, in the relay or in slots that were reserved for the worst case, and the CPU works with results that are one step old. One ordering rule, the fence, keeps the scheduler’s writes behind the reads of the running step.

Coming back to the question from the beginning: in the configurations that I measured, the GPU doesn’t have to wait between steps, for plain decoding and for MTP. On GLM-5.3 with MTP and one request, the gap between steps shrinks from 0.3 ms to 0.01 ms, and the server is 8% faster. On Qwen3-8B, where a step is short, the gain is 11 to 13%. The GPU still waits wherever a fresh result has to reach the CPU first, though I measured this only for NGRAM.

Overlap can’t remove the CPU’s own work per step. For GLM-5.3 with MTP and one request, that is 1.8 ms of an 8.2 ms step. By this count, the GPU would wait for the CPU again, with or without overlap, if a step took less than 1.8 ms. I haven’t run a server with steps that short.

In practice, this means three things. Leave overlap on for throughput, and expect the most from it where steps are short: small models and few requests per GPU. When you compare two runs, check that the accept length is the same in both. And when you change the scheduler, keep whatever the next step needs on the GPU, and never wait on the scheduler’s stream.