Keeping Speculative Decoding Fast During RL Training

Most inference engines today, including SGLang and vLLM, support speculative decoding, and many deployments turn it on for the models that have a drafter. The idea goes back to two papers from 2022 and 2023, one from Google and one from DeepMind, and it is pleasantly simple. A small model, the drafter, guesses the next few tokens, and the large model, the target, checks all of these guesses in one forward pass. Since checking a handful of tokens costs about as much as generating a single one, every guess that the target accepts is a token we get almost for free.

The drafter guesses well only because it was trained to imitate one particular target model, though. That’s fine in serving, where a released model stays the same for months. In reinforcement learning (RL), on the other hand, the model changes at every training step, because changing it is the whole point of the run. (RL papers call the model being trained the policy, and some of the figures below use that word.) What happens to the drafter’s guesses while the model it imitates keeps moving away from it?

They get worse, and the speedup fades over the course of the run. This would be a minor issue if generation were a small part of RL training, but on reasoning models, it is usually the largest part. That’s why several open-source RL frameworks have started to train the drafter alongside the model, and each of them goes about it in a different way.

Here, I’d like to explain why the drafter falls behind during RL and how current systems keep it up to date. Most of the space goes to verl-SpeCo, which trains the drafter inside the RL loop. After a detour into asynchronous RL, we will compare it with slime and Miles, which train a drafter that is built into the model, and with SpecForge, which trains drafters on separate GPUs.

(I will stick to drafters that read the hidden states of the target model, since those are the ones these systems train, and I will assume that decoding runs on a single node.)

All of the details below come from the systems’ papers, repositories, and documentation, linked at the end. I didn’t run any benchmarks myself, so every measured number is one that those sources report. For the arithmetic, the running example is Qwen3-8B, the model in the verl-SpeCo benchmark. (GB means 10⁹ bytes throughout.)

Before we start, Figure 1 below shows how a drafter attaches to the target.

So the drafter is a small network built on the target’s layers rather than a separate language model, and section 3.4 explains why that matters.

1. Why Rollouts Are the Bottleneck#

Speculative decoding only makes sense if decoding is what we are waiting for, so let’s first check that this is the case in RL. Two ideas from this section come back later: decoding speed is set by how fast the GPU reads the weights, and an RL step can’t finish before its slowest rollout does.

1.1 Decoding Is Limited by Memory Bandwidth#

An LLM produces one token per forward pass, and each forward pass reads every weight of the model from GPU memory. During decoding, the arithmetic per token is small compared to the amount of data that has to be read, so the speed comes down to how fast the GPU can stream its weights.

For Qwen3-8B, with about 8 billion parameters in 16-bit precision, that’s about 16 GB per token. An H100 SXM has a memory bandwidth of 3.35 TB/s on paper. If we assume that real kernels reach roughly 75 to 80 percent of it, one forward pass takes 6 to 6.5 milliseconds, so a single request can’t be decoded faster than about 160 tokens per second.

Larger models don’t change this limit much, because a Mixture-of-Experts (MoE) model only reads the weights of the experts that are active for the current token. For example, a model with a trillion parameters, of which roughly 32 billion are active, reads about 16 GB of weights per token in 4-bit precision, which is the same as Qwen3-8B above.

1.2 Batching and Latency#

The usual way around this limit is batching. Since the GPU reads the weights once per forward pass no matter how many requests are in it, decoding hundreds of requests together multiplies the number of tokens per second.

However, batching doesn’t make any one request finish sooner. A reasoning trace with 8,000 tokens still needs 8,000 forward passes in a row, regardless of how many other requests share those passes.

This matters for RL, because the rest of the step can only start once every rollout in the batch is finished. So even though RL generates a huge number of tokens in parallel, the generation time depends on how fast the slowest rollout can be decoded.

1.3 How Much Time Generation Takes#

To get a feel for the numbers, let’s take a GRPO run, where the model writes several responses per prompt and each response is scored against the others in its group. With 512 responses per step that average 4,000 tokens, each step generates about 2 million tokens. Even at 20,000 tokens per second on a node with 8 GPUs, which is fast, that’s over 100 seconds of generation per step, and a run has hundreds of steps. Together AI reports that the rollout phase takes about 70 percent of the total training time.

One synchronous RL step has four phases:

$$T_{\mathrm{step}} = T_{\mathrm{gen}} + T_{\mathrm{old}} + T_{\mathrm{ref}} + T_{\mathrm{update}}$$

Here, $T_{\mathrm{gen}}$ and $T_{\mathrm{update}}$ are the times for generation and the optimizer update. In between, $T_{\mathrm{old}}$ covers a forward pass that recomputes the log-probabilities of the generated tokens, and $T_{\mathrm{ref}}$ covers the same pass with a frozen copy of the starting model, for a KL penalty that keeps the model from drifting too far. (Runs without that penalty skip it.) In verl’s default setup, the same GPUs take turns: they generate the rollouts, then switch over to computing log-probabilities and updating the model, so the four phases add up rather than overlap.

Speculative decoding goes after the largest of these phases, $T_{\mathrm{gen}}$, because it gets more than one token out of each forward pass of the target.

2. Speculative Decoding#

Before we can see why RL causes trouble, we need to know where the speedup of speculative decoding comes from.

2.1 Draft, Then Verify#

Speculative decoding alternates between two stages. First, the drafter proposes a block of $K$ tokens. Then, the target runs one forward pass over all $K$ proposals and checks them from left to right. It keeps every proposal up to the first one it disagrees with, replaces that one with a token of its own, and throws away everything after it. (If it agrees with all $K$, it adds one more token itself.)

The verify stage is cheap for the same reason that decoding is slow. A forward pass over $K$ tokens reads the weights once, just like a forward pass over a single token, so the two cost about the same. Let’s say the drafter proposes 4 tokens and the target accepts 3 of them. Then we get 4 tokens (the 3 accepted ones plus the target’s replacement) for roughly the price of one regular decoding step.

I will write $\tau$ for the acceptance length, the number of tokens that one verify pass produces, counting the target’s own token. (Papers don’t all count it the same way, so a quoted acceptance length can be off by one.)

2.2 Why the Output Doesn’t Change#

What makes speculative decoding safe to use inside RL is that it doesn’t change what the model generates. Let’s write $p$ for the probability that the target assigns to a token $x$, and $q$ for the drafter’s probability. The target accepts a proposed token with probability $A(x)$ and otherwise draws a replacement from a residual distribution $r$:

$$A(x) = \min\!\left(1, \frac{p(x)}{q(x)}\right), \qquad r(x) = \frac{\max\big(0,\, p(x) - q(x)\big)}{\sum_{x'} \max\big(0,\, p(x') - q(x')\big)} \tag{1}$$

This is a form of rejection sampling, and it guarantees that every emitted token follows $p$ exactly, no matter how good or bad the drafter is. As a result, a model served with speculative decoding produces outputs from the same distribution as the model served without it. A bad drafter only makes generation slower.

(In practice, verifying several tokens at once changes the kernel shapes and therefore the rounding, which adds one more small difference to the many that RL pipelines already live with between training and inference.)

2.3 How Much Faster It Gets#

How many tokens does one verify pass give us on average? If we assume that the target agrees with each drafted token with the same probability $\alpha$, the answer is a geometric series:

$$\tau = \frac{1 - \alpha^{K+1}}{1 - \alpha} \tag{2}$$

Real drafters agree less often towards the end of a block, so with $\alpha$ set to the agreement at the first position, equation (2) is an upper bound. It still shows the trend well. With a block of $K = 4$, an agreement of 0.5 gives 1.94 tokens per pass, 0.7 gives 2.77, 0.8 gives 3.36, and 0.9 gives 4.10.

Going from an agreement of 0.7 to 0.9 makes decoding 1.48 times faster. In the other direction, a drop from 0.9 to 0.85 costs about 10 percent of the speed (4.10 to 3.71 tokens per pass), which is why it matters so much when the drafter falls behind the model.

The drafter itself also costs something. If drafting one block costs $\beta$ times as much as one target pass (one drafter pass for DFlash, $K$ of them for EAGLE-3), each draft-and-verify cycle gives us $\tau$ tokens for the price of $1 + \beta$ target passes, and the speedup over regular decoding is:

$$S = \frac{\tau}{1 + \beta} = \frac{1 - \alpha^{K+1}}{(1 - \alpha)(1 + \beta)} \tag{3}$$

For example, take $\alpha = 0.8$, a block of $K = 4$, and a block drafter like DFlash that costs a tenth of a target pass per block ($\beta = 0.1$). That gives 3.36 tokens per pass and a speedup of about 3.1. Doubling the block to 8 raises it only to 3.9, because as the block grows, $\tau$ approaches $1/(1 - \alpha)$, and the speedup can never pass $1/((1 - \alpha)(1 + \beta))$, which is about 4.5 here. At an agreement of 0.9, the same limit is about 9, so a longer block pays off only when the drafter agrees often enough to reach its later positions.

Modal checked how close a real system gets to this. It fixed the acceptance length in SGLang with a simulation option, served Qwen3.5-27B on one B200 for a single request, and measured speedups of 1.86 times at an acceptance length of 2, 3.57 times at 4, and 5.62 times at 8. That’s 7 percent below the ideal speedup of 2 times and 30 percent below the ideal of 8 times. Part of the gap is the drafting cost $\beta$, and the rest is that verifying 8 tokens costs more than one regular pass, which equation (3) ignores.

2.4 Speculative Decoding at RL Batch Sizes#

A common objection is that speculative decoding works best for small batches, whereas RL generates rollouts in large ones. As a batch grows, the target stops waiting on memory and starts waiting on compute, so the verify passes stop being cheap. At that point, speculative decoding can even lower the total throughput.

The public InferenceX benchmarks show how much per-user speed drops as the batch grows. In their 8K-input, 1K-output scenario, they serve DeepSeek-V4-Pro, a 1.6-trillion-parameter MoE model, in FP4 with speculative decoding on 8 B200 GPUs. A single user gets 302 tokens per second, and each of 256 concurrent users gets 89.

Still, two things make speculative decoding worth it for RL rollouts.

First, a rollout batch doesn’t finish all at once. Towards the end of the generation phase, only the longest rollouts are still running, so the batch shrinks to a handful of stragglers. That’s where speculative decoding works best, and since the whole step waits for these stragglers, it’s also where much of the time goes.

Second, a better drafter pays off under load. In DeepSeek’s production serving system, the DSpark drafter (section 3.3) replaced a multi-token prediction (MTP) drafter that drafts one token. At the same total throughput, it makes generation 57 to 85 percent faster per user, depending on the model. Below about 150 to 200 concurrent requests, its scheduler finds enough spare compute to verify 4 to 6 tokens per request instead of 2.

2.5 Summary#

To summarize, speculative decoding gets several tokens out of each target pass without changing the output distribution, and its speed depends mostly on how often the target agrees with the drafter. The rest of this article is about keeping that agreement high while RL changes the model.

3. Three Drafter Architectures#

Next, let’s look at the three drafter designs that come up later. They draft in different ways, but none of them works from the text alone. Each one reads the hidden states of the target model (Figure 1), which already say a lot about the tokens that are likely to come next.

3.1 EAGLE-3#

EAGLE-3 drafts one token at a time with a single transformer layer. At each position, it takes the hidden states from a low, a middle, and a high layer of the target, concatenates them, and projects the result back down to the model’s hidden size:

$$g_t = \mathrm{FC}\big([\,l_t \,;\, m_t \,;\, h_t\,]\big) \tag{4}$$

Here, $l_t$, $m_t$, and $h_t$ are the hidden states of the three layers at position $t$, and $g_t$ is the fused result. The draft layer combines $g_t$ with the embedding of the token that was just sampled, and the drafter reads off the next token through a small output layer of its own, which covers a smaller vocabulary than the target’s.

During training, EAGLE-3 feeds the drafter its own earlier guesses as inputs, so that it learns to cope with its mistakes when it drafts several tokens in a row. The paper calls this training-time test. With Llama-3.1-8B as the target, EAGLE-3 produces about 6 tokens per verify pass and runs 4.4 times faster than regular decoding for a single request on MT-bench, using a tree of candidate drafts rather than a single chain.

3.2 DFlash#

DFlash drafts a whole block at once. It uses a small block diffusion model that fills in all the positions of a block in a single forward pass, and it feeds the target’s hidden states into every draft layer as extra keys and values, so that each layer can attend to them.

To train it, we pick random positions in the response as anchors. Each anchor starts a block of blanks, and the drafter learns to fill in all of them in one pass, as shown in Figure 2 below.

Note the bars in the “Loss weight” row of Figure 2. If the drafter gets the first position of a block wrong, the target throws away the whole block, whereas a mistake at the last position only costs that one token. That’s why the early positions get larger weights. (Section 6.4 gives the exact formula.)

On Qwen3-8B, DFlash runs up to 6.1 times faster than regular decoding, and on most of the paper’s benchmarks its speedup is nearly 2.5 times that of EAGLE-3.

3.3 DSpark#

Filling a block all at once has one weakness: the drafter guesses each position without knowing what it guessed for the positions before it. DSpark reduces this problem by putting a small sequential head on top of a DFlash-style backbone. The head adds a correction to the scores at each position, based on the tokens drafted earlier in the block:

$$q_k(x_k \mid x_0, x_{<k}) \;\propto\; \exp\!\big(U_k(x_k) + b_k(x_k \mid x_0, x_{<k})\big) \tag{5}$$

Here, $U_k$ are the backbone’s scores at position $k$, $x_0$ is the anchor token, and $b_k$ is the head’s correction. The head comes in two variants. The simpler one, which verl-SpeCo uses by default, only looks at the previous token, through a lookup table compressed to rank 256. The other one runs a small recurrent unit that carries a state through the block.

DSpark has a second head, the confidence head, which predicts how likely each drafted token is to be accepted. When the server is busy, a scheduler uses these predictions to decide how much of each block is worth sending to the target, as shown in Figure 3 below.

In Figure 3, the scheduler spends the target’s time only on the proposals that have a real chance of being accepted.

3.4 The Shared Output Layer#

All three designs read the target’s hidden states, and all of them are trained to match the target’s next-token distribution, which is the target’s output layer applied to its last hidden state. DFlash and DSpark go one step further: they use the target’s token embedding and output layer as their own and train only their draft layers. EAGLE-3 shares the embedding and trains a small output layer of its own over a reduced vocabulary.

So if RL changes the target’s hidden states, the drafter falls out of step with the target unless we update it too.

4. How RL Moves the Target#

Everything up to this point assumed that the target stays fixed: we train the drafter once and ship it together with the model. RL, however, changes the model with every update.

4.1 How the Drafter Falls Behind#

As the RL updates add up, the model drifts away from the checkpoint that the drafter was trained against. It probably picks up new formatting habits, new ways of reasoning, and different token statistics. Cognition reports this drift for its SWE-2 model (section 8.4), and the slime documentation warns that it can make speculative decoding a net loss. The agreement $\alpha$ drops, $\tau$ drops with it, and the speedup shrinks a little more with every step.

The longest runs suffer the most, because their model ends up furthest from the drafter’s checkpoint, and they also spend the most time generating.

4.2 Measuring the Drift#

The good news is that this drift is easy to measure. If $q$ is the drafter’s distribution over the next token and $p$ is the target’s, as in equation (1), then the probability that the target accepts the drafted token at that position is

$$1 - \tfrac{1}{2}\,\big\lVert q - p \big\rVert_1 . \tag{6}$$

This follows directly from the acceptance rule in equation (1). With the target’s hidden states for any batch of text, we can run the drafter, compare the two distributions, and read off the agreement $\alpha$ without decoding anything. verl-SpeCo instead logs the mean acceptance length that vLLM observes during the rollouts, which shows the drift as it happens.

4.3 The Obvious Fixes#

There are three ways to deal with a stale drafter, and each has a real cost:

• Retrain the drafter offline every N steps. This means pausing training, generating a new dataset with the current model, and running a separate training job. We pay for inference twice and have one more pipeline to look after.
• Turn speculative decoding off. This gives up the speedup when the rollouts are at their longest.
• Keep the old drafter. Over a long run, this can end up slower than not speculating at all. When the target rejects every proposal ($\tau = 1$), we pay for drafting and verifying and get only the one token we would have gotten anyway.

5. Where the Drafter’s Training Data Comes From#

It turns out that the RL loop already computes most of what the drafter needs, and verl-SpeCo is built on this observation.

5.1 The Drafter’s Training Examples#

Let’s think about what the drafter learns from. All three designs from section 3 train on the same kind of example: the target’s hidden states at some position, together with either the token the target actually generated next or its full distribution over that token.

RL training computes exactly this, for every generated token, at every step. During generation, the rollout engine computes the hidden states of the current model on freshly generated text. And before the update, PPO-style algorithms, GRPO included, run one more forward pass over all the generated tokens to recompute their log-probabilities with the training engine. This is the old-log-prob pass ($T_{\mathrm{old}}$ in section 1.3). verl runs it even for fully on-policy GRPO, since the update needs the training engine’s log-probabilities, which differ slightly from the rollout engine’s. The pass produces both the hidden states and the model’s full distribution over the next token at every position, and verl then throws both away.

5.2 Why the Teacher Signal Is Almost Free#

As section 3.4 noted, the teacher signal is the target’s next-token distribution, which is just its current output layer applied to its last hidden state.

This means that we don’t need to store the target’s full distribution. It’s enough to copy the rows of the current output layer that cover the drafter’s vocabulary into its trainer, which can then compute the teacher distribution itself. For Qwen3-8B, the full output layer has 151,936 × 4,096 entries, or about 1.24 GB in 16-bit precision. Storing the full distribution instead would take 151,936 × 2 bytes per position, or about 2.5 GB for the 8,192 positions that one replica keeps in a round (16 samples of 512 tokens, section 6.3).

5.3 Effect on the RL Training#

You may wonder whether training the drafter inside the loop changes the RL training itself. It doesn’t, since speculative decoding leaves the distribution of the rollouts as it is (section 2.2), however good or bad the drafter is. Updating the drafter only changes how quickly the rollouts arrive. (verl-SpeCo even refuses lossy acceptance methods unless a run explicitly opts in.)

The design is then simple: keep the data that the RL loop computes, train the drafter on it every few steps, and swap the new drafter into the rollout engines.

6. verl-SpeCo#

verl-SpeCo implements this design on top of verl, a widely used RL training framework. It supports several drafters, including EAGLE-3, DFlash, and DSpark, with vLLM or SGLang as the rollout engine (DSpark only on vLLM).

6.1 Three Hooks Around the Training Loop#

verl-SpeCo wraps the training loop of section 1.3 from the outside instead of changing verl’s code, so with the drafter turned off, what runs is plain verl.

It adds hooks at three points in the loop:

1. Collect: save the hidden states during generation or during the old-log-prob pass.
2. Train: right after the model update, train the drafter on the same GPUs.
3. Publish: send the new drafter weights to the rollout engines together with the new model weights.

Figure 4 below shows where these hooks sit in one RL step, with the weight sync to the rollout engines drawn as a fifth column at the end.

As Figure 4 shows, generation runs as before, and the rollout engines don’t see the new drafter until the publish step. The drafter’s training round, however, adds to the step time.

6.2 Two Sources of Hidden States#

The hidden states can come from two places, and the choice of rollout engine mostly decides between them.

The first option is the rollout engine, which computes the hidden states during generation anyway. These are the tensors the drafter sees when it is served, since they come from the same engine and the same kernels. In verl-SpeCo, only SGLang can return them, except with its DFlash worker, so every vLLM run, including the benchmark in section 6.7, uses the second option.

The second option is the old-log-prob pass from section 5.1. Since that pass runs over every generated token in any case, saving its hidden states costs no extra forward pass. It depends on the training engine alone, so it always works.

The two options don’t give identical tensors. The engines use different kernels, and SGLang returns the last layer’s hidden states before the final normalization layer, whereas the training pass applies it. For a drafter that reads the last layer, verl-SpeCo therefore refuses SGLang collection unless the run says that both training and serving use the pre-norm values.

6.3 What Gets Kept#

Every 5 steps, verl-SpeCo keeps at most 16 samples per rollout replica, and from each one only the first 512 tokens of the response, skipping shorter responses. Because it takes the samples in batch order, a rerun picks the same ones. Each sample records which model, which layers, and which step produced it, so that a training round can recognize data from an older model and reject it.

6.4 Training the Drafter#

Training rounds also run every 5 steps, once enough new data has come in. Each round is small on purpose: 10 optimizer steps, a batch of 4 samples per GPU, a learning rate of 10⁻⁵, and by default only the data collected at the current step. The idea is to nudge the drafter towards the current model often, at little cost each time.

Doesn’t it matter that the drafter always lags a few steps behind? RL uses small learning rates and often a KL penalty, so I would expect the model to move only a little in 5 steps. If the agreement still drops faster than the schedule can keep up with, we can shorten the interval, at the cost of more training time.

The loss functions follow each drafter’s paper, with more weight on the positions that matter most for acceptance. For DFlash, the loss is a cross-entropy against the tokens that the model actually generated, and the weight of position $k$ in a block decays exponentially, as we saw in Figure 2:

$$\mathcal{L}_{\mathrm{DF}} = -\sum_{k=1}^{K} w_k \log q\big(x_{a+k} \mid x_{\le a}, c\big), \qquad w_k = \exp\!\left(-\frac{k-1}{\gamma}\right) \tag{7}$$

Here, $a$ is the anchor position, $c$ are the target’s hidden states, $q$ is the drafter’s distribution, and $\gamma$ is 7 by default. With these settings, the first blank of a block has a weight of 1 and the eighth blank a weight of about 0.37.

EAGLE-3’s trainer by default computes the current model’s full next-token distribution with the copied output layer (section 5.2), restricted to the drafter’s smaller vocabulary, and trains against it with a cross-entropy loss.

The DSpark loss is a nice example of training directly for acceptance. It mixes a cross-entropy term with weight 0.1 and an L1 term with weight 0.9, where the L1 term is the distance between the drafter’s and the target’s distributions, weighted by the same $w_k$ as above. If we look back at equation (6), the acceptance probability is one minus half of this distance, so shrinking it makes acceptance more likely. (verl-SpeCo turns off the paper’s confidence-head loss, because its vLLM integration always verifies a fixed number of tokens.)

6.5 Publishing the New Drafter#

After a successful round, one worker exports the new drafter weights and loads them into the rollout engines. This happens together with the model’s own weight update, so every rollout engine switches to the new drafter in the pause between two generation phases. With the default synchronous publish, all engines always use the same drafter.

On each engine, the swap aborts generation, clears the KV cache, writes the new drafter weights in place, clears the KV cache again, and resumes generating.

6.6 Version Checks#

The drafter trains on several workers at once. A round in which some workers start training while others skip it, or in which a worker reads data that has changed since the round was planned, would produce a drafter trained on the wrong data.

To prevent this, every round carries version numbers for its plan, for the collected data, and for each worker process. Before training, all workers check them, and if any number doesn’t match, every worker skips the round, nothing is published, and the rollout engines keep the previous drafter.

6.7 What It Costs and What It Gains#

Besides the training rounds, saving the hidden states adds a little time to the old-log-prob pass, and every publish pauses the rollout engines and clears their KV cache.

The reported benchmark is a 100-step GRPO run of Qwen3-8B with an EAGLE-3 drafter, using vLLM-Ascend on Huawei Ascend NPUs. Updating the drafter gives a higher mean acceptance length than keeping it fixed, and compared to the baseline, the rollouts get about 20 percent faster and the end-to-end training about 11 percent faster, with no drop in accuracy.

These gains are much smaller than the serving numbers in section 3 (4.4 times for EAGLE-3, 6.1 times for DFlash), and I see four reasons. First, the baseline may already include the fixed drafter. Second, the serving numbers are for a single request (EAGLE-3’s with a draft tree), whereas RL rollouts run in large batches and verl-SpeCo drafts a single chain of 3 tokens. Third, the RL numbers include the costs above. Finally, generation is only one of the four phases of a step. (If 20 percent faster rollouts give 11 percent faster steps, generation must have taken at least half of each step.)

7. Asynchronous RL#

Asynchronous RL removes the barrier between generation and training, so the trainer no longer waits for the slowest rollout. Doesn’t that solve the problem from section 1 on its own?

It helps, but only partly. The engine still has to decode each rollout one token after the other, and an asynchronous schedule doesn’t make that any faster. What changes is where the speedup shows up. In a synchronous step, speculative decoding shortens $T_{\mathrm{gen}}$ directly. In an asynchronous system, it raises the rate at which finished rollouts arrive in the training buffer.

I would expect the rollout engines to remain the limiting factor in most runs, since generating a token costs more than training on it. If they do, faster decoding shortens the whole run. Cognition, for example, trains SWE-2 with off-policy RL and a rollout system designed to limit staleness, and still lists an online-trained DSpark drafter among its rollout improvements.

There is a second effect. In asynchronous RL, the weights in the rollout engines lag behind the trainer by some number of steps, and the larger the lag, the more off-policy the training data becomes. Faster rollouts keep that lag short, because each rollout spans fewer trainer steps while it is being decoded, so it makes sense to combine asynchronous RL with speculative decoding. The Miles framework does both. In a published run with MTP enabled, it trains the 744-billion-parameter GLM-5.2 model on 64 GB300 GPUs, half of them for rollout, and the rollout weights lag 1.7 steps behind the trainer on average, with steps of about 4.5 minutes.

8. Other Approaches#

verl-SpeCo is not the only way to keep the drafter up to date during RL. Let’s look at two other designs that make quite different trade-offs.

8.1 Joint MTP Training in slime and Miles#

slime and its fork Miles take the most direct route. Some models, such as DeepSeek-V3 and GLM-4.7, come with a built-in MTP layer, which predicts one token further ahead than the model and can serve as the drafter. slime and Miles train this layer in the same backward pass as the model. There is no separate trainer, no pipeline for hidden states, and no publish step. All it takes is a checkpoint that includes the MTP weights and a few training flags.

This works because the drafter is itself a layer of the model. The training forward pass produces the hidden states the MTP layer needs, and the rollout tokens in the training batch provide its labels, as Figure 5 below shows.

In panel A, the MTP layer gets the model’s hidden state at position $t$ and the embedding of token $t+1$, and predicts token $t+2$, so its labels are just the rollout tokens shifted by one more position. The important detail is in panel B: the MTP layer’s inputs are detached from the model, so the MTP loss, scaled by 0.2 and added to the RL loss, trains only the MTP layer, and one optimizer step updates both. Panel C shows the payoff: the MTP layer reaches the rollout engines in the same weight update and always has the model’s version, even though it trained against the model from before that step.

The trade-off is that this only works for models with a built-in MTP layer. Separate drafters like EAGLE-3, DFlash, or DSpark can’t share the backward pass. slime lists them as work in progress, and Miles says they are not supported yet and recommends retraining them offline.

8.2 SpecForge#

SpecForge, the drafter training framework from the SGLang project, goes the opposite way and moves training off the RL GPUs entirely.

In its online mode, an SGLang server runs the target model, captures the hidden states during prefill, and writes them into Mooncake, a distributed object store that moves tensors over RDMA or TCP. The trainer reads them on its own GPUs, and only sample IDs and tensor shapes pass through the control plane. The drafter therefore trains against whatever weights that server holds.

SpecForge doesn’t run RL itself. It trains drafters for all three designs from section 3 and several more, and both slime and Miles document how to load them. To follow an RL run, an RL framework would have to keep loading the newest model into the SpecForge server.

8.3 Comparing the Three Designs#

The three designs differ in where the drafter’s data comes from, where the drafter trains, and how its weights reach the rollout engines. Figure 6 below puts them side by side.

Apart from where the drafter trains, Figure 6 shows how far each drafter can lag behind the model. The MTP layer in slime and Miles always has the same version as the model it serves. verl-SpeCo’s drafter can be up to about 5 steps behind with the defaults, because it trains and publishes every 5 steps. And SpecForge’s drafter follows whatever model its server holds, so its lag depends on how often the RL framework updates that server.

8.4 Experience from Industry#

Two teams report how they deal with a stale drafter during RL:

• Cognition describes the whole sequence for SWE-2, a coding model post-trained from Kimi K3. It used DSpark for the RL rollouts, saw acceptance drop as the model changed, retrained the drafter with SpecForge, which gave 15 percent longer acceptance lengths, and then switched to online drafter training.
• Together AI uses a training-free suffix-tree speculator that doesn’t read hidden states, and keeps it in step with the model by inserting each verified rollout into it. It reports over 50 percent less rollout time on math RL with DeepSeek-R1-Distill-Qwen-7B and about 25 percent on code RL with Qwen3-8B, with unchanged reward curves.

9. Topics I Skipped#

There is more to speculative decoding in RL than I could fit here, and I left out the following:

• Training objectives that aim at acceptance more directly, such as LK losses, which replace the KL-divergence loss, and D-PACE, which replaces the fixed position weights from section 6.4 with weights that focus on whichever positions currently limit acceptance. Both are built into SpecForge.
• Re-SpS, which works the other way around and uses RL to pick the shape of the draft tree (its depth, top-k, and token budget) instead of training the drafter.
• The other drafters that verl-SpeCo and SpecForge support, such as P-EAGLE, Domino, and DFlash2.
• GPU numbers for verl-SpeCo. Its benchmark ran on Ascend NPUs, and the repository says that more results, including DFlash, will follow.

I hope to come back to some of these in a later article.

10. Conclusion#

The question at the start was what happens to speculative decoding when RL changes the target model at every step. The short answer is that the speedup keeps shrinking, because each RL update moves the target further from the drafter, and small drops in agreement cost a lot.

The fix is cheaper than it first looks. The RL loop already computes the drafter’s training data from the current model, and then discards it. And since speculative decoding doesn’t change the output distribution, using that data to keep the drafter current doesn’t bias the training.

The systems put the drafter’s training in three places. verl-SpeCo trains several kinds of drafters in small rounds inside the loop and publishes them together with the model, which gives about 20 percent faster rollouts and 11 percent faster training on Qwen3-8B with EAGLE-3 on Ascend NPUs. slime and Miles train a built-in MTP layer in the same backward pass as the model. And SpecForge provides the piece for a third design, a drafter trainer on separate GPUs that reads hidden states from a live server.

My guess is that training the drafter online will become a standard feature of RL frameworks, much as asynchronous rollouts have become one. Which of the three designs ends up winning depends on whether future models ship with built-in MTP layers, and, for separate drafters, on whether spare GPUs for a dedicated trainer cost less than the time that training rounds add to each step.

References#

• verl-SpeCo, the system covered in sections 5 and 6, and verl, the RL framework it builds on.
• Fast Inference from Transformers via Speculative Decoding, by Leviathan et al., and Accelerating Large Language Model Decoding with Speculative Sampling, by Chen et al.
• EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test, by Li et al.
• DFlash: Block Diffusion for Flash Speculative Decoding, by Chen et al.
• DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation, by Cheng et al.
• slime, and Miles with its announcement post, which describes the GLM-5.2 run from section 7.
• SpecForge, drafter training for SGLang.
• LK losses, D-PACE, and Re-SpS, from section 9.
• Cognition on SWE-2.
• Together AI on accelerating RL rollouts with distribution-aware speculative decoding.
• Modal’s Speculation Is All You Need.
• The InferenceX DeepSeek-V4-Pro serving benchmarks by SemiAnalysis, retrieved on October 2, 2026.