How an MoE Model Picks Its Experts During LLM Inference

Almost every description of a Mixture-of-Experts (MoE) model contains a sentence like “a router selects 8 of the 288 experts for each token,” and then moves on to the experts. For GLM-5.3-Flash, the router of one layer computes 288 dot products of length 4,096 for each token, or 1.2 million multiply-adds. The experts that run afterward need 226 million, so the router does about half a percent of the layer’s feed-forward arithmetic.

Yet the developers of inference engines have written several kernels, functions that the GPU runs, for this step alone. The reason is that the router is cheap to compute and expensive to launch. On a B300, an NVIDIA Blackwell GPU, a kernel that does nothing takes 1.2 microseconds, and one that only reads the router’s weight matrix from memory takes 2.6. What the SGLang inference engine runs for GLM-5.3-Flash takes 7.0 microseconds per layer. Multiplied by the model’s 42 MoE layers, that is 0.3 ms (milliseconds, whereas I spell out microseconds). A decode step, in which the model produces the next token, takes 5.0 ms for a single request on my server, and about two thirds of a router timing shows up in it (section 5.2). The router is therefore worth about 4% of a step, and this article is about where that 4% comes from. More than half of that time is the fixed cost of running kernels and reading cold memory.

If we write the router the naive way, as five lines of PyTorch, it takes 26.9 microseconds per layer. SGLang’s fastest router for one token, a pair of kernels, takes 6.7 (section 5). Of the 20 microseconds in between, 13 come from a single change: computing the scores and the selection in one kernel instead of seven. Multiplying in 16-bit floats saves 4, and a faster multiplication kernel another 3. After that, three more designs merge kernels further. For a single token, none of them gains a full microsecond, and with more tokens, two of the three save several microseconds. The article goes in this order: separate kernels, a fused pair of kernels, a router absorbed into the kernel that runs the experts, a router absorbed into the matrix multiplication in front of the experts, and a single kernel.

A second thread runs through the article: a router with a selection bias (section 1.3) is the one place in the model where SGLang avoids rounding. The model computes in bf16, a 16-bit float format, and the expert weights are often quantized to 4 bits, but SGLang keeps the inputs to the router’s selection in fp32, the 32-bit format. With rounded inputs, many tokens select different experts, and fp32 costs almost nothing here. However, the effect that I could measure on the output is small, as section 2.3 shows. The fp32 requirement comes up three times below, in sections 5.1, 6.3, and 7.

The running example is GLM-5.3-Flash with 4-bit expert weights, served by SGLang on B300 GPUs, with kernels from the FlashInfer library. I chose it because I can serve it on two GPUs of a shared machine and because most designs below apply to it. For every design, I will say which models use it on this hardware. Unless I say otherwise, the timings are my own, measured on one B300 with the model’s real router weights and with hidden states that I captured from a running server.

I cover only inference on NVIDIA GPUs, and I stop where the experts start to run. I include the bookkeeping that the expert kernel needs beforehand, counting and sorting the tokens, since SGLang lets that kernel do the routing by default. Sending tokens to experts on other GPUs is covered in How GPUs Talk to Each Other During LLM Inference.

1. What the Router Computes#

1.1 Three Steps#

An MoE model replaces the feed-forward module of a transformer layer with many smaller ones, the experts, and only a few of them run for a given token. The router decides which ones and how much each contributes to the result.

It does this in three steps. First, it multiplies the token’s hidden state by the gate weight, a matrix that has one row per expert. This gives one number per expert, a logit. Second, it turns the logits into scores, for example with a sigmoid. Third, it selects the k experts with the highest scores and rescales their scores so that they sum to 1. The layer later adds up the experts’ outputs with these rescaled scores as weights.

To make this concrete, let’s take a toy layer with 8 experts and k = 2, as shown in Figure 1.

In Figure 1, experts 2 and 5 have the highest logits and therefore the highest scores, 0.881 and 0.870. After rescaling, expert 2 gets a weight of 0.503 and expert 5 gets 0.497. Note how little leaves the router: for each token, k expert ids and k weights.

GLM-5.3-Flash has 45 transformer layers. The first 3 have an ordinary feed-forward module, and the other 42 are MoE layers with 288 experts each, of which the router selects 8 per token. Every MoE layer also has one shared expert that runs for all tokens. The hidden state has 4,096 numbers, so the gate weight of a layer is a 288 × 4,096 matrix, or 2.4 MB in bf16.

1.2 Who Waits for the Router#

If the router is that small, why should we care how long it takes? The reason is its position in the layer, which Figure 2 illustrates.

In Figure 2, the selected experts can’t start before the router is done. The router and the selected experts therefore run one after the other, 42 times per forward pass. During decoding, SGLang runs the shared expert at the same time, so the router doesn’t have the GPU to itself. Section 5.2 measures how much of a router timing a server sees.

1.3 The Rank Key and the Weight#

It’s tempting to think of every router as “a softmax followed by a top-k.” That is true for softmax routers, such as those of Mixtral and Qwen3. There, one array does two jobs: the softmax probabilities are ranked to pick the experts, and the selected ones become the weights after rescaling.

DeepSeek-V3 changed two things, and the GLM-5 series and the Kimi models follow it. First, the score of an expert is the sigmoid of its own logit. Second, the model stores one bias per expert, which is added to the scores before they are ranked. The bias balances the load of the experts during training, and at inference time, it is a fixed vector in the checkpoint. The DeepSeek-V3 paper is explicit about where it is used:

Note that the bias term is only used for routing. The gating value, which will be multiplied with the FFN output, is still derived from the original affinity score s_i,t.

In other words, there are now two arrays. I will call the first one the rank key, which is the score plus the bias and is used only to pick the k experts. The second one is the score itself, which becomes the weight after rescaling. Figure 3 shows both kinds of router for the 8 logits of our toy example.

On the left of Figure 3, experts 2 and 5 are selected, as in Figure 1. On the right, the biases favor expert 0 a little: its rank key of 8.899 is now the second highest, and expert 2 drops out with 8.881. The weights, however, are computed from the scores 0.769 and 0.870, without the bias.

The boundary between the last selected rank key and the first rejected one is the cut, and the difference between these two keys is the gap at the cut. In Figure 3, the gap is 8.899 − 8.881 = 0.018. Here, each rank key is a bias near 8 plus a score between 0 and 1, so the selection is decided in the third and fourth significant digits. Most of the kernels below implement the right side of Figure 3. Other families vary the score or the bias. DeepSeek-V4, for example, replaces the sigmoid with the square root of a softplus.

2. Why Engines Don’t Round the Router#

Activations in these models are bf16, and nobody worries about that. A bf16 number has 8 significant bits, so it is accurate to about 0.4%, and such a small error in one activation barely changes the layer’s output. The router is the one place where this reasoning breaks down, because of its third step.

2.1 An Error Larger Than the Gap#

A top-k is not a continuous function of its input. As long as the order of the rank keys at the cut stays the same, the output doesn’t change, but as soon as two of them swap places, a different expert runs. What matters is whether a rounding error is larger than the gap at the cut.

How large is the bf16 rounding error of a bias? Between 8 and 16, neighboring bf16 values are 0.0625 apart, so a bias in that range can be off by up to 0.03. In the example from Figure 3, storing the 8 biases in bf16 turns 8.03 into 8.0, and the rank key of expert 5 drops from 8.900 to 8.870, below the 8.881 of expert 2. The router now selects experts 0 and 2 instead of 0 and 5.

The same thing happens in a real checkpoint. Figure 4 shows the 288 biases of layer 24 of GLM-5.3-Flash, as stored in fp32 and after a cast to bf16.

In Figure 4, the biases lie between 11.54 and 11.96, and in bf16, they collapse onto 7 values. Across the checkpoint’s layers, the median is 8 distinct values. Biases that the model keeps apart by a few thousandths become indistinguishable. This depends on their size and not on the format alone. Kimi K3’s biases lie between −0.76 and 0.15, where neighboring bf16 values are at most 0.004 apart, and a cast to bf16 keeps 302 to 644 of the 896 biases of a layer distinct.

2.2 The Gap, Measured#

The size of the error is only half of the comparison. To measure the gap, I served GLM-5.3-Flash on two GPUs and sent 8 requests at once, each asking it to explain a page of SGLang’s documentation. I saved the hidden states that arrive at the router of each of the model’s 42 MoE layers. Per layer, that’s about 16,400 hidden states from the prompts and about 680 from 86 decode steps. For every hidden state in every layer, I then recomputed the selection with different arithmetic and compared it to a reference computed in 64-bit floats.

This is a layer-by-layer recomputation on the hidden states of a normal run. It tells us how often a layer would choose differently, but not what the changed choices would do to the later layers.

Figure 5 shows the distribution of the gap at the cut, together with the median error of the rank key for three kinds of arithmetic.

In Figure 5, with fp32 logits and an fp32 bias, the median error of a rank key is less than a millionth, and 0.004% of the prompt tokens select a different expert set in an average layer. If we round the logits to bf16, the median error is 0.0002, and 2.5% of the tokens select a different expert set. (The error is that small because a logit passes through the sigmoid, whose slope is at most 1/4, before it becomes part of the rank key.)

With the bias in bf16, the median error is 0.012, more than twice the median gap of 0.005. Accordingly, 66% of the prompt tokens select a different expert set in an average layer, and 7.6% of the weight goes to experts that the reference wouldn’t have selected. During decoding, the gaps are somewhat wider, and the shares are 2.2% with bf16 logits and 55% with a bf16 bias.

A softmax router has no bias, but its logits can still be rounded. For Qwen3-30B-A3B, which selects 8 of 128 experts, bf16 logits change the expert set of 6.6% of the prompt tokens per layer in the same kind of experiment. SGLang nevertheless serves this model with bf16 logits, since its gate is an ordinary bf16 linear layer.

2.3 What the Flips Do to the Output#

Do these flips of the expert set matter? I tested that on GLM-5.3-Flash with speculative decoding as in its recipe, where a draft layer proposes several next tokens and the model verifies 6 of them per step. I ran four servers: one as shipped, one with a bf16 bias, one with bf16 logits, and one with both. On the 1,314 GSM8K problems that SGLang’s evaluation script scores, they scored 97.2%, 97.3%, 97.2%, and 97.0%, and a second run of the shipped server scored 97.0%, so these differences are noise. The acceptance length, the average number of draft tokens accepted per step, was 4.27 and 4.28 in the shipped runs, 4.28 with rounded logits, and 4.25 in both servers with a rounded bias.

From these averages, I calculate that the rounded bias costs about 0.6% of acceptance length, and I can’t show any other effect on this model. GSM8K is an easy benchmark, so this tells us only that the output doesn’t break. My guess is that the model tolerates the flips because they happen between experts with almost tied rank keys, which it rated as nearly equally suitable.

SGLang shipped the rounded bias for a while, for 4-bit checkpoints on Blackwell GPUs. The pull request that removed it reports a much larger effect on GLM-5.2, an acceptance length of 3.88 after its two fixes against 3.46 before, which I didn’t reproduce on my checkpoint.

2.4 Why fp32 Costs Almost Nothing#

A matrix multiplication with bf16 inputs adds up its products in fp32 anyway, and the output type decides only whether the finished sum is rounded before it is stored. For one row, it takes 6.5 microseconds if it writes fp32 logits and 6.3 if it writes bf16, and the bias is 288 numbers either way.

To summarize, a token changes its experts when the rounding error of a rank key is larger than the gap at the cut. That happens often with a bf16 bias. The measured effect is small, but avoiding it is nearly free. This is why the kernels in the rest of this article take bf16 hidden states and bf16 weights but hand fp32 logits and an fp32 bias to the selection.

3. Rows, Launches, and the Floor#

If the arithmetic of the router is negligible, what does its time consist of? To answer that, we need three things: how many tokens the router sees at once, what a kernel costs before it does any work, and how long a whole decode step takes.

3.1 Rows#

So far, we have followed one token. A real forward pass processes many tokens at once: the hidden states form a matrix with one row per token, and so do the logits, with 288 columns. During decoding, every request contributes one row per step, or several with speculative decoding. The low-latency recipe for GLM-5.3-Flash verifies 6 tokens per step, so a single request is 6 rows and 16 concurrent requests are 96.

During prefill, the whole prompt goes through the model, usually in chunks of several thousand tokens. I will use 8,192 rows as the example, the chunk size in a recipe from the InferenceX benchmark for GLM-5.2. Figure 6 puts these three operating points and a single row on one axis.

Most of the kernels below are built for one part of the axis in Figure 6.

3.2 What a Kernel Costs Before It Computes#

On a GPU, every operation runs as a kernel, which many threads execute in parallel. The threads are organized in blocks, and each block runs on one of the GPU’s processing units, the streaming multiprocessors (SMs), of which a B300 has 148. Inside a block, threads run in groups of 32, called warps.

I measured all kernel timings in this article inside a CUDA graph, a recorded sequence of kernel launches that the GPU driver replays without going through Python, since inference servers run the decode step this way. A timing is the span from the start of the first kernel to the end of the last one, as recorded by CUPTI, NVIDIA’s profiling interface.

A kernel runs warm when the data it reads is in the GPU’s L2 cache, and cold otherwise. For the router, a cold gate weight is the normal case, since all the other layers run between two calls of the same layer. I therefore flush the cache before each measurement, which also evicts the hidden state that a server would still have warm.

3.3 The Floor#

What is the least that a router can cost? It has to launch at least one kernel, and that kernel has to read the gate weight, 2.4 MB, from cold memory. I measured both with kernels that do no useful work.

An empty kernel with a single block takes 1.2 microseconds, and so does an empty kernel with 288 blocks of 512 threads. If each of these threads also loads 16 bytes of the gate weight, so that the kernel reads the whole matrix once, it takes 2.6 microseconds. I will use about 2.6 microseconds as the floor on this GPU. The two parts can be told apart with a warm cache: the empty kernel then takes 0.9 microseconds and the reading one 1.4. A launch costs about a microsecond with or without cold memory, and the first touch of 2.4 MB of cold memory costs about 1.1. The floor leaves out work: a real router also has to select and to write its results. Every additional kernel adds its own launch: a chain of 12 nearly empty kernels takes 13.9 microseconds, or 1.2 per kernel.

3.4 The Decode Step#

The other end of the scale is the decode step. With one request at a time and without speculative decoding, GLM-5.3-Flash takes 5.0 ms from one token to the next on two B300 GPUs. That’s the median a client sees over three sets of 6 ShareGPT conversations with 256 output tokens each, so it includes the server’s work outside the forward pass. Dividing by the 42 MoE layers gives 119 microseconds per layer for attention, experts, and everything else.

Against this, the floor of 2.6 microseconds is 2% of a layer. A router at 7 microseconds is 6%, and one at 27 would be 19% of the longer layer. These shares come from kernel timings, and section 5.2 will show that the server sees about two thirds of such a difference. I didn’t measure a decode step with speculative decoding.

4. Where the Time Goes#

With the floor and the decode step in place, we can ask where the router’s time goes. I do this for one row, where the fixed costs are easiest to see, and give the same numbers for the 6 rows of the low-latency recipe at the end of the section. Figure 7 shows six ways to build the router, from the naive one to the designs of the following sections.

The top bar of Figure 7 is the reference code of the model, which does everything in fp32 with plain PyTorch calls. No engine ships this naive version. Written out, it is five lines:

logits = hidden.float() @ gate_weight_fp32.t()
scores = torch.sigmoid(logits)
ids = torch.topk(scores + bias, k, dim=-1, sorted=False).indices
weights = scores.gather(1, ids)
weights = weights / (weights.sum(-1, keepdim=True) + 1e-20)

These five lines launch nine kernels and take 26.9 microseconds. Figure 8 shows them on a time axis, together with what SGLang runs for this model.

At the top of Figure 8, three kernels take 16 of this run’s 26 microseconds: the cast of the hidden state to fp32, the multiplication, and the top-k. The other six take between 1.3 and 1.8 microseconds each, which is close to an empty kernel’s 1.2. None of the nine has much to compute, but each reads its input from memory and writes its output back. The bottom of Figure 8 is what ships: two kernels that sections 5 and 6 explain.

The next three bars of Figure 7 remove this overhead one step at a time:

• Multiplying in bf16 and writing fp32 logits, as in section 2.4, saves the cast and brings the router to 23.0 microseconds. Nine kernels remain, because this multiplication launches two (section 5.1).
• The tiny GEMM of section 5.1, a multiplication kernel for few rows, brings it to 19.7.
• Replacing the seven kernels of steps 2 and 3 with one brings it to 6.7. This design is the fused pair, since two kernels remain.

Of the 20.2 microseconds between the naive version and the fused pair, 13.0 come from fusing steps 2 and 3. The matrix multiplication, the only part with real arithmetic, accounts for the other 7.2.

The fused pair is 4.1 microseconds above the floor. For the 6 rows of the low-latency recipe, the six designs take 35.5, 25.1, 21.3, 7.0, 8.2, and 8.0 microseconds, with what ships at 8.2, behind the fused pair’s 7.0. The last two bars of Figure 7 are slower for one row, at 7.0 microseconds for routing inside the expert kernel and 7.9 for a single kernel. They exist because they remove kernels elsewhere or because they are faster with more rows. Sections 5 to 8 cover four designs:

• The fused pair (section 5), which runs for our model when SGLang does the routing.
• Routing inside the expert kernel (section 6), the default for our model on a B300.
• Logits from the matrix multiplication in front of the experts (section 7), which only Kimi K3 uses.
• A single kernel (section 8), which SGLang doesn’t use for our model but which vLLM, another inference engine, uses for the DeepSeek-V4 models.

5. The Fused Pair#

The fused pair consists of one kernel for the logits and one for steps 2 and 3. On a B300, it runs for routers with a selection bias whenever SGLang does the routing. As far as I can tell, there are three cases: setups that send tokens to experts on other GPUs, DeepSeek-V4 checkpoints with 4-bit experts by default, and our model with a flag from the appendix.

5.1 The Logits: A Matrix Multiplication Near the Floor#

The first step is a general matrix multiplication, a GEMM: the hidden states, with one row per token, times the gate weight. GPU vendors ship tuned libraries for this, cuBLAS in NVIDIA’s case, and PyTorch calls cuBLAS for us. cuBLAS takes 6.5 microseconds for one row of our model and 6.7 for 64 rows. Almost all of its time is therefore fixed cost, and part of that is a second kernel: cuBLAS computes each dot product in several parts and adds them together in another launch.

A kernel that has to handle only a few rows can be organized differently. SGLang calls its version the tiny GEMM, and Figure 9 shows how it divides the work for our model.

As shown in Figure 9, the kernel starts 144 blocks, and each block handles 2 of the 288 experts, which keeps the number of blocks below the 148 SMs of the GPU. A block has 256 threads, and each thread owns 16 consecutive positions of the 4,096. It loads these 16 numbers from the 2 expert rows of its block and from every token row, multiplies the pairs, and adds the products in fp32. Finally, the block adds up the 256 partial sums for each combination of a token and an expert and writes the total to the logits.

The kernel also starts its cold reads early. A kernel normally starts when the previous one has finished, but with programmatic dependent launch (PDL), it may start earlier and waits only before it reads the previous kernel’s output. The gate weight doesn’t depend on that output, so the threads request it first and wait afterward.

For one row, the tiny GEMM takes 2.8 microseconds, and section 3.3 showed that reading the gate weight cold takes 2.6. The logits are already near the floor, and a faster multiplication has little to gain for one row.

Figure 10 shows how this continues along the row axis.

In Figure 10, the tiny GEMM slows by about 0.1 microseconds with every additional row, because each thread has more to load and to multiply, while cuBLAS stays flat. SGLang switches to cuBLAS above 16 rows. Compiled for up to 64 rows (the dashed line), the tiny GEMM is still ahead at 24 rows and level at 32, so the 16-row limit is conservative for this model. With the low-latency recipe’s 6 rows per request, the tiny GEMM serves up to 2 concurrent requests.

The gray line in Figure 10 is the multiplication of the naive version, with everything in fp32. It takes 9.5 microseconds for one row and 16 to 25 for 2 to 64 rows.

A related kernel comes from TensorRT-LLM, NVIDIA’s inference engine, and is available through FlashInfer. It uses the tensor cores, the GPU’s matrix-multiplication hardware, and takes 4.2 to 4.5 microseconds from 1 to 64 rows for our gate weight. It writes bf16 logits, though, so the fp32 requirement from section 2 rules it out for our model. This is where fp32 logits have a price: between 17 and 64 rows, cuBLAS is about 2 microseconds slower than this kernel.

5.2 How Much of a Kernel Timing Arrives in a Server#

Does the user see these microseconds? I served GLM-5.3-Flash on two B300 GPUs in three variants, without speculative decoding, and changed only the kernel behind the logits: SGLang’s default, cuBLAS at every row count, and the fp32 multiplication of the naive version.

With one request at a time, forcing cuBLAS adds 0.10 ms to the median time between two tokens, and the fp32 multiplication adds 0.22 ms, consistently over three sets of prompts. (I launched each variant once per run. Four default launches with these 6-prompt sets differ by 0.05 ms among themselves, so the 0.10 ms is twice that spread and the 0.22 ms four times. A second run with synthetic prompts gave 0.10 and 0.18 ms.) From Figure 7’s steps and the 42 layers, I had expected 0.14 and 0.30 ms, so about two thirds of a difference in kernel timings arrives in the server. I can’t say for certain why. My guess is that the benchmark is harder on a kernel than the server is, since it flushes the whole cache and lets nothing overlap. The tiny GEMM is then worth about 2% of a single user’s decode step, and bf16 inputs to the multiplication another 2%. With 16 concurrent requests, a step has up to 16 rows and takes 10.4 ms. The default still uses the tiny GEMM there, and cuBLAS is within 0.1 ms of it, whereas the fp32 multiplication adds 0.7 ms per step.

5.3 Steps 2 and 3 in One Kernel#

SGLang’s fused router kernel replaces the seven gray kernels of Figure 8. It is written in Triton, a Python-based language for GPU kernels.

For our model, the kernel assigns one token row to one warp, so its 32 threads hold the 288 logits, 9 each on average. As far as I can tell from the code, the row never has to leave the threads’ registers from this point on. The kernel computes the scores and the rank keys as two separate arrays, as in Figure 3, and then selects the experts in k rounds. Figure 11 shows one such round with 16 experts and 4 threads.

As shown in Figure 11, every thread first finds the largest of its own rank keys, and the threads then agree on the largest one overall. If several experts hold this value, the one with the lowest id wins. The kernel then reads the score of the winner from the score array, marks its rank key as used, and starts the next round. After k rounds, it rescales the k scores and stores the ids and the weights.

By itself and with a cold cache, this kernel takes 4.0 microseconds for one row and 4.4 for 96 rows, since different warps handle different rows.

5.4 What the Pair Costs#

Together, the two kernels take 6.7 microseconds for 1 row, 7.0 for 6 rows, and 11.2 for 96 rows, where cuBLAS replaces the tiny GEMM and adds its second kernel. That’s 4.1 microseconds above the floor for one row, and almost all of it belongs to the fused router kernel. Its launch explains 1.2 of that and its cold reads 0.5, since it still takes 3.5 microseconds with a warm cache. That leaves over 2 microseconds for the selection itself, so the remaining designs can remove only the launch and the cold reads.

How much of this gap could a better pair recover? I wrote a pair of my own for the DeepSeek-V4 shapes. It is not released, and I mention it only for one comparison. Its GEMM applies the scoring function before it stores the result, and the top-k is chained to it with PDL. For the gate weight of DeepSeek-V4.1-Flash (384 experts and a hidden size of 5,120), it takes 5.0 microseconds for one row, against 6.8 for SGLang’s pair and a floor of 3.1 for this shape. I measured these three numbers on the same B300 with random hidden states. Even a pair that is tuned for one shape stays 1.9 microseconds above the floor.

6. Routing Inside the Expert Kernel#

On a B300, GLM-5.3-Flash doesn’t run the fused router kernel by default. SGLang hands the logits to the expert kernel and lets it do steps 2 and 3, which puts the router 4.4 microseconds above the floor for one row. This is the default for the GLM-5 series on Blackwell GPUs as long as no tokens are sent to experts on other GPUs. Why would an engine that has a good kernel for steps 2 and 3 not use it?

6.1 What the Expert Kernel Needs#

The expert kernel receives a batch of tokens, each of which wants 8 different experts. Before it can multiply anything, it has to turn this around. For every expert, it needs to know how many tokens that expert received, and it needs the tokens sorted so that all tokens of one expert sit next to each other. Only then can it run one matrix multiplication per expert.

Both are computed from the expert ids, so an expert kernel that receives finished ids from the engine still needs bookkeeping kernels of its own to count and to sort. The expert kernels that FlashInfer adopted from TensorRT-LLM therefore offer a second interface, which takes the logits and the bias instead of the ids. It does the scoring, the top-k, the counting, and the sorting in one routing stage, as shown in Figure 12.

The bottom of Figure 12 shows that this routing stage is not one fixed kernel. Depending on the number of rows, FlashInfer picks a different arrangement: one small kernel for up to 4 rows, a slightly different one for up to 16 rows, and, for our 288 experts, two chained kernels beyond that. The source file gives the reason for the split: in a single kernel, every warp has to carry its top-k through the bookkeeping, most warps have no row to work on, and nothing can overlap. This is the trade-off of section 8.1 inside the expert kernel. Above 256 rows, the second kernel uses more blocks.

6.2 Is It Faster?#

FlashInfer exposes the routing stage as a separate function, so I could time it. Alone, it takes 4.5 microseconds for 1 row, 5.4 for 6 rows, and 8.0 for 96 rows. Together with the tiny GEMM, or cuBLAS above 16 rows, the router then takes 7.0, 8.2, and 14.2 microseconds for the same rows. (The sum of the two parts is a little more, because the second starts while the first is still running, as in Figure 8.) Of the 7.0 microseconds for one row, the two launches account for 2.4 and the cold cache for 1.5, since the two kernels take 5.5 with a warm cache. That leaves about 3 for the selection and the bookkeeping.

These totals are higher than the fused pair’s 6.7, 7.0, and 11.2 microseconds, but the two don’t do the same work, because the expert kernel behind a fused pair still has to count and sort. I couldn’t time that bookkeeping by itself, so I compared the two complete paths in a server. I ran GLM-5.3-Flash with the default and with the backend flag from the appendix, with which SGLang’s fused pair does the routing and the same expert kernel receives ids.

A first attempt with 6 prompts per set couldn’t tell them apart, because one launch of the second path behaved differently from all the others for reasons that I couldn’t find. I therefore repeated the comparison with 24 prompts per set and three launches per variant in mixed order. With one request at a time, the median time between two tokens is 5.10 ms in all three default launches and 5.19 ms when SGLang routes. With 16 concurrent requests and 48 prompts per set, it is 10.39 ms against 10.50 ms, where only two routed launches finished.

Most of that difference is unrelated to routing, though. For up to 192 rows, only the default path adds the shared expert’s output inside the expert kernel’s final weighted sum. With this merge switched off, the default takes 5.18 ms and 10.48 ms.

What remains for the routing is 0.01 ms per step with one request and 0.02 ms with 16, or 0.3 to 0.6 microseconds per layer, which is as much as the launches of one variant differ among themselves. By kernel timings, the fused pair should have been ahead, by 0.3 microseconds per layer for one row and by 3.5 for 16 rows (8.3 against 11.8). The server shows no such lead, so the bookkeeping from finished ids probably costs about as much, though I didn’t time it. In short, routing inside the expert kernel makes the complete layer neither faster nor slower by an amount that matters, and the default’s measured advantage of about 0.1 ms per step comes mostly from the merged final sum.

6.3 What the Interface Demands#

Handing the selection to another library has a price, and the fp32 requirement from section 2 is part of it. SGLang no longer computes the rank keys, so it has to pass its fp32 logits and its fp32 bias through FlashInfer’s interface, and FlashInfer now decides the tie rule and the guard against a sum of zero in the rescaling. This went wrong once. After SGLang had switched the bias to fp32, an issue reported that long prompts produced only exclamation marks and traced them to NaN weights from the routing kernel for certain inputs. SGLang went back to the bf16 bias until a FlashInfer release fixed the kernel.

7. A Gate Inside the Multiplication in Front#

In section 6, steps 2 and 3 disappeared into the kernel behind the router. Step 1 can disappear in a similar way, but only for a model that is built for it. In our running example, the selected experts read the hidden state inside the expert kernel. The only other matrix that reads it is the shared expert’s first, which SGLang doesn’t merge with the gate weight. This section is therefore about Kimi K3, with its own numbers.

In Kimi K3, the experts don’t work on the 7,168-dimensional hidden state directly. A projection first maps it to a smaller vector, and the experts work on that. In every MoE layer, this projection with 3,584 rows reads the same hidden state as the gate weight, which has one row per expert, 896 in total. (So does the shared expert’s first matrix, which SGLang merges as well when no tokens are sent to other GPUs.) Figure 13 shows the two-matrix case, which SGLang uses when tokens are sent to other GPUs.

As shown on the right of Figure 13, SGLang stacks the two weights into one matrix when the model is loaded and runs a single GEMM over it. The first 896 columns of the result are the logits. One more kernel then selects 16 experts per row and casts the other 3,584 columns to bf16 for the experts. (For 16 of 896, SGLang doesn’t select in rounds. It sorts the rank keys into bins by their leading bits and looks further only into the bin that contains the cut, which is known as a radix select.)

I timed the two-matrix forms with the real weights of one Kimi K3 layer and random hidden states, since I couldn’t serve this model on the shared machine. I didn’t time the three-matrix form. For a single row, merged and separate are close, 21.7 microseconds against 22.6, because the large projection dominates either way. For 16 rows, the merged form still takes 21.7 microseconds and the separate one 30.5, and for 1,024 rows, they take 51.6 and 62.3. By default, SGLang uses the merged form for up to 1,024 rows. I didn’t measure a floor for this shape.

The fp32 requirement shows up here as well. The experts want their input in bf16, so the natural choice would have been a GEMM that writes bf16. The source file explains why SGLang doesn’t do that:

bf16 rounds the router logits and moves the selected expert set on 2-25% of rows depending on T

Here, T is the number of rows. The merged GEMM therefore writes fp32 for all of its columns, and the cast to bf16 happens afterward. The same comment gives the price: a merged GEMM with bf16 output was faster by about 1.2 microseconds for one row and slower from 8 rows on. The file also contains a third strategy that merges nothing. It runs the gate and the top-k on a second stream while the projection runs on the first, and SGLang uses it for some row counts on a GB300.

8. A Single Kernel#

The last design is one kernel for all three steps. It removes the second launch, but as before, the expert kernel then has to do its own bookkeeping. SGLang doesn’t have such a kernel for our model, but vLLM uses one for the DeepSeek-V4 models. The obvious way to build it has a drawback.

8.1 Why Merging the Pair Doesn’t Pay#

Look at Figures 9 and 11 again. The tiny GEMM is parallel across experts, with a block for every two experts, whereas the top-k is parallel across token rows, with a warp for every row. A kernel that does both has to give up one of the two layouts. A post on the LMSYS blog about another model came to the same conclusion, in the vocabulary of kernel authors (a CTA is a block, M is the number of rows, and the gate matvec is our step 1):

Folding the gate matvec into the top-k kernel would collapse parallelism from 129×M CTAs to M/BLOCK_M. PDL chaining is the right stopping point for that path.

In our terms, a merged kernel leaves far fewer blocks for the logits, and going by section 5.1, the logits need many threads that read the gate weight at the same time. SGLang has both forms for another model, Inkling, and uses the pair there by default.

8.2 Mega-Gate#

The Mega-Gate kernel in DeepGEMM, DeepSeek’s kernel library, is built like a large GEMM instead. It loads the weights and the hidden states with the GPU’s bulk-copy hardware, multiplies them on the tensor cores, and writes the scores to a scratch buffer. A separate group of warps in the same kernel runs the top-k, one warp per row. Mega-Gate never returns logits, only ids and weights.

The kernel was written for the DeepSeek-V4 models, but it also accepts sigmoid scores with a bias, so I could run it on the gate weight and the hidden states of our running example. On up to 8,192 of these rows from one layer, it selects the same experts as the fp32 reference. Figure 14 compares it with the two designs from sections 5 and 6 along the row axis.

In Figure 14, Mega-Gate takes 7.9 microseconds for one row, three times the floor. For up to 4 rows, it’s about a microsecond slower than the other two, which is the price of the tensor-core machinery for a job this small. The same machinery explains why the line is flat: the kernel multiplies the tokens in tiles of 16 to 256 rows, and a tile with one row takes the same steps as a full one. Above 16 rows, it is ahead of both, since they have to switch to cuBLAS there. At 96 rows, it takes 8.4 microseconds while the fused pair takes 11.2 and the expert kernel’s routing 14.2. At 8,192 rows, where the work outweighs the launches, it takes 35 microseconds against 45 and 42. (My servers run prefill without a CUDA graph, so the 8,192-row numbers are lower bounds for it. I also didn’t measure how long a layer takes for a prefill chunk, so I can’t say what share of it 35 or 45 microseconds is.)

Against the fused pair, a single kernel is therefore worth about 3 microseconds per layer at 96 rows, but SGLang doesn’t call this one at the commit that I looked at. The caveat from section 6 applies: Mega-Gate returns ids, so an expert kernel behind it has to do its own bookkeeping. I didn’t measure that combination. A kernel that did the three steps and the bookkeeping in one launch would avoid this, and as far as I know, nobody ships one yet.

With its DeepGEMM expert backend, vLLM uses Mega-Gate for the DeepSeek-V4 models above 16 rows. Below that, it keeps a pair of kernels much like SGLang’s.

9. A Side Effect: The Batch Decides the Kernel#

Every design above covers a part of the row axis, and at each boundary, one kernel is swapped for another. The same token can get its logits from different kernels, depending on the batch it is in. With 6 rows, they come from the tiny GEMM, and with 96 rows from cuBLAS. Both add up the same 4,096 products in different orders, and the result of a floating-point sum depends slightly on the order.

On the captured hidden states of GLM-5.3-Flash, the logits of the two kernels differ by at most 0.00007. That’s enough to change the expert set of 0.003% of the tokens in an average layer, the ones whose gap at the cut is even smaller: 21 of about 687,000 cases in the prompts.

I would expect this to be far too little to affect the quality of a model. It matters for testing and debugging: a test that sends many requests at once exercises different kernels than a single user does, and two runs of the same prompt can differ because the batches around it differed. For this reason, a pull request for Kimi K3 removed a rounding path that depended on the number of rows:

it will be really difficult to diagnose bugs in the future (when running evals quickly with large batch sizes, vs real world use at small batch sizes)

To remove the boundary at 16 rows, SGLang’s deterministic mode (see the appendix) uses one library call for the logits at every row count, although cuBLAS still changes kernels with it. By section 5.1, that costs 3 to 4 microseconds per layer for one row.

10. What I Left Out#

To keep the length manageable, I skipped the following:

• The routers on AMD, Ascend, and Intel hardware. SGLang has kernels for all three, and several of them mirror the ones described here.
• Grouped selection as in DeepSeek-V3, which picks experts only from the best 4 of 8 groups and has a FlashInfer kernel of its own.
• A profile of one served decode step with the router kernels summed over the layers.

11. Summary#

Let’s come back to the subtitle: the router is cheap to compute and expensive to launch. For one row of GLM-5.3-Flash on a B300, one launch and one cold read of the gate weight take 2.6 microseconds, and the tiny GEMM alone is already at 2.8. Everything above that comes with the additional kernels: their launches, their reads and writes, and the selection itself.

That’s why the largest gain, 13 of 20 microseconds, came from fusing steps 2 and 3, and why the designs after the fused pair no longer compete on the router’s own time for a single row. Routing inside the expert kernel is 0.3 microseconds slower than the fused pair for one row and 1.2 for the recipe’s 6 rows, and in the server, the routing itself accounts for at most 0.02 ms per step. The gains from merging show up with more rows: Kimi K3’s merged multiplication saves 9 microseconds at 16 rows, and a single tensor-core kernel is flat up to about 100 rows, while both earlier designs switch kernels above 16.

By kernel timings, the default path takes 7.0 microseconds per layer for one row, which over the 42 MoE layers would be 6% of a single-request decode step without speculative decoding if the timings carried over in full. With the two thirds that section 5.2 found, it is about 4%, and what lies above the floor is less than 3% of a step. A better pair doesn’t close that gap: my own, tuned for another shape, stays 1.9 microseconds above its floor.

As for precision, a bf16 bias changes the experts of more than half of the tokens in an average layer, and the only effect I could measure was 0.6% of acceptance length. But fp32 is nearly free for the router, and both absorbed designs had to be built around it: the expert kernel’s interface takes an fp32 bias, and Kimi K3’s merged GEMM writes fp32 for columns that would otherwise be bf16.

I expect this merging to continue. FlashInfer already has a newer router for softmax models, not in a release yet, that does the top-k and the bookkeeping in one call. Speculative decoding multiplies the number of rows per request, and with 6 rows each, the third concurrent request exceeds the tiny GEMM’s limit of 16 rows. From there on, the logits come from cuBLAS, and at 96 rows, the whole router takes 14.2 microseconds with its bookkeeping against 8.4 for Mega-Gate without it. What remains is one launch for the three steps and the bookkeeping, which I haven’t seen shipped. For an operator, the conclusion is to change nothing: SGLang’s defaults already pick these kernels, and there is nothing to tune for a single request. The third concurrent request matters more: with 18 rows, the router takes 13.8 microseconds per layer, against 8.2 for a single request, and a single kernel would still take about 8.

Appendix: Flags and Kernel Names#

Almost nothing in this article needs a flag, since SGLang picks the kernels from the shape of the model, the GPU, and the number of rows. Figure 15 shows the choices along the row axis for the models that share our model’s router code: the GLM-5 series, DeepSeek-V3 to V4.1, and the Kimi models.

In Figure 15, the kernel counts in the gray band are for our 288 experts. As an example, these are the low-latency flags for GLM-5.3-Flash on 8 B300 GPUs from the SGLang documentation. (I ran them with --tp-size 2.)

python3 -m sglang.launch_server \
  --model-path nvidia/GLM-5.3-Flash-NVFP4 \
  --quantization modelopt_fp4 \
  --tp-size 8 \
  --dsa-prefill-backend trtllm \
  --dsa-decode-backend trtllm \
  --kv-cache-dtype fp8_e4m3 \
  --moe-runner-backend flashinfer_trtllm \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 5 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 6

The --moe-runner-backend flag selects the expert kernels, and this value is what SGLang picks by itself for this model. A single request runs 6 rows per step, so its logits come from the tiny GEMM, and the routing stage of section 6 computes the scores and the top-k.

For the rare cases where something has to change, I found these switches:

• With --moe-runner-backend flashinfer_trtllm_routed, SGLang’s fused pair routes and the expert kernel receives ids (section 6.2).
• --enable-deterministic-inference pins the logits to one library call at every number of rows (section 9).
• SGLANG_OPT_BF16_FP32_GEMM_ALGO selects the library for the logits above 16 rows. The default is cublas.
• SGLANG_K3_FUSED_FRONT=0 turns off the merged form of Kimi K3 from section 7, for comparing the two forms.

To check what your own deployment runs, look at the kernel names in a GPU profile: tiny_n_gemm_kernel or tiny_k_gemm_kernel is the tiny GEMM. A kernel with splitKreduce in its name after a GEMM is the cuBLAS path at decode sizes. The fused router kernel is _router_triton_kernel, and names containing routingCustom or routingIndices are the expert kernel’s routing stage.