Mixture-of-Experts forward pass

num_tokens (T) = tokens entering the MoE module this forward pass — the M dimension of every GEMM below (tech_blog/blog24). Each token fans out to top-k routed experts, then results are alpha-weighted and scattered back. Drag to orbit · scroll to zoom.

num_tokens T num_experts E top_k K speed

expert idle (m_e = 0 tokens)

m_e load: memory-bound → compute-bound

shared expert (dense, all T tokens)

weight plates (W1 gate|up, W2) — static params

hidden activations (appear at FC1, merge at SwiGLU)

slab depth ∝ m_e · each expert = its own MLP stack