DeepSeek-V4 (DSV4) — full network dataflow

Combined from the diagrams in flashinfer#3346 and sglang#23666 (author: nvpohanh). Blue = GEMM/FC (precision noted), teal = elementwise/norm, red = attention core, orange = KV/state caches, purple = hyper-connection ops. Drag to orbit · scroll to zoom · buttons fly to each board.
Key dimensions
hidden H = 7168 · HC width 4×7168 = 28672 (n=4 streams)
layers ×61 · vocab 129280
attn: 128 heads × 512 (fc_qb 1536×65536) · MQA w/ sink
o-proj: reshape [S,16,4096] → bmm_oa → fc_ob 16384×7168
KV: SWA window 128 + compressed pages (128 tokens/page)
MoE: 384 experts, router [S,384], top-6 (ArgTop6/tid2eid)
expert FFN 7168→3072→7168, MXFP4×MXFP8 · shared expert FP8
FC / GEMM (FP8 · BF16 · FP32 · MXFP4×MXFP8)
elementwise / norm / reshape
attention core (MQA)
KV cache / persistent state
hyper-connection (mHC) ops
embeddings / logits / block refs
─ solid: dataflow · ┄ dashed: control/decode-step · ┄ orange: cache update
✕ close


keys: 1–5 boards · ? all shortcuts · hover / click nodes




  
Keyboard controls

  1–5 fly to board  ·  0/H overview  ·  [ ] prev / next board

  ←↑↓→ or WASD pan  ·  Q/E or -/= zoom

  hover node → details  ·  click node → focus camera on it

  T transcript  ·  F flow pulses  ·  L bandwidth heatmap  ·  B bloom  ·  V trails  ·  K depth-of-field

  M mouse mode (orbit ⇄ pan-first)  ·  I ASCII render mode  ·  G guided tour  ·  U pin transcript in 3D

  P PNG  ·  ⇧P PNG 2× supersampled  ·  O SVG vector  ·  X .glb scene

  ? this help  ·  Esc close overlays