Neura saysMixture of Experts (MoE) activates only some of a huge model per token — scale without the full cost.
A Mixture of Experts model has many "expert" sub-networks but a router sends each token to only a few. So total parameters can be enormous while the compute per token stays modest (sparse activation). This is how some frontier models get huge capacity cheaply at inference. Trade-offs: complex routing, memory to hold all experts, and load-balancing challenges.
Power-ups you unlock
Many experts; router picks a few per token
Sparse activation: big capacity, modest compute
Powers some large frontier models
Costs: routing, memory, load balancing
The Overfit Ogre attacks — common mistakes
Confusing total params with active params
Assuming MoE always beats dense models
Ignoring memory cost of holding all experts
Boss battleExplain how MoE separates total parameters from per-token compute.
Example code
<!doctype html><html><head><meta charset="utf-8"></head>
<body style="background:#06040d;color:#e6e0ff;font-family:monospace;padding:20px"><pre>token → router → 2 of 64 experts active
total params huge, compute per token small
(sparse activation)</pre></body></html>