In a mixture-of-experts (MoE) transformer, a learned router decides, for each token at each layer, which expert feed-forward networks run. The Mixtral paper gives a concrete published case: each token is sent to two of eight experts. Routing lets total parameters grow while expert compute per token stays fixed, and it is where most of the practical costs arise.
What the router computes for each token
In Mixtral, the paper's gating formula is y = Softmax(Top2(x · Wg)) applied to the expert outputs, where x is the token's hidden state, Wg is the router weight matrix, and each expert is a feed-forward network. Top2 keeps the two largest router logits, meaning the raw scores before softmax. The softmax runs only over those two, so the two gate weights sum to one, and only those two experts are evaluated for that token. The other six experts contribute nothing to that token in that layer.
A gate weight is the softmax value assigned to a selected expert. Top-k means keeping the k highest-scoring experts. The choice is made per token, so tokens in the same batch can reach different expert pairs within the same layer.
Why per-token compute stays roughly flat
Total parameters are every weight in the model. Active parameters are those used in the forward pass for one token: the router, shared layers such as attention and embeddings, and the selected experts. The Switch Transformer abstract describes MoE as selecting different parameters for each example, yielding a sparsely activated model with constant computational cost (arXiv 2101.03961). Adding experts raises total parameters without changing how many expert networks a token touches.
That constant-cost claim concerns the expert layers. Attention and embeddings run for every token, so activating two of eight experts does not mean a quarter of the model runs. Whole-model active fractions depend on what is counted, and the secondary sources that give figures for newer models were not checked against primary reports, so this article does not reproduce them. For the cost side of deployment, see what inference cost per token depends on.
Top-1 versus top-2 routing
Switch Transformer routes each token to exactly one expert, which is top-1 routing (arXiv 2101.03961). According to a secondary summary on Emergent Mind, this reduces router computation and the batch each expert must process. Top-2, used by Mixtral, doubles expert evaluations per token relative to top-1 in exchange for combining two weighted outputs. This article does not quote a Switch speedup figure, because the conditions of the paper's comparison were not checked here.
Capacity factor, overflow and dropped tokens
Each expert gets a fixed buffer per layer. The Hugging Face MoE blog gives its size as roughly (tokens per batch divided by number of experts) multiplied by the capacity factor. Tokens routed to a full expert are dropped for that layer. Some implementations instead pass them through the residual connection without expert processing. A higher capacity factor reduces drops, at the cost of memory and compute.
The formula comes from that blog and was not checked against the equation in the Switch paper, and sources differ on where the factor sits. A drop rate is meaningful only when batch size, routing skew and capacity factor are all stated. As the author's arithmetic, not a finding: with 4,096 tokens over 8 experts, the formula gives 512 slots per expert at a capacity factor of 1.0, and 640 at 1.25.
Load balancing: auxiliary losses and bias-based alternatives
Without intervention, the router tends to concentrate tokens on a few experts. Those experts train faster and so get picked more, a self-reinforcing loop the Hugging Face blog describes. The usual remedy is an auxiliary loss. The Switch loss, as described in a Substack explainer at cameronrwolfe.substack.com, encourages equal fractions of tokens and of router probability across experts.
DeepSeek's auxiliary-loss-free method adds an expert-wise bias to routing scores before top-K selection and updates that bias from recent expert load, with no loss term (arXiv 2408.15664). Its reported results cover models up to 3B parameters trained on up to 200B tokens. They do not establish frontier-scale behaviour, and this article did not verify how widely bias-based balancing is now used.
Expert-choice routing: the reversed direction
Expert-choice routing flips the selection, so each expert picks its top-k tokens instead of each token picking experts. Google Research says this guarantees even load and allows a variable number of experts per token. Its reported 2x convergence gain is a company claim on one configuration, not a general result. Whether the method suits autoregressive generation was not verified here.
What sparsity does not save: memory and communication
Sparse routing reduces expert FLOPs per token. It does not reduce the parameters held in memory, because every expert must be loaded even though each token uses only a few. In expert-parallel setups, tokens must be exchanged across devices through all-to-all communication, and the Hugging Face blog notes the communication cost of larger capacity. Per-token FLOP savings therefore do not translate directly into latency or throughput. For the wiring side of this, see how network topology affects distributed training speed.
Measured figures relating memory footprint and communication to per-token compute were not found in the sources reviewed, so the size of these costs remains open.
Reported configurations and open questions
The eight-expert, top-2 setting for Mixtral is sourced to the primary paper. Secondary accounts give about 47B total and 13B active parameters for Mixtral 8x7B; these were not checked against the paper. Settings for DeepSeek-V3, Qwen3, GPT-OSS-120B and Kimi K2 come only from secondary blogs, and their primary technical reports were not retrieved, so they are omitted here.
Three questions remain open. Expert specialization is disputed: one secondary account reports no clear domain-specific routing in Mixtral, while other sources describe specialization by domain or by syntax, and the Mixtral paper should settle it. Gradients through the discrete top-k choice typically flow through the selected experts' softmax probabilities, while unselected experts receive none for that batch; this is common practice per secondary sources, not a result from a primary paper checked here. Current frontier capacity factors were not established.
Before comparing two MoE model cards, check three things: whether the parameter count is total or active, the capacity factor and drop rate together with their batch size, and whether balancing uses an auxiliary loss or a bias term. Those answers determine most of what a headline number means.