Image credit: Jiang and colleagues, Mixtral of Experts, Figure 1. Padded to 16:9.
A language model can contain hundreds of billions of parameters without using all of them to produce each piece of an answer. DeepSeek V2 is a documented example. Its research paper lists 236 billion parameters in total, but says only 21 billion activate for each token. The difference is central to understanding how a large model can limit its computing bill.
The architecture behind that gap is called Mixture of Experts, or MoE. Instead of sending every token through the same large block of calculations, a sparse MoE model selects a few smaller networks inside that block. Different tokens can take different routes. The model keeps a large collection of learned weights while using only part of that collection at a time.
That does not mean every large AI model uses MoE. Nor does it mean the selected networks are separate chatbots debating an answer. Published architectures such as Mixtral and DeepSeek V2 describe experts embedded within a single model, with routing decisions repeated as information passes through its layers.
Parameters are the learned numerical values that determine how a network transforms its inputs. Total parameters count the values across the model. Active parameters describe the subset used for an individual token, the units of text that a language model processes. The distinction separates how much learned machinery exists from how much participates in a particular calculation.
In a conventional dense model, tokens use the same main computational blocks and their weights. In a sparse MoE model, some blocks contain alternatives, and only selected alternatives run for a given token. Sparse here refers to conditional use of those blocks. It does not mean the model has simply erased most of its weights or stopped needing to store them.
NVIDIA's explanation also distinguishes sparse MoE from dense mixtures, where every expert processes every input. The savings discussed here come from sparse selection. Merely describing a network as a mixture of experts does not establish how many experts it activates or how much computation it avoids.
The common arrangement replaces the feed forward portion of a transformer layer. A transformer also has attention, which relates information across positions in the text. The feed forward block transforms each token's current numerical representation. In Mixtral, that block becomes a choice among several expert networks, while other parts of the layer remain shared.
A small learned network called a router or gate makes the choice. It takes the token's current representation and calculates a score for each expert. A mathematical function called softmax converts scores into relative weights. A selection rule then restricts the calculation to the highest scoring experts, with details varying between architectures.
The term top k simply specifies how many experts the rule selects. Switch Transformer selects one. Mixtral selects two out of eight in each layer. Mixtral's paper describes keeping the highest scoring entries before applying softmax. The chosen experts process the input, and their outputs combine according to the gate weights. Unselected experts do not need to run for that token.
The router does not choose one permanent expert for an entire conversation. Mixtral makes the decision for each token at each layer. The same token's representation changes as it travels through the model, so later layers can make different selections. This is one reason that imagining eight complete assistants, each responsible for a subject, gives the wrong picture.
Training also has to prevent the router from repeatedly favoring a few experts. If that happens, favored networks receive more training while others receive little. On a cluster of chips, the imbalance can leave some devices busy and others waiting. The Switch paper adds an extra training penalty that encourages a more even distribution of work across experts.
Load balancing does not mean routing every token randomly or forcing every expert to do identical work. It means discouraging a distribution that wastes capacity or creates a bottleneck. Switch also describes limits on how many tokens an expert can process in a batch. If an expert overflows, some token representations skip its calculation and continue through a residual connection.
The modern lineage includes a 2017 paper by Noam Shazeer and colleagues on sparsely gated expert layers. Their experiments placed expert networks between recurrent language model layers, rather than using the now familiar transformer layout. The paper demonstrated the central idea of adding substantial model capacity without a proportional increase in computation, while addressing communication and uneven expert use.
Switch Transformer followed in a paper first released in 2021. William Fedus, Barret Zoph and Shazeer simplified routing to one expert per token. Their work examined training stability and the practical cost of moving information between devices, as well as parameter counts. Its speed improvements were results from specific training experiments, not a promise that every MoE chatbot answers faster.
Google's GLaM paper, also released in 2021, supplied another clear example of the size gap. Its largest model had 1.2 trillion total parameters and activated 96.6 billion for each token. It selected two experts in its MoE layers. Those figures show that a huge total count and a much smaller active count were already distinct measures in published research.
Mistral's December 2023 announcement gives Mixtral 8x7B a total of 46.7 billion parameters and 12.9 billion used per token. The accompanying paper rounds those counts to 47 billion and 13 billion. Both descriptions refer to the same architecture. The name does not mean eight complete models with seven billion parameters each, which would imply a different total.
Shared components explain part of that arithmetic. Attention and other common layers do not multiply eight times just because the feed forward block has eight choices. They also remain part of the work performed for each token. Active parameter counts therefore cannot be calculated by dividing the total count by the number of experts and multiplying by the selected number.
DeepSeek V2 develops the design with DeepSeekMoE. Its paper describes smaller, more numerous routed experts alongside shared experts that always participate. The intention is to give routed experts more specific patterns to learn while shared experts handle common information, reducing repeated learning across the routed collection.
For the full DeepSeek V2 model, the paper specifies 160 routed experts and two shared experts in each MoE layer. Six routed experts activate for a token. Its reported 236 billion total and 21 billion active parameters cover the model, rather than just one expert layer. Those numbers should not be confused with the paper's separate, smaller Lite model.
The word expert still needs caution. These networks learn through training rather than receiving human job titles such as lawyer or programmer. Mixtral's authors found no obvious topic based division among experts in their analysis. They observed more apparent links to syntax, including patterns involving code indentation and repeated words.
NVIDIA's separate Mixtral experiment found that certain subjects favored some experts at deeper layers. That does not establish a fixed department for each subject. Together, the studies suggest routing can reflect learned patterns at different levels, and that any claim about what an expert knows needs evidence rather than a convenient label.
For laboratories, the attraction is more learned capacity within a limited computation budget. A model can offer many alternative transformations without evaluating every one for every token. NVIDIA notes that this can also reduce the delay before an answer starts when large prompts or batches make computation the limiting factor. Actual speed still depends on the serving system.
There are costs that an active count hides. Routers take work, and tokens may need to travel to experts on other chips before their results return. Mixtral's paper explicitly separates computation savings from memory costs and hardware use. A poorly balanced request can overload a subset of experts even when the average workload looks manageable.
GPT 4 requires a different standard of evidence. SemiAnalysis reported in 2023 that its architecture was sparse and discussed MoE in the article's visible material. That is unverified reporting, not an OpenAI architecture disclosure. OpenAI has not confirmed that claim in the allowed sources available at the time of this writing. Detailed counts behind the paid article are not needed to explain the documented designs.
When reading a model card, look for both total and active parameters. Check how many experts exist per layer, how many activate, and whether shared experts also run. For a speed claim, check the hardware, request sizes and whether the measurement concerns training or answering. A small active count alone does not tell a reader what a deployment will cost.
Memory sets a practical floor beneath the computation savings. Unselected weights still belong to the model and must remain available somewhere in the serving system. GLaM's authors note that a large parameter collection can require more devices and raise serving costs when traffic is low. Sparse selection reduces the work for a token. It does not make the rest of the model disappear.
Comments