Mixture-of-Experts is the architecture powering the largest production LLMs — and it works differently than most people think

When OpenAI shipped GPT-4, the company declined to publish a parameter count. Months later, leaked documents and corroborating benchmarks suggested it uses a Mixture-of-Experts (MoE) architecture with roughly 1.8 trillion total parameters spread across eight expert sub-networks — but activates only about 220 billion per forward pass. That single design choice explains both the model's capability ceiling and its inference economics in ways that a naive parameter count never could.

MoE is now the dominant architecture for frontier models. Google's Gemini 1.5 uses MoE. Mistral AI's open Mixtral 8x7B and 8x22B models made MoE accessible to self-hosters. Meta's internal research on MoE for Llama successors is well-documented. Understanding how it actually works — and where it genuinely helps versus where it just makes the slides look good — matters if you're deciding which models to deploy or how to evaluate new releases.

The core idea: conditional computation

A standard dense transformer like Llama 2 70B activates every one of its 70 billion parameters for every token it processes. That's computationally expensive but predictable. MoE replaces the feedforward layers (the layers that make up the bulk of a transformer's parameter count) with multiple parallel "expert" networks plus a lightweight router. For each token, the router picks the top-k experts — typically 2 out of 8 or 16 — and only those experts process that token. The results are weighted and combined.

The practical consequence: a Mixtral 8x7B model has ~47 billion total parameters, but each token only touches about 13 billion of them. You get most of the representational capacity of a 47B dense model while running inference at closer to 13B cost. Throughput roughly doubles compared to an equivalent dense model on the same hardware, for the same quality output.

What the router actually learns

The router is a small linear layer that produces a probability distribution over all available experts. It's trained end-to-end with the rest of the model using standard gradient descent — there's no separate pre-training or hand-labeling of which expert should handle which content. What emerges is roughly domain specialization: analysis of Mixtral's routing patterns shows experts developing soft preferences for code syntax, natural language reasoning, factual recall, and so on. But this specialization is imprecise and doesn't always align with human intuitions about subject matter.

A persistent engineering problem is load balancing. Without intervention, the router tends to collapse onto a small set of "popular" experts and starve others, wasting capacity. The standard fix is an auxiliary load-balancing loss added during training that penalizes uneven expert utilization. Getting the strength of this loss right is a hyperparameter that meaningfully affects both model quality and hardware efficiency — too little, and expert collapse; too much, and the router can't learn meaningful specialization.

The memory bottleneck that marketing ignores

Here is where MoE becomes complicated for deployers. All parameters must reside in memory even though only a fraction activate per token. A Mixtral 8x22B model — with ~141 billion total parameters — requires roughly 280 GB of GPU VRAM in BF16 precision before you account for KV cache. That means at least four H100 80GB GPUs just to hold the weights, even though inference throughput is similar to a much smaller dense model.

This creates an infrastructure split. In a data center where you can dedicate a 4-GPU node per model replica, MoE is genuinely cheaper per token. In a deployment where you're trying to co-locate multiple models on shared hardware, MoE's memory footprint makes it expensive. It's also why quantization matters more for MoE models: getting Mixtral 8x7B to 4-bit precision (roughly 25 GB) is what makes it practical to run on a single consumer-grade workstation or a dual-GPU server.

Expert parallelism as a scaling lever

For training very large MoE models, a technique called expert parallelism distributes different experts across different physical GPUs. When a token is routed to Expert #5, the computation happens on the GPU that holds Expert #5's weights, and the result is sent back. This turns all-reduce communication into more localized point-to-point transfers and allows training at scales that would otherwise require too much per-GPU memory.

Google's Switch Transformer paper from 2021 demonstrated this at 1.6 trillion parameters — the first publicly documented trillion-parameter model. The key finding: a 64-expert MoE with the same compute budget as a T5-XXL dense model achieved 4x speedup in training time while matching or exceeding quality on standard benchmarks. The paper also documented failure modes: training instability at high expert counts, the load-balancing collapse problem, and communications overhead in multi-node setups.

Where MoE genuinely underperforms dense models

Few-shot learning on highly domain-specific tasks is one area where MoE models can underperform similarly-sized dense models. Because the router assigns tokens probabilistically and different tokens in the same prompt can go to different experts, the model's "memory" of early context can be fragmented across experts in ways that hurt coherence on long, specialized documents. Anecdotal reports from enterprise deployments of Mixtral suggest that dense models of equivalent inference cost sometimes produce better results on legal or medical text where exact terminology consistency matters.

Batch size also matters. MoE architecture's throughput advantage is most pronounced at large batch sizes where all experts get roughly even utilization. At batch size 1 — a single user making a real-time query — you're activating two experts and waiting idle on the other six. The latency per token can actually be worse than a dense model of equivalent activated parameter count due to routing overhead. This is why production deployments batch requests aggressively and why streaming API endpoints have different latency profiles than batch inference endpoints.

Practical decisions for teams evaluating MoE models

If you're comparing a dense 70B model against a MoE model like Mixtral 8x22B for deployment, the right comparison isn't parameter count — it's memory footprint against quality on your specific workload. Run both on your actual task distribution. Mixtral 8x22B will consistently beat Llama 2 70B on reasoning benchmarks but the gap closes significantly on narrow retrieval-augmented generation tasks where the dataset is homogeneous.

For fine-tuning, MoE models present a particular challenge: LoRA fine-tuning applied only to dense layers won't touch expert weights, which contain the majority of the model's specialized knowledge. Full fine-tuning of MoE models is memory-intensive. MoE-specific LoRA variants that apply adapters to expert feedforward layers exist but are not yet standard tooling — check whether your fine-tuning framework supports them before committing.

The router weights themselves can be frozen during fine-tuning to preserve the specialization patterns learned during pre-training. This works well when fine-tuning for a task that's well-represented in the original training distribution. When adapting to a genuinely novel domain, unfreezing the router and accepting the longer fine-tuning run is worth it.

What comes next

Research directions actively being explored include sparse MoE with more than two activated experts per token (trading compute for quality), hierarchical routing where a coarse router selects expert "families" before a fine-grained router selects specific experts, and mixture-of-depths architectures that route tokens to different layers rather than different experts within a layer. Google DeepMind's 2024 paper on mixture-of-depths showed that not every token needs to go through every transformer layer, enabling further conditional computation gains.

The architectural lesson from MoE is consistent: scaling laws reward conditional computation. Spending all your compute on every token for every task is wasteful. The models that will matter over the next two years will increasingly be hybrid systems that route work intelligently — whether that's routing to experts within a model, routing to different models via orchestration, or routing to external tools. MoE is the first production-scale demonstration that this principle works at the weights level.