Mixtral of Experts

Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary et al. (+21)•View original

OverviewBalancedmiles & helen hosts

Miles: Forty-seven billion parameters in the model, with thirteen billion actually firing at any moment. That gap is not a rounding error; it represents the whole design. The question is, what is doing the selecting? Helen: Right, and that's the part I want to understand. When you say "a router picks two experts," I picture something that is rule-based, like a lookup table. But that's not what's happening here. Miles: Not at all. Jiang and colleagues built Mixtral eight times seven billion as a Sparse Mixture of Experts, or SMoE, where every feed-forward layer in the transformer is replaced by eight distinct expert networks. The backbone is otherwise identical to Mistral seven billion. It has the same attention and the same overall structure. The difference is that instead of one feed-forward block per layer, there are eight, and for every single token at every single layer, a router picks exactly two of them. Helen: Walk me through it. Miles: The router is a learned linear layer that is small and cheap to run. It takes the token's hidden state, which is the vector representation at that point in the network, and multiplies it by a weight matrix. This produces eight numbers, which represent one score per expert. Then you apply a top-two mask: keep the two largest scores and replace the other six with negative infinity. Running the softmax function over all eight, the six masked entries go to zero. This leaves you with two probabilities that sum to one. Helen: Those two probabilities weigh the expert outputs. Miles: Exactly. The token's output from that layer is the first probability times expert one's output, plus the second probability times expert two's output. Six experts contribute nothing; their parameters are not touched. Helen: Here's where I'd push back, though. You said the router is learned, so it adjusts its weights during training to minimize the language modeling loss. If that's true, why wouldn't it just learn to always route everything to the two best experts and ignore the rest? What prevents it from collapsing? Miles: The paper honestly acknowledges that this is a real failure mode. Jiang and colleagues highlight expert imbalance as a hardware problem specifically. Mixtral uses expert parallelism, where different experts reside on different graphics processing units. If the router sends eighty percent of tokens to the same two experts, those graphics processing units are saturated while the others remain idle. They refer to this as over-subscription. Helen: So, there's a structural pressure against collapse, not just a loss term? Miles: The paper emphasizes the hardware constraint directly. Balanced routing is essential for not overloading individual graphics processing units. The routing analysis they conduct supports this in an interesting way — the distributions across domains like ArXiv, PubMed, and PhilPapers are actually quite similar. No expert ends up as "the science expert" or "the philosophy expert." Helen: That surprised me when I read it. I would have expected domain specialization to be the whole point. Miles: The authors also find it surprising. When they analyze routing decisions on The Pile validation set, which is a large and domain-diverse corpus, the per-expert assignment distributions look nearly identical across ArXiv papers, PubMed abstracts, and PhilPapers philosophy. The only exception is DM Mathematics, which is highly synthetic and narrow, showing a meaningfully different distribution. Helen: So, the router isn't sorting tokens by topic. What is it sorting by? Miles: Syntax, at least partially. Jiang and colleagues show that tokens like Python's "self" keyword and English words like "Question" tend to route through the same expert even when they are split across multiple sub-tokens. Indentation tokens in code are almost always assigned to the same experts, particularly at the first and last layers. Helen: That's a cleaner pattern than I expected, actually. It's not semantic, but structural. Miles: There's also something striking about time. Consecutive tokens in a sequence share experts far more often than chance would predict. The paper quantifies this. At layer fifteen, for example, the probability that token i and token i plus one are assigned the same first-choice expert is around 28 percent on ArXiv and 28 percent on DM Mathematics. The random baseline, if the router picked uniformly among eight experts, would be twelve point five percent. Helen: So, it’s more than double the random rate. Miles: For first or second choice matches, the gap is even wider. The expected baseline is about 46 percent. Observed values at layer 15 reached 62 percent on ArXiv and 67 percent on DM Mathematics. Helen: That temporal locality is the part I don't know how to interpret. Is the router learning that adjacent tokens should go together because they are semantically similar? Or is it an artifact of how hidden states evolve across positions? Miles: The authors do not fully resolve this issue. They note that locality could be exploited for caching. If you know the next token is likely to hit the same expert, you can keep that expert's computation warm. However, they do not claim to know whether locality is the cause or the effect of the training dynamics. Helen: This means the router is doing something real and exploitable, but we cannot fully understand the logic. Miles: That's the honest summary. The structure is there, but the mechanism isn't fully clear. Helen: Okay. But does any of this matter for performance? Miles: Substantially. With 13 billion active parameters per token, compared to Llama 2 70B's full 70 billion, Mixtral matches or exceeds Llama 2 70B across most evaluated benchmarks. The gap is largest in mathematics and code. On the MBPP code generation benchmark, Mixtral achieves a pass rate of 60.7 percent, while Llama 2 70B has a pass rate of 49.8 percent and GPT-3.5 has 52.2 percent. In the math benchmark GSM-8K, Mixtral reaches a score of 58.4 percent, compared to 53.6 percent for Llama 2 70B. Helen: And multilingual? Miles: French MMLU is 70.9 percent for Mixtral compared to 64.3 percent for Llama two seventy billion. The paper trained on multilingual data with a context window of thirty-two thousand seven hundred sixty-eight tokens, and the gains are evident across the languages tested. Helen: The instruction-tuned version is where the comparison to commercial models becomes interesting. Miles: Mixtral eight by seven B Instruct is trained using supervised fine-tuning followed by Direct Preference Optimization, or DPO. This method aligns models to human preferences and scores eight point three on MT-Bench. On the LMSys Arena leaderboard, it achieves an Arena Elo rating of one thousand one hundred twenty-one. This outperforms Claude two point one and sits above the best reported GPT three point five Turbo variants at one thousand one hundred seventeen. Helen: All of that with approximately one fifth of the active compute per token. Miles: That's the trade Mixtral is making. The sparse routing gives you a higher parameter count without incurring the full inference cost of a dense model of that size. Helen: Here's what I'm still thinking about. The routing analysis shows structure, including temporal locality and syntax alignment, but there is no clear semantic specialization. If the experts are not specializing by topic or domain, what is the theory for why having eight of them helps at all? Miles: The paper doesn't provide a complete answer. They show that the router learns something, the benchmarks confirm that the system works, and the routing patterns are real but not fully interpretable. That is where the analysis ends. Helen: Which is not a failure of the paper; it's an honest accounting. Miles: The weights are publicly released under the Apache two-point-zero license. The routing analysis conducted by Jiang and colleagues is a starting point. Why top two is the right value of K, what exactly the router has learned to see in a token's hidden state, and whether the temporal locality reflects something semantic or something purely representational — those questions remain open, and the model is available for anyone who wants to pursue them. Helen: The finding that survives all of that uncertainty is the efficiency result. A model that processes 13 billion parameters per token outperforms one that processes 70 billion. It achieves this through a learned linear layer, making a hard choice among eight experts at every layer of the network. Miles: A small arithmetic decision with large consequences. This lecture was created by ennepō. Go to https://ennepo.ai to Discover, Create and Follow the latest research in your field. Read when you can. Listen when you want to.

Forty-seven billion parameters in the model, with thirteen billion actually firing at any moment. That gap is not a rounding error; it represents the whole design. The question is, what is doing the selecting?

Right, and that's the part I want to understand. When you say "a router picks two experts," I picture something that is rule-based, like a lookup table. But that's not what's happening here.

Not at all. Jiang and colleagues built Mixtral eight times seven billion as a Sparse Mixture of Experts, or SMoE, where every feed-forward layer in the transformer is replaced by eight distinct expert networks. The backbone is otherwise identical to Mistral seven billion. It has the same attention and the same overall structure. The difference is that instead of one feed-forward block per layer, there are eight, and for every single token at every single layer, a router picks exactly two of them.

Walk me through it.

The router is a learned linear layer that is small and cheap to run. It takes the token's hidden state, which is the vector representation at that point in the network, and multiplies it by a weight matrix. This produces eight numbers, which represent one score per expert. Then you apply a top-two mask: keep the two largest scores and replace the other six with negative infinity. Running the softmax function over all eight, the six masked entries go to zero. This leaves you with two probabilities that sum to one.

Those two probabilities weigh the expert outputs.

Exactly. The token's output from that layer is the first probability times expert one's output, plus the second probability times expert two's output. Six experts contribute nothing; their parameters are not touched.

Here's where I'd push back, though. You said the router is learned, so it adjusts its weights during training to minimize the language modeling loss. If that's true, why wouldn't it just learn to always route everything to the two best experts and ignore the rest? What prevents it from collapsing?

The paper honestly acknowledges that this is a real failure mode. Jiang and colleagues highlight expert imbalance as a hardware problem specifically. Mixtral uses expert parallelism, where different experts reside on different graphics processing units. If the router sends eighty percent of tokens to the same two experts, those graphics processing units are saturated while the others remain idle. They refer to this as over-subscription.

So, there's a structural pressure against collapse, not just a loss term?

The paper emphasizes the hardware constraint directly. Balanced routing is essential for not overloading individual graphics processing units. The routing analysis they conduct supports this in an interesting way — the distributions across domains like ArXiv, PubMed, and PhilPapers are actually quite similar. No expert ends up as "the science expert" or "the philosophy expert."

That surprised me when I read it. I would have expected domain specialization to be the whole point.

The authors also find it surprising. When they analyze routing decisions on The Pile validation set, which is a large and domain-diverse corpus, the per-expert assignment distributions look nearly identical across ArXiv papers, PubMed abstracts, and PhilPapers philosophy. The only exception is DM Mathematics, which is highly synthetic and narrow, showing a meaningfully different distribution.

So, the router isn't sorting tokens by topic. What is it sorting by?

Syntax, at least partially. Jiang and colleagues show that tokens like Python's "self" keyword and English words like "Question" tend to route through the same expert even when they are split across multiple sub-tokens. Indentation tokens in code are almost always assigned to the same experts, particularly at the first and last layers.

That's a cleaner pattern than I expected, actually. It's not semantic, but structural.

There's also something striking about time. Consecutive tokens in a sequence share experts far more often than chance would predict. The paper quantifies this. At layer fifteen, for example, the probability that token i and token i plus one are assigned the same first-choice expert is around 28 percent on ArXiv and 28 percent on DM Mathematics. The random baseline, if the router picked uniformly among eight experts, would be twelve point five percent.

So, it’s more than double the random rate.

For first or second choice matches, the gap is even wider. The expected baseline is about 46 percent. Observed values at layer 15 reached 62 percent on ArXiv and 67 percent on DM Mathematics.

That temporal locality is the part I don't know how to interpret. Is the router learning that adjacent tokens should go together because they are semantically similar? Or is it an artifact of how hidden states evolve across positions?

The authors do not fully resolve this issue. They note that locality could be exploited for caching. If you know the next token is likely to hit the same expert, you can keep that expert's computation warm. However, they do not claim to know whether locality is the cause or the effect of the training dynamics.

This means the router is doing something real and exploitable, but we cannot fully understand the logic.

That's the honest summary. The structure is there, but the mechanism isn't fully clear.

Okay. But does any of this matter for performance?

Substantially. With 13 billion active parameters per token, compared to Llama 2 70B's full 70 billion, Mixtral matches or exceeds Llama 2 70B across most evaluated benchmarks. The gap is largest in mathematics and code. On the MBPP code generation benchmark, Mixtral achieves a pass rate of 60.7 percent, while Llama 2 70B has a pass rate of 49.8 percent and GPT-3.5 has 52.2 percent. In the math benchmark GSM-8K, Mixtral reaches a score of 58.4 percent, compared to 53.6 percent for Llama 2 70B.

And multilingual?

French MMLU is 70.9 percent for Mixtral compared to 64.3 percent for Llama two seventy billion. The paper trained on multilingual data with a context window of thirty-two thousand seven hundred sixty-eight tokens, and the gains are evident across the languages tested.

The instruction-tuned version is where the comparison to commercial models becomes interesting.

Mixtral eight by seven B Instruct is trained using supervised fine-tuning followed by Direct Preference Optimization, or DPO. This method aligns models to human preferences and scores eight point three on MT-Bench. On the LMSys Arena leaderboard, it achieves an Arena Elo rating of one thousand one hundred twenty-one. This outperforms Claude two point one and sits above the best reported GPT three point five Turbo variants at one thousand one hundred seventeen.

All of that with approximately one fifth of the active compute per token.

That's the trade Mixtral is making. The sparse routing gives you a higher parameter count without incurring the full inference cost of a dense model of that size.

Here's what I'm still thinking about. The routing analysis shows structure, including temporal locality and syntax alignment, but there is no clear semantic specialization. If the experts are not specializing by topic or domain, what is the theory for why having eight of them helps at all?

The paper doesn't provide a complete answer. They show that the router learns something, the benchmarks confirm that the system works, and the routing patterns are real but not fully interpretable. That is where the analysis ends.

Which is not a failure of the paper; it's an honest accounting.

The weights are publicly released under the Apache two-point-zero license. The routing analysis conducted by Jiang and colleagues is a starting point. Why top two is the right value of K, what exactly the router has learned to see in a token's hidden state, and whether the temporal locality reflects something semantic or something purely representational — those questions remain open, and the model is available for anyone who wants to pursue them.

The finding that survives all of that uncertainty is the efficiency result. A model that processes 13 billion parameters per token outperforms one that processes 70 billion. It achieves this through a learned linear layer, making a hard choice among eight experts at every layer of the network.

A small arithmetic decision with large consequences. This lecture was created by ennepō. Go to https://ennepo.ai to Discover, Create and Follow the latest research in your field. Read when you can. Listen when you want to.

Related lectures