1. Implement a top-k MoE router in PyTorch.
The full question
Implement a top-k MoE router in PyTorch. Then contrast auxiliary-loss load balancing with a loss-free approach.
Model answer
Core routing plus the balancing loss, isolated:
import torch, torch.nn.functional as F
def moe_route(x, gate_w, experts, k=2):
# x: (T, D); gate_w: (D, E); experts: list of callables
logits = x @ gate_w # (T, E)
topv, topi = logits.topk(k, dim=-1) # (T, k)
w = F.softmax(topv, dim=-1) # normalise over the chosen k
out = torch.zeros_like(x)
for slot in range(k):
for e in range(len(experts)):
mask = topi[:, slot] == e
if mask.any():
out[mask] += w[mask, slot, None] * experts[e](x[mask])
# auxiliary load-balancing loss (Switch/GShard style)
probs = logits.softmax(-1) # (T, E)
importance = probs.mean(0) # fraction of prob per expert
onehot = F.one_hot(topi[:, 0], probs.size(-1)).float()
load = onehot.mean(0) # fraction of tokens per expert
aux = (importance * load).sum() * probs.size(-1)
return out, aux
Points that get probed: softmax over the selected top-k (not all experts), the causal detail that routing is per token per layer, and that production code dispatches with grouped gather/scatter rather than a Python loop.
Auxiliary-loss balancing adds aux (weighted) to the training loss to push token assignment toward uniform. It works but injects gradients that are not about the language objective, so too high a weight hurts quality and too low lets experts collapse - a tuning headache. Loss-free balancing (the direction more recent MoEs, including the GLM/DeepSeek lineage, favour) instead keeps a per-expert bias term added to the routing logits and nudges that bias up for under-used experts and down for over-used ones between steps, based on observed load. Balancing then happens through a controller on the bias, not through an extra loss term, so the main objective's gradients stay clean while utilisation still evens out.
Follow-ups: What breaks if an expert receives zero tokens for many steps? How does capacity factor / token dropping interact with balancing?