1. Implement top-k MoE routing with a shared expert in PyTorch, and point out where the efficiency and correctness traps are.
Model answer
A readable forward pass for one MoE layer:
import torch, torch.nn.functional as F
def moe_forward(x, router_w, routed_experts, shared_expert, k, bias):
# x: (T, d). bias: (E,) per-expert load-balancing bias, no grad.
scores = x @ router_w # (T, E) affinities
topv, topi = torch.topk(scores + bias, k, dim=-1) # bias steers selection only
gate = F.softmax(topv, dim=-1) # weights from UNBIASED-ranked scores' values
out = shared_expert(x) # shared expert runs for every token
for slot in range(k):
idx = topi[:, slot] # (T,) expert id chosen in this slot
w = gate[:, slot].unsqueeze(-1) # (T,1) its gate weight
for e in routed_experts.unique_ids(): # in practice: group tokens per expert
mask = idx == e
if mask.any():
out[mask] += w[mask] * routed_experts[e](x[mask])
return out
Traps interviewers probe. Correctness: the load-balancing bias must affect selection (the topk) but the gate value used in the weighted sum should come from the true affinity, not the biased one - mix these up and you distort expert contributions. The shared expert is unconditional and must not be double-counted in the routed loop. Efficiency: the Python for e in experts loop is the naive version; real code sorts/groups tokens by expert once and does one batched matmul per expert (a grouped GEMM), because launching many tiny per-token kernels destroys throughput. In distributed training this grouping is an all-to-all that ships tokens to the device holding each expert, then ships results back. Also handle the capacity case: if an expert is overloaded beyond its buffer, you either drop or overflow tokens, and that policy affects both quality and balance.
Follow-ups: Rewrite the inner loop as a grouped matmul. How does token dropping at capacity interact with the auxiliary-loss-free bias?