DeepSeek interview questions & answers

14 representative DeepSeek interview questions with full model answers — Technical, System design, Coding. Synthesised from DeepSeek's publicly known focus areas, not reported by candidates (14 DeepSeek questions in total).

CodingMediumDeepSeekAI Engineer

1. Implement top-k MoE routing with a shared expert in PyTorch, and point out where the efficiency and correctness traps are.

Model answer

A readable forward pass for one MoE layer:

import torch, torch.nn.functional as F

def moe_forward(x, router_w, routed_experts, shared_expert, k, bias):
    # x: (T, d).  bias: (E,) per-expert load-balancing bias, no grad.
    scores = x @ router_w                     # (T, E) affinities
    topv, topi = torch.topk(scores + bias, k, dim=-1)   # bias steers selection only
    gate = F.softmax(topv, dim=-1)            # weights from UNBIASED-ranked scores' values

    out = shared_expert(x)                    # shared expert runs for every token
    for slot in range(k):
        idx = topi[:, slot]                   # (T,) expert id chosen in this slot
        w   = gate[:, slot].unsqueeze(-1)     # (T,1) its gate weight
        for e in routed_experts.unique_ids(): # in practice: group tokens per expert
            mask = idx == e
            if mask.any():
                out[mask] += w[mask] * routed_experts[e](x[mask])
    return out

Traps interviewers probe. Correctness: the load-balancing bias must affect selection (the topk) but the gate value used in the weighted sum should come from the true affinity, not the biased one - mix these up and you distort expert contributions. The shared expert is unconditional and must not be double-counted in the routed loop. Efficiency: the Python for e in experts loop is the naive version; real code sorts/groups tokens by expert once and does one batched matmul per expert (a grouped GEMM), because launching many tiny per-token kernels destroys throughput. In distributed training this grouping is an all-to-all that ships tokens to the device holding each expert, then ships results back. Also handle the capacity case: if an expert is overloaded beyond its buffer, you either drop or overflow tokens, and that policy affects both quality and balance.

Follow-ups: Rewrite the inner loop as a grouped matmul. How does token dropping at capacity interact with the auxiliary-loss-free bias?

System designMediumDeepSeekAI Engineer

2. Sketch how you would serve a 671B-parameter MoE model with low latency under GPU-memory constraints.

Model answer

Start from the constraint: 671B parameters will not fit on one GPU even at FP8, so serving is inherently multi-GPU, and MoE means memory scales with total parameters (all experts resident) while compute scales with active ones (about 37B). So you shard. Use expert parallelism to spread experts across devices, plus tensor/pipeline parallelism for the dense parts, and accept that every token now triggers all-to-all communication to reach its expert. That communication is the latency enemy, so you overlap it with compute and co-locate frequently co-activated experts; DeepEP exists precisely to make this all-to-all fast.

Next, attack the KV cache, which at long context dominates memory: MLA already compresses it to a latent, and you can quantize the latent further. Use continuous (iteration-level) batching with paged KV so short requests do not wait on long ones and admission is cheap. Separate prefill and decode: prefill is compute-bound and batches well, decode is memory-bandwidth-bound, so many systems disaggregate them onto different pools sized differently. Add MTP/speculative decoding to cut per-token steps.

The MoE-specific serving headache is load imbalance within a batch: if many tokens in a step route to the same expert, that device becomes the bottleneck while others idle. So you need dynamic expert placement or replication of hot experts (an EPLB-style balancer) and you plan capacity around p95 routing skew, not the average. The whole design is a bandwidth-accounting exercise: weights, KV, and all-to-all traffic, budgeted against the interconnect.

Follow-ups: How do you decide the expert-parallel degree versus tensor-parallel degree? What metric tells you experts are imbalanced in production, and what is your mitigation?

TechnicalMediumDeepSeekAI Engineer

3. DeepSeek-V4 serves a native 1M-token context.

The full question

DeepSeek-V4 serves a native 1M-token context. MLA already shrinks the KV cache, so why was it not enough, and how does V4's hybrid compressed attention work?

Model answer

MLA compresses each token's KV entry, but it still stores one entry per token and every decoded token still attends over all of them. At 1M tokens the problem is the length of the sequence, not the width of each entry: KV memory and per-token attention FLOPs both grow linearly with context. V4 attacks the sequence axis directly, by compressing along it and by attending sparsely.

It interleaves two attention types across layers. Compressed Sparse Attention (CSA) pools KV entries into blocks at roughly 4x compression using learned, softmax-gated pooling, then a cheap lightning indexer (run in FP4, with ReLU-scored dot products) scores the compressed blocks for each query and keeps only the top-k. Heavily Compressed Attention (HCA) compresses about 128x and skips selection entirely: the compressed sequence is short enough that dense attention over it is cheap. Both keep a sliding window of recent uncompressed tokens so local detail is never lost. The indexer idea comes from V3.2's DeepSeek Sparse Attention. DeepSeek reports that at 1M tokens V4-Pro needs about 27% of V3.2's single-token inference FLOPs and about 10% of its KV cache, with most KV stored in FP8.

The trade-offs are what an interviewer will push on. Top-k selection is a hard, non-differentiable choice, so the indexer must be trained to agree with what dense attention would have attended to, and if it misses the block holding the evidence, the model never sees it. Compression blurs exact tokens, which hurts verbatim recall such as quoting an ID from 600K tokens back. Mixing CSA and HCA layers hedges both failure modes. Gathering top-k blocks is irregular memory access, so the win depends on kernels and on paged-KV layouts aligned to compression blocks. Evaluate with long-range retrieval and multi-hop tasks, not perplexity, which barely notices a missed needle.

Follow-ups: How would you detect the indexer silently dropping the block a query needed? How does sequence-axis compression interact with prefix caching for agent sessions that keep appending tool output?

TechnicalMediumDeepSeekAI Engineer

4. How do you build a training dataset without triggering model collapse when a lot of your data is synthetic?

Model answer

Model collapse is the failure where a model trained heavily on another model's (or its own) outputs progressively loses the tails of the distribution: rare facts, unusual phrasings, and diversity disappear, and quality degrades generation after generation. The core cause is that generated data is a lossy, mode-seeking sample of the real distribution, so recycling it amplifies the centre and forgets the edges. So the first rule is that synthetic data supplements real data, it does not replace it - keep a strong anchor of human/real corpus so the tails stay represented.

Concretely: (1) verify, do not just generate. For domains with checkable answers - math, code, formal reasoning - keep only synthetic samples that pass a verifier (unit tests, a solver, an execution check). Verified synthetic data is high-signal precisely because the filter, not the generator, decides what survives. (2) Deduplicate and diversity-filter aggressively; near-duplicate synthetic samples collapse variety and over-weight whatever the generator likes. (3) Filter for quality and difficulty, not volume - a smaller curated set beats a huge noisy one, and mixing in genuinely hard examples prevents the model from only seeing easy, self-confirming cases. (4) Track distribution drift: monitor n-gram/embedding diversity and per-domain coverage across data generations so you catch narrowing early.

The DeepSeek-flavoured version of this answer stresses the pipeline as a first-class research artefact: their reasoning data is built by generating candidate traces, verifying outcomes, and keeping the winners, which is exactly the "let a cheap objective filter do the curation" pattern.

Follow-ups: How would you measure diversity loss quantitatively before it shows up in benchmarks? When is self-generated data safe to train on and when is it poison?

TechnicalMediumDeepSeekAI Engineer

5. DualPipe overlaps computation and communication in training.

The full question

DualPipe overlaps computation and communication in training. Why is that overlap the whole game at this scale, and what is the trade-off?

Model answer

At 671B parameters across a large cluster, the GPUs spend a large fraction of every step waiting: pipeline parallelism leaves bubbles (stages idle while the pipeline fills and drains), and MoE adds heavy all-to-all traffic to route tokens to experts. If communication runs while compute is stalled, your expensive accelerators are underused and effective throughput craters. The single highest-leverage systems move is therefore to hide communication behind computation so the network transfers happen concurrently with matmuls rather than in series.

DualPipe is a bidirectional pipeline-parallelism schedule that runs micro-batches from both ends of the pipeline so that forward and backward phases interleave, shrinking the bubble, and it carefully arranges the schedule so that the all-to-all communication of one micro-batch overlaps the computation of another. Combined with a communication library tuned for the cross-node MoE traffic (DeepEP), most of the communication cost disappears into time the GPU was going to spend computing anyway. That overlap is a big part of how they hit their reported training-cost efficiency on constrained hardware.

The trade-off is memory and complexity. Keeping two directions of the pipeline in flight and overlapping stages means holding more activations and more in-flight state simultaneously, so DualPipe trades higher peak memory for lower idle time, and the schedule is far more intricate to implement and debug than naive 1F1B. It is worth it only when communication is genuinely a large share of step time, which at this scale it is.

Follow-ups: Where does the extra memory in DualPipe come from, and how would you bound it? When is pipeline parallelism the wrong tool versus more tensor or data parallelism?

TechnicalMediumDeepSeekAI Engineer

6. DeepSeek-V4.1-Flash splits 40 layers into a 20-layer causal encoder and a 20-layer decoder whose KV cache is built from the encoder's final hidden…

The full question

DeepSeek-V4.1-Flash splits 40 layers into a 20-layer causal encoder and a 20-layer decoder whose KV cache is built from the encoder's final hidden states. Why does that roughly halve the active parameters in prefill, what does the decoder lose, and what does it do to KV memory and serving?

Model answer

In a standard decoder-only stack, every layer computes its own K and V for every prompt token, so prefill runs the full depth over the whole prompt and the cache stores one entry per layer per token. V4.1-Flash changes where the cache comes from. The encoder half processes the prompt causally; the decoder half does not compute per-layer KV from its own hidden states for the prompt, but projects its global KV from the encoder's final hidden states.

Why prefill gets cheaper. For prompt tokens, the expensive per-layer work is concentrated in the encoder: the decoder's prompt-side KV is a projection of encoder outputs rather than a full pass through the decoder's own layers. The model card gives 8B active parameters per token in prefill versus 16B in decode, which is roughly the encoder-only path versus both halves. Since prefill dominates cost for long prompts and agent sessions that keep appending tool output, halving the active parameters there is a direct cut in time to first token and in prefill compute.

Why KV memory drops. The decoder layers share KV derived from one representation instead of each storing an independent cache, which is a form of cross-layer KV sharing. Combined with an FP4 main KV cache, the card reports about 890 bytes per token of global KV, roughly a quarter of V4-Flash. At 1M tokens that is under 1 GB of global KV per sequence, which changes how many long sessions fit per node.

What the decoder loses. Its layers can no longer build prompt representations of their own: every decoder layer attends over the same projected view of the prompt. Fine-grained features that a deep layer would normally compute from the prompt must already exist in the encoder's final states. Expect risk on tasks that need deep, layer-specific processing of distant context (multi-hop reasoning over a long document, exact recall), and an FP4 cache adds quantization error on top. Generated tokens still pass through both halves, so decode quality on short prompts is less affected.

Compared with other designs. Cross-layer KV sharing (several adjacent layers reusing one cache) and YOCO-style designs, where later layers reuse a single global cache produced by earlier layers, make a similar trade: fewer independent caches in exchange for less layer-specific context processing. V4.1-Flash goes further by also changing the active compute in prefill, and pairs it with CSA2 sparse attention so each decoded token reads only selected blocks.

Serving implications. Prefill pools need less compute per token, so the prefill-to-decode capacity ratio in a disaggregated setup shifts. Prefix caching gets cheaper to store. KV transfer between prefill and decode nodes is smaller. Engine support matters: an encoder-decoder layout with FP4 KV needs kernels and a scheduler that understand it, so check the serving engine's support before planning a deployment.

Worth sketching. Optional, but a quick diagram makes the shared-cache point obvious.

flowchart LR
    A["Prompt tokens"] --> B["Causal encoder<br/>20 layers"]
    B --> C["Final encoder states"]
    C --> D["KV projection<br/>FP4 cache"]
    D --> E["Decoder layers<br/>20 layers"]
    F["Generated token"] --> B
    E --> G["Next token"]
Diagram

Follow-ups: How would you test whether the shared prompt representation hurts multi-hop retrieval at 500K tokens? What does this design do to the prefill-to-decode GPU ratio in a disaggregated cluster?

TechnicalMediumDeepSeekAI Engineer

7. Explain Multi-head Latent Attention (MLA).

The full question

Explain Multi-head Latent Attention (MLA). What problem does it solve and what does it cost?

Model answer

MLA attacks the KV-cache bottleneck. In standard multi-head attention you cache a full K and V per head per token, and at long context and large batch that cache, not the weights, becomes the memory and bandwidth ceiling on decode. MLA instead projects the input down into a single low-rank latent vector per token (dimension far smaller than the concatenated head dimensions) and caches only that. At attention time it up-projects the latent back into per-head K and V. So you store one small vector per token instead of full K/V for every head, which is where the large reported KV-cache reductions come from (roughly a few percent of MHA for the big MoE models), directly buying you longer context and bigger batches per GPU.

The complication is RoPE. Rotary position embedding has to be applied to K in a way that does not commute cleanly with the low-rank up-projection, so you cannot just compress the RoPE-carrying part. DeepSeek's fix is decoupled RoPE: split into a position-agnostic component that gets compressed into the latent, and a small position-carrying component (extra query dimensions plus a shared key) that carries RoPE separately and is concatenated in. That preserves positional sensitivity while keeping the cache small.

Costs: extra up/down projection matrices and a more intricate kernel than vanilla attention, which is exactly why they wrote and open-sourced FlashMLA to make MLA decode fast on Hopper GPUs. Naive implementations can lose the win to projection overhead.

Follow-ups: Why does decode become memory-bandwidth-bound rather than FLOP-bound, and how does that change your kernel priorities? How does MLA interact with tensor parallelism across the head dimension?

TechnicalMediumDeepSeekAI Engineer

8. DeepSeek claims frontier-class results at a fraction of the usual training cost.

The full question

DeepSeek claims frontier-class results at a fraction of the usual training cost. If an interviewer asks "how is that even possible," what is your structured answer?

Model answer

Frame it as compounding efficiency at every layer, not one magic trick, and be careful about what the cost number does and does not include. Architecture: MoE means you train and serve with about 37B active parameters while holding 671B of capacity, so you pay far less compute per token than a dense model of equivalent quality; MLA shrinks the KV cache so long-context training and inference fit; MTP squeezes more signal from each step. Numerics: FP8 training roughly halves memory and doubles matmul throughput, and fine-grained quantization makes it stable at scale. Systems: DualPipe plus a tuned all-to-all library hide the MoE communication behind compute, so the cluster stays busy; custom kernels (FlashMLA, DeepGEMM) mean they are not leaving hardware performance on the table. Method: verifiable-reward RL with GRPO (no critic network) gets reasoning cheaply, and heavy data curation means fewer wasted tokens.

The honest caveats a good candidate adds: the widely cited figure is a final training run compute cost, not total cost of ownership - it excludes prior research, failed runs, data work, and salaries. It also reflects the discipline of a team that co-designs model, kernels, and cluster together, which is hard to replicate by copying any single component. The real lesson is organisational: efficiency came from owning the whole stack, from the attention variant down to the GEMM and the file system, so no layer silently wastes the others' work.

Follow-ups: Which of these levers generalises to a lab with different hardware, and which is specific to their GPU setup? What is missing from a single "training cost" number?

TechnicalMediumDeepSeekAI Engineer

9. DeepSeek-V3 uses auxiliary-loss-free load balancing.

The full question

DeepSeek-V3 uses auxiliary-loss-free load balancing. What was wrong with the auxiliary loss, and how does the bias trick work?

Model answer

Left alone, MoE routers collapse onto a few favourite experts: those get all the tokens and all the gradient, the rest starve, and you waste both capacity and hardware because expert-parallel devices sit idle. The classic fix is an auxiliary load-balancing loss that penalises uneven expert utilisation. It works, but it fights the main objective: you are adding a gradient term whose only purpose is balance, and it bleeds into the model's quality. Tuning its weight is a lose-lose - too small and you get imbalance, too large and you hurt the language-modelling loss.

DeepSeek's auxiliary-loss-free approach removes that gradient term entirely and instead keeps a per-expert bias added to the routing scores only for the top-k selection decision. If an expert is over-subscribed, its bias is nudged down; if under-used, nudged up. The bias is updated by a simple rule based on observed load, not by backprop, so it steers routing toward balance without contaminating the training gradient. The expert's actual output weighting still uses the unbiased affinity, so you get balance in who is selected without distorting how much they contribute.

The payoff they report is better performance and cleaner expert specialisation than the auxiliary-loss baseline, because the model is no longer paying a quality tax for balance. Naming the separation - bias affects selection, not the gate value used in the weighted sum - is the detail that shows you actually understand it.

Follow-ups: How would you set the bias update step size, and what happens if it is too aggressive? Does this remove the need for any balancing signal during inference?

TechnicalMediumDeepSeekAI Engineer

10. What is multi-token prediction (MTP) and why train with it?

Model answer

Standard language-model training predicts only the next token. MTP adds auxiliary heads (or sequential modules) that also predict the next few tokens at each position, so the training signal per step is denser: the model is pushed to plan slightly further ahead rather than being purely greedy about position t+1. DeepSeek-V3 uses this as a training objective and reports it improves quality, the intuition being that forcing the representation to be predictive of t+2, t+3 encourages it to encode more about the future trajectory, which regularises and strengthens the main next-token head.

There is a second, practical payoff at inference. The extra prediction heads give you a natural speculative decoding mechanism: the MTP head proposes future tokens, the main model verifies them in one pass, and accepted tokens are emitted without a full forward step each. That turns a training-time objective into a decode-time speedup, which fits DeepSeek's efficiency-everywhere pattern.

Trade-offs to name: extra parameters and compute during training for the additional heads, and the depth-vs-benefit question - predicting too far ahead adds noise for diminishing return, so the horizon is small. At inference, the speculative win depends on acceptance rate; if the draft tokens are usually rejected you have added latency, not removed it. The honest framing is that MTP is a modest, cheap training improvement that also happens to hand you a compatible draft model for free.

Follow-ups: How does MTP-based speculation compare to using a separate small draft model? What acceptance rate do you need for speculative decoding to be a net win?

TechnicalMediumDeepSeekAI Engineer

11. FP8 training at 671B scale is hard.

The full question

FP8 training at 671B scale is hard. What actually breaks in low precision, and how do you make it stable?

Model answer

FP8 has very few mantissa bits and a narrow dynamic range, so the failure mode is that a handful of large-magnitude values (activation or gradient outliers) either overflow or force the scale so high that everything small underflows to zero. Do it naively across a whole tensor and training diverges or silently loses signal. The reason FP8 is worth the pain is that it roughly halves memory and doubles tensor-core throughput versus BF16, which at 671B parameters is the difference between feasible and not.

DeepSeek's approach is fine-grained quantization rather than per-tensor: they scale in small blocks - reported as tile-wise 1x128 for activations and block-wise 128x128 for weights - so an outlier only blows up the scale of its own small block instead of the entire tensor, keeping precision where the values are well-behaved. They also keep the numerically sensitive parts in higher precision: master weights, optimizer state, and certain accumulations stay in BF16/FP32, and they promote accumulation out of FP8 to avoid error building up over a long dot product. The GEMMs run in FP8 (they open-sourced DeepGEMM for exactly this), but the accumulation and the delicate reductions do not.

The general principle to state: quantize the bandwidth- and compute-heavy matmuls, protect the low-count, high-sensitivity state, and choose a quantization granularity fine enough that outliers are contained. Prove it with loss-curve parity against a BF16 baseline, not vibes.

Follow-ups: Why is per-tensor scaling insufficient and per-block enough? Which tensors would you never put in FP8, and why keep FP32 master weights?

TechnicalMediumDeepSeekAI Engineer

12. R1-Zero was trained with reinforcement learning and essentially no supervised fine-tuning first.

The full question

R1-Zero was trained with reinforcement learning and essentially no supervised fine-tuning first. What did that show, and why did the full R1 add SFT back?

Model answer

R1-Zero's result is the striking one: starting from the base model and applying RL with simple, rule-based verifiable rewards (is the final answer correct, is the format right), reasoning ability emerged on its own. The model learned to produce long chains of thought, to self-verify, to backtrack, and reasoning benchmark scores climbed sharply (for example AIME pass@1 rising from the mid-teens to the seventies over training). The point it proved is that you do not necessarily need human-written reasoning traces to get reasoning; you can incentivise it with reward alone, and sophisticated behaviours like re-checking work appear without being explicitly demonstrated.

The catch is that pure-RL output is a mess to use. R1-Zero suffered from poor readability and language mixing - it would switch languages mid-trace or produce chains that were effective but hard to read. So the full R1 pipeline reintroduces a small amount of curated cold-start SFT data before RL to give the model a clean starting style and format, then does RL for reasoning, then a further round to fold in general helpfulness and safety. SFT here is not teaching it to reason - RL does that - it is fixing presentation and stability so the reasoning is legible and well-behaved.

The lesson for a candidate: separate the capability question (RL can induce reasoning) from the product question (you still need alignment and formatting passes to ship it).

Follow-ups: What reward-hacking failure modes would you watch for with rule-based rewards? Why does language mixing emerge, and how would you penalise it without hurting reasoning?

TechnicalMediumDeepSeekAI Engineer

13. Explain GRPO. Why did DeepSeek drop the value network that PPO uses?

Model answer

GRPO (Group Relative Policy Optimization) is the RL algorithm behind R1. In PPO you train a separate value network (a critic) to estimate the baseline expected reward at each state, and the advantage is reward minus that baseline. That critic is roughly the same size as the policy, so it doubles memory and adds its own training instability and hyperparameters. GRPO removes it. For each prompt you sample a group of outputs from the current policy, score them all with the reward function, and use the group's mean (and standard deviation) as the baseline: an output's advantage is how much better than its peers it did, normalised within the group. No learned critic, just the empirical group statistics.

That is a big efficiency win, which is on-brand for DeepSeek: you cut the critic's memory and compute, and for verifiable-reward tasks (math, code) where reward is a cheap, objective check, the group baseline is a perfectly good advantage estimate. You keep PPO's clipped surrogate objective and a KL penalty toward a reference policy to stop the model drifting off distribution.

The costs: you need multiple samples per prompt (the group), so more inference per update, and the baseline is only as good as your group size and reward signal. It shines when rewards are verifiable and cheap to compute at scale; it is shakier when reward is a noisy learned model, where a proper critic might reduce variance more.

Follow-ups: How does group size trade off against variance and cost? Where does the KL-to-reference term matter most, and what breaks if you drop it?

TechnicalMediumDeepSeekAI Engineer

14. Walk me through DeepSeekMoE.

The full question

Walk me through DeepSeekMoE. How is it different from a standard top-2 MoE like Mixtral?

Model answer

Two ideas: fine-grained experts and shared experts. Standard MoE (Mixtral-style) splits each FFN into a handful of full-size experts and routes each token to top-2. DeepSeekMoE instead uses many smaller experts and routes to more of them, which increases the number of possible expert combinations per token and lets specialisation be finer-grained without raising active parameter count. On top of that, a few experts are designated shared and always active for every token, so common, cross-cutting knowledge lives there and the routed experts do not each have to re-learn it, reducing redundancy.

The economics are the usual MoE trade: total parameters are huge (V3 is 671B) but active parameters per token are small (about 37B), so you pay dense-13B-class compute for a much larger knowledge capacity. Memory still scales with total parameters, so you must hold all experts in VRAM across the cluster, and training needs expert parallelism with all-to-all communication to ship tokens to the device holding their expert. That communication is a first-order cost, which is why they built DeepEP and overlapped it with compute via DualPipe.

The interview signal is that you see MoE as a systems problem, not just a modelling trick: quality-per-FLOP goes up, but you now own load balancing, all-to-all bandwidth, and expert-placement decisions that dense models never face.

Follow-ups: Why do fine-grained experts make load balancing harder? Where would you place shared vs routed experts across devices to minimise communication?

Practice these out loud, don't memorise them

Reading an answer is not the same as being able to give one under pressure. ChannelPulse plays the interviewer, asks the follow-ups, and scores each answer with feedback and a model answer so you can hear the gap between what you said and what lands.

Get ChannelPulse Browse all questions