Alibaba (Qwen) interview questions & answers

19 Alibaba (Qwen) interview questions with full model answers — Coding, Behavioral, Technical. Drawn from the same bank ChannelPulse drills from (19 Alibaba (Qwen) questions in total).

BehavioralMediumAlibaba (Qwen)AI Engineer

1. Alibaba open-sources Qwen under Apache 2.0 while running a commercial cloud business.

The full question

Alibaba open-sources Qwen under Apache 2.0 while running a commercial cloud business. Walk me through the strategy, and tell me about a time you owned an ambiguous technical decision end to end.

Model answer

This is a hybrid strategy-plus-behavioural question, and both halves matter here. On strategy, the honest business case for open weights from a cloud company: open models are demand generation for the cloud. Every developer who builds on Qwen is a candidate to run it on Alibaba Cloud (Model Studio / DashScope) or to buy the hosted, larger, or managed variants. Open weights build ecosystem gravity, standardise the world on your architecture and tokenizer, attract external contributions and talent, and generate an enormous free evaluation and bug-finding loop. It also reframes competition: against closed frontier labs, being the default open choice is a defensible position, especially internationally and in markets sensitive to vendor lock-in. The tradeoff is that you give away the weights, so your moat shifts to scale, serving efficiency, the newest/biggest closed variants, data, and the cloud platform around the model. Being able to argue this crisply signals you understand why your work exists commercially, not just technically.

For the ownership half, use STAR and pick a real story where the requirements were genuinely unclear and you drove to a decision: frame the ambiguity (conflicting goals, no obvious right answer), the options you weighed with their tradeoffs, the data or experiments you used to decide, the call you made, and - critically - how you measured whether it worked and what you would change. Large AI orgs weight ownership, cross-team collaboration, and incident handling heavily in the hiring-manager round, so land the "I owned it, including the parts that went wrong" note rather than a tidy success story. Keep it concrete and quantified.

Follow-ups: Where is the open-weight strategy weakest, and what would change your mind? Tell me about a decision of yours that turned out wrong and how you caught it.

CodingMediumAlibaba (Qwen)Software Engineer

2. Split Concatenated Strings (LeetCode)

Model answer

The task

Given strings in a fixed order, you may reverse each one, concatenate them, then split the result and swap the two halves. Return the lexicographically largest string you can obtain.

Key idea

The split falls either between strings or inside one string. For each possible split string, use the lexicographically larger orientation for every other string, then try both orientations and every split position in the chosen string.

Approach

  1. Replace each string with the larger of itself and its reverse.
  2. For each string, precompute the other strings in the order they appear after a split inside it.
  3. Try both orientations of the split string.
  4. For each split position, form the swapped result: suffix of the split string, remaining strings in cyclic order, then its prefix.
  5. Keep the lexicographically largest candidate.
from typing import List

class Solution:
    def splitLoopedString(self, strs: List[str]) -> str:
        if not strs:
            return ""

        # For strings not containing the split, the larger orientation is best.
        best_orientations = [
            max(s, s[::-1]) for s in strs
        ]

        answer = ""

        for i, original in enumerate(strs):
            # After the split, the other strings appear cyclically from i + 1.
            middle = "".join(best_orientations[i + 1:] + best_orientations[:i])

            # The split string itself may use either orientation.
            for current in (original, original[::-1]):
                for cut in range(len(current) + 1):
                    candidate = (
                        current[cut:]
                        + middle
                        + current[:cut]
                    )
                    if candidate > answer:
                        answer = candidate

        return answer

Complexity:

  • Time: O(n · m · N), where n is the number of strings, m is their maximum length, and N is the total output length; each candidate is assembled and compared.
  • Space: O(N) for the prepared strings and candidate construction.

Common follow-ups

  • What if strings can be reordered? Sort the other strings by the concatenation comparator a + b > b + a before building candidates.
  • Can candidate construction be made faster? Use rolling hashes or suffix-array-style comparisons to avoid repeatedly materializing full candidates.
  • What if the split must be strictly inside the concatenation? Exclude the two boundary cuts that would leave an empty half.
CodingMediumAlibaba (Qwen)AI EngineerCoding

3. A standard Alibaba coding round.

The full question

A standard Alibaba coding round. Design a data structure for a fixed-capacity cache that supports get and put in O(1) and evicts the least-recently-used entry when full. Implement it in Python.

Model answer

This is the classic LRU cache, and it is worth knowing cold because the algorithmic rounds gate the whole loop regardless of how senior the model work is. The requirement is O(1) for both operations, which rules out scanning for the least-recently-used item. The standard answer is a hash map plus a doubly linked list: the map gives O(1) lookup by key, and the linked list keeps entries in recency order so eviction is O(1) at the tail and promotion on access is O(1) by splicing.

class Node:
    __slots__ = ("k", "v", "prev", "next")
    def __init__(self, k=0, v=0):
        self.k, self.v, self.prev, self.next = k, v, None, None

class LRUCache:
    def __init__(self, capacity: int):
        self.cap = capacity
        self.map = {}                      # key -> Node
        self.head, self.tail = Node(), Node()   # sentinels
        self.head.next, self.tail.prev = self.tail, self.head

    def _remove(self, n):
        n.prev.next, n.next.prev = n.next, n.prev

    def _push_front(self, n):              # most-recently-used at front
        n.prev, n.next = self.head, self.head.next
        self.head.next.prev = n
        self.head.next = n

    def get(self, key: int) -> int:
        if key not in self.map:
            return -1
        n = self.map[key]
        self._remove(n); self._push_front(n)
        return n.v

    def put(self, key: int, value: int) -> None:
        if key in self.map:
            n = self.map[key]; n.v = value
            self._remove(n); self._push_front(n)
            return
        if len(self.map) == self.cap:
            lru = self.tail.prev            # evict at tail
            self._remove(lru); del self.map[lru.k]
        n = Node(key, value)
        self.map[key] = n; self._push_front(n)

Sentinel head/tail nodes remove the null-checking edge cases that trip people up under time pressure. In Python you could also use collections.OrderedDict with move_to_end and popitem(last=False) for a shorter answer, but interviewers usually want the underlying mechanism, so I would code the linked list and mention the shortcut. Complexity: O(1) time per operation, O(capacity) space.

Follow-ups: Extend it to LFU (least-frequently-used) - what changes? How does this relate to KV-cache eviction policies in long-context serving?

CodingMediumAlibaba (Qwen)Software Engineer

4. Valid Parenthesis String (LeetCode)

Model answer

The task

Given a string containing (, ), and *, decide whether the * characters can be interpreted so the resulting parentheses are balanced. Each * may represent an opening parenthesis, a closing parenthesis, or nothing.

Key idea

Track the range of possible counts of unmatched opening parentheses after each character. The lowest count assumes stars close parentheses when possible; the highest assumes stars open them. If the highest count becomes negative, no interpretation can work.

Approach

  1. Initialize the minimum and maximum possible unmatched opening counts to zero.
  2. For (, increment both counts.
  3. For ), decrement both counts.
  4. For *, decrement the minimum and increment the maximum.
  5. Clamp the minimum to zero, since unmatched openings cannot be negative.
  6. Reject if the maximum is negative; after scanning, accept only if the minimum is zero.
class Solution:
    def checkValidString(self, s: str) -> bool:
        # Range of possible unmatched opening-parenthesis counts.
        low = 0
        high = 0

        for ch in s:
            if ch == "(":
                low += 1
                high += 1
            elif ch == ")":
                low -= 1
                high -= 1
            else:  # "*" may be "(", ")", or empty
                low -= 1
                high += 1

            # No interpretation can have fewer than zero unmatched opens.
            low = max(low, 0)

            # Even interpreting every star as "(" cannot fix this prefix.
            if high < 0:
                return False

        # A valid interpretation must be able to leave no unmatched opens.
        return low == 0

Complexity:

  • Time: O(n), where n is the string length; each character is processed once.
  • Space: O(1), using only two counters.

Common follow-ups

  • How would you return one valid interpretation? Track choices or use a stack of parenthesis and star indices, then match remaining openings with later stars.
  • Can dynamic programming solve it? Yes; track reachable balance counts, though it uses O(n²) time and O(n) space.
  • What if there are multiple wildcard types with different allowed meanings? Update the reachable balance range or use a more general DP if the possible balances are not contiguous.
CodingMediumAlibaba (Qwen)Software Engineer

5. Kth Largest Element in an Array (LeetCode)

Model answer

The task

Find the value that would appear in position k when the array is sorted from largest to smallest. Duplicate values count as separate positions.

Key idea

The answer is the element at index len(nums) - k in ascending order. Quickselect partitions the array around pivots, then continues only in the partition that contains that index.

Approach

  1. Convert k to its target index in ascending order.
  2. Choose a random pivot to reduce the chance of consistently poor partitions.
  3. Partition into values smaller than, equal to, and greater than the pivot.
  4. Narrow the search to the partition containing the target, or return the pivot if the target lies among the equal values.
import random
from typing import List

class Solution:
    def findKthLargest(self, nums: List[int], k: int) -> int:
        target = len(nums) - k
        left, right = 0, len(nums) - 1

        while left <= right:
            pivot = nums[random.randint(left, right)]

            # Three-way partition:
            # nums[left:lt] < pivot
            # nums[lt:gt + 1] == pivot
            # nums[gt + 1:right + 1] > pivot
            lt, i, gt = left, left, right
            while i <= gt:
                if nums[i] < pivot:
                    nums[lt], nums[i] = nums[i], nums[lt]
                    lt += 1
                    i += 1
                elif nums[i] > pivot:
                    nums[i], nums[gt] = nums[gt], nums[i]
                    gt -= 1
                else:
                    i += 1

            if target < lt:
                right = lt - 1
            elif target > gt:
                left = gt + 1
            else:
                return pivot

        raise ValueError("k must be within the array's valid range")

Complexity:

  • Time: O(n) on average, since each partition discards the portions that cannot contain the answer.
  • Space: O(1) auxiliary space; the partitioning is in place.

Common follow-ups

  • What if worst-case time must be bounded? Use a min-heap of size k for O(n log k) time.
  • How do duplicates affect the result? Each occurrence occupies its own rank; the three-way partition handles equal values together.
  • Can the input remain unchanged? Make a copy before partitioning, at the cost of O(n) extra space.
CodingMediumAlibaba (Qwen)AI Engineer

6. Qwen2.5-Coder trains with repository-level fill-in-the-middle using special tokens like < fim_prefix >, < fim_suffix >, < fim_middle >, < repo_name…

The full question

Qwen2.5-Coder trains with repository-level fill-in-the-middle using special tokens like <|fim_prefix|>, <|fim_suffix|>, <|fim_middle|>, <|repo_name|>, and <|file_sep|>. Write the function that formats a repo-level FIM training example, and explain why repo-level beats file-level for real code completion.

Model answer

FIM teaches the model to fill a gap given both left and right context, which is what an editor autocomplete actually needs (you have code before and after the cursor). The trick is to reorder: put prefix and suffix first, then have the model predict the middle. Repo-level FIM extends this across files so the model can use imports and definitions from elsewhere in the project.

def format_repo_fim(repo_name, files, target_idx, cut_a, cut_b):
    # files: list of (path, content). Split the target file into pre/mid/suf.
    path, content = files[target_idx]
    pre, mid, suf = content[:cut_a], content[cut_a:cut_b], content[cut_b:]
    parts = [f"<|repo_name|>{repo_name}"]
    for i, (p, c) in enumerate(files):
        if i == target_idx:
            parts.append(
                f"<|file_sep|>{p}\n"
                f"<|fim_prefix|>{pre}<|fim_suffix|>{suf}<|fim_middle|>{mid}"
            )
        else:
            parts.append(f"<|file_sep|>{p}\n{c}")   # context files verbatim
    return "".join(parts) + "<|endoftext|>"

At training time the loss is on the whole sequence (or focused on the middle); at inference you feed everything up to <|fim_middle|> and let the model generate the missing span. The special tokens matter because they are unambiguous structural signals the model learns to key on: <|repo_name|> and <|file_sep|> mark project and file boundaries so cross-file references are learnable, and the FIM markers tell the model this is an infilling task, not left-to-right continuation.

Why repo-level wins: real completions depend on symbols defined in other files - a function you call, a type you import, a config constant. File-level training caps the model's world at one file, so it hallucinates signatures. Repo-level pretraining (reported around 32K-token context with long-context repo data) lets the model actually read the surrounding project, which is the difference between guessing an API and using the real one. The cost is longer sequences and careful data construction (ordering files, avoiding leakage of the answer into context).

Follow-ups: How would you order context files so the most relevant ones survive context truncation? Why can naive FIM masking accidentally leak the middle into the suffix, and how do you prevent it?

CodingMediumAlibaba (Qwen)Software Engineer

7. Path Sum IV (LeetCode)

Model answer

The task

Each integer encodes a node’s depth, position within that depth, and value in a small binary tree. Return the sum of the values of all root-to-leaf paths.

Key idea

Store node values by (depth, position) so children can be found directly. A depth-first search carries the running path sum and adds it to the answer whenever it reaches a leaf.

Approach

  1. Decode each integer and put its value in a hash map keyed by depth and position.
  2. Start a depth-first search at the root with a path sum of zero.
  3. Add each visited node’s value to the running sum.
  4. Find the next-level children using their positions.
  5. If neither child exists, add the running sum to the total; otherwise visit each existing child.
from typing import List

class Solution:
    def pathSum(self, nums: List[int]) -> int:
        # Map each (depth, position) to its node value.
        values = {}
        for encoded in nums:
            depth = encoded // 100
            position = (encoded // 10) % 10
            value = encoded % 10
            values[(depth, position)] = value

        total = 0

        def dfs(depth: int, position: int, path_sum: int) -> None:
            nonlocal total

            path_sum += values[(depth, position)]
            next_depth = depth + 1
            left = (next_depth, 2 * position - 1)
            right = (next_depth, 2 * position)

            has_left = left in values
            has_right = right in values

            if not has_left and not has_right:
                total += path_sum
                return

            if has_left:
                dfs(left[0], left[1], path_sum)
            if has_right:
                dfs(right[0], right[1], path_sum)

        dfs(1, 1, 0)
        return total

Complexity:

  • Time: O(n), where n is the number of encoded nodes; each node is visited once.
  • Space: O(n), for the node map and the depth-first search stack.

Common follow-ups

  • What if the tree is much deeper? Use an iterative stack to avoid recursion-depth limits.
  • What if nodes arrive as a stream? Store them as they arrive, then traverse once the full tree is available.
  • How would you return each path sum, not just the total? Collect sums at leaves in a list instead of adding them to one total.
CodingMediumAlibaba (Qwen)AI Engineer

8. Implement grouped-query attention in PyTorch and explain where the KV-cache savings come from, since Qwen relies on GQA for serving efficiency.

Model answer

GQA sits between full multi-head attention (one KV head per query head) and multi-query attention (one KV head total): query heads are split into groups, and each group shares one KV head. Qwen uses this to shrink the KV cache without the quality hit of full MQA.

import torch, torch.nn.functional as F

def gqa(x, wq, wk, wv, wo, n_heads, n_kv_heads):
    B, T, D = x.shape
    hd = D // n_heads
    q = (x @ wq).view(B, T, n_heads, hd).transpose(1, 2)       # B, H, T, hd
    k = (x @ wk).view(B, T, n_kv_heads, hd).transpose(1, 2)    # B, Hkv, T, hd
    v = (x @ wv).view(B, T, n_kv_heads, hd).transpose(1, 2)
    rep = n_heads // n_kv_heads              # query heads per KV head
    k = k.repeat_interleave(rep, dim=1)      # expand KV to match query heads
    v = v.repeat_interleave(rep, dim=1)
    att = (q @ k.transpose(-2, -1)) / hd**0.5
    mask = torch.triu(torch.ones(T, T, dtype=torch.bool, device=x.device), 1)
    att = att.masked_fill(mask, float("-inf"))
    out = F.softmax(att, dim=-1) @ v          # B, H, T, hd
    return out.transpose(1, 2).reshape(B, T, D) @ wo

The savings: the KV cache stores keys and values for every past token, sized 2 n_kv_heads hd * seq_len per layer. With MHA that is n_kv_heads == n_heads; GQA sets n_kv_heads much smaller (say 8 KV heads for 32 query heads), a 4x cache reduction. That matters because decode is memory-bandwidth bound: at each step you read the whole KV cache to attend, so a smaller cache means less bandwidth per token, larger batch sizes, and longer contexts on the same GPU. Note the cache is stored before the repeat_interleave (you keep the compact KV), and production kernels avoid materialising the repeat by indexing. Real serving uses F.scaled_dot_product_attention or FlashAttention for the fused kernel; this version is for clarity.

Follow-ups: Add the single-token decode path with a growing KV cache. Where does RoPE get applied, and why to q and k only, not v?

CodingMediumAlibaba (Qwen)Software Engineer

9. Given an even number (greater than 2), return two prime numbers whose sum will be equal to the given number.

The full question

Given an even number (greater than 2), return two prime numbers whose sum will be equal to the given number. A solution will always exist. See Goldbach’s conjecture. Example: Input: 4 Output: 2 + 2 = 4 If there are more than one solution possible, return the lexicographically smaller solution. If [a, b] is one solution with a <= b, and [c, d] is another solution with c <= d, then [a, b] < [c, d] If a < c OR a==c AND b < d.

Model answer

function goldbachPair(n) {
  if (!Number.isInteger(n) || n <= 2 || n % 2 !== 0) {
    throw new RangeError("n must be an even integer greater than 2");
  }

  // Sieve all primes up to n so each candidate can be checked in O(1).
  const isPrime = new Uint8Array(n + 1);
  isPrime.fill(1);
  isPrime[0] = 0;
  isPrime[1] = 0;

  for (let p = 2; p * p <= n; p++) {
    if (isPrime[p]) {
      for (let multiple = p * p; multiple <= n; multiple += p) {
        isPrime[multiple] = 0;
      }
    }
  }

  // The first valid a gives the lexicographically smallest [a, n - a].
  for (let a = 2; a <= n / 2; a++) {
    const b = n - a;
    if (isPrime[a] && isPrime[b]) return [a, b];
  }

  // The problem guarantees a solution exists.
  throw new Error("No prime pair found");
}

const [a, b] = goldbachPair(4);
console.log(`${a} + ${b} = 4`); // 2 + 2 = 4
  • Sieve primes up to n, then scan a in ascending order and test n - a.
  • Since a <= n / 2, each pair is ordered; the first valid pair is lexicographically smallest.

Complexity: Time O(n log log n); space O(n).

CodingHardAlibaba (Qwen)Software Engineer

10. Split Array with Equal Sum (LeetCode)

Model answer

The task

Determine whether three indices can be removed from an array so that the four nonempty sections left behind all have the same sum.

Key idea

Fix the middle removed index. Record the sums that work for the first two sections, then check whether a later third index produces that same sum for the last two sections.

Approach

  1. Build prefix sums so any section sum can be computed in constant time.
  2. Iterate over each valid middle index.
  3. For that index, examine valid first indices and store sums where the first two sections match.
  4. Examine valid third indices and check whether the last two sections match a stored sum.
from typing import List

class Solution:
    def splitArray(self, nums: List[int]) -> bool:
        n = len(nums)
        if n < 7:
            return False

        # prefix[t] is the sum of nums[0:t].
        prefix = [0] * (n + 1)
        for i, value in enumerate(nums):
            prefix[i + 1] = prefix[i] + value

        # j is the middle removed index.
        for j in range(3, n - 3):
            first_pair_sums = set()

            # i is the first removed index.
            for i in range(1, j - 1):
                left_sum = prefix[i]
                second_sum = prefix[j] - prefix[i + 1]

                if left_sum == second_sum:
                    first_pair_sums.add(left_sum)

            # k is the third removed index.
            for k in range(j + 2, n - 1):
                third_sum = prefix[k] - prefix[j + 1]
                right_sum = prefix[n] - prefix[k + 1]

                if third_sum == right_sum and third_sum in first_pair_sums:
                    return True

        return False

Complexity:

  • Time: O(n²), from checking pairs of cut indices for each middle index.
  • Space: O(n), for prefix sums and the set of candidate sums.

Common follow-ups

  • What if values can be negative? The prefix-sum and hash-set approach still works.
  • Why require at least seven elements? Four nonempty sections need three removed elements between them.
  • Can the extra space be reduced? Prefix sums can be computed incrementally, though candidate sums still need to be stored.
TechnicalMediumAlibaba (Qwen)AI Engineer

11. Qwen ships open weights that top public leaderboards.

The full question

Qwen ships open weights that top public leaderboards. As the engineer responsible for a release, how do you make sure the benchmark numbers are trustworthy and not contaminated?

Model answer

Contamination - test data leaking into pretraining - is the single biggest threat to a leaderboard-topping open release, because your credibility is the product. The defence is layered.

Prevention first. Run decontamination on the pretraining and post-training corpora: n-gram or embedding-based overlap detection against the eval sets you care about (maths, code, knowledge benchmarks), and remove or flag matches. This is imperfect (paraphrases and translations slip through, especially for a multilingual model where a benchmark may exist in several languages), so treat it as reducing, not eliminating, risk.

Detection and honesty. Report results on held-out and freshly-collected evals, not just static public sets that have been on the internet for years and are almost certainly in everyone's training data. Use benchmarks with private test splits or rolling/time-gated versions where possible. Run contamination probes: check whether the model can complete verbatim held-back test items, or whether performance collapses on perturbed versions of the same questions (rename variables, change numbers) - a big drop signals memorisation rather than capability.

Broaden the signal. No single number should carry a release. Combine automatic benchmarks with human evaluation and, ideally, third-party arena-style comparisons the team does not control, plus real task evals (agentic, long-context, tool-use) that are harder to game than static multiple-choice. Watch for the classic tell: a model that is amazing on the reported benchmark and mediocre in the wild.

Process. Freeze eval code and prompts, publish the methodology and settings (few-shot count, decoding, chat template) so results are reproducible, and version everything. For an open-weights team, reproducibility by outsiders is itself the trust mechanism: if the community can rerun your numbers, contamination gets caught fast.

Follow-ups: Why is contamination especially insidious for a multilingual model? How would you design an eval that stays meaningful six months after release?

TechnicalMediumAlibaba (Qwen)AI Engineer

12. Qwen3.5 interleaves Gated DeltaNet linear-attention layers with full gated-attention layers at about 3:1.

The full question

Qwen3.5 interleaves Gated DeltaNet linear-attention layers with full gated-attention layers at about 3:1. Why build a hybrid, and what does it change for long-context serving?

Model answer

Full softmax attention is the expensive part at long context: prefill compute grows quadratically and the KV cache grows with every token at every layer. Linear-attention layers keep a fixed-size state instead. A hybrid lets the cheap layers do most of the sequence mixing and keeps a minority of full-attention layers for precise recall over the whole context.

How Gated DeltaNet works. Each head maintains a matrix-valued state rather than a cache of all past keys and values. The delta rule updates it like online regression: read what the state currently predicts for the new key, then write a correction toward the new value, scaled by a learned rate. A learned gate decays the state so the layer can forget stale content. Memory and per-token decode cost are constant in sequence length, and chunked parallel forms keep training efficient on GPUs.

Why not all linear. A fixed-size state is lossy compression. Exact retrieval of one specific token far back (a name, a number, a needle) is where pure linear models fall down. One full-attention layer every few blocks restores that. The model card layout is 15 repeats of (3 Gated DeltaNet + 1 gated attention), each followed by an MoE layer.

Serving consequences. Only about a quarter of the layers hold a KV cache, so long-context memory drops roughly 4x against an all-attention stack of the same depth, which is what makes a 262K native context practical. The engine now manages two kinds of state: paged KV blocks for attention layers and recurrent states for DeltaNet layers. Prefix caching gets harder because a recurrent state is a snapshot at one position, not blocks you can share at any prefix boundary, so engines checkpoint states at chunk boundaries. Speculative decoding needs state rollback when draft tokens are rejected, and kernels are less mature than FlashAttention.

Follow-ups: How would you show the hybrid has not lost long-range recall relative to a full-attention baseline? How would you implement prefix caching when a quarter of your layers hold KV and the rest hold recurrent state?

TechnicalMediumAlibaba (Qwen)AI Engineer

13. Qwen2.5 extends context to 128K (and up to about 1M for the Turbo variant) using YaRN plus Dual Chunk Attention, mostly training-free.

The full question

Qwen2.5 extends context to 128K (and up to about 1M for the Turbo variant) using YaRN plus Dual Chunk Attention, mostly training-free. Explain how that works and why post-hoc extension is attractive.

Model answer

The core problem is RoPE: a model pretrained at, say, 32K tokens has never seen the rotary phases that correspond to positions far beyond that, so naive extrapolation to 128K produces garbage attention. The Qwen approach combines two training-free tricks.

YaRN rescales the RoPE frequencies so the effective positions at inference stay inside the range the model was trained on. Rather than uniformly stretching all frequencies (which blurs local, high-frequency structure), YaRN interpolates the low frequencies (long-range) more than the high ones and adds an attention-temperature correction, so long-range positions become representable without destroying short-range resolution.

Dual Chunk Attention (DCA) attacks the same problem from the attention side. It splits the long sequence into chunks and remaps relative positions so that no two tokens ever appear farther apart than the pretraining length. It uses distinct patterns: intra-chunk attention (original relative positions within a chunk), inter-chunk attention (positions across chunks kept in-range), and successive-chunk handling for continuity at boundaries. Together they let a model trained short attend coherently over very long inputs.

Why post-hoc is attractive: training natively at 128K or 1M is brutally expensive - attention memory and compute grow with sequence length, so long-context pretraining is often the single most costly thing you can do. Training-free extension gets most of the capability by adjusting positional handling at inference plus light long-context fine-tuning, at a fraction of the cost, and lets one base model serve multiple context tiers. The tradeoff is that extrapolated long-context quality still trails a natively long-trained model, and effective recall past the window degrades, so you validate with retrieval-style long-context evals, not just perplexity.

Follow-ups: Why does KV-cache memory, not FLOPs, often become the real 1M-token bottleneck? How would you test whether the model truly uses information 800K tokens back rather than just not crashing?

TechnicalMediumAlibaba (Qwen)AI Engineer

14. Qwen3 unifies a "thinking" mode and a "non-thinking" mode in a single model, with a thinking budget the caller can set.

The full question

Qwen3 unifies a "thinking" mode and a "non-thinking" mode in a single model, with a thinking budget the caller can set. How would you train that, and how would you serve it?

Model answer

The product goal is one checkpoint that can do fast, cheap answers for easy queries and long chain-of-thought for hard ones, with the caller trading latency for accuracy via a budget. Qwen3's reported recipe is a four-stage post-training pipeline: (1) a long chain-of-thought cold start that teaches the model to produce structured reasoning traces; (2) reasoning RL that rewards correct final answers on verifiable maths and code so the reasoning actually pays off; (3) thinking-mode fusion, blending the reasoning model back together with non-thinking behaviour so one model supports both styles, typically gated by a control token or template; and (4) a general RL stage to polish helpfulness, safety, and formatting across the mixed distribution.

Serving: expose the mode as a flag plus a thinking budget (a token cap on the internal reasoning, reported up to tens of thousands of tokens). When the budget is small or zero, the model skips or truncates the thinking span and answers directly; when it is large, it can reason further before the answer. Practically the reasoning tokens sit between control markers so you can bill or hide them, and you can route by difficulty: cheap classifier or heuristics decide whether to spend budget at all.

The hard part is not regressing the fast path. Fusing a heavy-reasoning model with a terse one risks the model over-thinking trivial prompts (latency and cost blow up) or leaking reasoning style into simple answers. That is exactly why a later general-RL stage and careful mode gating exist. Alibaba publicly noted the unified toggle had rough edges, and the mid-2025 Qwen3-2507 refresh shipped separate Instruct and Thinking checkpoints instead; Qwen3.5 then returned to one model that thinks by default with an enable_thinking switch. That back-and-forth is a useful, honest thing to be able to discuss.

Follow-ups: How would you stop the model from "thinking" on trivial prompts? How do you evaluate whether extra budget actually buys accuracy rather than just tokens?

TechnicalMediumAlibaba (Qwen)AI Engineer

15. Qwen's reasoning models are trained with reinforcement learning using verifiable rewards on maths and code.

The full question

Qwen's reasoning models are trained with reinforcement learning using verifiable rewards on maths and code. Explain that setup and why it is preferred over PPO with a learned reward model for these domains.

Model answer

The core idea (RL with verifiable rewards, RLVR) is that for maths and code you do not need a learned reward model at all: correctness is checkable. For a maths problem you compare the final answer to ground truth; for code you run unit tests. That check is the reward - 1 if the answer is right or the tests pass, 0 otherwise (sometimes with partial credit). The policy generates a full reasoning trace and answer, gets the binary/graded reward, and RL pushes it toward reasoning paths that lead to correct answers.

Why this beats PPO with a learned reward model here: a learned reward model is a proxy, and policies are ruthless at exploiting proxy errors (reward hacking) - producing confident, well-formatted, wrong answers the reward model happens to like. A verifier that actually executes the code or checks the number is grounded truth, so there is far less to hack, and it removes the cost and drift of training and maintaining a separate reward model. It also scales cleanly: you can generate huge amounts of maths/code problems with automatic checkers.

On the algorithm side, group-relative methods (sampling several completions per prompt and using their relative rewards as the advantage) are attractive because they drop the value/critic network: with a verifiable 0/1 reward and a group baseline, you avoid the instability and memory of a learned value head that plain PPO needs. That makes large-scale reasoning RL cheaper and more stable.

Limits worth naming: verifiable rewards only exist where you can check answers, so this does not directly cover open-ended writing or subjective helpfulness (hence a separate general-RL stage). Sparse 0/1 rewards on very hard problems give little signal, so curriculum and reward shaping matter, and you must guard against the model gaming the checker (for example printing the expected output).

Follow-ups: How would you stop a code policy from hard-coding test outputs to pass the checker? When does a learned reward model still earn its keep despite the hacking risk?

TechnicalMediumAlibaba (Qwen)AI Engineer

16. Qwen2.5-VL uses a native dynamic-resolution ViT with window attention and multimodal RoPE (MRoPE).

The full question

Qwen2.5-VL uses a native dynamic-resolution ViT with window attention and multimodal RoPE (MRoPE). Why native resolution instead of fixed-size tiling, and what does MRoPE encode?

Model answer

Traditional vision-language models resize every image to a fixed square (224 or 336) or tile it into fixed crops. That throws away information: a wide document, a tall screenshot, or a high-resolution chart gets squashed or chopped, and the model loses aspect ratio and real-world scale. Qwen2.5-VL trains a native dynamic-resolution ViT that processes images at their actual spatial scale, producing a variable number of visual tokens proportional to image size. Big detailed images yield more tokens (more detail preserved), small images yield fewer (less waste). This is what lets the model do fine-grained tasks like reading small text, grounding boxes, and parsing documents where absolute and relative size matter.

The cost of variable, potentially large token counts is compute, so the ViT uses window attention in most layers: attention is restricted to local windows rather than full-image global attention, keeping cost roughly linear in the number of patches instead of quadratic, with a few full-attention layers to mix globally. That is the standard efficiency lever that makes native high resolution affordable.

MRoPE (multimodal RoPE) generalises rotary position embeddings from a single 1D text position to multiple coordinate axes: it encodes position along temporal, height, and width dimensions. For an image, a patch's position carries its 2D (row, column) location; for video, it also carries a temporal index, and Qwen2.5-VL couples that with dynamic FPS sampling so long videos are sampled sparsely and short clips densely. The payoff is that the language backbone gets genuine spatial and temporal structure - it can reason about where things are and when they happen, not just an unordered bag of visual tokens - while sharing one positional scheme with the text stream so the modalities align.

Follow-ups: How does variable token count per image complicate batching and serving? Why is native resolution especially important for OCR and grounding versus, say, image captioning?

TechnicalMediumAlibaba (Qwen)AI Engineer

17. Through Qwen3, Qwen used a byte-level BPE tokenizer with a vocabulary around 151K, augmented for multilingual coverage and with digits split into s…

The full question

Through Qwen3, Qwen used a byte-level BPE tokenizer with a vocabulary around 151K, augmented for multilingual coverage and with digits split into single characters. Why those choices for a multilingual, maths-capable model, and what are the tradeoffs?

Model answer

Start with the goal: one tokenizer that compresses Chinese, English, and dozens of other scripts reasonably well while not sabotaging arithmetic. Byte-level BPE guarantees there is no out-of-vocabulary failure: any Unicode text falls back to bytes, so rare CJK characters, emoji, and code symbols always encode. A large vocab (~151K) buys compression: more multilingual merges mean fewer tokens per sentence, which is fewer forward passes per character and better effective context for non-Latin scripts. Qwen's reported ratios are roughly 3-4 characters per token for English and denser for Chinese, so a bigger vocab directly lowers serving cost for Chinese-heavy traffic.

Digit splitting (each digit its own token) is a deliberate maths choice. If "1234" is one merged token, the model has to memorise arithmetic on opaque chunks; splitting into 1 2 3 4 exposes place value consistently, so carrying and long addition generalise far better. It costs a few extra tokens on numbers, which is a cheap price.

Tradeoffs: a large vocab inflates the embedding and output-projection matrices (vocab x hidden), which grows parameters and the softmax cost, and it needs enough multilingual data so rare tokens are actually trained. Byte fallback can fragment a rare script into many byte tokens, hurting compression for the least-resourced languages. From Qwen2.5 onward the family moved to a unified vocabulary shared across text, code, and multimodal subfamilies, which simplifies serving and lets special tokens (roles, tool calls, FIM markers) live in one namespace. Qwen3.5 pushed the same tradeoff further: roughly 250K tokens (248,320 padded) to cover 201 languages and dialects, buying compression for more scripts at the price of bigger embedding and output matrices.

Follow-ups: How would you measure whether the tokenizer is fair across languages? Why does adding a new language after pretraining rarely mean just retraining the tokenizer?

TechnicalMediumAlibaba (Qwen)AI Engineer

18. Qwen3 uses "strong-to-weak" distillation, bootstrapping the smaller models from flagship ones instead of running the full heavy post-training on ev…

The full question

Qwen3 uses "strong-to-weak" distillation, bootstrapping the smaller models from flagship ones instead of running the full heavy post-training on every size. How does that work and why is it cheaper?

Model answer

The insight: the expensive part of post-training is the search - long-CoT construction and reasoning RL that discovers good reasoning behaviour. Once a flagship model has paid that cost, you do not need every small model to rediscover it from scratch. Strong-to-weak distillation transfers the behaviour of the large "teacher" (thinking and non-thinking) into the smaller "student" checkpoints.

Concretely it usually combines two signals. On-policy / logit distillation: the student learns to match the teacher's output distribution (soft targets carry more information than hard labels, teaching relative preferences over tokens, not just the argmax). And data distillation: the teacher generates high-quality reasoning traces and responses that become supervised fine-tuning data for the student, including both mode styles so the student inherits the hybrid thinking capability.

Why it is cheaper: reasoning RL needs rollouts, reward computation, and many optimization steps at each model scale, and it is unstable and compute-heavy. Distillation replaces most of that with straightforward supervised or logit-matching objectives, which converge faster and more stably, and you generate the teacher data once and reuse it across all student sizes. Reported outcome: the small Qwen3 models reach strong performance for far less compute than running full RL at each size would take.

Risks to name: the student is capped by the teacher (you cannot distill capability the teacher lacks), distillation can transfer the teacher's biases and mistakes, and pure imitation without any RL can leave the student worse at exploration-sensitive tasks. In practice you often do a light RL or preference stage on top to recover the last bit.

Follow-ups: When does logit distillation beat SFT on teacher outputs, and when is it not worth the extra plumbing? How would you detect that the student has simply memorised teacher traces?

TechnicalMediumAlibaba (Qwen)AI Engineer

19. Qwen ships both dense and MoE models (for example Qwen3's 30B model with ~3B active parameters, and a 235B model with ~22B active).

The full question

Qwen ships both dense and MoE models (for example Qwen3's 30B model with ~3B active parameters, and a 235B model with ~22B active). Explain the tradeoff, and when you would pick the 30B-A3B MoE over a 32B dense.

Model answer

In a sparse MoE, each transformer block's feed-forward layer is replaced by many expert FFNs plus a router that sends each token to a small top-k of them. So total parameters are large (all experts stored) but active parameters per token are small (only the routed experts run). The 30B-A3B model holds ~30B weights but executes ~3B per token, so its decode FLOPs and latency resemble a ~3B dense model while its quality benefits from the far larger parameter pool.

The catch is asymmetric economics. FLOPs scale with active params; memory scales with total params. You must hold all ~30B weights in VRAM even though you only compute 3B worth per token. So MoE buys quality-per-FLOP and throughput, not quality-per-gigabyte.

When to pick 30B-A3B over 32B dense: when you are compute or throughput bound with memory to spare - high-QPS serving where you want the accuracy of a bigger model but cannot afford 32B of active compute per token, and you have the VRAM (or expert-parallel sharding) to hold all experts. Pick the 32B dense when memory is the binding constraint (single smaller GPU, edge, tight VRAM), when you want predictable per-token cost without router load-imbalance surprises, or when simplicity of deployment and fine-tuning matters. Dense is also friendlier to naive quantization and LoRA. Training MoE adds a load-balancing auxiliary loss (or the router collapses onto a few experts) and all-to-all communication for expert parallelism, a systems tax dense models never pay.

Follow-ups: Why is batched inference harder for MoE (expert imbalance inside a batch)? How does expert-choice vs token-choice routing change the load-balancing story?

Practice these out loud, don't memorise them

Reading an answer is not the same as being able to give one under pressure. ChannelPulse plays the interviewer, asks the follow-ups, and scores each answer with feedback and a model answer so you can hear the gap between what you said and what lands.

Get ChannelPulse Browse all questions