1. Write code to measure a tokenizer's fertility across languages, and explain what you would do with the result.
Model answer
Fertility is mean tokens per word. The measurement must be per language on comparable content, and word counting has to respect scripts that do not delimit with spaces the same way.
from collections import defaultdict
def fertility(tokenizer, corpus_by_lang):
# corpus_by_lang: {"hi": [sentence, ...], "en": [...], ...}
results = {}
for lang, sentences in corpus_by_lang.items():
n_words = n_tokens = 0
for s in sentences:
words = s.split() # crude; see caveats
n_words += len(words)
n_tokens += len(tokenizer.encode(s, add_special_tokens=False))
results[lang] = n_tokens / max(n_words, 1)
return results
# Interpreting: fertility ~1.1 (English) vs ~5 (an unseen Indic script)
# means that language costs ~5x the tokens per word -> cost, latency,
# and effective-context penalty all scale with that ratio.
Caveats I would raise unprompted: str.split() is a poor word boundary for many Indian languages and for scripts without spaces, so use a language-aware segmenter or at least a consistent Unicode-aware rule, and compare on parallel content (the same sentences translated) so you are measuring the tokenizer, not differences in what each corpus talks about. Also report token-per-character or token-per-byte alongside fertility, since word definitions vary.
What I do with it: rank languages by fertility to find where the tokenizer is starving. High fertility on a target language flags either retraining the tokenizer on a balanced corpus or extending the vocabulary with that language's frequent morphemes. It also feeds cost and context-budget planning: if Kannada runs 3x English fertility, your RAG chunker, few-shot budget, and pricing all need adjusting for Kannada users. This is exactly the analysis behind Sarvam-1's low-fertility design.
Follow-ups: Why can two tokenizers with the same vocab size have very different fertility on Tamil? How would parallel-corpus fertility differ from fertility measured on independent monolingual corpora?