Tokenization — BPE, WordPiece, SentencePiece

Tokenization converts raw text into a sequence of token IDs that the model can process. The tokenizer defines the vocabulary — the set of all tokens the model knows. Challenges: • Word-level: Large vocab (millions of words). OOV problem. • Character-level: No OOV, but sequences are very long. • Subword: Balance between both — the standard approach.

Used by GPT, RoBERTa, LLaMA. 1. Start with character-level vocabulary 2. Iteratively merge the most frequent adjacent pair of tokens 3. Repeat until vocabulary size is reached (e.g., 50,257 tokens for GPT-2) Result: Common words are single tokens. Rare words are split into frequent subwords. 'tokenization' → ['token', 'ization'] or ['t', 'oken', 'ization']

Used by BERT. Similar to BPE but merges pairs that maximize language model likelihood rather than raw frequency. 'playing' → ['play', '##ing'] '##' prefix marks continuation of a word.

Used by T5, LLaMA, multilingual models. Language-independent — treats text as a sequence of Unicode characters with no word-boundary assumptions. Works on any language without preprocessing.