10. LLM - Foundation

Model that assigns probabilities to sequence of words

N-gram models predict the next word using the last N words.

Examples:

• Estimate probabilities by counting occurrences: P(word | beginning of sentence) ~ C(complete sentence) / C(beginning of sentence)

• Use pseudo-words for sentence beginning <S>

Sequence of n tokens

Challenges:

1. Probabilities are ≤ 1, so multiplying many small probabilities → Numerical underflow.

2. Rare or unseen N-Grams → Probability becomes zero, affecting generation & evaluation.

Solution:

• Store log probabilities instead of raw probabilities.

• Logarithms prevent numerical underflow while preserving ranking order.

1. Extrinsic evaluation:

   • Test LM in an actual application (e.g., speech recognition).

   • Expensive but measures real-world impact.

2. Intrinsic evaluation:

   • Evaluate LM using a gold standard test set.

   • Prevent test data from appearing in training.

1. Start with a seed word or phrase.

2. Predict the next word based on conditional probabilities.

3. Sample from the probability distribution.

4. Repeat until completion.

Minimizing perplexity -> maximizing probability

Problems caused by OOV words:

1. During evaluation: Test set contains unseen words → Probability becomes zero → Perplexity undefined.

2. During generation: LM cannot generate words it has never seen.

Solutions:

• Replace rare words with <UNK> (unknown token).

• Use smoothing techniques to assign probabilities to unseen words like Laplace-Smoothing

Purpose: Assign some probability to unseen N-Grams to prevent zero probabilities.

Smoothing techniques:

1. Use <UNK> -> Convert OOV word and rare words to <UNK>

2. Laplace Smoothing → Add 1 to every count.

3. Add-k Smoothing → Add a smaller fraction kkk (e.g., 0.5, 0.05).

4. Backoff → Use smaller N-Grams when larger ones are missing.

5. Interpolation → Combine probabilities from different N-Gram sizes.

🚀 Effect: Prevents zero probabilities & improves generalization.

Word-level tokenization problems:

• Struggles with out-of-vocab (OOV) words

• Fails to generalize well to unseen or rare terms

• Requires a larger vocab to account for all word variations

Subword tokenization solution:

• Breaks words into smaller units -> flexible tokenization

• Guarantees near-complete coverage, even for unseen text

WordPiece Tokenization:

• Similar to Byte Pair Encoding -> Initalizes vocabulary with every char in the training data

• Learns a fixed number of merge rules.

Difference from BPE:

• Select symbol pairs that maximizes training data likelihood, not just frequency.

• Avoids merging symbols that reduce overall probability.

SentencePiece Tokenization:

• Does not assume spaces separate words.

• Treats spaces as characters.

• Uses _ (underscore) to represent spaces and add this to the base vocabulary

🚀 Advantages:

• Works for languages without spaces (e.g., Chinese, Japanese).

• No need for pre-tokenization.