04. IR- Word Representation and Data Collection

One-hot encoding represents words as binary vectors where only one position is 1, and all others are 0.

Inefficiencies:

• Very sparse (high-dimensional).

• No semantic relationships between words.

• Not scalable for large vocabularies.

Word embeddings are dense vector representations that capture semantic relationships between words.

Advantages:

• Lower dimensionality (100-300 dimensions instead of thousands).

• Encodes meaning (similar words have closer vectors).

• Mathematical operations can be performed on vectors (e.g., king - man + woman = queen).

• Skip-gram: Predicts context words given a word. +Works well for rare words.

• CBOW: Predicts a word given its context words.+ Faster and works better with smaller datasets.

Negative sampling speeds up training by only updating a small subset of weights instead of the entire vocabulary.

Instead of computing probabilities for all words, it selects a few negative words (random words unlikely to be in the context).

Subword tokenization breaks words into smaller units (e.g., unhappiness → un + happy + ness).

Advantages:

• Helps with rare words

• Allows for better generalization to unseen words

• Reduces vocabulary size while preserving meaning

BPE merges the most frequent character pairs iteratively to form subwords.

Example: t h a t -> t h at -> th at -> that

• Compresses text efficiently.

• Works well for morphologically complex languages.

• Compression efficiency: common sequences are treated as single tokens

• Adaptability: Encoding can optimize for different types of text

• Flexibility: Can handle out-of-vocabulary text

1. Pretraining: Train embeddings on large, general datasets (e.g., Wikipedia, Common Crawl).

2. Fine-tuning: Adapt embeddings to specific tasks (e.g., medical terminology, legal documents).

1. Multiple senses per word -> no contextualization after training

2. Does not preserve word order / context -> cannto create n-grams (Sparsity problem, not enough training data)