02. Foundations of Text Classification

• Rule-Based – Uses handcrafted linguistic rules.

• Supervised Machine Learning – Trains a model using labeled data.

• Unsupervised Learning – Uses clustering or topic modeling without labels.

+ Precision can be high, human-readable rules

- Very expensive to build and maintain

Pros: More accurate, easier to maintain than rule-based systems.

Cons: Requires labeled training data.

Limitation: Assumption that indivisual cases are disconnected and independent

-> Hidden Markov Models

• Each token in a sequence is assigned a label

• Labels of tokens are dependent on the labels of other tokens in the sequence, particularly their neighbors

• Example: POS

• 
  

• Not easy to integrate information from category of tokens on both sides

• Difficult to propagate uncertainty between decisions and “collectively” determine the most likely joint assignment of categories to all of the tokens in a sequence

• Fixed set of states

• State transition probabilities

• Fixed set of possible outputs

• For each state: a distribution of probabilities for every possible output -> Emission probabilities

Transition Probabilities

Emission Probabilities

Viterbi algorithm

The Viterbi algorithm efficiently finds the most likely sequence of hidden states by:

• Computing emission and transition probabilities.

• Backpropagating to reconstruct the best sequence.

• Reducing complexity to O(sm) compared to O(ms²).

1. One-Hot Encoding – Sparse vector representation.

2. TF-IDF – Weighs words based on importance.

3. Word Embeddings (Word2Vec, GloVe) – Dense, semantic representations.