06. IR-Transformer Architecture

The Transformer is a neural network architecture that processes sequences without recurrence by using self-attention mechanisms.

Advantages:

• More efficient than RNNs since it doesn’t process data sequentially.

• Better at capturing long-range dependencies in text.

• Foundation for state-of-the-art NLP models (BERT, GPT)

Self-attention allows each word to attend to every other word in a sequence, capturing relationships regardless of distance.

Key Properties:

• Helps with contextualization (understanding word meaning based on surrounding context).

• Enables parallel processing (unlike RNNs, which process data sequentially).

Self-attention has complexity of O(n^2) due to the pairwise word interactions.

-> This makes processing long sequences expensive, requiring high memory and computing power.

Multi-head self-attention allows the model to focus on different parts of a sentence simultaneously.

Benefits:

• Helps capture multiple relationships between words.

• Improves contextual understanding.

1. Self-attention layers → Capture word relationships.

2. Feedforward layers → Process attended features.

3. Layer normalization → Stabilizes training.

4. Positional encoding → Adds order information to words.

Transformer:

• Uses self-attention instead of recurrence.

• Processes entire sequences in parallel.

• More scalable for large datasets.

RNNs:

• Process sequences step by step (slower).

• Struggle with long-range dependencies due to vanishing gradients.

1. [CLS] → Classification token (used to represent entire sequences).

2. [MASK] → Used in masked language modeling (MLM) for training.

3. [SEP] → Used to indicate a next sentence.

1. Masked Language Model (MLM) – Predicts masked words in a sentence.

2. Next Sentence Prediction (NSP) – Determines if one sentence follows another.

After pretraining, BERT is fine-tuned for specific NLP tasks.

SPLADE combines BERT’s semantic understanding with sparse, interpretable representations.

-> It generates sparse vectors by activating only relevant terms from a vocabulary.

• Uses BERT embeddings to encode each input word

• Expands dense vectors into mix of tokens

• Enforece sparsity of result vectors through activation function -> Keeping only the most relevant terms

• Regularization to further control the sparsity

-> This makes SPLADE more explainable and efficient than fully dense retrieval models.