05. IR - Simple Neural Techniques

CNNs use filters with a sliding window that merge values in filter region into an output value to detect local patterns.

Key Features:

• Filters parameters are learn from data during training.

• Multiple filters are applied in parallel.

• 1D CNNs are used for text processing.

1. CNNs focus on local patterns with filters and work well for text classification.

2. RNNs process sequences recursively, handling context-dependent tasks like translation.

1. Translation

2. Question answering

3. Summarization

4. Chatbots

5. Email auto-response

In sequence modeling, an RNN encodes an input sequence into a single vector representing the meaning of the input.

1. Vanishing gradients → Hard to learn long-range dependencies.

2. Exploding gradients → Large weight updates can destabilize training.

3. Slow training compared to CNNs and Transformers.

• Allow for arbitrarily sized inputs (Model global patterns in seuquence data)

• Recursion gets unrolled to from a standard computation graph -> Trainable with backpropagation & gradient descent

• Allows for conditional generation models (Ability to condition the next output on an entire sentence history)

The Encoder-Decoder model processes sequence input & output for different tasks.

Features:

• Encoder processes the input into a vector.

• Decoder generates the output based on the encoded information.

Attention helps the model focus on relevant parts of the input while making predictions.

Benefits:

• Improves accuracy in long-sequence tasks.

• Provides some interpretability by highlighting important words.

1. Computes a weighted average of input.

2. Uses softmax to assign importance scores.

3. Attention is parameterized and trained e2e with the model

   
   

The fully connected layer:

• Contains trainable parameters.

• Applies non-linear activation.

• Helps combine weighted input features for the final output.