14. LLM - Long Context, RAG

Reinforcement Learning (RL) involves an agent interacting with an environment by taking actions.

Key components:

• State → Representation of the world at a moment.

• Action → Decision made by the agent.

• Reward → Feedback from the environment.

• Policy function → Defines which action the agent takes in a given state.

RLHF optimizes language models by using human preferences to guide training.

Reward function R(s, prompt) [s: output]

Reward is higher when humans prefere the output

Process:

1. Estimate reward function R(s; prompt)

Find the best generative model p that maximizes the expected reward

Two main approaches:

1. Get humas to provide absolute scores for each output -> Human judgement on different instances / by different people can be noisy or mis-calibrated

2. Ask for pairwise comparisons -> Can be more relibale

3. Scaling Reward Model: Large enough reward trained on large enough data approaching human performance

1. Collect a dataset of human preferences ->Human annotators rank outputs based on preferability

2. Use this data to train a reward model -> Reward model returns a scalar reward which should numerically represent the human preference

3. We want to learn a policy (a Language Model) that optimizes against the reward model

4. Periodically train the reward model with more samples and human feedback

5. -> PROBLEM: System will learn to “cheat”, be producing gibberish / irrelevant outputs -> SOLUTION: Add penalty term penalizing deviations from distribution of pretrained LM

Challenges:

• The model may learn to "cheat" by generating gibberish outputs that maximize reward scores.

• Difficult to train good reward models.

• Requires a lot of human annotations.

• 
  

Reward models can enforce desired behaviors, such as:

• Avioiding bias.

• Avoiding toxic outputs.

• Staying within the model’s knowledge scope.

• Tricky to get right

• Training a good reward may require a lot of annotations

1. Scale up the model and train it on longer context window size -> Bottleneck: Memory usage and number of operations in self-attention increase quadratically

2. Sparse Attention Patters -> Improve efficiency: Make attention operations sparse

Sparse Attention reduces the number of attention computations by focusing only on selected tokens.

Implementation strategies:

• Different layers and attention heads follow different sparsity patterns.

• Earlier layers with sparser attention.

RAG combines LLMs with external data to improve responses.

Why is it useful?

• LLMs cannot memorize all knowledge.

• LLM knowledge can be outdated and hard to update.

• LLM output is challenging to interpret and verify

• LLMs are expensive to train; reducing their size while retrieving information is more efficient.

Key design decisions in RAG:

1. Memories: What should be stored as memory? (Documents, databases, etc.).

2. Memory retrieval method: How should retrieval work? (Pretrained retriever or off-the-shelf search).

3. Retriever memories usage: How should retrieved information be used? (Fusion with prompt or model input).

Two major failure modes:

1. Underutilization → The model ignores retrieved memories.

2. Overreliance → The model depends too much on retrieved information.

Limitations of storing all knowledge in LLMs:

• Cannot store long-tail knowledge efficiently.

• Difficult to update (LLM retraining is expensive).

• Hard to verify generated facts.