13. LLM - Prompting, Alignment, Instruction Tuning

In-Context Learning (ICL) enables a model to perform a task by conditioning on input-output examples without updating weights.

Key Differences from Fine-Tuning:

• No weight updates → The model remains unchanged.

• Works immediately without extra training.

• Practically useful → Avoids costly data labeling and fine-tuning complexity.

• Intellectual intriguing -> Genralization from few examples as test for “intelligent behaviour”

ICL avoids costly data labeling & fine-tuning:

1. Labelling data is costly

   • Requires domain expertise.

   • Emergent, time-sensitive scenarios.

2. Fine-tuning is challenging

   • Training is sensitive to hyperparameters.

   • Not enough validation data.

   • Expensive in memory and time.

   • We dont quite understand how fine-tuning works

Key biases:

• Encoding Sensitivity → The format of prompts affects performance (e.g., stars, sentiment labels).

• Majority Label Bias → Frequent training answers dominate predictions.

• Recency Bias → Examples near the end of the prompt influence predictions more.

Types of reasoning problems:

1. Arithmetic Reasoning

2. Symbolic Reasoning (Letters)

3. Commonsense Reasoning

Approaches to enhance reasoning:

• Train with 100x more data.

• Increase model size for better knowledge retention.

Specific techniques:

• Chain of Thought (CoT) → Step-by-step problem solving.

• Few-Shot CoT → Step-by-step demonstration and response.

• Zero-Shot CoT → Two-stage prompting + step-by-step answer.

• Self-Consistency → Majority voting for answer selection.

Problem: There is a mismatch between pre-training and user intent.

Solution: Alignment → Adjusting models to ensure they do what we want.

Goal: Make the pre-trained model follow human instructions reliably.

Instruction-Tuning fine-tunes LLMs on a collection of datasets that involve mapping language instructions to their corresponding desirable generations.

Steps:

1. Collect example (instruction, output) pairs across multiple tasks.

2. Fine-tune the model on these examples.

3. Test on unseen tasks to check generalization.

Frequent tasks in instruction-tuning:

• Sentiment classification

• Text summarization

• Question answering

• Machine translation

• Textual entailment

Challenges:

• Expensive → Requires ground-truth data for tasks.

• Hard for open-ended tasks → No clear right answer for creative outputs.

• Prone to hallucinations → Generates incorrect but confident responses.

Advantages:

• Improves zero-shot performance.

• Scaling up instruction data improves accuracy.

• Minor costs compared to pretraining

Limitations:

• Hard to collect diverse labeled data.

• Limited creativity → Generates token-by-token, not holistic ideas.

• Encourages hallucinations → Model does not verify its outputs.