C 1.5 - Recent Deelopments LLM

• combines several natural language processing methods in single service (speecth synthesis, speech recognitoin, …)

• cannot engage in sophisticated dialogues

• only limited to set of user commands

• mainly a language-based user interface to web services

• based on transformer architecture

  • -> specifically designes for dialouge generation (i.e. chatbots)

• incorporates metrics to improve quality of conversaiton

  • sensibleness

  • specifity

  • interestingness

  • helpfulness

• training include dialogues that were evaluated by humans according to these mstrics

• system can additionally access external data sources to answer questions

  • e.g. calculator, web search

• i.e. GPT-4 is multmodal

• -> can process both text and images…

• language models scale well with their number of parameters

• => rapid growth of SotA language models…

1. training of a generative model

   • self-supervised training alleviates need for manual labeling

2. new neural network architecture for text called transformer

   • equivalent of CNNs for text…

3. new training procedure for learning dialogues

   • optimization of model for human preferences

4. massively large scaling of the model

   • improving performance with data and computing power

• discriminative:

  • classify data

• generative:

  • generate new data (i.e. iamges, text, audio, etc.)

=> ChatGPT is generative for natural language

• T9 - text input method for phones

• -> instead of pressing digit key multiple times -> press once

• -> use interlan dictionary to determine words matchgin sequence of pressed keys…!

• at begining -> frequency of use

• frequencies adapted over time according to user preferences

• learn statistical relations between words

• => words are in general characterized by their context (i.e. word2vec…)

• GPT: uses mechanism that is called self-attention to learn contextual relationships between words…

• similar to CNN

• -> convolutions capture locality of features

• self-attention captures semantic relationships between words in texts

=>

• deep CNN learn increasingly abstract image features

• -> transformer learn increasingly abstract concepts

• pre trained for general text prediction

• -> to answer questions dialogue-based

• => needs training with data specific to that task…

• 3 steps

• Step 1:

  • collect demonstration data and train a supervised policy

• Step 2:

  • collect comparison data and train a reward model

• Step 3:

  • optimize a policy against the reward model using the PPO reinforcement learning algorithm

• sample prompt from prompt dataset

  • i.e. “explain reinfocement learnin got a 6 year old”

• labeler demonstrated the desired output behavior

  • i.e. labeles writes answer

  • e.g. “we give treats and punishments to teach…”

• this data is used to fine-tune GPT with supervised learning

• promt and several model outputs are sampled

  • i.e. prompt: “explain reinforcement learning to a 6 year old

  • model 1 answer: i.e. in reinfocement learning, the agent is…

  • model 2 answer: i.e. explain rewards…

  • model 3: we give treats and punishemnets to teach…

• a labeler (human) ranks the outputs from best to worst

  • i.e. model 3 > model 1 > model 3…

• data used to train reward model

• are statistical models

• => can only approximate exact computation to some extent (ie.e. arithmetic, algorithms, rules,….)

• new prompt is sampeled from dataset

  • i.e. write a story about otters…

• PPO model is initialized from the supervised policy

• policy generates an output

• reward model calculates a reward for the output

• reward is used to update the policy using PPO

=> train reward model on some samples

=> use policy to train on more models…