SyntaxPilot
Courses/AI: Zero to Agents/Lesson 17

AI: Zero to Agents · Lesson 17 of 40 · 11 min

From raw model to assistant: fine-tuning and RLHF

queued

These are the notes for AI episode 17. Last episode you learned the knobs that control what a model writes. But a model fresh out of training is not an assistant: ask it a question and it may ask you another one. Here you invent the steps that turn a base model into a helpful assistant: fine-tuning, supervised fine-tuning (SFT) with a chat template, human preferences, a reward model, reinforcement learning from human feedback (RLHF), reward hacking and the KL penalty, and the simpler DPO. Every Python example runs as-is in the SyntaxPilot compiler; outputs are real (Python 3.13).

Real-life example: the bookworm at the enquiry counter

At the railway counter In an LLM
Someone who has read every book in a huge library a base model after pretraining on huge amounts of text
A passenger asks "When does the Rajdhani leave?" a user's prompt
They read out the next line of the timetable: "When does the Shatabdi leave?" the base model continues the text instead of answering
On-the-job training: how to answer, politely fine-tuning and RLHF

Attempt: ask a base model a question

A tiny language model like the one from episode 15: it learns which word follows each pair of words. Its training text is a page of quiz questions, plus one answer.

import random
from collections import defaultdict
random.seed(5)
web = """Q: what is the capital of india ? Q: what is the capital of france ?
Q: what is the largest planet ? Q: how do i make chai ? Q: what is
the capital of japan ? the capital of india is delhi ."""
model = defaultdict(list)             # pretraining: last 2 words -> next
w = web.split()
for i in range(len(w) - 2):
    model[(w[i], w[i + 1])].append(w[i + 2])

out = "what is the capital of india ?".split()
for _ in range(10):                   # generate: predict, append, repeat
    out.append(random.choice(model.get(tuple(out[-2:]), ["<end>"])))
    if out[-1] == "<end>": break
print(" ".join(out))

Output:

what is the capital of india ? Q: how do i make chai ? Q: how do

It didn't answer: it asked another question. Learning from a big pile of text is pretraining, and the result is a base model. A base model only learned to continue text the way its training text continues, and on a quiz page a question is followed by another question.

This happens with big models too. In OpenAI's 2022 InstructGPT paper (Ouyang et al., arXiv 2203.02155), GPT-3 was given a French instruction to write a short story about a frog, and replied with more instructions instead of the story. The paper opens: "Making language models bigger does not inherently make them better at following a user's intent." Getting a model to do what people actually want is called alignment.

Fine-tuning and supervised fine-tuning (SFT)

  • Start from the pretrained base model (it already knows a lot).
  • Keep training it, the same way, on a small new dataset: that is fine-tuning.
  • The dataset is demonstrations: a prompt plus the ideal answer, written by a person.
  • Mark who is speaking with a fixed pattern (User: ... Assistant: ...): a chat template.
  • Learning from demonstrations is supervised fine-tuning (SFT): "supervised" because the right answer is given.
import random
from collections import defaultdict
random.seed(5)
web = """Q: what is the capital of india ? Q: what is the capital of france ?
Q: what is the largest planet ? Q: how do i make chai ? Q: what is"""
demos = """User: what is the capital of india ? Assistant: delhi . <end>
User: how do i make chai ? Assistant: boil tea leaves with milk . <end>"""
model = defaultdict(list)
for text in [web, demos]:             # pretrain, then fine-tune
    w = text.split()
    for i in range(len(w) - 2):
        model[(w[i], w[i + 1])].append(w[i + 2])

for q in ["how do i make chai ?", "what is the capital of india ?"]:
    out = ("User: " + q + " Assistant:").split()
    while out[-1] != "<end>" and len(out) < 30:
        out.append(random.choice(model.get(tuple(out[-2:]), ["<end>"])))
    print(q, "->", " ".join(out[len(q.split()) + 2:]))

Output:

how do i make chai ? -> boil tea leaves with milk . <end>
what is the capital of india ? -> delhi . <end>

It answers now, and stops at the end token <end>. Fine-tuning taught the format and the job (when you see Assistant:, answer, then stop), not much new knowledge.

Honest note: our toy only looks at the last 2 words, so after ? Assistant: it can't tell the two questions apart; with random.seed(5) both happened to land on the right answer. A real transformer reads the whole prompt with attention. Practice question 1 shows the fix for the toy.

Why demonstrations are not enough

Problem Why it matters
Slow and costly to write a person writes every ideal answer (InstructGPT used about 13k prompts for SFT)
One answer per prompt the model copies it, but never learns which answers are worse, or why
Judging is easier than doing at a cooking contest the judges taste two dishes; they don't need to be chefs

So: show people two answers and ask which is better. Those picks are human preferences.

The reward model

from collections import defaultdict
# people compared two replies to "how do i make chai ?" and picked one
prefs = [("boil tea leaves with milk and sugar .", "figure it out yourself ."),
         ("sure ! boil water , add tea and milk .", "chai is a drink ."),
         ("add ginger , boil with milk and tea .", "no .")]
weight = defaultdict(float)           # the reward model: a score per word
def reward(reply):
    return sum(weight[w] for w in reply.split())

for _ in range(5):                    # learn: better reply must score higher
    for better, worse in prefs:
        if reward(better) <= reward(worse):
            for w in better.split(): weight[w] += 0.1
            for w in worse.split(): weight[w] -= 0.1
for reply in ["boil milk and tea leaves .", "figure it out .", "chai ."]:
    print(f"{reward(reply):5.1f}  {reply}")

Output:

  0.5  boil milk and tea leaves .
 -0.3  figure it out .
  0.0  chai .

A model that gives any reply a score, trained from comparisons, is a reward model. It scored three replies it had never seen in a sensible order. Now we have an automatic judge that can score millions of replies without a person.

Reinforcement learning from human feedback (RLHF)

import math, random
random.seed(1)
replies = ["boil tea leaves with milk .", "figure it out .", "chai ."]
reward = [0.5, -0.3, 0.0]             # from our reward model
score = [0.0, 0.0, 0.0]               # the model's preference for each reply

def probs():
    e = [math.exp(s) for s in score]
    return [round(x / sum(e), 2) for x in e]

print("before:", probs())
for step in range(300):               # reinforcement learning
    i = random.choices(range(3), weights=probs())[0]   # model answers
    score[i] += 0.5 * reward[i]                        # reward nudges it
print("after: ", probs())

Output:

before: [0.33, 0.33, 0.33]
after:  [1.0, 0.0, 0.0]
  • Reinforcement learning (RL): learn by trying and getting a reward, instead of being shown the right answer.
  • Policy: the model being trained by RL (it decides what to say).
  • RLHF: RL where the reward comes from a reward model trained on human preferences.

How InstructGPT was trained (Ouyang et al., 2022)

GPT-3 (base) ──► 1. SFT ──► 2. Reward model ──► 3. RL with PPO ──► InstructGPT
                 written      ranked answers      reward model
                 answers                          as the judge
Step Data What it learns
1. Supervised fine-tuning about 13k prompts with answers written by labelers the job and the format
2. Reward model 33k prompts, labelers ranked several model answers to score answers like people do
3. Reinforcement learning (PPO, proximal policy optimization) 31k prompts to write answers the reward model scores highly
  • About 40 contractors (labelers) wrote and ranked the answers.
  • People preferred outputs of the 1.3B-parameter InstructGPT over the 175B GPT-3, a model over 100 times bigger.
  • ChatGPT (OpenAI, 30 November 2022) was described as "a sibling model to InstructGPT".

Reward hacking

from collections import defaultdict
prefs = [("boil tea leaves with milk and sugar .", "figure it out yourself ."),
         ("sure ! boil water , add tea and milk .", "chai is a drink ."),
         ("add ginger , boil with milk and tea .", "no .")]
weight = defaultdict(float)
def reward(reply):
    return sum(weight[w] for w in reply.split())
for _ in range(5):
    for better, worse in prefs:
        if reward(better) <= reward(worse):
            for w in better.split(): weight[w] += 0.1
            for w in worse.split(): weight[w] -= 0.1

for reply in ["boil tea leaves with milk and sugar .",
              "milk milk milk milk milk milk milk milk"]:
    print(f"{reward(reply):5.1f}  {reply}")

Output:

  0.7  boil tea leaves with milk and sugar .
  0.8  milk milk milk milk milk milk milk milk

Nonsense beats the real recipe. The reward model is only a rough copy of what people like: it learned "milk" is a good word, so more milk means more points. RL hunts for the highest score and finds this hole: reward hacking.

The KL penalty: a kite on a string

Flying a kite In RLHF
The wind pulls it higher the reward pulls toward higher scores
The string keeps it near your hand the KL penalty: a cost for drifting from the fine-tuned model
It flies high but never flies away better replies that stay sensible
import math
reward = [0.7, 0.8]                   # recipe vs spam: spam wins!
ref = [0.9, 0.1]                      # the fine-tuned model before RL
def rlhf(beta, steps=3000):
    score = [math.log(x) for x in ref]
    for _ in range(steps):
        e = [math.exp(s) for s in score]
        p = [x / sum(e) for x in e]
        # reward, minus a penalty for drifting away from the old model
        r = [reward[j] - beta * math.log(p[j] / ref[j]) for j in range(2)]
        avg = sum(p[j] * r[j] for j in range(2))
        for j in range(2):
            score[j] += 1.0 * p[j] * (r[j] - avg)   # nudge toward better
    return [round(x, 2) for x in p]
print("no leash:  ", rlhf(beta=0.0))
print("with leash:", rlhf(beta=0.5))

Output:

no leash:   [0.0, 1.0]
with leash: [0.88, 0.12]

log(p / ref) measures how far the model has drifted from the reference model; on average it is the KL divergence, a measure of how different two probability distributions are. beta is the strength of the leash. With no leash the model always says "milk milk milk"; with beta = 0.5 it keeps the recipe 88% of the time.

DPO: skip the separate judge

RLHF DPO (direct preference optimization)
Stages SFT, reward model, RL SFT, then one training stage on the preferences
Separate reward model yes no
RL loop (sampling during training) yes no
Data pairs: answer A is better than answer B the same pairs

DPO was proposed by Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning and Chelsea Finn (arXiv 2305.18290, May 2023): "Your Language Model is Secretly a Reward Model".

Glossary

Term Meaning Example from this episode
Pretraining training on a huge pile of text to predict the next token the quiz-page text
Base model the model right after pretraining; it continues text asks another question back
Alignment making a model do what people actually want answering, politely
Fine-tuning continuing to train an already trained model on new data training on the demonstrations
Demonstration a prompt plus an ideal answer written by a person "how do i make chai ?" → "boil tea leaves with milk ."
Supervised fine-tuning (SFT) fine-tuning on demonstrations (the right answer is given) step 1 of InstructGPT
Chat template a fixed pattern marking who speaks User: ... Assistant: ...
End token a special token that means "stop" <end>
Human preference a person's pick of the better of two answers recipe beats "figure it out yourself"
Labeler a person hired to write or rank answers about 40 for InstructGPT
Reward model a model that scores replies, trained on preferences our score-per-word model
Reinforcement learning (RL) learning by trying and getting a reward the 300-step loop
Policy the model being trained by RL our three-reply model
RLHF reinforcement learning from human feedback SFT → reward model → RL
PPO proximal policy optimization, the RL algorithm InstructGPT used step 3
Reward hacking the policy gaming the reward model "milk milk milk" scores 0.8
KL penalty a cost for drifting from the starting model (beta sets its strength) the kite string
KL divergence a measure of how different two probability distributions are average of log(p / ref)
DPO direct preference optimization: train on preference pairs directly no separate reward model

Common mistakes

Code Exact error Why Fix
random.choice(model[("hello", "there")]) with a defaultdict(list) IndexError: Cannot choose from an empty sequence that word pair never appeared in training model.get(key, ["<end>"]): stop on unseen text
math.log(p / ref) with p = 0 ValueError: math domain error the log of zero is undefined never let a probability hit exactly 0
math.exp(1000) OverflowError: math range error scores grew too large: too big a step use a smaller step size

Mini project: a tiny instruction-following bot

Write 5 demonstrations in the chat template (User: ... Assistant: ... <end>) about your city (famous food, a monument, the best time to visit). Fine-tune the 4-word model from practice question 1 on them and check every question gets its own answer. Then write 3 preference pairs and score 3 new replies with the reward model.

Practice

  1. Add a third demonstration ("what is the largest planet ?" → "jupiter .") to the SFT demo. What goes wrong with the 2-word model, and how do you fix it?
  2. Add your own preference pair to the reward model and score a reply it has never seen.
  3. In the leash demo, try beta = 0.1 and beta = 2.0. What changes?
  4. Find a reply that scores even higher than "milk milk milk" on the reward model.
Solutions

1. With three demonstrations, ? Assistant: is followed by three different answers, so the 2-word model mixes them up. With random.seed(5) it answered:

how do i make chai ? -> jupiter . <end>
what is the capital of india ? -> jupiter . <end>
what is the largest planet ? -> delhi . <end>

Look at the last 4 words instead:

import random
from collections import defaultdict
random.seed(5)
web = """Q: what is the capital of india ? Q: what is the capital of france ?
Q: what is the largest planet ? Q: how do i make chai ? Q: what is"""
demos = """User: what is the capital of india ? Assistant: delhi . <end>
User: how do i make chai ? Assistant: boil tea leaves with milk . <end>
User: what is the largest planet ? Assistant: jupiter . <end>"""
model = defaultdict(list)
for text in [web, demos]:             # pretrain, then fine-tune
    w = text.split()
    for i in range(len(w) - 4):        # now: last 4 words -> next
        model[tuple(w[i:i + 4])].append(w[i + 4])

for q in ["how do i make chai ?", "what is the capital of india ?",
          "what is the largest planet ?"]:
    out = ("User: " + q + " Assistant:").split()
    while out[-1] != "<end>" and len(out) < 30:
        out.append(random.choice(model.get(tuple(out[-4:]), ["<end>"])))
    print(q, "->", " ".join(out[len(q.split()) + 2:]))
how do i make chai ? -> boil tea leaves with milk . <end>
what is the capital of india ? -> delhi . <end>
what is the largest planet ? -> jupiter . <end>

More context lets the model tell the questions apart, which is exactly why transformers look at the whole prompt.

2. Add ("boil milk , add tea and cardamom .", "google it .") to prefs and score new replies:

for reply in ["boil milk and tea leaves .", "google it .", "add cardamom ."]:
    print(f"{reward(reply):5.1f}  {reply}")
  0.5  boil milk and tea leaves .
 -0.1  google it .
  0.0  add cardamom .

"google it ." is now judged a poor reply.

3.

beta 0.1: [0.77, 0.23]
beta 2.0: [0.9, 0.1]

A weak leash (0.1) lets spam grow to 23%; a strong leash (2.0) keeps the model exactly where it started (0.9, 0.1), so it also can't improve. beta is a trade-off.

4. Repeat every positive word:

for reply in ["boil tea leaves with milk and sugar .",
              "milk milk milk milk milk milk milk milk",
              "boil boil boil tea tea tea milk milk milk"]:
    print(f"{reward(reply):5.1f}  {reply}")
  0.7  boil tea leaves with milk and sugar .
  0.8  milk milk milk milk milk milk milk milk
  0.9  boil boil boil tea tea tea milk milk milk

The score keeps rising with nonsense, which is reward hacking in action.

Quick quiz 1

People see two answers and click the better one. Which step uses this data? A: pretraining · B: writing demonstrations · C: training the reward model · D: the context window

Answer

C: training the reward model. Pretraining uses raw text; SFT uses written ideal answers.

Quick quiz 2

After RL, a model starts every reply with "Great question! Great question! Great question!". What went wrong? A: reward hacking · B: too little pretraining · C: a chat template bug · D: top-p too low

Answer

A: reward hacking. The reward model likes that phrase, so RL found a way to game it. A stronger KL leash and better preference data help.

Interview questions

  1. Base model vs instruct/chat model? A base model is only pretrained and continues text; an instruct model is fine-tuned (SFT, then RLHF or DPO) to follow instructions in a chat template.
  2. What are the three stages of RLHF in InstructGPT? Supervised fine-tuning on demonstrations, a reward model trained on human rankings, and RL (PPO) against the reward model.
  3. Why use preferences instead of only demonstrations? Comparing is cheaper and easier than writing ideal answers, and comparisons teach what makes one answer better than another.
  4. What is reward hacking? The policy finds outputs that score high on the (imperfect) reward model without being good.
  5. Why is there a KL penalty in RLHF? It keeps the policy close to the SFT model, which limits reward hacking and keeps the language fluent.
  6. How is DPO different from RLHF? DPO trains directly on preference pairs in one stage, with no separate reward model and no RL sampling loop.
  7. Does fine-tuning add knowledge? Mostly it changes behaviour and format; most knowledge comes from pretraining.

Summary

  • Pretraining gives a base model that only continues text.
  • Fine-tuning keeps training it; SFT uses demonstrations in a chat template.
  • Demonstrations are costly and show one answer each, so people compare answers: human preferences.
  • A reward model learns to score replies; reinforcement learning pushes the policy toward high reward: RLHF (InstructGPT: SFT → RM → PPO).
  • The policy can game the judge (reward hacking), so a KL penalty keeps it near where it started.
  • DPO does the same job in one stage, from the same preference pairs.
  • Next episode: multimodal AI: how models see images and hear audio.