Fine-Tuning — From Base Model to Assistant

What does a base model actually do when you prompt it? In What Is a Large Language Model?, these models are trained on a single objective: predict the next token. They learn grammar, facts, reasoning, and even code — all as a side effect of that pressure. But nothing in the training objective says “be helpful” or “answer questions.”

The difference between a base model and the ChatGPT or Claude you interact with daily is post-training — a set of techniques applied after pre-training that reshape the model’s behavior without fundamentally altering its knowledge.

SFT is the simplest and most foundational step: take your pre-trained model and train it further on a curated dataset of (instruction, desired response) pairs. The loss function is the same cross-entropy we covered in Pre-Training — but now the training data looks very different.

Instead of raw internet text, SFT data consists of conversations. Each example has a system prompt (setting the model’s persona), a user message (the instruction), and an assistant response (the ideal output). The model learns to predict the tokens of the assistant response — while the loss on prompt tokens is masked so the model is not rewarded for predicting the instruction itself (Meta, 2024).

The limitation of SFT is that it trains on a single “gold” response per prompt. But in practice, there are many ways to answer a question — some more helpful, more accurate, or more concise than others. How do you teach a model that subtle quality gradient?

Reinforcement Learning from Human Feedback (RLHF) answers this with a three-step process that was first demonstrated at scale in the InstructGPT paper (Ouyang et al., 2022).

Step 1: Collect preference data. Human labelers are shown two or more model responses to the same prompt and asked to rank them. “Response A is better than Response B.” This is fundamentally easier than writing ideal responses — comparing is cognitively cheaper than creating.

Step 2: Train a reward model. A separate neural network (often a smaller LLM) learns to predict the human preference scores. Given any (prompt, response) pair, the reward model outputs a scalar score representing “how good a human would rate this.”

Step 3: Optimize with PPO. The LLM generates responses, the reward model scores them, and Proximal Policy Optimization (PPO) adjusts the LLM’s weights to maximize the reward. A KL-divergence penalty prevents the model from drifting too far from the SFT baseline — without this anchor, the model can “hack” the reward model by producing degenerate outputs.

In 2023, Rafailov et al. asked a sharp question: if the reward model is just learning a function of the LLM’s own probabilities, why not skip it entirely?

The answer was Direct Preference Optimization (DPO) — a method that uses the same preference data as RLHF but optimizes the LLM directly, without a separate reward model and without reinforcement learning (Rafailov et al., NeurIPS 2023).

The key insight: there is a closed-form mapping between the optimal reward function and the optimal policy. DPO exploits this by treating the LLM itself as an implicit reward model. The training objective becomes a simple classification loss: given a pair of responses (one preferred, one rejected), increase the probability of the preferred response and decrease the probability of the rejected one.

The impact was immediate. Meta adopted DPO as a core component of the Llama 3 / 3.1 post-training pipeline — six rounds total, with fresh preference annotations each time. During DPO, only preference pairs where the chosen response was rated “significantly better” or “better” were used; “similar” pairs were discarded (Meta, 2024).

In January 2025, DeepSeek published a result that shifted the trajectory of the field. DeepSeek-R1-Zero — trained with pure reinforcement learning and no supervised fine-tuning — spontaneously developed chain-of-thought reasoning, self-verification, and reflection behaviors (DeepSeek-R1, Jan 2025; Nature, 2025).

The algorithm behind it was GRPO — Group Relative Policy Optimization. GRPO is conceptually simple: for each prompt, generate a group of responses. Score each response with a verifiable reward — did the math answer match the ground truth? Did the code compile and pass tests? Then compute advantages relative to the group rather than using a learned critic model.

This is the critical difference from PPO. In PPO, a separate critic network estimates the value of each state. In GRPO, there is no critic — the group itself serves as the baseline. If your response scored higher than the group average, your advantage is positive; if lower, negative. This cuts memory requirements by roughly 50% compared to PPO (arXiv, Mar 2025).

Every technique discussed so far has a practical problem: full fine-tuning means updating every parameter in the model. For a 70-billion-parameter model, that requires north of 100 GB of VRAM just for the weights, optimizer states, and gradients — the domain of multiple A100 or H100 GPUs.

LoRA (Low-Rank Adaptation) changed this equation entirely. The core insight, published by Hu et al. in 2021, is that the weight updates during fine-tuning have a low “intrinsic rank.” You do not need to change every dimension of every weight matrix — a small, low-rank approximation captures the behavioral shift.

In practice: freeze the entire pre-trained model. For each target weight matrix W (of size d × k), inject two small matrices: A (d × r) and B (r × k), where r is the rank — typically 4 to 64, compared to original dimensions of 4,096 to 12,288. The effective weight becomes W + AB. Only A and B are trained.

You now understand the full toolkit. The harder question is: should you fine-tune at all?

The answer depends on what you are trying to fix. There are three levers for customizing LLM behavior, and they form a natural escalation path:

Prompt engineering is your first move. It requires no training data, no GPUs, and no model modifications. If you can get the behavior you need by crafting better prompts — with few-shot examples, system instructions, or structured output formats — stop here.

Retrieval-Augmented Generation (RAG) is your second move. If the model lacks specific, current, or proprietary knowledge, don’t fine-tune it in — retrieve it at inference time. As we covered in RAG From the Ground Up, RAG keeps the knowledge layer separate from the model’s behavior layer.

Fine-tuning is your third move — and only when the first two are insufficient. Fine-tuning excels at changing behavior, not injecting knowledge. Use it when you need:

• A consistent output format across thousands of requests (always return valid JSON with specific fields)
• Domain-specific tone or persona (legal writing, medical communication, customer support)
• Reliable policy adherence (always cite sources, never reveal system prompts, refuse specific categories)
• Latency reduction (encode instructions into weights so you need shorter prompts)