Training refers to iteratively adjusting model parameters using data to fit the mapping relationship from inputs to outputs. Current large models typically employ a two-stage process: the first stage performs pre-training on massive amounts of text with the objective of predicting the next token; the second stage performs post-training using a small amount of high-quality data, shifting the model’s behavior from text continuation to instruction following.

In 2022, Ouyang et al. proposed the classic post-training pipeline in the paper 《Training language models to follow instructions with human feedback》: first, annotators write demonstration answers; then, reward signals are constructed based on human preference rankings of multiple outputs for the same question, and the model is further optimized accordingly. The results showed that the outputs of InstructGPT with 1.3B parameters were preferred by humans over those of GPT-3 with 175B parameters; the former had approximately one percent of the parameters of the latter, while performing better on truthfulness and toxicity metrics. This result demonstrates that post-training can significantly improve model usability without increasing parameter scale. Since then, “pre-training + post-training” has become the standard pipeline for training large models.