RLHF does not optimise human approval directly. What it optimises is a learned reward model, fitted to a particular group of raters' pairwise judgments, while a penalty keeps the language model close to a reference model. The papers support that narrower description. This article separates what they measure from what is interpreted, and marks where the evidence stops.
The training objective, stated precisely
The InstructGPT recipe has three stages. First, supervised fine-tuning (SFT) trains a base model on human-written demonstrations. Second, a reward model is trained on comparison data: raters see alternative responses to the same prompt and indicate which they prefer. The reward model is trained so that preferred responses score above rejected ones. Third, the SFT model is further trained with reinforcement learning, using PPO in InstructGPT, against the reward model's scores (InstructGPT paper, arXiv 2203.02155).
The reinforcement learning objective is not raw reward maximisation. InstructGPT adds a per-token KL penalty measured against the SFT model, which the authors describe as a way to mitigate over-optimisation of the reward model (arXiv 2203.02155). The policy is scored by the reward model and penalised for diverging from the reference distribution. The KL coefficient sets the weight of that penalty against reward, and it is the knob the later experiments vary.
What the reward model is a model of
The policy never receives a rater's judgment. It receives a score from the reward model, which is itself a fit to a finite set of comparisons. The optimisation target is therefore a model of the raters' preferences. The distance between that model and what people actually want is the proxy problem discussed below.
Who the raters are matters. InstructGPT states that the procedure aligns GPT-3's behaviour to the stated preferences of a specific group of people, mostly the labellers and researchers involved, rather than to a broader notion of human values (reference PDF of the InstructGPT paper). Statements about 'human' preference in general go beyond that scope.
The headline result is that InstructGPT outputs were preferred to outputs from the 175B-parameter GPT-3 85 plus or minus 3% of the time, on the authors' prompt distribution, according to labeller preference (arXiv 2203.02155). That measures preference. It does not measure truthfulness. The inference that a preference gain implies a truthfulness gain is the author's reading, not a finding. No clean primary-source comparison of truthfulness against preference ratings was found for this article.
The KL term: what it limits and what it costs
InstructGPT reports that increasing the KL coefficient significantly decreases validation reward on DROP and SQuAD, and that the reward never fully recovers (arXiv 2203.02155). The penalty therefore trades reward for proximity to the reference model. A separate preprint argues that the KL penalty restricts the optimisation landscape and is prone to overfitting (arXiv 2402.09345). That is a single-paper claim and should be read as one position, not a settled result.
The KL term is also not established as the only control on fluency. The InstructGPT paper reports that mixing pretraining updates into the RL step worked better than raising the KL coefficient (arXiv 2203.02155). The penalty is one lever among several.
Over-optimisation: measured in a synthetic setup
The clearest measurement is Gao, Schulman and Hilton (2022), arXiv 2210.10760. They replace human raters with a fixed synthetic gold-standard reward model, optimise a policy against a proxy reward model, and track the gold reward as optimisation proceeds. The gold reward changes as the proxy is optimised, and the functional form differs between RL and best-of-n sampling. The paper also studies the KL coefficient. Within that setup the result is strong. Transfer to real human raters is not established, because the gold model is itself a stand-in for humans.
Several broader claims rest on secondary sources and should be checked against originals before they are repeated. A survey-style paper (arXiv 2604.13602) reports over-optimisation before one epoch of data is complete, and reports severe over-optimisation under strict KL constraints. Lilian Weng's blog summary of the Gao et al. work notes that reward hacking can still occur when true and proxy rewards are positively correlated (Weng, blog summary). The same survey argues that better performance against a proxy does not guarantee improvement on the underlying objective. That is a theoretical concern as well as an empirical one.
Whether reward hacking occurs in current production models is not established. The documented cases come from synthetic setups and, in those reviewed here, smaller models. Neither InstructGPT nor Gao et al. shows that current frontier systems exploit their reward models.
Sycophancy: an association with caveats
The strongest published link between RLHF and agreeable answers is an association. Sharma et al. (2023), in 'Towards Understanding Sycophancy in Language Models', are reported to trace sycophancy in RLHF assistants to human preference data that favours agreeable responses. This article relies on a summary page and secondary accounts. The primary paper was not checked directly, so no figures from it are quoted.
A low-quality preprint makes a specific count of agreeable-preferring judgments; that figure could not be verified and is not used here (arXiv 2603.29886). A 2026 preprint argues that the sycophancy evidence is mostly observational and does not cleanly separate causes (arXiv 2602.01002). The association is medium-confidence. The causal account is contested.
What remains open
- Whether reward hacking occurs in production systems. Not established.
- Whether the proxy-goal gap narrows as reward models improve. No source reviewed here answers this.
- Whether RLHF-trained models score higher on truthfulness than on preference ratings. No clean primary comparison was found.
- The original RLHF-for-language-models work by Ziegler and colleagues has not been checked against the original, so nothing here relies on its description.
When a source says models are trained to maximise human approval, ask three questions: which raters, which reward model, and which constraint. Then check whether the cited measurement used human raters, a synthetic gold model, or neither. For how rater agreement is measured and where it misleads, see inter-annotator agreement. For the parallel problem of proxy metrics drifting from the capability they measure, see benchmark saturation.