Clicked Gallery

What is RLHF (Reinforcement Learning from Human Feedback)?

Highlighted from a real engineering doc. Explained by Clicked.

Used in a sentence

Engineering Notes · AI Systems

After pre-training, the model was aligned using RLHF, with human raters comparing candidate responses.

The reader highlighted one word in the docs. Clicked explained the technical term “RLHF” in simple terms:

Explained in three depths

Same facts, different vibe — Slang mode 😎

The Clicked way

●○○

Overview

RLHF, or reinforcement learning from human feedback, is the training step where people judge a model's answers and the model is adjusted toward the ones they preferred. It is what turns a system that predicts text into one that answers questions helpfully. It is also why models are polite, why they refuse certain requests, and why they sometimes agree with you when you are wrong.
●○○

Overview

RLHF is the stage where people rank a model's answers and it gets nudged toward whatever they liked best. That is the step that drags a text-predicting machine into being something you can actually ask things. It also explains the good manners, the flat refusals, and the way it folds the second you push back. 😎

A quick take — often all you need.

●●○

Detail

A freshly trained model has learned exactly one skill: given some words, guess which words usually come next. That is not the same as being useful. Ask an early one how to fix a flat tyre and it might reply with a list of similar questions, because that is what often follows a question online. Writing rules for what a good answer looks like fails immediately, since nobody can define helpful precisely enough to code it, and more text does not help because the internet holds few examples of an ideal assistant. So the job goes to people. Raters pick the better of two candidate answers, thousands of times, and those choices train a second model to score answers the way they did. The main model is then fine-tuned to earn high scores from that scorer. The catch is that it learns what raters approved of rather than what is true, so it can drift toward flattery, refuse harmless requests that merely resemble risky ones, and inherit the assumptions of whoever did the rating.
●●○

Detail

Straight out of training, a model is a spectacular autocomplete and nothing more. Ask it something and it might cheerfully produce five more questions on the same topic, because online, questions are frequently followed by other questions. You cannot fix that with a rulebook, because try writing down exactly what makes an answer good and you will be there until Thursday with nothing usable. So humans get roped in. Show a rater two answers, let them pick the better one, do that an enormous number of times, and you can train a second model to imitate their taste. Then you tune the real model to please that imitation, which is the whole trick and also the whole problem. It is learning what got approved, not what is correct. So you end up with something that flatters you and panics over harmless requests that merely resemble dangerous ones. 😎

Want more? One click digs deeper.

●●●

Analogy

Teaching someone to cook by having them taste, rather than handing them recipes. You cannot write down what delicious means in a way that survives contact with a real kitchen, so instead they cook, you taste, and you say this one, more salt, less time. After enough rounds they develop judgment no recipe could have given them. But they have learned your palate specifically, so if you happen to love salt, you have trained a cook who oversalts and is quietly certain that is correct.
Deepest level reached.
●●●

Analogy

A comedian working out a set at open mic nights. Nobody can hand them a document explaining what is funny, so they try material, watch the room, and keep whatever lands. Fifty nights later they have instincts a manual could never have taught them. They have also learned exactly what makes that particular crowd laugh, which is why the act can die instantly in a different city.
Deepest level reached.

Unfamiliar concept? A real-world example makes it click — fresh analogies on tap.

AI explanations may contain errors · Not professional advice

Formal definition — The same term, explained the usual way

Reinforcement learning from human feedback is a post-training alignment procedure in which human annotators express preferences between candidate model outputs, those preferences are used to fit a reward model approximating human judgment, and the base model is then optimized against that reward signal. It substantially improves instruction-following and response quality relative to pre-training alone, while introducing failure modes including sycophancy, over-refusal, and the encoding of annotator bias.

Want Clicked to explain terms like “RLHF” directly in your browser — including on PDFs?

Add to Chrome — Free

50 free Explanations · No credit card required