Reinforcement Learning in LLM Alignment: From Reward Signals to Advantage Estimation

For a long time, we talked about enhanced learning in LLM alignment, often in algorithmic names: PPO, DPO, RLOO, GRPO, REINFORCE++. It is certainly convenient to remember this, but it is also easy to lose a comprehensive understanding of the field, as if the progress in this area is mainly about changing abbreviations, changing a loss, changing a training formula, increasing points (and not necessarily really rising), pub, and then going to the next. The strategies are perfect, but it seems that outsiders don't know what they're up to.

This article defaults that readers already know SFT, preference data and classic RLHF processes. If you need to see the training first, then you can go back to the training, LoRA and the visual estimates.Post-Language Training and Micro-Adjusting Practice; here is the special tracking of how reward becomes an optimised signal.

It'll be clearer from another angle.LLM is constantly changing in Quri, the way to reward signal entry to the optimizer. A primitive reward, how it was bound, distributed, reduced, consolidated and eventually transformed into a stable and updated strategy, is the main disagreement of these approaches.

Once we have this main line, the way we cut each other apart can be linked:

PPO provides early LLM RLHF project templates with high impact reward-to-advantage;
DPO bypassing the online RL to write preferences directly into the optimization target;
GRPO and REINFORCE++ re-examine:No critic after, baseline and numaration should come from.

Here is the conceptual framework for how to optimizer consumption; if you care about TRL GRPOTrainer The code-level principle and training interface that allows comparison Re0-04: TRL GRPO Trainer (principle)。

Plugin: Why is it not the RL in LLM that has no reward, but rather whether it can be a reliable training signal?

Reward in traditional intensive learning, at least in many classical environments, is relatively clear: game scores, goal hits, collisions, success in mission. But in LLM-System, reward is not usually the same.

First of all, it's often...SlightYeah. In many scenarios, the reward model gives you a total score after the model produces a complete section of the answer, or the certifier tells you the answer.Learning to Summarize from Human Feedback and InstructGPT This type of classic RLHF process is largely addressing this final feedback.

Second, it's often...DelayYeah. The value of a single answer token of the dozens that lie ahead is probably only judged after the entire answer has been completed. One of the difficulties of the strategy gradient is to redistribute the late signal back to the process of generation.Williams 1992 / Policy Gradient Theorem

And third, it's often...Noisey.I'm sorry. The reward model is not a truth judge, it's just a sorrogate. It can learn from some patterns of human preference, or from others. OpenAI has long observed in the wrap-up mission that if strategies are over-optimised, the end result is a departure from the real human preference.

Fourth, it could be.UtilizationI'm sorry. Models may not really be more helpful for open missions, but they may be better at following reward models. And that's why many RLHF articles introduce mechanisms like KL constraint, length control, reference models: not because they look more elegant, but because without them, rewards are too easy to hack.

So, the RL in the LLM is never just a reward, but:

This is a training signal that can be processed into a low differential, but not a perfect, not easily expluited, and can keep the scale stable across the prompt.

TLDR

Write the judgment I want to put at the top of the line.

LLM-RL evolution is constantly rewrite reward -> advantage This chain. It is not the algorithm that changes, but who is better at translating raw feedback into stable estimates of strengths.PPO / Back to Basics / REINFORCE++
Classic RLHF reward, not a nudity score from the beginning, but a composite signal of "mission reward plus behavioral restraint". Learning to Summarize from Human Feedback / InstructGPT
The clitic-free method does not eliminate the problem of advantage estability, but simply transfers it from the clitic network to the reward processing and statistical construction. You don't train value head, but you have to answer: where is baseline from, how do you measure it, how do you control the local noise?Back to Basics / DeepSeekMath / GRPO
The key innovation of GRPO is not only the use of critić, but also the adaptation of learning signals to group comparative advantage. This is very effective in the reasoning task, but it also exposes the problem: what about the small difference in local standards? Is better in the group equal to better in the whole world?DeepSeekMath / GRPO
REINFORCE++ deserves separate attention because it brings the issue of metrics to the global level. It is no longer merely a matter of baseline, but rather: whether the scale of advantage should be defined in the prompt or in the whole of the bat.REINFORCE++
If you read the new RL alignment paper later, the first thing to ask is not what it is called PO, but five questions: where did you get it from, where did you put it, where did you get it, where you did it, what was the ultimate optimization. This is more useful than the acronym of the back calculation.

Dissociate the concept first: reward, return, baseline, advantage, KL, normalization

If you do not open these words first, you will almost certainly be confused.

Reward: Raw feedback, not necessarily suitable for training

The most original source of feedback for the mission is reward. In LLM paired up, it could come from:

Scores of reward models;
(a) The determination of the correctness of the rule certifier;
Step points of the process reward model;
Evaluation of LLM as Judge;
Signs of success/failure in the return of the external environment.

But from the history of RlHF,Reward is not usually used directly. It also needs to be combined with mechanisms such as KL restraint, length punishment, and cessation of punishment to become a signal of excellence. While the quality of reward itself is critical, the processing of signals at the RL algorithm level is equally a major investment to make rewards available for training.

Return: Rewarding the future and the return

The strategy gradient is not just about the immediate reward, but about whether the current action will bring the rewards behind it. The most simple cumulative return is:

$$ G_t = \sum_{t' = t}^{T} \gamma^{t' - t} r_{t'} $$

If there's only a final reward for an LLM mission, this formula means:Although the reward was given at the end, the front token still gets the training signal through the cumulative return.

The idea is simple, but it is the basis for retrofitting the thin incentive to a token level signal for optimization. And then, with the return, the next step is to update the parameters with it -- The simplest thing is that REINFORCE is going to be: $G_t$ Multiply the gradient of log probability, high return token increases the probability, low return pressure lower the probability. But this rough distribution also leads to a lack of quality of the signal itself (the whole trajectory is randomly stacked), a wide range of differences, and each step later — baseline, advantage, nonmalization — is compensating for this rough signal.

Baseline: not an additional incentive rule, but a reference to the reduction margin Yes

A number of beginners understand the baseline as another incentive function, but more precisely it is more like a statistical reference system. You're not asking how much you scored this time, but how much you're asking how much higher this time than usual.

Greensmith et al. 2004 This is clearly illustrated: the main function of Baseline is to do control variate, which is the drop margin, not to change the gradient expectations. The simple reward is of limited significance, and the update is on advantage, and the baseline is on it.

Advantage: The real newer is not usually the reward itself

Once the baseline is introduced, the trainers are really interested in not just nudity but in advantages:

$$ A_t = G_t - b_t $$

In introducing the policy gradient method for baseline / advantage, the updated signal is usually not a nudity score, but rather an advantage over the reference level.

In the PPO era, the most common method of estimation is that of the old vantage. GAE（Generalized Advantage Estimation）I'm sorry. The idea is to see only one step of the TD error equation but rely heavily on clitic. The Monte Carlo is no bias but the difference is big. GAE goes through one parameter. $\lambda$ Inserting values between the two, and balancing them.

KL: It's not an attachment, it's a behavioural boundary.

Another particularly easy-to-penetrating target in RLF is KL. Many people would understand it as a simple addition to the regularizer, but from the early RLHF work, KL's role is closer to the behavioral boundary:

I'm gonna tell you where you want to go.
KL tells you not to deviate from the reference model too far.

Without this layer of boundary, reward model can be easily over-optimised. In the LLM scenario, it may be behavioural drift, preference for degradation, speculation in format or length, degradation of capacity, or, more typically, rewarding. If understood from an optimised perspective, it can be seen as a strategy that moves away from the high-quality areas of behaviour represented by reference mode; but it is not the only mechanism, nor is it necessary to attribute all problems to catastrophic oblivion or to a lost base.

It's not KL that needs to be watched here, it's KL, the super-parameter itself.Optimizing pressureI'm sorry. The same reward signature may be reliable in light reranking or short-wheeling preferences; once you enter the line RL, multi-spectrum sample, low KL or high weight rewarding, the border of this signal is detected more frequently. DPO, rejection sampling, online RL, use different ways of rewarding, but they may all increase the selective pressure on proxy signal defects; the real risk is that strong online optimization continues to target a blind area as a signal.

So the point of rewarding is not that "rewarded is wrong" but:The policy explores beyond the definition and generalization of the reward signature, but the optimizer continues to use this signature as a target. This is first KL/ Optimised Risk Volkage; more systematic uncomfort to Title III. Training monitoring, KL, reward distribution, human rating or validation indicators, and length of behaviour, should be seen together; average reward is easy to miscalculate as an improvement in capability.

It determines who the model compares with.

Normaration looks like a project detail, but it determines the scale of comparison. Because it determines the scale at which the above- and below-average are defined:

Is it more than average for a sample of the same prompt?
Or is all the whole of the bats above average?

This difference, the center of the GRPO divide with REINFORCE++.

And the next general map is the first one I think I've ever been able to build.

Link	What's it answering?	Common objects	Representative practice	Common failure patterns
original	What's the better answer?	R.M., Verifier, Environmental Feedback	Awarding model scores, rule validation, process awards	Reward noise, mislabelling, non-prescriptive.
Binding Item	How far away can we go to the reference strategy?	KL, length constraints, end of punishment	KL Penalty, length/cut-off penalty	Rewarding, shifting style
credit assignment	How do we get the final signal back to the front?	return、token reward	Cumulative return, end end reward refill	Scatter, delay, long-range noise
baseline / advantage	How much higher than the baseline this time?	critic、group mean、leave-one-out	GAE、leave-one-out、group mean	High variance, baseline distortion
normalization	Where do you define the margin of advantage?	local group、global batch	group std、batch std、z-score	Local explosions, cross-prompt.
policy update	How to hold each parameter up to date	actor / old policy / reference policy	PPO clip、KL loss、critic-free update	Tactical shakes, training unstable.

If these layers are separated, much of the debate that follows will be clear. To be honest, many paper-based rewards, the real ones are the fourth and fifth lines, and this should be the algorithm itself.

We'll use a line to tie the whole chain of signals:

raw reward → +KL约束 → return(分配到token) → -baseline → advantage → normalize → ×∇log π → 更新参数

The difference between each algorithm in the end can be compared to the line by which it makes a different choice.

First generation answer: how the PPO RLHF spell reward model, critic and KL as a training system

PPO It was not originally designed for LLM, but it gave RLHF a particularly important project template:

An estimate of the size of the chemical;
Do not over-renew with clipping control tactics... Specifically, the PPO is usually used $\epsilon \approx 0.2$ (b) Crop the probability ratio of old and new strategies, remove incentives to continue moving in a more favourable direction, and prevent single updates from taking too many steps;
Provides a stable reference to training with old policy/reference policy.

Here we are. Learning to Summarize from Human Feedback and InstructGPT This RLHF route, reward Model and PPO are beginning to really integrate. A good enough mind model is:

$$ R(x, y) = r_\phi(x, y) - \beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\mathrm{ref}}(y \mid x)} $$

It does not have to be understood as a uniform formula that is used word for word in all the papers, but it captures the spirit of the classic RLHF:

Part I $r_\phi(x, y)$ (b) Expressing reward models or mission feedback;
The second part shows the cost of deviating from the reference strategy.

This is a crucial step because it shows thatClassic RLHF from the beginning making composite reward designsI'm sorry. And not all of them, KL, and not all of them, together, constitute a real goal of optimization.

The role of clitic here is also important. It is not telling the model what is true, but it is estimating as steadily as possible the advantage so that training does not spread out because the reward noise is too loud. This is the first generation answer for the PPO RLHF:

Provide direction with reward Model, boundaries with KL, low differential advantage estimates with critic, and a PPO update mechanism to stabilize the system.

The program is strong, but it is not light:

Structure heavy - under LLM scenario, clitic usually shares backbone with an actor plus a value head, plus reference policy and reward model, training with the need to maintain a network of four large models at the same time, with almost doubling of the amount of presence and calculation;
When critic is bad, it becomes a new source of error;
The training formula is complex (learning rate, GAE parameters, clipping range, KL coefficients).

And it's because of these problems that the back of the critic-free route is back on the rise.

Sideline: DPO why is not the main line of the article

DPO A key relationship was found: the most effective strategy is closed to hidden rewards, with KL constraints, so that it can be directly optimized in classification on preferences, bypassing online rollout, critic and advantage estimates. It makes a lot of people realize for the first time thatAlignment does not have to be done online through RLs.I'm sorry. But it's because it's bypassed. reward -> advantage This chain, rather than continuing to rewriting it, is used here as a cross-reference. If you're concerned about the online RL itself, how to handle the reward signal, the main line is still PPO, GRPO and REINFORCE++.

clitic-free main line: comparative advantage within the group from the REINFORCE prototype to GRPO

Once you remove the critic, the problem doesn't disappear, just reposition it on the table. The new issue becomes:

No, no, no, no, no. Where's the baseline from?
No GAE. What do you think?
No critic to smooth your scale, nomalization.

REINFORCE: Minimum prototype, directly passing the results back to the trajectory

Williams 1992 The ReINFORCE is the starting point for all of this. Its most simple new instinct is that the high return trajectory rises logprob and the low return trajectory is down.

$$ \nabla_\theta J(\theta) \approx \sum_t G_t \nabla_\theta \log \pi_\theta(a_t \mid s_t) $$

On LLM, this can be understood in a visual sense as the token option that makes up a whole paragraph that achieves a high final score should be more easily sampled again in a similar context.

REINFORCE's value is not that it can still be used today to train large models, but that it makes the point clear:Strategic learning is the reverse attribution of results to the whole sample trajectory.

But it also immediately exposed the oldest problem: the square difference is too wide, the fraction itself has noise, and the allocation token will introduce more noise later; training will often be difficult to stabilize if the baseline is not reduced, but rather if absolute scores are directly influenced by the output probability. The simple baseline REINFORCE is rarely directly used for modern LLM alignment, but the REINFORCE-style critic-free method is re-enacted and continues to be adapted on this very prototype of the idea.

GRPO: Push local relativization to group average + group standard Bad

Back to Basics This kind of work re-places the REINFORCE style approach back to the LLM RLHF discussion center, and reminds us that the critic-free is not going to baseline, but instead moving baseline from value head to sample statistics. For example, a multiple sample of the same question gives the average of the other answers under the same prompt as the reference system. GRPO continues the road: it not only subtracts the group average but divides it by group deviation.

DeepSeekMath / GRPO The typical way to sample the same prompt $k$ One answer, a set of rewards. $r_{1:k}$and then construct:

$$ A_i = \frac{r_i - \mathrm{mean}(r_{1:k})}{\mathrm{std}(r_{1:k}) + \epsilon} $$

The idea behind this is intuitive:For the same topic, what is worth learning is not absolute scores, but advantages relative to the level within the group. If you take it apart, GRPO has done three things at the same time:

Group means baseline, which no longer compares with zero points or global averages, but rather with the average of candidates for the same subject;
The group std is local networking, the margin of advantage is determined by this group of candidates of its own;
Multiple sampling of the same subject defines comparative space, and the training signal first answers “Who among these candidates is more worthy of increasing probability”.

This is the most critical shift in the critic-free phase:Less training on a critic does not mean that the advantage improvement is gone, but it is being pushed into the statistical structure of the Rollout group. You no longer need a value head, and you have reduced a possible training estimate; the cost is more reliance on the quality of the same subject multiple samples, more reliance on the verifier / reward model to create differences within the group and more reliance on the group statistics themselves to be sufficiently stable.

This is effective in the case of tasks such as mathematical reasoning, which are so diverse and clearly sequenced as to the same subject. And that's why GRPO is fast becoming an influential representational method in the reasing scene. It is particularly appropriate for training settings that can generate multiple candidates, reuser certifications or incentive models for the same subject matter.

But if you look at the reward-signal perspective, the GRPO also exposes the problem of local relativization:

The group is too hourly, the average and standard deviations are per se unstable, and the advantage is heavily dependent on the candidate for this sample.
If the reward in the group is almost the same, group std will get the signal right. $\epsilon$ / Clipping is very sensitive; in some settings close to zero signals, in others the noise may be magnified.
This prompt is better than the candidate, not necessarily equal to "better distribution of the data as a whole".

I'd rather understand these questions as the historical value of GRPO: it's not the end of the problem, it's the end of it. local normalization This thing pushed into the middle of the stage. The subsequent REINFORCE++, which follows this line, continues to follow.

REINFORCE++: From local Normalization Debate to Global Scale

If the key to GRPO is local relativization, then REINFORCE++ The key is that it is:It's not just the baseline itself, it's a measure of partial integration.

In the author ' s narrative, there are at least three risks to prompt-level regulation such as GRPO: small local groups, uneven averages and standard differences; molecular and denominators from the same group, statistically linked; models learn more like winning themselves in the group than necessarily forming a global scale across the prompt.

So REINFORCE++ gives a response not to replace a new model, but to pull the range of the local group from the global watch:

$$ A^{\text{norm}} = \frac{A - \mathrm{mean}{\text{batch}}(A)}{\mathrm{std}{\text{batch}}(A) + \epsilon} $$

This step seems to be just a change of location, but it recast the training signal for a comparator:

In GRPO, the model is mainly a comparison with other candidates on the same subject;
In REINFORCE++, the model is compared with the distribution of the advantage of the whole bat.

The normal REINFORCE++ and REINFORCE++-Baceline in the paper can be understood in this framework: the former places greater emphasis on the integration of the same topic into the "advantage" at the catch scale, even without multiple samples; the latter retains the average value impairment of the group, allowing the same issue to continue to be taken together, but moves the std from the local group to the catch level, avoiding using the same group to decide both "who is higher" and "how much higher".

What is more worth remembering here is not terminology, but three dimensions of decision-making:

- Where did you get that?
Where does that happen?
KL is either blending into advantage, or alone into loss.

And that's exactly the value of REINFORCE++ in this main line: it's not adding a complex branch, but it's continuing the core debate of the clitic-free debate from “baseline where” to “where does the scale define”?

End to end: how a signal from prompt was converted from reward to advantage

A number of concepts have been removed, and the chain is being followed by a specific numerical example.

Suppose there's a prompt:"One sentence explains quantum tangles."We took it three times, and the reward model scored it:

Answer!	RRM Score $r_i$
Answer A.	8.2
Answer B.	6.5
Answer C.	9.1

Return/Return: In the traditional RL intuition, if only the final reward, you can put $G_t = \gamma^{T-t} \cdot r_T$ Different steps are allocated. But in the common LLM RLHF / GRPO realization, it is more common to convert the security-level response to the same command advantage and broadcast it to the updated version of the protocol tokens, with token-level KL or radio constraints. The concept of return is retained here to explain "how the result is attributed in reverse to the trajectory" and not that all LLM realizations are allocated on a discount basis.

Baseline (average / leave-one-out style): When looking at answer A, a reference system can take the average of the remaining answers, baseline = $(6.5 + 9.1) / 2 = 7.8$I'm sorry. If written as GRPO Group Mean, the average of three answers is used directly $7.93$ As this prompt's baseline. The details are different, but in common: baseline is no longer from critic, but rather from the same subject matter sample.

Advantage: If written in leave-one-out, answer A $A = 8.2 - 7.8 = +0.4$(lightly better than reference level for the same subject; small increase probability); answer B $A = 6.5 - 8.65 = -2.15$(Specificly different from other samples, press down probability). If GRPO is used, the average of the group is then reduced and then the standard deviation within the group is standardized.

Normalization (GRPO style vs REINFORCE++ style)：

GRPO: With these three answers to their average and standard differences z-score,$A_i^{\text{norm}} = (r_i - 7.93) / 1.07$I'm sorry. This is the local scale of the prompt, which is not stable when there are only 3 samples in the group.
REINFORCE++: Calculate the average and standard deviation of this three responses with the whole bat (e.g. 256 prompt x 3 = 768 answers) and the average and standard deviations, with a more stable scale.

Policy Update: Finally, REINFORCE-type update multiplied by $\nabla_\theta \log \pi_\theta$— The probability of a token answering A is slightly increased (advantage positive), the probability of a token answering B is significantly lowered (advantage negative), and the answer is C obtains the greatest positive update.

That's the whole thing. reward → return → baseline → advantage → normalization → policy update Chain. All the differences in the algorithms that follow can be returned to this example, and see where they make different choices.

Read the paper checklist: which five interfaces should be asked first

If only one reusable frame is left at the end of the article, this section is probably the case.

See the new RL alignment algorithm later, so don't hurry up and remember what it's called. The following five questions are first asked:

Problem	What does it really decide?	Typical answer.
Original where did you come from?	What are you perfecting?	Incentive models, rule-certifiers, process rewards, environmental feedback
KL writes "reward" or "single"?	Whether or not to bind directly to the advantage construction	Classic RLHF for the former, GRPO / REINFORCE++ for some variants
- Where's the baseline from?	How do we lower the difference?	critic、moving average、leave-one-out、group mean
What range does that have to be?	Who are the models comparing themselves to?	local group、global batch、running statistics
What was the final update?	What is the object that is really being optimized?	sequence reward、cumulative return、normalized advantage

These five issues can almost reorder all mainstream approaches:

PPO RLHF: critic + KL + PPO update, reward mode signal;
DPO: Just bypass the online advantage assessment;
GRPO: group mean baseline + group std, defining signals as local comparative advantage;
REINFORCE++: Move the integration scale to the catch/ global level.

Ultimately, the discussion is not about complete rewarding, but rather about rewarding signal shaping / rewarding consummation: how to be restrained, compared, normalized and interpreted before and after the reward signal enters the optimist. It includes at least:

How the original feedback was processed through a range of statistics and optimizations was eventually interpreted as a powerful signal that could be trained in strategy.

In other words, the main line is... reward-to-advantage design: nominally rewardAnd actually is improving the RL algorithm itself.

Summary

If LLM alignment is seen as a series of algorithms, then what happens over the years is like changing abbreviations; but if it is seen as a signal chain from reward to advantage, many things suddenly become consistent.

The PPO RLHF addresses how to combine reward Mode, critic and KL into a system template that can be used on a large scale;
DPO tells us that there are some problems that can be bypassed from the online RL;
GRPO Note: After no critic, baseline and numaration are transferred to the group for statistical purposes;
REINFORCE++ further narrows the issue to a specific debate:The measure of advantage should be defined locally or globally.

This line of study can be summed up in one sentence:

At least on the RL algorithm line, the focus of the improvement is not on constantly creating new incentives, but on how to be explained before and after rewrite the reward signal into the optimizer.

If the chain is sufficiently mature, the subsequent bottlenecks will shift further towards rollout selection, sample efficiency and system ingestion. The training system problems of the PODS are placed in the appendix; the main line goes back to the next step: how did the reward actually produce itself?

Appendix: PODS and the side support of the training system - not all rollout is worth updating

PODS It's the side of the training-system, not the main line in the next chapter of this series. It's interesting, not to rewrite, not to rewrite, but to ask a question closer to the training system itself:Which rollout is worth sending in an expensive parameter update. On the online RL, generating rollout tends to be easier to create in parallel, and the real cost is the reverse transmission, cross-calorized and optimised state maintenance, so the problem starts with"How do you define the signal?"Turn"Which samples are worth consuming the updated budget"。

It is also a simple idea: Mr. Rollout becomes more, sifts less informative samples, and only gives more training-worthy subsets to the original GRPO/PPO target. It is not just who is the top, but who is better able to pull the training signal; the max-variance down-sampling highlighted in the paper prefers to retain a portion of both high and low incentive samples instead of a top-k-only reservation.

If you return to the main line of this article, the PODS is not about the chain itself, but rather how the bottlenecks continue to shift to rollout selection, sample efficiency and system ingestion, when the chain is mature enough. The main line remains how the signal is explained, and the PODS discussion is about what samples are worth updating the model after the explanation has been completed.