Re0-04: TRL GRPOTrainer (Theory)

This English version was machine-translated from the Chinese original. Technical terms may require verification.

Auxiliary code:grpotrainer.py

I'm going to take it from here.

The first three chapters have been completed:

• Chapter I: using native Trainer for SFT, combing the basic concepts of Los Masking
• Chapter II: automated SFT training was achieved using SFTTrainer to simplify processes
• Chapter III: use DPO to align preferences and allow models to learn about humans Prefer

Through the DPO, models can learn:

• How to distinguish between good and bad.
• Generate a response that is more in line with human preferences.
• Improved quality while maintaining stability

But..., original / offline DPO limits are also clear: it consumes offline preferences (chosen vs reprojected). If you have only a valid signal like mathematical correctness, code test results, you will usually have to change the result to a positive one or to an online/valitable incentive.

Introduction to this chapter— an enhanced learning approach that supports the customised reward function. If GRPO algorithms are to be understood in a systematic way, they should be read in the original language; this focuses on the need for GRPTrainer to understand reward signature, group advantage, KL / access to binding and online sampling.

This article is based on how GRPOTaire uses the prompt online sampling to form a group of copies, then calculate group advantage using reward signature and update the policy with the ratio clipping and optional KL/ access constraints.

Learning objectives of this chapter

This chapter covers:

1. Verifiable incentive Concepts and values
2. GRPO Distinction from DPO/PPO
3. Online sampling and Group comparative advantage The principles of the
4. Incentive Function Design of the Convention on the Elimination of All Forms of Discrimination against Women
5. GRPOTrainer Use of ideas (realism)Next）

Note: This is the rationale; see the examples of the use, training monitoring and complete use of GRPO TrainerNext.。

1. Limitations of the DPO and advantages of the GRPO

1.1 scenes that cannot be handled by DPO

See first two typical scenarios: the DPO input is offline, and GRPO can directly consume the results of the online production.

• Math questions.: can verify whether the signal is correct or not. DPO needs pre-formation chosen/rejected, GRPO can use multiple answers to the same prompt sample, and translate correctness directly into reward.
• Code Generation: The signal can be verified as whether the unit test passed. The test results cannot be entered directly into the DPO data format, but you can reward each candidate code and then do group advantage.

DPO questions:

• Could not directly use the test results
• Manual label required chosen/rejected

1.2 GRPO Core Advantages

GRPO = online sampling + customised reward + group comparative advantage

GRPO vs DPO contrast

DPO:

• Data: Offline (prompt, chosen, reprojected)
• Incentives: Invisible (by preference over learning)
• Application: Subjective preference for task

GRPO:

• Data: only prompt
• Incentives: Visible, customizable
• Application: Any verifiable task (metametry, code, reasoning, etc.)

1.3 GRPO application scene

Task Type	Incentive Function Design	Example:
Mathematical reasoning	Correct answer	GSM8K, MATH
Code Generation	Unit pass rate	HumanEval, MBPP
Logical reasoning	The final answer is correct.	ReClor, LogiQA
Factual	External Knowledge Base Validation	TriviaQA
Multiple objectives	Weighted portfolio incentive	Correctability + Efficiency + Readability

2. GRPO Albula

2.1 Core thinking: comparative advantages of clusters

The previous section addressed “Why GRPO”: DPO cannot directly consume online authentication signals. The next question is how GRPO consumes this signal: it does not give a straight-on-one rating, but rather a set of answers to the same prompt sample and puts each one back in the group for comparison.

Traditional PPO issues：

• Need an independent value network (Value Network/Critic) and combine with reward signals to estimate advantages
• Training links are heavier, more sensitive to clitic, KL, rollout and watch

GRPO Innovation：

• Generate for each programpt k each ring Reactions(e.g. k=4)
• Use group internalization / standardization reward to estimate advantages
• No extra value network required

flowchart LR
    P[Prompt] --> G[生成 k 个 responses]
    G --> R[Verifier / reward signal 打分]
    R --> A[组内中心化或标准化]
    A --> U[ratio clipping + 可选 KL/reference 更新]

The same example would make the values clearer. For prompt "Calculate 2+3 =?" sample 4 answers, the average value for reward is 0.125:

Response	reward	advantage = reward - mean	Update Direction
The answer is five.	1.0	0.875	Increase probability
The answer is six.	-0.5	-0.625	Reduce probability
The answer is five.	1.0	0.875	Increase probability
I don't know.	-1.0	-1.125	Reduce probability

2.2 An optional background: mathematical forms and core differences

To understand the innovations of GRPO, we need to compare the mathematical forms of PPOs and GRPOs.

If you only want to use GRPOTaire, you can first capture the three conclusions of this section: GRPO does not need critic/ GAE; the advantage comes from a comparison of the group with the same prompt; the same group is updated several times to understand the old-policy ratio, climping and KL/reference constraints. The formula below is designed to help readers put these conclusions back in the PPO context.

2.2.1 The mathematical form of PPO

PPO Loss Function (full version):

$$ L_{\text{PPO}}(\theta) = \mathbb{E}_t\left[L_t^{CLIP}(\theta) - c_1 L_t^{VF}(\theta) + c_2 S\pi_\theta\right] $$

Of which:

Strategic losses (CLIP losses)： $$ L_t^{CLIP}(\theta) = -\min\left(r_t(\theta)\hat{A}_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_t\right) $$

• $r_t(\theta) = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_{old}}(a_t \mid s_t)}$ ○Softscale sample ratio
• $\hat{A}_t$ ← Empirical Functions Estimates
• $\epsilon$ ← Crop parameters (usually 0.2)

Value Function Loss： $$ L_t^{VF}(\theta) = \left(V_\theta(s_t) - V_t^{\text{target}}\right)^2 $$

• $V_\theta$ • Value network (needs separate training)
• $V_t^{\text{target}}$ Target value

Arsenal Regularization： $$ S\pi_\theta = -\sum_a \pi_\theta(a \mid s_t) \log \pi_\theta(a \mid s_t) $$

PPO Advantage Function Calculation (GAE)：

$$ \hat{A}t^{\text{GAE}}(\gamma, \lambda) = \sum{l=0}^{\infty} (\gamma \lambda)^l \delta_{t+l}^V $$

of which $\delta_t^V = r_t + \gamma V(s_{t+1}) - V(s_t)$ is a trade disability.

Critical observations：

• [Facilitation] Need for trainingValue Network $V_\theta$(Additional parameters)
• [Facilitation] Need** broad advantages estimate (GAE)** to calculate complex
• [Facilitation] DependenceSampling ratio of importance $r_t(\theta)$
• [Facilitation] Need for a cut mechanism to prevent over-up

2.2.2 Methodological form of GRPO

GRPO Loss Function (Intuitive Version)：

$$ L_{\text{GRPO}}(\theta) = -\mathbb{E}{s \sim \mathcal{D}, a_i \sim \pi\theta(\cdot \mid s)}\left[\hat{A}(s, a_i) \log \pi_\theta(a_i \mid s)\right] + \beta \cdot \text{KL}(\pi_\theta \Vert \pi_{\text{ref}}) $$

Of which:

This is written in the policy-gradient instinct to see how rewards can enter the update; actual realization requires a specific trainer. Original DeepSeekMath GRPO uses clipped subside and KL; as of 2026-05-05, TRL GRPOTrainer Default beta=0.0、loss_type="dapo"、scale_rewards="group"、num_iterations=1,Reference/KL, Loss, reward scaling and multiple rounds of climped update should be used by GRPOConfig Yes.

Group Relative Advantage： $$ \hat{A}(s, a_i) = \frac{R(s, a_i) - \frac{1}{k}\sum_{j=1}^{k} R(s, a_j)}{\operatorname{std}({R(s, a_j)}_{j=1}^{k}) + \epsilon} $$ If not, it can be understood that: $$ \hat{A}(s, a_i) = R(s, a_i) - \text{mean}(R(s, \cdot)) $$

• $R(s, a_i)$ ← Reward function or reward signal echo Reactions $a_i$ Ratings
• $k$ Group size (number of responses per programpt)
• Chile: The advantage isRelative to Group Mean; above the same group average is positive and below the same group average is negative

KL Scatter Normalization： $$ \text{KL}(\pi_\theta \Vert \pi_{\text{ref}}) = \sum_{a} \pi_\theta(a \mid s) \log \frac{\pi_\theta(a \mid s)}{\pi_{\text{ref}}(a \mid s)} $$

• $\pi_{\text{ref}}$ ← Reference Policy (Freezing SFT Model)
• $\beta$ KL coefficient
• It's conceptually binding to limit policy not to be far away from reference; in the realization of DeepSeekMath/ TRL, the common estimation is that $\frac{\pi_{\text{ref}}}{\pi_\theta} - \log \frac{\pi_{\text{ref}}}{\pi_\theta} - 1$, not necessarily the full distribution of the visible number.

Critical observations：

• No value network required(savings on visibility and calculation)
• No GAE/ Value Network required(Incentives for direct construction of centralised advantage; std scaling is a common realization option)
• Still possible to use old-policy rateo and shear(The same rollout is particularly important when it is updated several times)
• KL / restraint is one of the stability tools(Concrete concretisation, not as the only mechanism for stabilization)

2.2.3 Core differentiation

The PPO's advantage estimates usually depend on GAE:

$$ \hat{A_t} = \sum_l(\gamma \lambda)^l \delta_{t+l}^V $$

GRPO's advantage comes from the group under the same programt.

$$ \hat{A}(s, a_i) = R(s, a_i) - \text{mean}(R) $$

Common realization also divides the group by std, bringing different prompt reward scales closer.

• Value Network:PPO needs $V_\theta$;GRPO does not need extra value head.
• Sample Policy:PPO using rollout and old-policy radio; GRPO online $k$ Response, it is still possible to use radio / climping when updating the same rollout several times.
• Stability mechanisms: PPO common combination is radio clipping + KL; GRPO can use radio clipping or can be bound by configuration KL / reference.
• Visible occupancy: PPO is a strategic network + value network + reward signal/ reward model; GRPO is a strategic network + reward signal without additional value network.

2.2.4 Why is the comparative advantage of the cluster effective?

Math intuition.：

For each programt $s$, Generate $k$ A response. ${a_1, a_2, ..., a_k}$：

Nature of zero average value： $$ \mathbb{E}{a_i \sim \pi\theta}[\hat{A}(s, a_i)] = \frac{1}{k}\sum_{i=1}^{k} (R(s, a_i) - \text{mean}(R)) = 0 $$

• Naturalization of the Advantage Functions
• Reduced variance and increased training stability

Internalize / Scale：

• The incentive scale for different prompts may vary (easiness, difficulty)
• Internalization and std scaling helps stabilize training
• But it does not guarantee natural comparability across the prompt scale; in TRL scale_rewards Optional group、batch or none, the selection needs to be validated by task and reward distribution

No absolute value.：

• PPO: "What's the value of this response?" Yes, I do. $V(s)$
• GRPO: "How much better is this response than the other responses from the same group?" ♪ Just a group comparison ♪

2.3 Why?"Group relative"？

Traditional RL: Need to estimate absolute value,"How's that for a response?" ♪ Need value networks ♪

GRPO: Just a relative comparison,"Is this better than the other responses from the same team?" → Directly use reward function

Advantages:

• Simple: no additional network required
• Stability: relatively better
• Efficiency: reduced training costs

3. Incentive system in GRPO

3.1 Incentive signals: core components of enhanced learning

In GRPO and modern enhanced learning algorithms, what must be provided is that Reward signalIt's not necessarily a neural network. Reward modelI'm sorry. This is important: DeepSeek-R1-Zero uses anacity reward + format reward and does not use non-eurolal RM; complete DeepSeek-R1 follow-up RL also adds reward model/ reference reward, and the paper also mentions that the reference reversion was introduced only at the last 400 steps to reduce the risk of rewarding.

What's the reward signal?

The role of the reward signal is to:

• Input:prompt + restone (or other information relevant to the task)
• Output: a standard incentive value, which indicates the quality of the response
• Form: A function R(prompt, response, context) → reward

This function can be achieved in many ways:

• Rule function or verifier: e.g. answer matching, unit testing, SQL execution results, database finality;
• LLM / rubric judge: lets strong models be evaluated by rubric or pairwise comparison;
• Neuro-reward models: training a dedicated RM to predict human preferences or mission quality;
• Mixed incentives: combine the verifier, judice, RM, costs and safety constraints.

Key points：

• In GRPO, we optimized the strategy model. $\pi_\theta$
• But it needs a reward signal. $R$ Tell us what's better in the team's response.
• Neural RM is just a realization of the reward signature, not a condition of GRPO

3.2 Two types of training for neuro-reward models

If you choose to use the reward signal as a nerve RM, the reward model can usually be trained in two ways:

Modalities 1: Rules-based training

Apply scene: Tasks with clear and correct answers (e.g., mathematical questions)

Training process：

# 示例：数学题奖励模型训练
training_data = [
    {
        "prompt": "计算 2+2",
        "response": "答案是 4",
        "outcome": +1.0  # 答案正确
    },
    {
        "prompt": "计算 2+2",
        "response": "答案是 5",
        "outcome": -1.0  # 答案错误
    },
]

训练奖励模型：学习预测 outcome
reward_model.train(
    inputs=[(d["prompt"], d["response"]) for d in training_data],
    targets=[d["outcome"] for d in training_data]
)

Rationale：

• Rule functions (e.g. check_answer_correct()) Generate training labels
• Incentive model learning to imitate this rule function
• After training, models can be extended to similar problems.

Advantages：

• [Effects] No manual labelling: rules automatically generate training data
• [Effects] Probable: The answer is objective
• [Effects] Efficient: A large number of training samples can be generated

Limits：

• [Facilitation] Need for clear rules for certification
• [Facilitative] Only judgmental."Right and wrong."Quality assessment difficult

Mode 2: Based on manual labelling training (Preference General)

Apply scene: subjective tasks (e.g. quality of dialogue, writing style)

Training process：

# 示例：人工偏好数据
training_data = [
    {
        "prompt": "写一首关于春天的诗",
        "response_A": "春风拂面暖人心...",
        "response_B": "春天很好...",
        "preference": "A"  # 标注者认为 A 更好
    },
]

训练奖励模型
方法 1: 成对排序损失
loss = -log(sigmoid(r_A - r_B))  # 确保 r_A > r_B
方法 2: Bradley-Terry 模型
P(A > B) = sigmoid(r_A - r_B)

Rationale：

• Manual tabifier compares two responses
• Incentive models to learn to predict human preferences
• After training, model outputs match human judgment.

Advantages：

• [Effects] Capture complex quality dimensions: fluidity, creativity, utility
• [Effects] Broader capacity: various tasks can be addressed

Limits：

• [Facilitation] High cost
• [Facilitative] Subjectivity: Different labelers may not be consistent

3.3 Incentive signal in GRPO: rules vs model

Significant clarifications: In GRPO,"Incentive Functions" This could be:

Type A: pure rule function / verifier

def rule_based_reward(prompt, response):
    """直接使用规则计算奖励，无需训练"""
    if check_correct(response):
        return 1.0
    else:
        return -1.0

• [Advanced] Simple, efficient, suitable for rapid experiments
• [Effects] Recoverable, auditable, suitable for verifiable tasks such as mathematics, code, SQL, tool terminal
• [Less] Coverage depends on the rules themselves, and open missions are easy to literally fit.
• [Careful] It's not a nerve."Incentive model", but it can be fully reward signature of GRPO

Type B: Incentive model for pre-training

# 预先训练好的神经网络
reward_model = load_pretrained_reward_model("path/to/model")

def model_based_reward(prompts, responses):
    """使用奖励模型推理"""
    rewards = []
    for prompt, response in zip(prompts, responses):
        reward = reward_model(prompt, response)  # 模型推理
        rewards.append(reward)
    return rewards

• [Advanced] That's standard."Incentive model"
• [Advanced] Capture complex patterns
• [Facilitative] Need for pre-training
• [Licens] Fixed RM may be used by policy to optimize on-line, reward hacking

Type C: LLM / Rubric Judge

def rubric_judge_reward(prompt, response, rubric):
    """让 LLM judge 按 rubric 输出评分"""
    score = llm_judge.evaluate(
        prompt=prompt,
        response=response,
        rubric=rubric,
    )
    return score

• [Effects] for tasks that are difficult to verify directly, such as open questions and answers, writing, complex angent tracks, etc.
• [Advanced] can mitigate absolute fraction drift by rubric, pairwise comparison, two-way rating
• [Facilitation] Possible systemic deviations such as verbosity bias, position bias, self-preference

Type D: Mixed incentive

def hybrid_reward(rule_score, judge_score, cost, has_hard_violation):
    """先处理红线，再聚合软指标"""
    if has_hard_violation:
        return 0.0
    return 0.7 * rule_score + 0.3 * judge_score - 0.05 * cost

• [Effects] Closer to real industrial systems, which can address correctness, expression, cost and security boundaries simultaneously
• [Facilitation] If all items are weighted linearly, the model may offset key errors with some additional sub-sections
• [Recommendation] Hard numbers should start with veto, then weighted totals.

3.4 DeepSeek/RLVR-style Incentive System

In the DeepSeek-R1-Zero verifiable resonating settings, the GRPO incentive system can be very simple: direct rule reward to assess the correctness of the answer and can be added to the format reward. There is no need to train outcome/process nonural RM, because the core results of the math, code, etc. are already verifiable. Full R1 pageline may still use a mobile-based reward/ treatment reward for more general data; here is the RLVR mainline for a verifiable task.

flowchart TD
    Start[GRPO 训练循环开始]

Step1["策略模型生成响应<br/>策略 π_θ → k 个响应"]

Step2["verifier 评估<br/>answer / test / state → reward"]

Step3["可选格式奖励<br/>format / protocol"]

Step4["组合 reward signal<br/>不一定有神经 RM"]

Step5["计算群组优势<br/>优势 A = r - E r"]

Step6["更新策略模型<br/>π_θ ← π_θ + Δπ_θ"]

Decision{"是否继续<br/>采样训练?"}

UpdateYes["继续生成新 rollout"]

UpdateNo["结束当前训练轮次"]

End[结束当前训练轮次]

Start --> Step1
Step1 --> Step2
Step2 --> Step3
Step3 --> Step4
Step4 --> Step5
Step5 --> Step6
Step6 --> Decision

Decision -->|继续| UpdateYes
Decision -->|停止| UpdateNo

UpdateYes --> Step1
UpdateNo --> End

The core elements of this approach：

1. "reward signatureal from verifeer": Math answer, cell test, SQL execution results, database finality can be given a direct score.
2. Format reward only supportsFormat binding helps model formation to interpret output, but is also the case when weight is too high.
3. Neurological RM is optional, not a GRPO requirement: Open or subjective preference tasks may require RM/judge; a pre-validation of the task directly.
4. The focus of protection against rewarding is on interface design.: Do not place the average score for hard veto errors; do not give all black boxes judge where you can use active outcome.
5. Dynamic update of incentive models: The incentive model is continuously updated during GRPO training and adapts to the strategy model distribution.

3.5 When reward signature is neuro-RM

This section is not a GRPO/RLVR process required. The pre-use rule or the certifier gives a reward signature directly; the reward signal is required to be a nerve RM or LLM judge only if the task is open, subjective or difficult to verify directly.

If you are only trying to read GRPTrainer, you can jump to section 4; the following paragraphs only keep the background of the nerve RM and avoid misreading the "GRPO must train reward models" as the main line.

3.5.1 Construction of training data

Rule-based data generation：

def generate_reward_model_training_data(math_dataset):
    """
    从数学题数据集生成奖励模型训练数据
    """
    training_data = []

for sample in math_dataset:
    prompt = sample["question"]
    ground_truth = sample["answer"]

    # 生成多个响应（包含正确和错误的）
    responses = [
        generate_correct_response(prompt, ground_truth),
        generate_incorrect_responses(prompt, ground_truth, n=3),
    ]

    # 为每个响应生成标签
    for response in responses:
        outcome = check_answer_correct(response, ground_truth)
        training_data.append({
            "prompt": prompt,
            "response": response,
            "reward": 1.0 if outcome else -1.0
        })

return training_data

3.5.2 Incentive model structure

Neuro-reward models usually add headers to the pre-trained language model, allowing them to export a standard incentive, which is fine-tuned with the incentive data collected. The following is a toy-level example to illustrate the structure, which does not represent the achievement of the only or standard RM in LLM RLHF/ RLVR.

import torch
import torch.nn as nn
from transformers import AutoModel

class RewardModel(nn.Module):
    """
    简单的奖励模型实现
    """
    def init(self, base_model_name="bert-base-uncased"):
        super().init()
        self.encoder = AutoModel.from_pretrained(base_model_name)
        hidden_size = self.encoder.config.hidden_size
    # 奖励头：输出单个标量
    self.reward_head = nn.Sequential(
        nn.Linear(hidden_size, hidden_size // 2),
        nn.ReLU(),
        nn.Dropout(0.1),
        nn.Linear(hidden_size // 2, 1)
    )

def forward(self, input_ids, attention_mask):
    # 编码
    outputs = self.encoder(
        input_ids=input_ids,
        attention_mask=attention_mask
    )

    # 使用 [CLS] token 的表示
    cls_embedding = outputs.last_hidden_state[:, 0, :]

    # 计算奖励
    reward = self.reward_head(cls_embedding).squeeze(-1)

    return reward

训练循环
def train_reward_model(model, training_data, epochs=3):
    optimizer = torch.optim.AdamW(model.parameters(), lr=1e-5)
    loss_fn = nn.MSELoss()  # 回归损失
for epoch in range(epochs):
    for batch in training_data:
        prompts = batch["prompt"]
        responses = batch["response"]
        targets = batch["reward"]

        # 前向传播
        rewards = model(prompts, responses)

        # 计算损失
        loss = loss_fn(rewards, targets)

        # 反向传播
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

return model

3.5.3 Assessment of incentive models

def evaluate_reward_model(reward_model, test_dataset):
    """
    评估奖励模型的质量
    """
    correct_rankings = 0
    total = 0

for sample in test_dataset:
    prompt = sample["prompt"]
    better_response = sample["better"]
    worse_response = sample["worse"]

    # 计算奖励
    r_better = reward_model(prompt, better_response)
    r_worse = reward_model(prompt, worse_response)

    # 检查排名是否正确
    if r_better > r_worse:
        correct_rankings += 1
    total += 1

accuracy = correct_rankings / total
print(f"Ranking Accuracy: {accuracy:.2%}")

return accuracy

3.6 Common traps for training in reward models

Trap 1: Reward Hacking

Problem: The model finds a loophole in the reward function, generating an unnatural but highly rewarded text

Example:：

• Incentive function: The answer to which the reward contains a number
• Modelling Institute: Generating a large number of unrelated figures

Solutions：

def robust_reward_function(response):
    reward = 0.0

# 基础奖励：答案正确性
if check_correct(response):
    reward += 0.7

# 辅助奖励：格式规范（但要小心！）
if has_proper_format(response):
    reward += 0.2

# 惩罚：异常模式
if has_repetitive_patterns(response):
    reward -= 0.5  # 严重惩罚

if is_too_short(response):
    reward -= 0.3

return reward

Trap 2: Distributional deviation

Problem: later in the training, resulting in responses that exceed the distribution of training in reward models

Solutions：

# 定期评估奖励模型的校准性，重新微调奖励模型
def check_reward_distribution(reward_model, validation_set):
    rewards = []
    for sample in validation_set:
        r = reward_model(sample["prompt"], sample["response"])
        rewards.append(r)

# 检查奖励分布是否异常
if abs(np.mean(rewards)) > 10:  # 奖励值异常大
    print("Warning: Reward model may be miscalibrated!")
    return False

return True

Trap 3: Overcompatibility of training data

Problem: reward models perform well in training packages, but they are general Bad

Solutions：

• Use data enhancement
• Add regularization
• Periodically evaluated on test set

4. GRPO vs DPO vs PPO

4.1 Three-by-one

The following table is a graphic-spectrum inspiration for the project selection, not a fixed ranking set across the horizon, across hardware and missions; actual visibility, speed and stability will be influenced by the trainer, the Batch Organization, whether or not the reference/RM is loaded, parallel strategies and reward patterns.

Dimensions	DPO	GRPO	PPO
Data requirements	- Offline. - Yeah.	Just prompt	Just prompt
Incentive Functions	Invisible	Visible, customizable	Visible, customizable
Training stability	[High]	[High]	[Chamber]
Achieving complexity	[Low]	[Chamber]	[High]
Visible occupancy	[Low]	[High]	[High]
Apply scene	Subjective preference	Could not close temporary folder: %s	Complex awards
Training speed	Come on.	Medium	Slow

4.2 Selection of recommendations

When to use DPO：

• High-quality preferences for data
• Subjective tasks (style, preference)
• Hope training is steady, fast.

When to use GRPO：

• There are clear verifiable standards (metametric, code)
• Customised incentive function required
• Logical intensive tasks
• Application of DeepSeek-R1 style

When to use PPO：

• It takes complex reward plastics.
• Multi-purpose optimization
• Sufficient engineering resources

5. Incentive functions and uniform paradigm

By this point, the main line of the GRP Trainer doctrine is over: online sampling, reward signature, group advantage, radio/clipping and optional KL/reference. The following is a broader extension of research to help readers understand the direction of the reward design and online RL, rather than the pre-existing knowledge necessary for the next exercise.

Current: artificially designed incentive function

• Knowledge in need
• There may be a loophole.
• Many complex issues are hard to reward by the rules.

Future: AI Supplementary Incentive Design

• Use of strong models (e.g. GPT-4) as incentive models
• Autodiscover incentive function
• Multimodular Incentive (text + image + code execution)
• Self-improvement incentive function

A unified paradigm and insight for enhanced learning and training

Based on the original GRPO paper, we can understand the interlinkages and differences in the various methods of intensive learning training from a more macro-level perspective.

A unified and enhanced learning and training framework

The GRPO paper proposed a unified framework for intensive learning training that incorporated different training methods (SFT, RFT, DPO, PPO, GRPO, etc.) into the same analytical framework.

Core Gradient Formula：

$$ \nabla_\theta J_\pi(\theta) = \mathbb{E}{(q, o) \sim \mathcal{D}} \left[ \frac{1}{\lvert o \rvert} \sum{t=1}^{\lvert o \rvert} GC_\pi(q, o, t, \pi_f) \nabla_\theta \log \pi_\theta(o_t \mid q, o_{<t}) \right] $$

Of which:

• $q$: Query or hint
• $o$:output or response
• $t$Time steps
• $\pi_f$: Reference policy or incentive function for assessing quality
• $GC_\pi$Gradient Coefficent, determining the direction and range of the updated parameters

Three key components：

1. Data Sources $\mathcal{D}$: source of training data
2. Incentive Functions $r_{\pi_f}$: provide training incentive signals
3. Algebra $\mathcal{A}$: processing data and reward signals, generating gradient coefficients $GC$

Classification of different training methods

Based on this unified framework, we can group common training methods into several categories:

Offline Methods

RFT（Rejection Sampling Fine-tuning）：

• SFT-based sample output
• Filter based on correctness of answer
• Just fine-tune the right response.
• Feature: Simple and effective, but unable to use information that responds to a mistake

DPO（Direct Preference Optimization）：

• Use as a pair of preferences to optimize loss
• fine-tune on enhanced outputs
• No need for a visible reward model
• Feature: Stable and efficient, suitable for subjective preference

Online Methods

Online RFT：

• Initializing policy model with SFT model
• fine-tuning using enhanced output from real-time policy models
• Online sampling to explore new responses
• Feature: Early training is comparable to RFT and later is significantly better than RFT

PPO/GRPO：

• Initializing policy model with SFT model
• Enhanced output policy through real-time strategy models
• Use gradient factor for variance update
• Feature: Performance in relevant experimental settings is strong, especially in the case of GRPOs, which achieves a good balance between efficiency and effectiveness

Core Insight: Online vs offline

Differences in training dynamics

Initial phase：

• Actor (tactical model) is similar to SFT
• Small differences in sampling data
• The advantages of online methods are not clear.

Later：

• Actor sample data and SFT data are significantly increasing
• The advantages of online sampling have become obvious.
• Real-time data exploration has brought about more significant performance improvements.

Performance Comparison

In the DeepSeekMath 1.3B GSM8K/MATH Experimental Set, several training developments were reported:

• Online RFT has significantly outperformed RFT in the advanced stages of training.
• GRPO exceeded Online RFT in two baseline tests
• The iterative RL was the most significant increase in the first round.
• GRPO + PS (step gradient factor) is better than GRPO + OS

Key role of the mechanism of the gradient coefficient

Core differences of GRPO vs Online RFT：

Methodology	Gradient coefficient Policy	Characteristics
Online RFT	Unite and strengthen all sound. Reactions	No punishment for wrong response, no difference.
GRPO	Adjustments based on incentive value dynamics	Increased differential/punishment by response magnitude

Critical Insight：

Importance of dynamic gradient factor：

• GRPO adjusts the gradient factor based on the incentive function or the incentive signal
• Achieved"Increased differentiation/punishment": higher quality response obtained a higher positive gradient
• Low-quality response received negative gradient (punishment)

Gains from fine particle gradient design：

• GRPO + PS (step perception, Step-aware) performance is better than GRPO + OS
• Description of the finely designed gradient factor that can lead to additional performance improvements

Value of iterative training：

• RL continuously enhances performance in two-way iteratives
• The first round of the upswing was the most significant.
• Multiple-wheeled iteratives can sustain cumulative gains

Data source classification: online sampling vs offline sampling

Online Sampling：

• Training data from research results from real-time training strategy models
• Dynamic generation, continuous improvement
• Advantages: to explore new and better responses
• Method of representation: Online RFT, PPO, GRPO

Offline Sampling：

• Training data from pre-collected data sets
• Static data, fixed and unchanged
• Strengths: training is stable, easy to control
• Representational method: SFT, RFT, DPO

Practical recommendations

Based on the above insights, in practical application:

Select the scene of the online method：

• The mission has clear verifiable criteria (e.g., mathematics, code)
• Need for continuous exploration and improvement
• Sufficient computing resources to support online sampling
• I'm looking for higher performance.

Select the scene for the offline method：

• High-quality pre-data collection
• Subjective preference (style, tone)
• Limited computing resources
• Need for rapid iterative and stabilization training

Mixed Policy：

• Initial development of basic capacity using offline (SFT/DPO)
• Targeted optimization using the online method (GRPO) at a later stage
• The iterative RL can accumulate up and down.

Role of the unified framework

The unified framework has several main uses:

1. Theory Understanding: Help us understand the structural differences of different approaches
2. Method Selection: Guidance on the selection of appropriate training methods for specific tasks
3. Algorithm Design: directs the design and optimization of the new algorithm
4. Performance Analysis: explain why some methods are more effective in a given context

This framework allows for a clearer comparison of the use of several techniques for enhancing learning in language model fine-tuning and facilitates subsequent new methods of interpretation in the same set of coordinates.

Summary of this chapter

This paper completes the rationale for GRPOTaire: online sampling, reward signature, group advantage, radio clipping and optional KL / access. Next one will be in. GRPOTrainer fieldRun through the code, monitor and debug.

References

• TRL Document: GRPO Trainer
• GRPO Paper: DeepSeekMath
• DeepSeek-R1: rule-based reward with GRPO
• PPO Paper: Legal Policy Optimization
• DPO Paper: Directing Practice Optimization: Your Language Mode is Secretly a Reward Mode
• TRL