Re0-05: TRL GRPOTrainer (Practice)

This English version was machine-translated from the Chinese original. Technical terms may require verification.

Auxiliary code:grpotrainer.py

The questions in this article can also be addressedRe0-04: TRL GRPO Trainer (principle)、Re0-02 : HuggingFace TRL SFTTrainerHow the concept of a relatively close read together is developed in different contexts.

This section focuses on the real battle of GRPO: how to use TRL GRPOTrainer Run-through training, monitoring and debugging, and a full example of mathematical tasks. The rationale analysis is in the last section.

1. GRPO Trainer Parsing

1.1 The rationale for GRPOTrainer

GRPOTrainer carries training in GRPO algorithms; it is more straightforward to see its working mechanisms and to debug and optimize.

1.1.1 Full training cycle

GRPO Trainer One Step Training Process

Input1 bats prompts (e. g. watch size=2)

Step 1: Online Sampling

• For each prompt: Generate k=4 different responses (using current policy model π)
• Result: 2 prompts x 4 results = 8 sound Reactions

Example:

• Prompt 1："Calculate 5+3"

  • Response 1.1: "5+3=8"
  • Response 1.2: "First five plus three is eight, the answer is eight."
  • Response 1.3: "Counted 8"
  • Response 1.4: "Result is 8"

• Prompt 2："Calculation 7-2"

  • Response 2.1: "7-2=5"
  • Response 2.2: "Seven minus two equals five."
  • Response 2.3: "The answer is five."
  • Response 2.4: "Calculated result 5"

Step 2: Assessment of awards

• Call reward_function(prompts, responses)

Example reward:

• Prompt 1 award: [0.5, 0.9, 0.6, 0.4]
• Prompt 2 awards: [0.7, 0.8, 0.5, 0.6]

Step 3: Compute group comparative advantage (Group Relative Advantage)

Response group for each prompt:

• Prompt 1：

  • mean_reward_1 = (0.5+0.9+0.6+0.4)/4 = 0.6
  • advantages_1 = [0.5-0.6, 0.9-0.6, 0.6-0.6, 0.4-0.6] = [-0.1, +0.3, 0.0, -0.2]

• Prompt 2：

  • mean_reward_2 = (0.7+0.8+0.5+0.6)/4 = 0.65
  • advantages_2 = [+0.05, +0.15, -0.15, -0.05]

Step 4: Calculate strategic loss (Policy Los)

• For each (response, advantage):
  • log_prob = log π_θ(response | prompt)
  • policy_loss += -advantage * log_prob

Understanding:

• advantage > 0 ~ Increase the probability of generating the response
• advantage < 0 • Reduce the probability of generating the response

Step 5: KL Regularization

• Calculates the KL diffusion of the current model and the reference model:kl_penalty = β * KL(π_θ || π_ref)
• Effect: limit the excessive deviation of the model from the reference model

Step 6: Total losses and reverse transmission

• total_loss = policy_loss + kl_penalty
• total_loss.backward()
• optimizer.step()

Step 7: Update reference models (optional)

• Some of the EMAs that will regularly update the reference model as the current model

1.1.2 The mathematical rationale for the comparative advantage of the cluster

Traditional PPO issues：

• Value network estimated status value required
• Additional training value network required (costed)
• Value estimates may not be accurate (impacting training stability)
• Complicated

GRPO Innovation: No value networks needed! Use the group comparative advantage

Generate k responses for each programpt:{r_1, r_2, ..., r_k}

Advantage estimate：

For each prompt generation $k$ A response. ${a_1, a_2, ..., a_k}$, the advantage function is defined as:

$$ A(s, a_i) = R(s, a_i) - \text{mean}(R(s, \cdot)) $$

of which：

• $R(s, a_i)$: Response $i$ ♪ The prize ♪
• $\text{mean}(R(s, \cdot))$: Average incentive for all responses from the same group

Intuition.：

• "How much better is this response than the other responses from the same group?"
• I don't need to know."Absolutely."Standards
• Just the relative ranking.
• Use group average as baseline

Mathematics certificate (simplified)：

$$ \begin{aligned} \mathbb{E}[A(s,a)] &= \mathbb{E}[R(s,a) - \text{mean}(R(s,\cdot))] \ &= \mathbb{E}[R(s,a)] - \mathbb{E}[\text{mean}(R(s,\cdot))] \ &= \mathbb{E}[R(s,a)] - \mathbb{E}[R(s,a)] \ &= 0 \end{aligned} $$

• The advantage function is zero average.
• Reduced variance and increased training stability

Why does it work?

1. Automaticization: different incentive scales for prompt, and comparative advantage eliminates this problem
2. Reduction of the variance: relative stability compared to absolute assessment
3. Value-free network: substantial simplification achieved
4. Adaptive: auto-adaptation to different difficult tasks

1.1.3 GRPO Trainer vs PPO Trainer Comparison

Dimensions	PPOTrainer	GRPOTrainer
Value Network	Yes.	No need.
Advantage estimate	GAE（Generalized Advantage Estimation）	Group comparative advantage
Achieving complexity	[High] It's complicated.	[Chamber]
Training stability	[C]trained for referral	[High] More stable
Visible occupancy	[High] Large (tactical model + value network)	[High] Large (tactical model)
Costing	High (two networks required)	Medium (strategy network only)

1.1.4 Role of reference models

Reference Model：

• Usually a copy of the SFT model before training.
• Parameters frozen, not involved in training
• Effect: Calculate KL dispersion to prevent model deviations too far

Why do you need reference models?

Problem: Over-optimization If only incentives are optimized, models may:

1. Generate strange, unnatural text
2. Use incentive function leaks (reward hacking)
3. Knowledge acquired during loss of pre-training
4. The same function as the reference model in the DPO, and the same is true of the parameters involved

Beta (kl coef) controls balance：

• Beta, big, big, more conservative, near reference model.
• Beta, little twig, more radical models, higher rewards.

Actual: GRPTrainer automatically creates reference models:

1. Parameters for copying the current model
2. Freeze parameters (requires grad=False)
3. For KL only

Monitoring indicators：

• objective/kl: KL diffusion value with ideal range 0.1~5.0
• Too small (<0.1: Models are almost non-updated
• Too big.>10: Models are too far away, potentially unstable

1.2 Basic configuration

from trl import GRPOConfig, GRPOTrainer

GRPO 配置
training_args = GRPOConfig(
    output_dir="./grpo-output",
# GRPO 核心参数
num_generations=4,  # [关键] 每个 prompt 生成几个响应
temperature=0.7,    # [关键] 采样温度
kl_coef=0.05,       # [关键] KL 正则化系数

# 生成参数
max_new_tokens=256,       # 生成的最大 token 数
max_prompt_length=512,    # prompt 最大长度

# 训练参数
num_train_epochs=3,       # GRPO 可能需要更多 epochs
per_device_train_batch_size=1,  # 通常较小
gradient_accumulation_steps=8,   # 通过累积增加有效 batch
learning_rate=1e-6,       # GRPO 用很小的学习率

# 其他
gradient_checkpointing=True,
bf16=True,

)

1.3 Detailed key parameters

Parameter 1: Num generations

num_generationsNumber of responses per programpt

Impact：

• Training signal stability (larger and more stable)
• Costing (larger and more expensive)
• Quality of strength estimates (larger and more accurate)

Recommended value：

• num_generations = 4-8(Standard)
• num_generations = 2-4(Quick Experiment)
• num_generations = 8-16(Quality training)

Attention.：

• Actual batch_size = per_device_batch_size * num_generations
• Need to consider the obvious limitations

Parameter 2: temperature

temperature: Sample temperature, control diversity of generation

Impact：

• high (e. 1.0): Generate more samples and explore more possibilities but may have lower quality
• Low (e.g. 0.5): Generate more certainty, use known good models, but may lack exploration

Recommended value：

• temperature = 0.7-0.9(Standard)
• temperature = 0.5-0.7(Conservative, quality first)
• temperature = 0.9-1.2(radical, exploration priority)

Adjustment Policy：

• Initial training: higher temperatures (exploration)
• Post-training: lower temperature (utilisation)

Parameter 3: kl coef

kl_coefKL RPV, beta similar to DPO

Role：

• Limiting excessive deviations from reference models
• KL(π_θ || π_ref) • Punishment

Impact：

• kl coef Large (e.g. 0.1): The model is more conservative, the improvement is slow but stable
• kl coef Small (e.g. 0.01): Models are more radical, improve fast but potentially unstable

Recommended value：

• kl_coef = 0.05(Standard)
• kl_coef = 0.01-0.03(radical)
• kl_coef = 0.1-0.2(Conservative)

Monitor：

• objective/kl Should be kept in < 10
• If KL is too big, add kl coef

1.4 Data set preparation

# GRPO 只需要 prompt，不需要 response
# 因为响应会在训练时动态生成

from datasets import Dataset
方式 1: 从现有数据集加载
dataset = load_dataset("gsm8k", "main")
train_dataset = dataset["train"]
转换为 GRPO 格式
def format_for_grpo(sample):
    return {
        "query": format_prompt(sample["question"]),
        "ground_truth": sample["answer"]  # 用于奖励计算
    }
train_dataset = train_dataset.map(format_for_grpo)
方式 2: 自定义数据集
data = {
    "query": [
        "Question: What is 2+2? Answer:",
        "Question: What is 3+3? Answer:",
    ],
    "ground_truth": [4, 6]
}
train_dataset = Dataset.from_dict(data)

1.5 Create GRPO Trainer

from trl import GRPOTrainer

定义奖励函数
def reward_function(prompts, responses):
    """
    计算响应的奖励
Args:
    prompts: List[str] - prompt 列表
    responses: List[str] - 响应列表

Returns:
    List[float] - 奖励列表
"""
rewards = []
for prompt, response in zip(prompts, responses):
    reward = compute_reward(prompt, response)
    rewards.append(reward)
return rewards

创建 trainer
trainer = GRPOTrainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    processing_class=tokenizer,
    reward_function=reward_function,  # [关键] 自定义奖励函数
    peft_config=lora_config,
)
开始训练
trainer.train()

2. Training for monitoring and commissioning

2.1 Key indicators

Indicators	Meaning	Trends in expectations
rewards/mean	Average incentive	♪ ♪ Up ♪
rewards/best	Best reward.	♪ ♪ Up ♪
rewards/worst	Best reward.	↑ Up (but can be slower)
objective/kl	KL Scatter	• Stay stable (< 10）
objective/entropy	Policy entropy	Keep it steady.
loss	Training losses	♪ ♪ Down ♪

2.2 Training phase analysis

Typical training curve：

Phase 1: Rapid improvement period（Step 0-500）

• rewards/mean：0.2 → 0.5 → 0.7
• The phenomenon: modeled basic model of rapid learning

Phase 2: Steady upgradation period(Step 500 to 2000)

• rewards/mean：0.7 → 0.8 → 0.85
• Event: Model optimization details

Phase 3: Depression period（Step 2000+）

• rewards/mean：0.85 → 0.87 → 0.88
• Symptoms: slow improvement, near ceiling

Warning signal：

• rewards/mean Check for reward functions or learning rates
• objective/kl Too big.>20) Increase kl_coef
• loss No contraction, no reduction in learning rates.

2.3 Common problems and solutions

Problem 1: Average incentive does not rise

Possible cause：

1. Inappropriate design of reward functions (all responses are equal)
2. Over- or over-leaving
3. num_generations Too small.
4. The temperature is not set properly.

Solutions：

• Check reward function: Ensure differentiation degrees
• Adjusting the learning rate: trying 5e-7 or 2e-6
• Increase num_generations To 8
• Adjustment temperature to 0.8-1.0

Problem 2: KL spent a lot of time

phenomena：objective/kl > 20

Reason: Model deviation from reference model too far

Solutions：

• Increase kl_coef(if from) 0.05 → 0.1）
• Reduced learning rate
• Reduction in number of training steps

Issue 3: Discretionary deficiencies

GRPO notable occupancy ≈ base model + num generations x watch size

Solutions：

• Decrease num_generations（8 → 4）
• Decrease per_device_train_batch_size（2 → 1）
• Enable gradient_checkpointing
• Quantify with 4-bit
• Decrease max_new_tokens

Question 4: Slow training

GRPO reason for being slower than SFT/DPO：

1. Need to generate an online response
2. Calculate incentive function required
3. Generate multiple response increases

Accelerated programme：

• Enable Flash Attention
• Use bf16 Mixed precision
• Optimizing incentive function calculations
• Use less num_generations

3. Full example: solve a mathematical problem

3.1 Data readiness

from datasets import load_dataset

加载 GSM8K 数据集
dataset = load_dataset("gsm8k", "main")
格式化为 GRPO 格式
def format_math_problem(sample):
    prompt = f"""Question: {sample['question']}
Please solve this step by step and provide your final answer after ####.
Answer:"""
# 提取正确答案
answer = extract_number(sample['answer'])

return {
    "query": prompt,
    "ground_truth": answer
}

train_dataset = dataset["train"].map(format_math_problem)

3.2 Incentive function

import re

def math_reward_function(prompts, responses):
    """数学问题奖励函数"""
    rewards = []
for prompt, response in zip(prompts, responses):
    reward = 0.0

    # 检查格式：是否包含 ####
    if "####" in response:
        reward += 0.2

    # 检查推理：是否包含数学运算
    if any(op in response for op in ['+', '-', '*', '/', '=']):
        reward += 0.2

    # 检查答案（如果有 ground_truth）
    predicted = extract_answer(response)
    if predicted is not None:
        reward += 0.2

        # 这里需要从 prompt 中恢复 ground_truth
        # 实际应用中需要维护映射
        # if predicted == ground_truth:
        #     reward += 0.4

    rewards.append(reward)

return rewards

3.3 Training configuration

from trl import GRPOConfig, GRPOTrainer

配置
config = GRPOConfig(
    output_dir="./grpo-math-output",
    num_generations=6,        # 每题生成 6 个解答
    temperature=0.8,          # 适中的温度
    kl_coef=0.05,            # 标准 KL 系数
    max_new_tokens=300,      # 数学题需要较长推理
    learning_rate=5e-7,      # 较小学习率
    num_train_epochs=3,
    per_device_train_batch_size=1,
    gradient_accumulation_steps=16,
    gradient_checkpointing=True,
    bf16=True,
    logging_steps=10,
)
创建 trainer
trainer = GRPOTrainer(
    model=model,
    args=config,
    train_dataset=train_dataset,
    processing_class=tokenizer,
    reward_function=math_reward_function,
    peft_config=lora_config,
)
训练
trainer.train()

4. Operational codes

4.1 Environmental readiness

# 安装依赖
pip install transformers datasets peft bitsandbytes accelerate trl

可选：安装 Flash Attention 以加速训练
pip install flash-attn –no-build-isolation

4.2 Fast Start

# 运行训练
python grpotrainer.py

监控训练（如果使用 tensorboard）
tensorboard –logdir ./grpo-training-output/logs

4.3 Custom Configuration

Yes. grpotrainer.py , and amend:

# 模型
MODEL_NAME = "Qwen/Qwen3-8B"

数据集
DATASET_NAME = "gsm8k"  # 或其他数据集
GRPO 参数
NUM_GENERATIONS = 4     # 每个 prompt 生成数
KL_COEF = 0.05         # KL 正则化系数
TEMPERATURE = 0.7      # 采样温度
修改奖励函数（在代码中）
def reward_function(prompts, responses):
    # 你的自定义奖励逻辑
    …

Summary of this chapter

Concept	Annotations
GRPO	Group relative strategy optimized and supported customised incentives
Verifiable incentive	Incentives based on clear criteria (e.g. correctness)
Group comparative advantage	Use relative ranking advantages of the group response
Online sampling	Dynamic response generation during training, rather than offline data
num_generations	Key parameters, control the sample for each prompt

Summary

These five Blogs are a series of successive articles that have been made of the various stages of the Finetune around HF ecology and TRL. Start with the most basic SFT, gradually introducing the use of Trainer, SFTTrainer, DPOTrainer and GRPOTrainer, with code references.

LLM Finetune has many tools that deserve to be continued, such as Unsloth, which focuses on single card optimization, DeepSpeed, FSDP, Accerate, and the vLM and Online RL tools Verl.

If this set of articles continues to be expanded, it would be more appropriate to enter from external awards API and Verl such as Online RL tools, and to integrate custom reward, rollout and training movement control into a more complete process.

Recalling

Chapter	Main elements	Core skills
Chapter I: Trainer	Losing Masking, Quantification, LoRA	[Full completed] Manual SFT realization
Chapter 2: SFTTrainer	TRL library, automated SFT	[Facilitated] Efficient SFT training
Chapter 3: DPO	Preference alignment, offline RL	[completed] Human preference for learning
Chapter 4: GRPO (Presentation)	Online sampling, group comparative advantage, incentive system	[completed] Methodology and Intuitiveness
Chapter V: GRPO (Practical)	GRPOTrainer, M.C. debugging, full example	[Final] Run-through training and referral

Common problems

What if GRPO is slow?
A:1) Reduce num generations 2) Enable Flash Attention 3) Optimize incentive function calculations 4) Use smaller data sets to quickly overlap Subaru

Q: Average reward does not rise?
A: 1) Check for difference in incentive functions 2) Increase num generations 3) Adjust learning rate 4) Check for temperature settings

Q: Can GRPO and DPO be used in combination?
A: Yes! The recommended process: SFT → DPO → GRPO, so that models are generic and task-specific optimized

Q: How do we evaluate the effectiveness of GRPO training?
A:1) Monitor the upward trend of rewards/mean 2) Assess mission indicators (e.g., mathematical accuracy rate) on test set 3) Manual assessment of generation quality

References

• TRL Document: GRPO Trainer
• GRPO Paper: DeepSeekMath
• PPO Paper: Legal Policy Optimization
• FlashAttention paper: Fast and Memoory-Efficient Exact Actment with IO-Awarenescence
• TRL