Re0-03: HuggingFace TRL DPOTrainer

Hyacehila
  2025-12-29 14:00 2025-12-29 14:00 Created      

Auxiliary code:dpotrainer.py

The questions in this article can also be addressedRe0-02 : HuggingFace TRL SFTTrainer、Re0-01 : HuggingFace Transformers TrainerHow the concept of a relatively close read together is developed in different contexts.

I'm going to take it from here.

The first two chapters have been completed:

  • Chapter I: using native Trainer for SFT, combing the basic concepts of Los Masking
  • Chapter II: automated SFT training was achieved using SFTTrainer to simplify processes

Through SFT, the model learned:

  • Answer the question in the command format
  • Generate a syntax-compliant response
  • Understanding the user ' s intentions

But...The limitations of SFT are that it lets models learn."How?"But not directly."What's the better answer?"。

Introduction to this chapter- A training method that allows models to learn human preferences. If the math theory of the DPO is to be understood in a systematic way, the original text should still be read; only the necessary extrapolation is retained below.

Learning objectives of this chapter

This chapter covers:

  1. Preferred Alignment Concepts and values
  2. DPO Differentiating from traditional RLHF
  3. Preferences data sets Format and Build
  4. DPOTrainer Methods of use of the Convention
  5. Reference Models Role and configuration of the

1. Why should preferences be aligned?

1.1 Limitations of SFT

SFT cannot distinguish the quality of the answer

User questions: "How can programming be improved?"

Possible answer A for SFT models (better):

Improved programming efficiency can be achieved by:

  1. Learn Shortcuts
  2. Use code clips
  3. Resting regularly

Possible answer for SFT models B (poor):

Just write more codes.

Question: Both answers are correct, but A is clearly more detailed and practical! SFT cannot distinguish between answers."Quality"Variance.

1.2 Optimum alignment of targets

Let the model learn:Give the same question, which answer is more in line with human preferences?

偏好数据格式:
{
    "prompt": "如何提高编程效率？",
    "chosen": "提高编程效率可以通过以下方式：1. 学习快捷键...",   ← 更好的回答
    "rejected": "多写代码就行了。"                               ← 较差的回答
}

2. RLHF vs DPO

2.1 Traditional RLHF (PPO) process

Traditional RLHF (PPO) process

  1. Collecting preference data Human Marker
  2. Training incentive model • Training in rewarder models with preferred data, which will require the construction of a value model to ensure the stabilization of the gradient in baseline, resulting in significant significant savings
  3. PPO Optimizing Policy • Sample from the strategy model, rate it with the reward model, update the strategy

Four models are required: Strategies, references, rewards, value models

Question:

  • It's complicated. It takes multiple models.
  • Training is unstable, and it requires a lot of involvement.
  • Costed (sampling required)

2.2 DPO: Simplified Preference Alignment

DPO Process

  1. Collecting preference data → Same as RLHF
  2. Direct Optimization Policy ← Major changes

Only 2 models: Policy + Reference Model

Advantages:

  • It's simple, it's easy to achieve.
  • Training is stable, no complex involvement.
  • Computer efficiency
  • The code logic is close to SFT, still to optimize the loss.

2.3 Maths (from intuition to formulae)

2.3.1 Core objectives of RLHF

In the traditional RLHF, our goal is simple:Make models more likely to produce human preferences.。

In mathematical terms, for a group of preferences, $(x, y_w, y_l)$(of which) $y_w$ It's better to answer that.$y_l$ The answer is poor. We want:

$$P(y_w > y_l \mid x) > 0.5$$

That is, the model should think that the answer is better than the answer was given.

2.3.2 Traditional RLHF practices: training reward models

To achieve this goal, the traditional RLHF has two steps:

Step 1: Training incentive model $R(x, y)$

The incentive model is designed to score each answer. We train it with preference data, making:

$$R(x, y_w) > R(x, y_l)$$

Step 2: Optimizing strategies with enhanced learning

Adjust policy models using PPO equations $\pi_\theta$ the parameters that give it higher rewards for the answers it generates:

$$\text{目标：最大化 } E[R(x, y)]$$

ProblemThis requires training in an independent incentive model, complex enhanced learning algorithms, cumbersome and unstable processes.

2.3.3 Insight of DPO: The reward function can interpret expression

The main lines of the DPO are:If we know the best strategy of the incentive function, we can go around the reward model and directly optimize the strategy.

Key Theorem: The best policy incentive function can be expressed in RLHF settings as:

$$R^(x, y) = \beta \cdot \log\left(\frac{\pi^(y \mid x)}{\pi_{\text{ref}}(y \mid x)}\right) + Z(x)$$

Of which:

  • $\pi^*(y \mid x)$ = Best strategy.
  • $\pi_{\text{ref}}(y \mid x)$ = Reference policy (memode after SFT, as starting point)
  • $\beta$ = temperature parameter, control deviation
  • $Z(x)$ = Normalized constant (and $y$ Not relevant, negligible when optimising)

Intuition.：

This formula tells us:The size of the incentive depends on the strategy model versus the reference model."Preferance Change"。

  • If $\pi^*$ Compare $\pi_{\text{ref}}$ More likely to generate an answer → an incentive for the answer High
  • If $\pi^*$ Compare $\pi_{\text{ref}}$ And it's less likely to generate a certain answer.

2.3.4 Optimization objectives of the DPO

With the expression of the reward function, we can write the target for optimization.

Targets for RLHF (with incentive models)：

$$\mathcal{L}_{\text{RLHF}} = \log \sigma\left(\beta \left[R(x, y_w) - R(x, y_l)\right]\right)$$

of which $\sigma$ is the sigmaid function. This loss encourages $R(x, y w) > R(x, y_l)$。

DPO objectives (using a strategy model)：

Adds the expression of the reward function to the message, and removes it. $Z(x)$, gets:

$$\mathcal{L}{\text{DPO}} = \log \sigma\left(\beta \cdot \log\frac{\pi\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \cdot \log\frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\right)$$

Simplified:

$$\mathcal{L}{\text{DPO}} = \log \sigma\left(\beta \cdot \log\frac{\pi\theta(y_w \mid x) \cdot \pi_{\text{ref}}(y_l \mid x)}{\pi_\theta(y_l \mid x) \cdot \pi_{\text{ref}}(y_w \mid x)}\right)$$

What's the meaning of this loss?：

  • Molecular(What we want):$\pi_\theta(y_w \mid x)$ Big.$\pi_{\text{ref}}(y_l \mid x)$ It's the benchmark.
  • Factor(We're avoiding):$\pi_\theta(y_l \mid x)$ Small.$\pi_{\text{ref}}(y_w \mid x)$ It's the benchmark.

In training, we maximize this loss by increasing the gradient, making:

  • $\pi_\theta(y_w \mid x)$ The model is more likely to generate chosen answers than the reference model.
  • $\pi_\theta(y_l \mid x)$ The model is less inclined to generate reprojected answers than the reference model is.

2.3.5 Why reference models?

You might ask:Why not just maximize it? $\pi_\theta(y_w \mid x)$Minimize $\pi_\theta(y_l \mid x)$？

Problem: this will lead to model collapse (Mode Collapse)

If there is no reference model constraint, the model may learn to:

  • High probability of output for chosen (close 1)
  • Respond to the very low probability of output (close to 0)

But it makes the model:

  • Forget the SFT phase.
  • Generate strange, unnatural answers.
  • I'm losing my ability to generalize.

Role of reference models：

$\pi_{\text{ref}}$ Just like one."anchor", ensures that:

  • $\pi_\theta$ You can't deviate from the original model at will.
  • Only adjust the orientation of preferences
  • Maintaining linguistic capacity and stability of models

Maths, that's through. KL Scatter NormalizationAchieved:

$$\text{KL}(\pi_\theta \Vert \pi_{\text{ref}}) = \sum_y \pi_\theta(y \mid x) \log\frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)}$$

The loss function of the DPO already contains the KL binding in a hidden way (by $\beta$ Parameter control).

2.3.6 Summary: DPO vs RLHF

In terms of Traditional RLHF (PPO) DPO
Step one. Training incentive model $R(x, y)$ [No] Not required
Step two. Use enhanced learning optimization strategy [Yes] Directly optimized strategy
Optimizing objectives Maximize $E[R(x, y)]$ Maximize $\mathcal{L}_{\text{DPO}}$
Reference Models For KL Regularization Directly embedded loss function
Compute complexity Sample generation required, large amount of calculations SFT-like, computing-efficient.

Main strengths: DPO translated the issue of enhanced learning into oneMonitoring learning issues, no training incentive model, no complex RL algorithm, code near SFT.

3. Optimistic data set formats

3.1 Standard format

# 格式1: 简单格式
{
    "prompt": "The sky is",
    "chosen": " blue.",
    "rejected": " green."
}


格式2: 对话格式
{
    "prompt": [{"role": "user", "content": "What color is the sky?"}],
    "chosen": [{"role": "assistant", "content": "It is blue."}],
    "rejected": [{"role": "assistant", "content": "It is green."}]
}

3.2 Common preferred data sets

Dataset Description Size
trl-lib/ultrafeedback_binarized TRL Officially provided ~60k
Anthropic/hh-rlhf Useful + non-hazardous data for Anthropic ~170k
argilla/ultrafeedback-binarized-preferences-cleaned #ltraFeedback after cleaning ~60k

3.3 Self-build preferred data sets

# 偏好数据的构建方式：
# 1. 人类标注：让标注者选择更好的回答
# 2. AI 辅助：用强模型（如 GPT-4）判断偏好
# 3. 启发式规则：如选择更长/更详细的回答


preference_data = {
    "prompt": "解释什么是机器学习？",
    "chosen": "机器学习是人工智能的一个分支，它使计算机系统能够从数据中学习和改进，而无需进行明确的编程。主要类型包括：监督学习、无监督学习和强化学习…",
    "rejected": "就是让机器学东西。"
}

4. Guide for the use of DPOTrainer

4.1 Basic configuration

from trl import DPOConfig, DPOTrainer


DPO 配置
training_args = DPOConfig(
    output_dir="./dpo-output",


# DPO 核心参数
beta=0.1,  # [重要] KL 正则化强度
loss_type="sigmoid",  # 损失函数类型

# 训练参数
num_train_epochs=1,  # DPO 通常只需 1-3 个 epoch
per_device_train_batch_size=4,
learning_rate=5e-7,  # DPO 通常用更小的学习率

# 其他
gradient_checkpointing=True,
bf16=True,



)

4.2 Key parameters: Beta

beta is the most important ultra-parameter for the DPO:

beta 的作用：控制模型偏离参考模型的程度


beta 大 (如 0.5):
  └─→ 模型更保守，不敢偏离参考模型太远
  └─→ 适用于：偏好数据噪声较大时


beta 小 (如 0.05):
  └─→ 模型更激进，可能偏离参考模型较远
  └─→ 适用于：偏好数据质量高时


推荐起始值: 0.1
调整范围: 0.05 - 0.5

4.3 Loss function type

The focus of the DPO is on how to convert enhanced learning alignment to loss function optimization, and therefore there are many derivative forms where choices can be made.

loss_type = "sigmoid"  # 默认，标准 DPO


其他可选值：
"hinge"    - 来自 RSO 论文
"ipo"      - 来自 IPO 论文，解决过拟合
"robust"   - 鲁棒 DPO，处理噪声标签

4.4 Create DPO Trainer

from trl import DPOTrainer
from datasets import load_dataset


加载偏好数据集
dataset = load_dataset("trl-lib/ultrafeedback_binarized")


创建 DPOTrainer
trainer = DPOTrainer(
    model=model,                    # 策略模型
    ref_model=None,                 # 参考模型（None 则自动创建）
    args=training_args,             # DPO 配置
    train_dataset=dataset["train"],
    processing_class=tokenizer,     # 分词器
    peft_config=lora_config,        # LoRA 配置
)


开始训练
trainer.train()

5. Treatment of reference models

5.1 Role of reference models

DPO needs two models:

  • Policy Model

    • We're going to train the model.
    • Parameters will be updated

  • Reference Models

    • The KL diffusion regularization used to calculate KL
    • Usually a copy of the model after SFT.
    • Parameters frozen, not updated

5.2 Three reference model strategies

flowchart TB
    subgraph Strategies["参考模型配置策略"]
        direction TB


    subgraph S1["策略1: 自动创建 推荐"]
        S1_Code["ref_model = None"]
        S1_F1["→ DPOTrainer 自动创建参考模型副本"]
        S1_F2["→ 使用 PEFT 时，会卸载适配器进行参考推理"]
        S1_F3["→ 最省显存"]

        S1_Code --> S1_F1
        S1_F1 --> S1_F2
        S1_F2 --> S1_F3
    end

    subgraph S2["策略2: 手动创建"]
        S2_Code["ref_model = AutoModelForCausalLM.from_pretrained(...)"]
        S2_F1["→ 完全独立的模型副本"]
        S2_F2["→ 显存占用翻倍"]

        S2_Code --> S2_F1
        S2_F1 --> S2_F2
    end

    subgraph S3["策略3: 适配器切换 (使用 PEFT 时)"]
        S3_F1["→ 同一个基础模型加载两个适配器"]
        S3_F2["→ 训练时切换适配器"]
        S3_F3["→ 显存占用较低"]

        S3_F1 --> S3_F2
        S3_F2 --> S3_F3
    end

    S1 -.-> S2
    S2 -.-> S3
end

5.3 Example of codes

# 策略1: 自动创建（推荐）
trainer = DPOTrainer(
    model=model,
    ref_model=None,  # ← 自动处理
    ...
)


策略2: 手动创建
ref_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-8B",
    dtype=torch.bfloat16,
    device_map="auto",
)
ref_model.eval()  # 冻结


trainer = DPOTrainer(
    model=model,
    ref_model=ref_model,
    …
)

6. Training monitoring indicators

6.1 Key indicators

Indicators Meaning Trends in expectations
rewards/chosen Chosen, average reward for answering. Up
rewards/rejected Average reward for responding to the question Down
rewards/margins margin of chosen - reprojected Up
rewards/accuracies chosen prize > Reprojected rate of reward Close 1.0

6.2 Examples of surveillance

训练日志:
  Step 100: rewards/margins = 0.5, rewards/accuracies = 0.65
  Step 200: rewards/margins = 1.2, rewards/accuracies = 0.78
  Step 300: rewards/margins = 2.0, rewards/accuracies = 0.89
  ...


分析:
  margins 在上升 → 模型学会区分 chosen 和 rejected
  accuracies 接近 1.0 → 模型几乎总能正确排序偏好

7. Training process for DPO versus SFT

Full training process comparison

Phase SFT (chapters I and II) DPO (This chapter)
Objective Let the model learn to follow the instructions. Let the model learn to be positive.
Data (prompt, restone) (prompt, chosen, reprojected) Group
Result It's a question-and-answerer, but not necessarily the best quality. The quality of the answers has improved significantly.
Time 1-3 epochs 1-3 epochs
Order Let's go first. After SFT

Important: The DPO should be done after SFT!

8. Operational codes

8.1 Configuration Parameters

Yes. dpotrainer.py , and can be modified:

MODEL_NAME = "Qwen/Qwen3-8B"  # 模型
DATASET_NAME = "trl-lib/ultrafeedback_binarized"  # 偏好数据集
BETA = 0.1  # KL 正则化强度
MAX_LENGTH = 2048  # 最大序列长度

8.2 Operational training

python dpotrainer.py

8.3 Assessing the effectiveness of training

After training, observe:

  1. rewards/margins Is it rising?
  2. rewards/accuracies Close to 1.0
  3. Generation test: whether model answers are more responsive to expectations

9. Summary of this chapter

Concept Annotations
Preferred Alignment Let the model learn."What's better?"
DPO Direct optimisation, simplified RLHF alternatives
Preferences (prompt, chosen, reprojected) Group
Reference Models For KL Regularization to prevent model deviations too far
beta Key super-parameters to control conservative/radicality

10. Limitations and prospects of the DPO

10.1 Limitations of the DPO

Dependence on offline data

  • Precollected preferences only
  • No online exploration and learning

Incentive function hidden

  • DPO hides the reward function through mathematical changes
  • We can't control the reward signal directly.
  • Difficulty in achieving complex incentive designs

Only for preferences

  • I can't handle a mission that requires a clear reward signal.
  • For example: mathematical reasoning, code generation, etc., require valid tasks

10.2 Next: GRPO and verifiable incentives

Problem: Can we design better rewards if our mission has a clear correct/mistakeful judgement (e.g., mathematical questions)?

Answer: Yes! This is where advanced RL methods like GRPO are used.

Appendix: Common problems

Q: DPO has changed the model since training?
A:1) Check beta values, which may lead to excessive deviation (2) Check preference data quality (3) Reduce training epochs

Q: Never lifts/accumulations?
A:1) Checks for correct data format 2) Attempts to increase learning rate 3) Ensures that chosen is better than reprojected

What if there's not enough to show?
A:1) Use ref_model=None Allows the DPOTrainer to autoprocess 2) to reduce the size of the watch size 3) to enable the original checkpointing

References

  • Title: Re0-03: HuggingFace TRL DPOTrainer
  • Author: Hyacehila
  • Created at : 2025-12-29 14:00:00
  • Link: https://hyacehila.github.io//blog/2025/12/29/Re0HF-03/
  • License: This work is licensed under CC BY-NC-SA 4.0.
Comments
On this page
Re0-03: HuggingFace TRL DPOTrainer
  1. I'm going to take it from here.
  2. Learning objectives of this chapter
  3. 1. Why should preferences be aligned?
    1. 1.1 Limitations of SFT
    2. 1.2 Optimum alignment of targets
  4. 2. RLHF vs DPO
    1. 2.1 Traditional RLHF (PPO) process
    2. 2.2 DPO: Simplified Preference Alignment
    3. 2.3 Maths (from intuition to formulae)
  5. 3. Optimistic data set formats
    1. 3.1 Standard format
  6. 格式2: 对话格式
    1. 3.2 Common preferred data sets
    2. 3.3 Self-build preferred data sets
    3. 4. Guide for the use of DPOTrainer
      1. 4.1 Basic configuration
  7. DPO 配置
    1. 4.2 Key parameters: Beta
    2. 4.3 Loss function type
  8. 其他可选值：
  9. "hinge" - 来自 RSO 论文
  10. "ipo" - 来自 IPO 论文，解决过拟合
  11. "robust" - 鲁棒 DPO，处理噪声标签
    1. 4.4 Create DPO Trainer
  12. 加载偏好数据集
  13. 创建 DPOTrainer
  14. 开始训练
    1. 5. Treatment of reference models
      1. 5.1 Role of reference models
      2. 5.2 Three reference model strategies
      3. 5.3 Example of codes
  15. 策略2: 手动创建
    1. 6. Training monitoring indicators
      1. 6.1 Key indicators
      2. 6.2 Examples of surveillance
    2. 7. Training process for DPO versus SFT
    3. 8. Operational codes
      1. 8.1 Configuration Parameters
      2. 8.2 Operational training
      3. 8.3 Assessing the effectiveness of training
    4. 9. Summary of this chapter
    5. 10. Limitations and prospects of the DPO
      1. 10.1 Limitations of the DPO
      2. 10.2 Next: GRPO and verifiable incentives
    6. References