LLM Post-Training and Fine-Tuning Practice: From SFT and LoRA to Human Alignment

Post-training is not an algorithm.

At the end of the pre-training exercise, we got a base model that's good at predicting the next token. It may have a large body of language and code models, but it may not necessarily answer in the way the user expects: unstable formats, vague mission boundaries, and not necessarily what should be the priority when faced with conflicting instructions. What is to be done after training is to re-engineer this behaviour with smaller and more targeted data.

I had two earlier sessions of fine-tuning of instructions, human alignment, visible estimates, data formats and framework options. The problem is that when training methods and engineering conditions are removed, it is easy to conclude that LoRA is only discussed without discussion of activation values, compared with data formats without looking at cat template, or that QLora is considered to be a quantitative deployment. Now put them back in the same chain.

graph LR
    A["预训练模型"] --> B["持续预训练"]
    A --> C["监督微调 SFT"]
    B --> C
    C --> D["偏好数据"]
    D --> E["DPO 等直接偏好优化"]
    D --> F["奖励模型 + RL"]
    C --> G["评估 / 安全 / 领域测试"]
    E --> G
    F --> G
    G --> H["部署与数据回流"]

The routes are different:

Ongoing pre-training(b) Continuing to study the text distribution of the fields, suitable for language, knowledge and code style;
SFT(a) Simulate quality input pairs and create command compliance, format and task behaviour;
PEFT(b) Determining which parameters are involved in updating is a training resource programme and not a stand-alone data target;
Prefer OptimizationMore multiple responses to make models more selective;
Enhanced learningConverting incentives or environmental feedback into strategic updates that are appropriate for interactive, verifiable or issues to explore.

There is no fixed streaming line for post-training. A field completion model may require only continuous pre-training, a taxonomy may be only SFT, and Agent may use SFT tracks and environmental incentives in turn. The order cannot be reversed before identifying what behaviour is to be changed and deciding on data and algorithms.

Data Engineering: Define the target for training first

The training script ends with token sequences and a loss mask, but the team should maintain a structured sample on a daily basis. The more common target groups are three.

Ongoing pre-training text

Ongoing pre-training usually retains the continuous structure of natural text or code:

{"text": "领域文档、代码或其他连续语料……"}

There are no natural data for this type of data. The training target remains the projection of the next token, so it is closer to pre-training, except for more centralized data distribution and computing. If the goal is to teach models to adhere to a certain output format, continuous pre-training alone is often not direct enough.

Command and single-wheel samples

Alpaca styles are common instruction、input、output For a single task:

{
  "instruction": "找出日志中的直接故障原因",
  "input": "{{日志内容}}",
  "output": "连接池耗尽导致请求持续排队。"
}

It is suitable for data production and manual review, but these fields are not the format that the model eventually reads. They are still token sequences of target models to be rendered before training.

Multiple rounds of information

Modern dialogue data usually use message arrays:

{
  "messages": [
    {"role": "system", "content": "只依据给定材料回答。"},
    {"role": "user", "content": "这次故障影响了哪些服务？"},
    {"role": "assistant", "content": "记录确认支付和订单查询受到影响。"}
  ]
}

ChatML, ShareGPT, Alpaca are common agreements, not uniform standards across models. ShareGPT Common conversations/from/valueMessage format is common messages/role/contentThe different training frameworks would also accept their own listing. Before entering tokenizer, you should convert to an internal uniform scheme, then replay it with the model's cattemplate.

Chat template decides what the model actually sees.

Chat texture uses characters, messages and special tokens as strings for modeling training. The following two messages are synonymous with different graphs and can be given completely different starting and starting marks and role token after different tokenizers. When templates do not match, the model sees controlrs that were never seen during the training phase, and performance is often more hidden than the problem of the data content itself.

The Hugging Face Chat Templates The document is given. apply_chat_template - How it's used. The training and reasoning should be re-enacted with the same template and make it clear whether to add a reference protocol, where to place EOS, how to encode the tool message. Do not manually guess a format that looks like the model.

Los Mask decides who the model is.

A conversation can calculate the loss of all tokens or only train antsant to respond. The latter are often called resonese-only or "assistant-only loss: system " and "user token " , which are still entered as conditions without requiring models to predict them.

There is no correct answer for all the tasks. Training only asistant is more in line with the goal of “generated answers on request” and avoids wasteful model capacity restatement users; some training that requires learning complete interactive structures or special controls token may also leave teams with more positions to lose. It's important to record the mask clearly. The two texts are identical, and the training objectives are different as long as the los mask is different.

Packing up utilization and changing sample boundaries

A large number of short samples, if article by article, would waste a significant portion of the calculations. Packing toss multiple samples into the same sequence to increase the effective token ratio. This is done by confirming whether attention is allowed to cross the sample, how the location ID is handled, whether EOS is correctly inserted, and whether the loss mark is at the wrong place at the border.

Packing is a stale optimization, and should not change the semantic of the data. If a model emerges after training that brings the last answer to the next question, the sample boundaries and templates are checked instead of the immediate learning rate.

High quality is not a model rating

Post-training data should be checked at least for:

Whether the Directive is enforceable and whether the entry contains the information required to complete the task;
The answer is correct, complete and in the same way as the desired model;
(a) Coherence of roles, tools and environmental status in the multi-cycle dialogues;
Whether to mix the rating collection, template leaks, error references and unrecoverable external state;
Whether data sources, generation models, filtering rules and history modification can be traced.

Synthetic data reduce the cost of writing samples and make it easier to scale up the same error. See more complete data synthesis process.Data synthesis is becoming a project: from Terminal-CorpusI'm sorry. The article discusses how to reverse training data from mission world, environment and verifier; here is the concern about how data is coded and calculated for the loss.

Visible estimate: Discount first, then training.

The "7B model requires a certain amount of visibility" without a fixed answer that is off the configuration. The full parameter is also LoRA, weight accuracy, optimizer, sequence length, bat size, whether to save master weights, checkpointing and fractions. It would be more stable to break the apparent deposits into several books.

Weights

Arguments in $N$, store accuracy is $b$ When bit, the theoretical size of weight is:

$$ M_{weights} = N \times \frac{b}{8} $$

7B parameters are about 14 GB, INT8 about 7 GB, 4 bit about 3.5 GB using BF16/FP16. This is only parameter data per se, and quantitative scale, zero point, grouping metadata and running buffers add additional occupancy.

Gradient and Optimizer State

If the training parameter is $P$, the gradient is approximated to:

$$ M_{grad} = P \times bytes_{grad} $$

Adam W, usually, keeps the first and second steps for each training parameter, and if both are used FP32, this is only part of it. $8P$ Bytes. Some blending precisions will retain FP32 master weights, and add more. $4P$ Bytes. The framework, optimizer and fractional policy are different and cannot be considered a "fixed byte number per parameter" as a common constant.

Take the full 7B BF16 training as an example, without consideration for fraction and activation: weights of about 14 GB, BF16 gradients of about 14 GB, two FP32 Adam states of about 56 GB; if FP32 master weights are saved, add about 28 GB. That is, these states alone could be 84 GB or 112 GB, active values, communication buffers and CUDA workspaces that have not yet started to be counted.

The difference between LoRA is clear here: the base weight still needs to be loaded, but only low-skull parameters are involved in gradients and optimizer updates, and therefore $P \ll N$I'm sorry. But the "Landitude and Optimizer Smaller" does not mean that the active value disappears, and long sequences and large bats may still fill the display.

Activate Value

Activated values relate to the size of the bat, the length of the sequence, the hidden dimensions, the layers, the attention achieved and the intermediate results that are saved. It will not be determined by the amount of parameters alone and it will be difficult to estimate using a fixed factor across structures.

These techniques are commonly used in practice to control the activation of the presence:

(a) Gradient checkpointing does not save all intermediate results and recalculates when they are disseminated in reverse;
(a) The physicalization of a matrix of reduced attention, such as FlashAttention;
(a) Increase the effective token ratio by setting a limit ton-up, but also by activation with the total token increase;
A few small micro-batchs simulated by a larger, effective watch;
Shortening the length of the sequence usually reduces activation costs more directly than fine-tuning a few LoRA rank.

Split and Frame Costs

FSDP, Zero, takes parameters, gradients and optimizer status fractions to multiple cards. They reduce the use of single cards, but increase the complexity of communications, movement control and configuration. CUDA constex, Kirnel workspace, temporary load, dataloader pre-emission and visible debris also require residual capacity.

So the visual memory estimate should be written in a compartment and validated by a short run with a target configuration. It is not enough to complete forward/backward, to see if peaks are visible, swallowed, offload, and if they are frequent, and if they are saved by checkpoint, OOM will not be repeated.

SFT: Make models imitate target behavior

Monitors fine-tuning the calculation of cross-cabins using input and target output, allowing the model to increase the probability of target token.InstructGPT、FLAN and Self-Instruct Shows the role of manual commands, task mixing and synthetic commands in the context of integration.

SFT is best suited to learn exemplary behaviour: answer structure, tool trajectories, domain terminology, refusal mode, code modification mode and conversation style. It does not guarantee reliable inclusion of facts in training samples in models, nor does it allow models to stabilize tasks that are completely impossible for the base model. If training data are themselves guessed, models can only be more skilled in imitating such guesses.

Learning rates, bats and epoch do not exit the data.

Post-training usually uses a learning rate lower than pre-training, but the specific range is influenced by model size, full parameters/PEFT, data volume, optimizer and target mission. What is more useful than observation of the default set of parameters:

Training for loss of productivity and the absence of improvement in validation tasks;
Whether the model is rapidly losing its original generic capability;
Whether or not a few templates occupy the output, causing style collapse;
Whether long and short answers receive different weights because of the token difference;
After a mixture of multiple data sources, which sample dominates the gradient.

Small data repeats multiple epochs easily. When data is big, a part of the training epoch may be sufficient. The final choice should be made by the set of tasks left, the pass rate and the regression test, rather than by training only, the loss.

PEFT: Reduced number of updated parameters without changing training objectives

Efficient fine-tuning of parameters (PEFT) freezes most of the base parameters and only trains a small number of additional parameters or the selected parameters. It reduces the presence of gradients and optimizers and allows the preservation of adapter for multiple tasks. More complete technology spectrum.PEFT: Technology Evolution from Adapter to LoRA。

LoRA

LoRA It is assumed that the weights needed to update downstream adaptations have a lower inherent value. For original weights $W_0$It's frozen. $W_0$, with two low-swipe matrices:

$$ W = W_0 + \Delta W, \qquad \Delta W = BA $$

If $W_0 \in \mathbb{R}^{d_{out} \times d_{in}}$♪ I'm so sorry ♪ $r$, and $A \in \mathbb{R}^{r \times d_{in}}$、$B \in \mathbb{R}^{d_{out} \times r}$I'm sorry. When? $r$ Training parameters are significantly reduced when much smaller than input output dimensions.

The main decisions of LoRA include target Modeles, rank, alpha, dropout and training for embedding, lm head or bias. The only word for "use" is still lacking enough information. The amount of argument may be significantly different from the effect of the attention projector, the effect of the profile, the effect of the profile and the addition of the profile to all linear layers.

Adapter can combine base weights before reasoning, or maintain independent loading. The consolidation does not normally add additional matrix calculations, but loses the convenience of running multiple adapter; the non-merger is more flexible and requires the correct support of the reasoning framework.

QLoRA

QLoRA Store and calculate the frozen base weights in 4 bits, while training higher precision LoRA parameters. It further subpresss the weight, but training still requires counterquantification of the calculation, activation and LoRA optimization.

QLora is not " train all parameters with INT4 " , nor is it a generic term for deployment of a quantitative format. It addresses the low resource fine-tuning; the decision whether to merge an adapter after the training is completed, what precision to export, and whether the target service supports the corresponding quantitative kernel remains another group.

Adapter and parameter selection

Classic Acapter inserts small bottlenecks networks in Transformer layers; prompt turning, prefix turning trains continuous vectors, and puts learning information in input or attention prefixes. Another method selects a partial layer or parameter update directly. They're all diminishing. $P$, but the placement, reasoning costs and multi-task combinations are different.

PEFT saves on training status and does not automatically fix data problems. If full fine-twining does not learn about targeted behaviour, LoRA is not usually solved in a vacuum; in turn, the task requires only light-scale behaviour and full-parameter updates may simply increase costs and forget risks.

Framework selection: group by function, rather than choosing a universal framework

The fine-tuning tools are often placed in the same ranking table, but they are at different levels. Before comparing them, look what they are responsible for.

Component	Main duties	Official entrance
Transformers	Models, tokenizer, chat template, basic training interface	Transformers
Datasets	Data loading, mapping, fluid processing and cache	Datasets
PEFT	Efficient method for parameters such as LoRA, IA3, prompt/prefix turning	PEFT
TRL	SFT, DPO, incentive model, PPO/GRPO etc.	TRL
Accelerate	Single-line Doca, hybrid precision and distribution startup	Accelerate
PyTorch FSDP	Parameters, Gradients and Optimizer Status Spectrometers	FSDP
DeepSpeed	ZeRO, offload, parallel training and reasoning component	ZeRO
Unsloth	Optimizing the performance of the seals for common models and the LoRA/RL workflow	Unsloth Docs

A common combination is Transformers for models and tokenizer, Datasets for data, PEFT for LoRA, TRL for SFT Trainer or preferred trainers, and Accelerate, FSDP or DeepSpeed processing equipment and fractions. Unsloth provides this eco-based set with optimized models for loading, kernel and training portals.

The selection of the frame depends on whether the target model supports, the chatch testmate is correct, the trainers are able to express the loss of the mark and the packing, the distributed checkpoint is restored, and the final weight is readable in the deployment frame. API writes short just as part of the experience. The higher the abstract layer, the more it needs to be prepared to go back to the bottom to confirm what it did when it met with new models and special ross.

Base is still a model that has been incorporated-tuned

The Base model retains a more primitive pre-training distribution, suitable for teams with sufficient data to want to redefine interactive behaviour; the interstasis-tuned model already has dialogue and guidelines to follow, and a small number of field data are usually more readily available, and may inherit original templates, refusals and style preferences.

There is no "how many data above must select the Base" common threshold. More practically, two starting points are compared with the same validation missions: if the INSTRUCTION model is already in place, it will be necessary to supplement the field behaviour and continue to fine-tune it, which is usually more economical; and if the original alignment seriously interferes with target formats or language distribution, then the Base model and the more complete data formulation will be assessed.

Human alignment: from imitation of answers to comparative behavior

SFT data tells the model "Here is a target answer." The preferred data give multiple answers and their relative selection under the same prompt, and allow training objectives to move from imitating a single answer to adjusting the probability relationship between the answer.

Classic RLHF

InstructGPT The processes frequently cited include three phases:

(a) Manual demonstration of SFT to obtain the initial strategy to follow the directive;
(b) Sorting of multiple model responses and training of incentive models;
Use the PPO optimization policy to give higher rewards to answers while using KL binding to avoid a deviation from the reference model.

The reward model is not a “human values function”. It learns about preference agents in specific labelling norms, samples and model distributions. The strategy may exploit the incentive gap when it is continually targeted to optimize it, and therefore there is a need to retain manual assessments, stand-alone task sets and behavioural constraints.

The PPO's hardness is not just an algorithm formula. The system is complicated by the generation of samples, computing incentives, estimations, training in value mode, control of KL, processing of long deviations and maintenance of multi-machine ingestion. For a full discussion of the issues of reward, baseline, advantage and numaration, seeIntensive Learning in LLM Alignment: From Incentive Signals to Estimation of Strengths。

Direct optimisation

DPO The goal of changing preference modelling to a direct optimization strategy and reference model does not require separate training incentive models and online PPOs. It brings training closer to regular supervisory learning, but does not eliminate data problems: Whether the difference between the size of the box indicates the preference of the labeler, whether the length of the response is short and whether the training distribution covers the line request, still determines the final effect.

The names of DPO, KTO, ORPO will continue to be added. When reading these methods, three questions can be asked: what feedback data are used, how reference models or hidden incentives enter the loss, whether training is offline or does online sampling are needed. This makes it easier to discern differences in the project than to group them simply by “Is it RL or not”.

SFT, Prejudice and RL. How to divide the job.

SFT is used first when there is a clear standard answer or a demonstration trajectory;
Multiple responses are available, but preference data are used when learning style, security or quality sequencing;
The award is from the environment, from testing or from interactive results, and is then considered online RLs when different strategies need to be explored;
In any case, independent assessments are maintained to prevent training signals from becoming more skilled.

Many projects are entering complex preference training too early, and the practical problems are SFT data format errors, chat test inconsistencies or weak assessment sets. The more sophisticated the post-training methodology, the more it is necessary to prove that the previous phase has stabilized.

An implementation checklist

Problem	Decisions requiring recording
Objective	What kind of behaviour is expected to change and which capabilities must remain intact?
Base	Base is still intract mode, whether license matches tokenizer
Data	How are schema, source, ratio, weight, quality doors and assessment sets defined
Templates	Chat text, special token, EOS and tool message code
Loss	Which token calculates the loss, how long the sample is added Rights
Training	Full/LoRA/QLoRA, target modules, precision, bat, serial length
Organisation	What are the weights, gradients, optimizers, activated, fractions and residuals?
Evaluation	Mission quality, pass rate, security, return, delay and cost
Export	Whether or not to merge and whether the final precision is compatible with the deployment framework
Roll back	Checkpoint, complete copy and training configuration

Post-training is often described as the selection of a framework, preparation of a JSONL, initiation of training. Actual results depend on consistency between objectives, data, templates, Loses, resources and assessments. LoRA can reduce the number of parameters updated, and cannot define tasks for you; RLHF can use preference signals or turn unreliable labels into reliable values.

If only one order of work is retained, I choose: to write an assessment before sorting out the data; to run through SFT before deciding whether to opt for optimization; to unload the visible accounts before choosing the frame. This is not a fancy exercise, but it can make most of the failures more visible earlier.