Parameter-Efficient Fine-Tuning (PEFT): From Adapter to LoRA

This English version was machine-translated from the Chinese original. Technical terms may require verification.

Why do you need to fine-tune the parameters efficiently?

The questions in this article can also be addressedRe0-01 : HuggingFace Transformers Trainer、Post-linguistic training and fine-tuning of practice: from SFT, LoRA to human alignmentHow the concept of a relatively close read together is developed in different contexts.

Full Fine-Tuning is usually a good effect of large language models, but computing and storing costs will increase rapidly with model size. To fit downstream missions with limited resources, researchers proposed the high-efficiency fine-tuning of parameters (Parameter-Efficient Fine-Tuning, PEFT) technology: freeze most pre-training parameters, train only a few newly introduced or re-parametric modules and use lower-cost effects to approach full fine-tuning.

This paper summarizes the design ideas and technical features of several representative methods in the PEFT field.

Adapter: Insert training modules into layers

Houlsby et al., Parameter-Efficient Transfer Learning for NLP, ICML 2019.

Adapter can be an early representation in the PEFT field. At this stage, although large-scale pre-training models have emerged, causal language models have not yet taken precedence and many downstream tasks still need to be fine-tuned. Adapter was raised in this context to reduce the cost of fine-tuning.

Adapter has inserted a training structure between the model layers. In the author 's experiment, Adapter was inserted after the FNN of the coding device, before the Layer Norm, and the internal FFN itself contained non-linear activation functions. But in the actual scene, Adapter is inserted in a more flexible position, with no defined norm.

The authors consider that the main advantages of Adapter include:

• Light Quantification and High Performance: fine-tuned only a smaller number of parameters (to a full measure), but very good.
• Model reuse: the main model only needs one copy, without having to copy the full amount of parameters for each task
• As a means of fine-tuning: no full training required on new assignments, consistent with the pre-training + fine-tuning paradigm

Prefix-Tuning: Optimizing Continuous Prefixes

Li and Liang, Prefix-Tuning: Optimizing Continuous Prompts for Generation, ACL 2021.

In order to obtain better off-the-road mission effects, we would like to fine-tune the pre-training model. The introduction of additional parameter adapters, in addition to the direct freezing of the original parameter layers, is also discussed in detail by scholars, given the high full fine-tuning costs. Prefix-Tuning is one such lightweight fine-tuning.

The idea of prefix fine-tuning comes from context learning (In-Context Learning). A small number of hints (token pre-input) can be used to acquire field performance, so the author considers adding prefixes to the token.Prefix is a continuum that needs to be optimized, during Prefix-Tuning, the main parameter layer of the model is frozen, and the replacement of Prefix is sufficient to allow the model to be converted.

All Encoder layers were added to Prefix, so the prefix actually formed a matrix, the total number of parameters, and the number of prefixes was the same. $\text{length}(\text{prefix}) \times \text{num_layers}$I'm sorry. Prefixes are left-hand to ensure that all Mask-attents have sufficient information.

This technology is also applied to NLG and NLU tasks, i.e., can be used to fine-tune the BERT structure and class GPT structure.

The advantages the author believes are:

• LightQuantification: fine-tune only a smaller number of parameters (as compared to the insertion of Adapter Layer)
• Model integrity: Without changing the model structure, only one copy is required for the core model
• Application level: Modeling is feasible because of its lightness and it also benefits privacy protection

LoRA: Low-fitness and grace

Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, ICLR 2022.

LoRA's idea is derived from the low-level breakdown of the matrix, which is based on the belief that changes to the parameter matrix can actually be summarized by a low-level matrix, and therefore uses a small learning matrix instead of the original full-parameter modification.

The main advantages identified by the authors include:

• Storage space advantage: original parameters frozen completely, reducing training visible consumption and expenditure on model storage and task switching
• Calculate efficiency: significant improvement in computing efficiency due to the need to optimize only the new low-caliber matrix
• Zero reasoning delay: The low-swirl matrix can be directly merged into the original parameter without any delay in reasoning as a result of the new matrix
• Flexible combinations: It can be used in combination with multiple methods
• Full fine-tuning of the generalization: Increase low tumble $r$ The size of the size will be nearly fully fined.

Comparison of other programmes

• Adapter's strategy, although it has rarely increased, still produces Inference Latecy
• Prefix-Tuning prefixes are hard to learn, performance optimization is low

The lab found out.

• QKVO's four attention matrices are better all optimized. Even if overall optimization takes a low runk, the four matrices are broken down by lower runks over only one of the runks.
• It doesn't need a high-level runk to achieve enough performance in the experiments that the author considers.$r=8$ and $r=64$ The performances do not differ significantly, because they share a dimension of the subspace.
• The mechanism for the central role of the Low Spectrofit Matrix - it may be enhanced by enhancing the pre-training modelLearning but not focusedand thereby effectively match the needs of specific downstream missions

LoRA and its derived fine-tuning techniques based on parameter freezing and matrix decomposition are among the most important SFT technologies at present.

Prompt Tuning: A hint from dispersing to continuous

Lester et al., The Power of Scale for Parameter-Efficient Prompt Tuning, EMNLP 2021.

Prompt Tuning freezes all original model parameters and adds learning-able soft prompts to the new Prompt. The idea is to make models fit for specific tasks by adding soft prompts to Prompt. Prompt Tunning can be considered a special form of Prefix-Tunning.

Prompt Tunning's idea comes from Prompt Engineering. The idea of using fine-tuning model parameters to replace artificial tips has emerged as a result of the need for more manual involvement in fine-tuning of the phrases. Only the additional learning soft programt is trained from end to end to enrich information about the type of task.

Prompt Tuning, compared to manual hints, is generally known as aContinuous prompt technology, because the hints are continuous throughout the embedded space, the method of the artificial hints is discrete in the embedded space and therefore generally perform better than the artificial hints.

Valuable conclusions

• As the size of the model grows, the gap between Prompt Tuning and the whole model fine-tuning is gradually narrowing.And better than artificially designed tips.
• The prefix token should not be too short, too long, and it would be more appropriate to control at dozens.
• Less effective than Prefix-Tuning, which leads to subsequent improvements in P-Tuning v2
• Prompt Tuning may be more robust because of the few changes to the structure of the original parameter, which is distributed at different stages of the training and assessment (i.e., field deviation)
• Freezing of common language understanding parameters and limiting downstream learning to lightweight parameters would help avoid over-representation of specific areas - Yeah.

Prompt Tunning is much more valuable in thought than it is in application.At the same time, this technology is generally used only for NLU tasks and not for the type GPT structures that are now in the mainstream.

P-Tuning v2: Deep tip return

Liu et al., P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks, ACL 2022.

The main contribution of this paper is an innovative empirical finding:Prompt Tuning In the context of the various model scales and natural language understanding (NLU) missions,Performance can be comparable to full-parameter fine-tuning。

P-Tuning v2 has abandoned the option of adding soft programt only to the embedded layer, but has added learning prefixes to the multiple Encoder layers. P-Tuning v2 and Prefix-Tuning are close, but Prefix-Tuning's high-dimensional prefix is obtained by a lower-dimensional weight parameter from an MLP layer, and P-Tuning v2 directly optimizes the high-dimensional vector.

P-Tuning v2 Clear applicationStandard Header, which can very naturally support:

• Text Classification
• Named Entity Identification (NER)
• Semantic Role Description (SRL)

. The task of NLU is covered. Compared to the LoRA, which is suitable for natural language generation tasks, it requires a variety of tips to activate its capabilities in other NLU missions.P-Tuning v2 is fit for this type of task。

P-Tuning v2 strengths and discoveries

• For small language models: The current fine-tuning focuses on the study of the production language model of 10B or more, while P-Tuning v2 has a good effect on the fact that it does not require so many parameters in the NLU field
• Value of Depth Tips: The original Prompt scheme is still inadequate here
• P-Tuning v2 is generally comparable to a whole fine tune for all tasks Beautiful.

As with Prompt Tunning, P-Tuning v2 is also a structure that applies to a similar BERT. The large cause-effect language model (GPT) has had a very strong effect on NLU missions, and the real application value of considering using these fine-tuning techniques is questionable.

Adalora: Self-adaptation distribution

Zhang et al., AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning, ICML 2023.

This is an improvement on the LoRA method. Unlike the visual modifications of QLora, LoRA+ (quantitative original parameters and modified learning rates for the LoRA part), Adalora wishes to introduce an Adaptive LoRA Rank, which reduces the low-value portion of LoRA, thereby increasing fine-tuning efficiency. For the importance of the distinction, Adalora was achieved using the oddly foreign decomposition (SVD).

SVD Adaptation

First, LoRA decomposition based on SVD, which converts the original simple matrix decomposition into SVD form:

$$ W = W^{(0)} + \Delta = W^{(0)} + P \Lambda Q $$

And use $\mathcal{G}i = { P{i}, \lambda_i, Q_{i} That indicates a strange value against a strange direction, where the odd value is used to initialize, the vector is to initialize with Gaussian and add a positive-soft constraint:

$$ R(P, Q) = | P^\top P - I |_F^2 + | Q Q^\top - I |_F^2 $$

All these operations are designed to ensure that normal processes are optimized when making the rank adaptation and reduce the calculation costs of the high-dimensional matrix for SVD breakdown.

We will apply the SVD-based appliance to each ownership matrix in the Transformer layer, including $W_q, W_k, W_v, W_{f_1}$ and $W_{f_2}$I'm sorry. In order to control the budget of the parameters, the branches are cut in order to be divided over time during the training process, based on the importance of the odd values.

Importance-based Rank Allocation

Use $k$ Indexing incremental matrix $\Delta_k = P_k \Lambda_k Q_k$（$k = 1, \dots, n$) , will $\Delta_k$ No. No. $i$ Three-dollar group to $matcal{G}{k,i} = {P{,i}, \lambda_{k,i}, Q_{i,}}$，其重要性得分为 $S_{k,i}$。

The training target function for adding regular items is:

$$ \mathcal{L}(\mathcal{P}, \mathcal{E}, \mathcal{Q}) = \mathcal{C}(\mathcal{P}, \mathcal{E}, \mathcal{Q}) + \gamma \sum_{k=1}^n R(P_k, Q_k) $$

At the end of the day $t$ Step, first to implement a random gradient step to update the parameters:

$$ \tilde{\Lambda}k^{(t)} = \Lambda_k^{(t)} - \eta \nabla{\Lambda_k} \mathcal{L}(\mathcal{P}^{(t)}, \mathcal{E}^{(t)}, \mathcal{Q}^{(t)}) $$

Then, you score the importance. $S_k^{t}$, the odd value is cut as follows:

$$ \mathcal{T}(\tilde{\Lambda}k^{(t)}, S_k^{(t)}){ii} = \begin{cases} \tilde{\Lambda}{k,ii}^{(t)} & \text{if}S\text{ before \t\text{}b^t}\text{name},\ Photo by Flickr user @un.org & \\text{Other circumstances} The next thing I know, I'm not sure. I'm sorry.

of which $S^{(t)}$ The first one was a three-dollar score, which was a three-dollar score.$b^{(t)}$ No. No. $t$ The remaining odd budget. In this way, cuttings are less important and leave more budgets to higher-priority incremental matrices.

The material measure

Range of the odd valueIt is the most direct quantitative method, but it is not possible to quantify appropriately the contribution of parameters to model performance. The author has proposed a basis for theSensitivityThe importance of the scoring:

$$ S_{k,i} = s(\lambda_{k,i}) + \frac{1}{d_1} \sum_{j=1}^{d_1} s(P_{k,ji}) + \frac{1}{d_2} \sum_{j=1}^{d_2} s(Q_{k,ij}) $$

Sensitivity using gradient-weight multiplier $I(w_{ij}) = \left\lvert w_{ij} \nabla_{w_{ij}} \mathcal{L} \right\rvert$and adoptedSensitivity SmoothandQuantified UncertaintyAddressing fluctuations:

$$ \begin{aligned} \bar{I}^{(t)}(w_{ij}) &= \beta_1 \bar{I}^{(t-1)}(w_{ij}) + (1 - \beta_1) I^{(t)}(w_{ij}) \ \bar{U}^{(t)}(w_{ij}) &= \beta_2 \bar{U}^{(t-1)}(w_{ij}) + (1 - \beta_2) \left\lvert I^{(t)}(w_{ij}) - \bar{I}^{(t)}(w_{ij}) \right\rvert \end{aligned} $$

The final definition of importance is the product of both:$s^{(t)}(w_{ij}) = \bar{I}^{(t)}(w_{ij}) \cdot \bar{U}^{(t)}(w_{ij})$。

Global budget movement

Budget to be budgeted $b^{(t)}$ Defines the sum of all incremental matrices (i.e. the total oddly foreign number). From a slightly higher than target budget $b^{(T)}$ Initial budget $b^{(0)}$ Start (e.g. 1.5 times) with the initial lid of each incremental matrix as $r = b^{(0)}/n$I'm sorry. Preheating training $t_i$ After that, the budget is gradually reduced following a three-way movement strategy $b^{(t)}$ Until the goal is achieved.

Special observation: Adalora always allocates more budget to the LM Head of the FNN and top floors, which is also consistent with the finding in the LoRA fine-tuning study that the concentration level is less important than the linear layer, and should provide the entire layer with the LoRA adaptor as far as possible, subject to the conditions.

Adalora is currently a fine-tuning framework as useful as traditional LoRA, used in PEFT libraries AdaLoraConfig To call.

Summary and outlook

This post has been used to review the representative approach of the PEFT field from Adapter to Adalora. These technologies together illustrate one thing:The parameter space of the pre-training model is heavily reusable and is sufficiently effective for many downstream missions, either low-dimensional or locally.

From the technical route, the PEFT approach can be broadly divided into three directions:

• Insert(Adapter): Insert training modules between models
• Prefix/tip(Prefix-Tuning, Prompt Tuning, P-Tuning v2) Injecting learning-able continuous vectors at input or at all levels
• Reparatic(LoRA, AdaloRA): Directly modify the weight matrix by low-stealing decomposition

Of these, LoRA and its variants have become one of the most commonly used PEFT programmes at present, with low additional reasoning costs and flexibility.

Looking ahead, the PEFT research may need to be directed from simple efficient adaptation to a shift from a more efficient to a more efficient one.Optimizing RL dynamics for adaptationI'm sorry. As Meta discussed in Three-Gate Theory, the gradient update for the RL post-training may follow a different geometric path than SFT: It prefers to modify the non-main directional subspace with a lower curvature in the pre-training parameter space rather than the main ingredient direction. Thus, the traditional LoRA, which is based on low sovereignty renewal, does not always match RL ' s optimisation dynamics.How to design efficient methods of protecting and using parameters that can update this non-main orientation will be a matter of concern for the next phase of PEFT.