From Principal Component Regression (PCR) to Partial Least Squares (PLS)

This English version was machine-translated from the Chinese original. Technical terms may require verification.

The core views and parts of this paper are based on the post of the Fau Fumi:Return of the main ingredient to the lowest 2x2。

The questions in this article can also be addressedLinear regression base: linear model, minimum 2x2 estimate and regression diagnosis、Linear regression step: proposed alignment, model selection and co-line SexHow the concept of a relatively close read together is developed in different contexts.

Returns from main ingredient (PCR) to minus 2 (PLS)

In Multilinear Regression, MLR, the minimum binary is the commonly used parameter estimation method. However, when data is centralizedMulticollineabilityor the number of variables exceeds the sample ($p) > n$）时，OLS 估计量会变得不稳定，甚至无法计算（因为设计矩阵 $X^TX$ is irreversible or close to extraordinary.

To address this problem, the concept of Democratization is common. **Regression of main constituents (PCR)andThe minimal 2x2 regression (PLSR)** is the two representatives of this approach. They all return by building a new " Potential Variables " instead of the original variables, but the logic of both building the potential variables is different and should therefore be used with caution.

1. Primary Component Return (Prircipal Component Regresion, PCR)

1.1 The mathematical rationale

The PCR can be summarized as follows:First, the PAA, then the OLS.。

Suppose we have a centralized variable matrix. $X \in \mathbb{R}^{n \times p}$ and cause variables $Y \in \mathbb{R}^{n \times 1}$。

1. Spectrolysis: Yes. $X$ Disaggregation of odd-value (SVD) or coordination matrix $X^TX$ Conducting characterization. $$ X = U \Sigma V^T $$ of which $V$ , the main ingredient.

2. Construct the main ingredient: Scores$Z$。 $$ Z = XV $$ Because the main ingredient is active, that is, $Z^TZ$ It's an angle matrix, which eliminates the problem of multiple co-linearity.

3. Cut and return: Usually we just take the front. $k$ Main component of the maximum characteristic value (most variance of explanation) $Z_k$I'm sorry. Do it. $Y$ About $Z_k$ Minimum 2x2 returns: $$ \hat{Y} = Z_k \hat{\gamma} $$ $$ \hat{\gamma} = (Z_k^T Z_k)^{-1} Z_k^T Y $$

4. Revert Parameters: Map the regression coefficient back to the original space: $$ \hat{\beta}_{PCR} = V_k \hat{\gamma} $$

1.2 Methodological deficiencies

PCR sounds tempting: it's eliminated the cosmopolitanity, it's preserved. $X$ . But it has a logical fatal wound:

The process of extracting the main ingredient is based only on the variable $X$ The co-ordinated structure, completely ignoring the variables. $Y$。

In the PCA, we select the main ingredient by "maximizing the variance". But...The difference doesn't mean we're going to be together. $Y$ RelevantI'm sorry. There may be a situation where:$X$ The direction is very small (and therefore discarded in PCR), but the direction contains $Y$ Most of the information.

As Ali S. Hadi and Robert F. Ling (1998) are The American Statistician As noted above, if the main components of the explanation of the variable are not related to the response variable, the PCR may not even be as effective as the discarding of the variable. The article gives an example: the former p-1 PCs have nothing to do with variables, while the last PC explains all variations due to variables. The reason is that the PCA relies only on X ' s co-conforming structure, while ignoring Y ' s information.

2. Minimum 2x2 regression (Partial List Review, PLSR)

In order to solve the problem of PCR “Only X does not look Y”, the lowest 2-fold return (PLSR) was created. Its core idea is:In searching for potential variables, let it explain as much as possible $X$ And the mutation of it makes it as explained as possible. $Y$ ..the variation of。

2.1 Optimizing objectives

Suppose we're looking for a weight vector. $w$(Fulfilled) $|w|=1$) , construct potential variables $t = Xw$I'm sorry. The target function for PLSR is maximized $t$ and $Y$ Other Organiser

$$ \max_{w} \text{Cov}(Xw, Y)^2 = \max_{w} \text{Var}(Xw) \cdot \text{Corr}(Xw, Y)^2 \cdot \text{Var}(Y) $$

PCR and PLSR:

• PCR: Maximize $\text{Var}(Xw)$。
• PLSR: Maximize $\text{Var}(Xw) \times \text{Corr}(Xw, Y)^2$I'm sorry. (Note:$\text{Var}(Y)$ (A constant)

So this is more intuitive: the PLSR is trying to find a balance that it's found to contain both. $X$ Main structure (large difference) and $Y$ Highly relevant. PLSR brought The main change is that the extraction of the ingredient is based not only on variance but also on the correlation with the variable.。

2.2 Summary of algorithms (NIPALS thought)

PLSR solvers usually use NIPALS algorithms or SIMPLS algorithms. For the single variable $Y$The iterative process is broadly as follows:

1. Calculate $X$ and $Y$ Accompany vector $w = X^T Y$。
2. Normalization $w \leftarrow w / |w|$。
3. Calculate score vector (Score Victor)$t = Xw$。
4. Calculate $Y$ Yeah. $t$ the load of the $c = Y^T t / (t^T t)$。
5. Calculate $X$ Yeah. $t$ the load of the $p = X^T t / (t^T t)$。
6. Detachment: From $X$ and $Y$ minus the explanation of this ingredient: $$ X_{new} = X - t p^T $$ $$ Y_{new} = Y - t c^T $$
7. The above steps are repeated using the disability matrix until sufficient ingredients are extracted.

Ultimately, we have modeled in the form of: $$ X = TP^T + E $$ $$ Y = TQ^T + F $$ (of which) $Q$ Often directly associated with regression coefficients).

2.3 Geometric interpretation

• OLS The search is for $X$ Column Space Distance $Y$ Recent projection.
• PCR First, find the largest square subspace in this space, then project it in the subspace.
• PLSR It's a rotational axis, which points to the longest-spreading of data, while the new axis is biased towards the same. $Y$ The largest direction of the gradient.

3. Summary and comparison

Dimensions	Return of main ingredient (PCR)	Minimal 2x2 regression (PLSR)
Basis for extracting constituents	Only rely $X$ Difference ($\text{Var}(X)$)	Consider $X$ Differences $X,Y$ Relevance ($\text{Cov}(X,Y)$)
Monitoring learning	No (no monitoring of first step)	Yes (using target variable Y)
Variable Selection	It's actually a hard cut.	It's equivalent to a soft weight.
Apply scene	The noise is mostly in the $X$ and $Y$ Association $X$ ..when the main variation is	The prediction is directed,$X$ When multiple co-linears exist internally

In most practical applications (especially chemical metrology, spectrometry) PLSRs tend to perform better or evenly than PCRs. By introducing information on variables, PLSRs often achieve the same prediction precision with fewer components, thus obtaining a simpler model (Parsimonious).

We suggest PLSR as a replacement for PCR for a straightforward reason: the former is statistically more superior, and it uses information from X and Y instead of using its own coordination structure only. PCR has the advantage of being able to deal with multiple co-linearity, but statistically, the statistical nature of the PCR is not reliable.

When actually used, PCRs are not always easily susceptible to the situations described above (the former p-1 ingredient is not related to Y, the last ingredient). In many cases (especially chemical data) the main ingredient can explain some of the built-in structures of the variables, which are often associated with the variables. In theory, however, PCR lacks the logic of return. PLSR retains the PCR's “goods” (the principle being similar) while taking account of the variables, which are more consistent with the purpose of return.