Advanced Linear Regression: Goodness of Fit, Model Selection, and Collinearity

Evaluation of the regression equation

Proposed eugenicity indicator

RSS

The difference of the model error item$\sigma^{2}$ Reflects the size of the error and observation error of the model

Definitions:$\hat{e}=y-x\hat{\beta}$It's an estimate of the difference vector. Definitions: $RSS=e^{\Lambda}\acute{e}^{\Lambda}=\sum_{i\cdots1}^{n}e_{i}^{\Lambda2}$ It's the squared balance of the cripple. RSS reflects the ability of models to aggregate data.

Because$RSS$The more you incorporate the variables, the smaller they are, the less we can use the RSS for comparison between the same selected models of the number of variables, and the less meaningful the rest (and not for comparison of RSS in multiple regression equations)

Theoretically:

$RSS=y^{\prime}(I-X(X^{\prime}X)^{-1}X^{\prime})y;$
$\hat{\sigma}^2=\frac{RSS}{n-p}$Yes.$\sigma^{2}$A non-event estimate
$RSS = y'y-\hat{\beta}'X'y.$ is the difference squared and equals the total squared and the reduction of the squared and the reduction of the sum of the squared

$R^{2}$

Definition: Returns squared $SS_{\text{回}}=\hat{\beta}^{\prime}X_{c}^{\prime}y=y^{\prime}X_{c}(X_{c}^{\prime}X_{c})^{-1}X_{c}^{\prime}y$ Definitions: Amending the total squared to $SS_{\text{总}}=(y-\hat{\alpha}^{\wedge}\mathbf{1})^{\prime}(y-\hat{\alpha}^{\wedge}\mathbf{1})=(y-\bar{y}\mathbf{1})^{\prime}(y-\bar{y}\mathbf{1})$ Attention, we need to go into advanced centralization here to start our research here, so all the tags are centralized tagging formats.

Definitions: Decision coefficients$R^{2}$Yes $R^2=\frac(mathrm{SS}{\cHFFFFFF}{\cH00FF00} What's wrong with you?\text{total}, $ Actually, we'll use it in front.$RSS$Explain the degree of model alignment.$R^{2}$It's the same thing. It's just...$R^{2}$The bigger the better.

I can see that. $R^2$The most important value is one, the closer one is the better the match, and of course he's not below zero.

There are amendments.$R^2$ Because we're thinking about it.$R^2$ Changes in (positive) sample capacity$R^2$ We usually use both volumes in statistical analysis.

In essence, it's not different from RSS, but it's comparable in multiple models, which is the advantage of RSS, and we're now almost off the balance and assessing the degree of alignment.

$R^2$No value applied when the regression equation does not have a cutout

$R^2$Quantitatively describes the extent to which the variation of the explained variable is described or explained by the regression variable, so it is used frequently in applications$R^2$ 。

But it's too dependent.$R^2$ It could put people in a difficult position, and in practical application, "maximize"$R^2$ There is little economic or statistical significance. On the contrary, whether or not to add a variable to the multiple regression depends on whether the inclusion of this variable will allow a better estimate of the causal effects of interest, i.e., the reference behind the reference.Linear regression base and the “Selection of Model Indicators” section Linear regression base and the “Information quantity indicators” section of the

Return standard error SER

Standard regression error (Standard Error of the Regresion, SER) is the standard error estimate for the regression, with the same error for SER and the regression result, so we also measure the degree of fragmentation of the regression error with SER;

We use the disability to estimate the SER formula as $$SER=s_{\hat{u}}\text{,其中 }s_{\hat{u}}^2=\frac{1}{n-2}\sum_{i=1}^{n}\hat{u}_{i}^2=\frac{\mathrm{SS}R}{n-2}$$

SER measured the size of the deviation from the regression line, i.e. the size of the regression error Too big for the SER. It's very bad.

Coefficient S.E.

For a sample of a whole number of times, each sample size is$n$So each sample has its own average, and the standard deviations for these averages are called standard errors.

For the regression coefficient, SE reflects the estimated effects of our coefficient in the same unit, and too large SE means that the coefficient estimate is not accurate and therefore meaningless.

The standard error is itself a term, with different interpretations of coefficients and regressions, and the standard error is a difference in the standard estimate used to measure the effects of the estimate

Select Model Indicators

We'll use it in this chapter.MSEPThe average prediction error instead of the normal average error is the measure of the prediction, just to make a distinction, and there is no change in the actual definition.

Full and selected models

The whole model is the regression model we were thinking of. $$y_{i}=\beta_{0}+\beta_{1}x_{i1}+\cdots+\beta_{p-1}x_{i,p-1}+e_{i}$$ And we're just choosing a part of the model that we're going to choose from to make a new model. $$Y=\beta_0+\beta_1X_1+\cdots+\beta_{q-1}X_{q-1}+e$$ It's perfectly normal that when we switch from a whole model to a model, the variables associated with the prediction deviations change, and we need to focus on the average E, the difference Var, the average projection error MSP. Same as the RSS that was used to evaluate errors. $$MSEP(\hat{y})=E(y-\hat{y})^2=E(\hat{e}^2)$$ We can do the full model and the selection of the equation error.

Even if the whole model is correct, the chosen model can get smaller MASS and smaller regression factors, but it's cost-effective.
Even if the whole model is correct, the selection of models sometimes increases the accuracy of the prediction, often because a regression factor is difficult to estimate accurately. So, it's good for us to have a few of the self-variant elements that are not very active or difficult to observe in the regression equation, so we're going to analyze the different sub-variant sets.Select the best subsetIt's what we do behind us. Now we're going to start looking at the criteria for evaluating the regression equation.

$RMS_q$Guidelines

$RMS_q$The idea of the code is to use the difference squared and RSS because RSS reflects the actual data and theoretical predictions of deviations, and even smaller RSS means a better alignment.

We'll take the model RSS as$RSS_q$ Based on the formula, here are two models.$RSS_q$ $$RSS_q=y^{\prime}(I-X_q(X_q^{\prime}X_q)^{-1}X_q^{\prime})y$$ $$RSS_{q+1}=y^{\prime}(I-X_{q+1}(X_{q+1}^{\prime}X_{q+1})^{-1}X_{q+1}^{\prime})y$$ Directly deviating can be done. $$RSS_{q+1}\leq RSS_q$$ The more the model expands, the better? This is clearly contrary to the point of our selection model, so we're used to introducing a penalty factor. $$RMS_q\overset{}{\operatorname*{=}}\frac1{n-q}\overset{}{\operatorname*{RSS}}_q$$ I can do it now.$RMS_q$The smallest rule is to choose the model we want. It's called average disability squared and standard.

$C_p$Guidelines

$C_p$ Guidelines are considered in terms of the accuracy of the projection of the variables $$MSEP(\hat{y})=E(\hat{y}-y)^2=Var(x_q^{\prime}\hat{\varphi}_q)+(E\hat{e})^2$$ It's the statistics we want to use. $$C_p=\frac{RSS_q}{\hat{\sigma}^2}-(n-2q)$$ $C_p$ The smaller the model, the more worthwhile it is to choose.

Indicators of volume of information

$AIC$Guidelines

We'll give it straight here.$AIC$He's based on the most obvious model code. $$AIC=-2\ln(\text{模型似然度})+2(\text{模型自由参数个数})$$ Akaike, a Japanese statistician, proposed a code for the volume of information of Akaike. Get the smallest one.$AIC$And that set of parameters is the best, and he's using very broad statistical guidelines, and a lot of areas are using them to study best models. For selecting models $$Y=\beta_0+\beta_1X_1+\cdots+\beta_{q-1}X_{q-1}+e,~e\sim N(0,\sigma^2I)$$ The function appears to be $$L(\varphi_q,\sigma_q^2|y)=(2\pi\sigma_q^2)^{-\frac n2}\exp\left{-\frac1{2\sigma_q^2}\sum_{i=1}^n(y_i-\sum_{j=0}^{q-1}\beta_jx_{ij})^2\right}$$ The logarithmic function is $US$ (\,\x\^2|)=-\frac{((22(2)=((2)=((2piv)=(1})=(})=(})=(})= (=)= (= \ ^ ^ ^)= (= = ^ ^)= (=)= (= = = ^ ^ ^ ^)= (= } })= (= = } } })= (1} } } } } } } })= (1} } } } } })= (} } } } } })'(y-X_q\varphi_q)$$ 代入 $$\begin{aligned}\hat{\varphi}q&=(X_q^{\prime}X_q)^{-1}X_qy\\hat{\sigma}q^2&=\frac{RSS_q}n\end{aligned}$$ 得到 $$\ln L(\hat{\varphi}{q},\hat{\sigma}{q}^{2}\left|y\right)=\left[-\frac{n}{2}+\frac{n}{2}\ln(\frac{n}{2\pi})\right]-\frac{n}{2}\ln(RSS_{q})$$ 因此 $$AIC = n\n(RSS {)+2q$ We'll use this directly from now on.$AIC$The formula is fine. Akaike's information we haven't learned, so it's hard to understand his mind.

BIC Guidelines

Bates Information Guidelines Calculate formulae as $$\mathrm{BIC}=-2\cdot\ln(\hat{L})+k\cdot\ln(n)$$ We don't have much of a presentation here. We'll do it later.

Assumptions test

The purpose of this chapter is to study the problem of prediction of variables in the linear statistical model, or at the core of the method of the lowest two-fold estimation, and then we'll give you a little bit of a little bit of the prediction of variables in the linear statistical model, which is the common way of organizing it. We're still studying the lowest-based two-fold estimate of the experience that we've got back to the equation. In fact, given the speciality of the hypothetical test, it's difficult to do the research in other cases. The main issues addressed in this chapter are as follows:

Re-entry equations for re-entry.
Visibility test for regression coefficients
Anomalous Point Test The hypothesis test is introduced because the experience that has been gained is not really the relationship between the equation and the variable.

General linear assumptions

Basic thinking

We're starting with our core linear regression model. $$y=X\beta+e,e\sim N(0,\sigma^2I)$$ of which$X$Yes.$n\times p$The matrix, which is...$n$Sub-observation $p$Parameters For general linear assumptions, we add linear assumptions for parameters to be estimated $$\boldsymbol{H}{:}\boldsymbol{A\beta}=\boldsymbol{b}$$ It's a minimum 2-fold estimate. of which$A$Yes.$m\times p$. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . $m$The number of bound equations We can easily give the balance of the difference before we increase the constraint. $$RSS=(y-A\hat{\beta})^{\prime}(y-A\hat{\beta})=y^{\prime}(I-X(X^{\prime}X)^{-1}X^{\prime})y$$ And of course we can give the balance of the post-restriction balance. $$\hat{\beta}_{H}=\hat{\beta}-(X^{\prime}X)^{-1}A^{\prime}(A(X^{\prime}X)^{-1}A^{\prime})^{-1}(A\hat{\beta}-b)$$ $$RSS_H=(y-A\hat{\beta}_H)^{\prime}(y-A\hat{\beta}H).$$ 增加了约束条件后非常明显的会增大残差的平方和也就是 $$RSS& US$US$ If the parameters of the model do satisfy the constraints, then the ordinary and limited amount of the post-conditional disability should be increased, so when the squares of the residual increase are increased to a certain degree, we can reject the assumption.

Core Theorem

For normal linear regression models$y=X\beta+e,e\sim N(0,\sigma^2I)$ Yes.

${RSS}/\sigma^{2}\sim\chi_{n-p}^{2}$
Assumptions${A\beta}={b}$ $(RSS {)H}-RSS{}^{})/\sigma^2 \sim \chi_{_m}^2;$
$RSS$and$RSS-RSS_H$ Independence
Assumptions${A\beta}={b}$ $F_{H}=\frac{(RSS{H}-RSS)/m}{RSS/(n-p)}\sim F{_{m,n-p}}$

Brief form

We gave a simple calculation of RSS, which is the difference squared and equals the total squared and the reduction of the reduction of the squared. $RSS = y'y-\hat{\beta}'X'y$$ 使用类似的手法还是可以计算$RSS_H$ 带入前面的$F_H$有 $$F_{H}=\frac{(A\hat{\boldsymbol{\beta}}-b)^{\prime}(A(X^{\prime}X)^{-1}A^{\prime})^{-1}(A\hat{\boldsymbol{\beta}}-b)/m}{RSS/(n-p)}$$

Quit Fields Confirm

The basic idea before us is that when the balance is squared and increased to a certain degree, we can reject the assumption. And that's what we're testing for, you know, the numbers are so big that we can just say no to the assumptions. So we should choose a single-side hypothetical test and reject the field selection as $F H}>\mathbf{F}_{m,n-p}(\alpha)$$

Concluding remarks

Why do you call it a general linear hypothesis? It's because many of the assumptions that follow are converted to some particular case of this general linear hypothesis, and then we apply the formula here to solve it. For example: $ \begin{aligned}y 1&=\beta_1+e_1\y_2&=2\beta_1-\beta_2+e_2\y_3&=\beta_1+2\beta_2+e_3\end{aligned}$$ 检验 $${H:\beta 1=beta 2} $ You can get a shape change. $\beta_1-\beta_2=0$ Which means...$n=3,p=2,m=1$General linear assumptions

Or his promotion. $ \begin{aligned}y 1&=X_1\beta_1+e_1,e_1\sim N(\theta,\sigma^2I_{\mathrm{n}1})\y_2&=X_2\beta_2+e_2,e_2\sim N(\theta,\sigma^2I{\mathrm{n}2})\end{aligned}$$ 检验 $$\boldsymbol{H}:\boldsymbol{\beta}1=\boldsymbol{\beta}2$$ 能看出参数这里等价于 $$I{p}\beta{1}-I=$0 So this is...$n=n_1+n_2;p=2p;m=p$General linear assumptions

Re-entry equations for re-entry.

Basic thinking and testing statistics

What are we trying to prove? It is certainly proof that the regression equation is really significant; so based on the idea of the hypothetical test, our original assumption is that all the regression coefficients are zero.$\beta_0$It's constants that don't add zero to the test here. $H:\beta scdots=beta sc{p-1}}=0$$ 如果我们同意了原假设就意味着所有自变量的影响都不重要此时我们的约束可以看作如下的形式 $$A=(0,I{p-1}),b=0\quad(A\beta=\mathbf{b})$$ 这就化简成我们上一节研究的一般线性假设问题了通过一系列化简运算我们得到原本的检验$F$统计量变形为如下 $$F \text{back}=\frac{SS text{back}/(p-1)} We can judge whether we reject the original hypothesis by formula.

Analysis of the perspective of the equation

Under the original assumptions of this section, $RSS H}=y^{\prime}y-\beta{0}^{^{\star}}\mathbf{1}^{\prime}y=\sum_{i=1}^{\mathrm{n}}(y_{i}-\overline{y})^{2}$$ 也就是在本节中 $$RSS_H=TSS$$ 又 $$TSS=RSS+SS \text{ So this section's test statistics can be considered as The test of statistics compares the two parts when the return to squares and the greater error of the test, rejecting the original hypothesis that the return equation is significant.

If we choose to accept the original hypothesis, This means that the error in the model is the one that's different from the one that's different.$Y$The impact can be ignored. The possibility is as follows:

The model is very different, and it's possible that some of the variables have been missing, some of the variables that have returned to the variables are not linear.
Back to the variable is true.$Y$The impact is small.

Visibility test for regression coefficients

We can determine the re-entry equation's re-entry profile.$Y$It depends on a range of self-variant variables that we have chosen, but it is not possible to rule out that some of the self-variant variables are not actually being used.$Y$Dependency, that's hypothetical. $US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$$US$US$US$US$US$US$US$US$$US$$US$US$US$US$$US$US$US$$US$$US$$US$US$US$$US$US$US$US$$US$$US$US$$US$$US$$US$$US$$$$$US$$$US$$$US$$$$US$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$...$$$$$$$$$...$$$$$...$$$$$$$$$...$$$$$...{{i}}:{\beta}Other Organiser Here we're going to look at the hypothetical test of this hypothesis.

Basic thinking

We can still choose to see this as a special case of a general linear hypothesis. $$\beta_i=0\Leftrightarrow A\beta=\mathbf{0},A=(0,...,0,1,0,..,0)$$

Then bring in the formula we're using in the first section. $$F_{H}=\frac{(A\hat{\boldsymbol{\beta}}-b)^{\prime}(A(X^{\prime}X)^{-1}A^{\prime})^{-1}(A\hat{\boldsymbol{\beta}}-b)/m}{RSS/(n-p)}$$

Core thinking

The calculation is still too complicated. Let's do something simple; look at the obvious test of regression coefficients in a different perspective. We know the minimum 2x2 estimate of the unknown parameter is satisfied. $$\hat{\beta}\sim N(\beta,\sigma^2(X^{\prime}X)^{-1}).$$ When the disability is simple, you can turn back to the nature. If I remember $$C_{p\times p}=(c_{ij})=(X^{\prime}X)^{-1}$$ Then it is. What's that?i\sim N(\beta_i,\sigma^2c{ii}).$$ 因此当假设$H_i:\beta_i=0$成立的时候有 $$\frac{\hat{\beta}i}{\sigma{\sqrt{c{ii}}}}={N}(0,1)$$ 又因为我们知道 $${RSS}/\sigma^{2}\sim\chi_{n-p}^{2}$$ 因此有 $$t_i=\frac{\hat{\beta}i}{\hat{\sigma}\sqrt{c{ii}}}\sim t_{n-p}$$ 其中$\hat{\sigma}^2=RSS/(n-p)$ 非常自然的我们应该使用双侧拒绝域的$t$检验选择的拒绝域为 $$|t_i|\geq t_{n-p}(\alpha/2).$$

Special analysis

We know. $ \\mathrm{Var} (hat(beta})i)=\sigma^2c{ii}$$ 所以可以给出标准误差 $$I'm sorry, I'm sorry, but I'm sorry, but I'm sorry, but I'm sorry, but... All you have to do is change the estimate.$\hat{\sigma}$ That's ours.$t$The test is... The lowest 2x2 estimate and the difference between the standard error The very natural thought is that if you're a variable,$X$For regression variables$Y$Without significant influence, you can be detached from the regression equation. And then you can use the rest of the regression from the variable to re-entry, and that's the updated coefficient, and that's the idea of chapter 5 to re-entry from the variable to the selection of the variable.

Anomalous Point Test

The sum of the anomaly is the point of departure from the main data, and this section is the one where we study how to study and judge the anomaly. Here we use the idea of a return equation to study anomalies, which is to use the handicapped mind. Use the size of the difference in the estimated value of the entire model to determine whether this point is abnormal The idea of the anomaly test in pre-processing of data is different.

Mean drift linear regression model

The normal linear regression model is as follows: $$y_{i}=x_{i}^{\prime}\beta+e_{i},e_{i}\sim N(0,\sigma^{2})$$ If one of the data is abnormal, he's bound to be off the equation, which is, $$y_{j}=x_{j}^{\prime}\beta+\eta+e_{j}$$ His matrix is... $$y=X_i\beta+d_j^{\prime}\eta+e$$ We call this model the mean drift linear regression model. Directly gives an estimate of the minimum two-fold of the average drift linear regression model. $$$200,000}=\hat{\beta}_{(j)},\eta^{}=\frac1{1-h_{jj}}\hat{e}$ Where's the £2500beta?{(j)}$ 为从原回归模型中剔除第$Minimum 2x2 estimate for j$ group data $h_{jj}$Yes. $H=X(X^{\prime}X)^{-1}X^{\prime}$ No. No.$j$Diagonal $e_j$It's the first one from the original regression model.$j$A cripple.

Test statistics

We're dropping the complicated extrapolation and giving the test statistics directly. If you're assuming$H:\eta=0$ There is. $$F_j=\frac{(n-p-1)r_j^2}{n-p-r_j^2}\sim F_{1,n-p-1}$$ Of which$r_j$The student disability we want to test. $n$It's the number of observations. $p$is the number of the variable that returns to the variable. For random$\alpha$ We use a single-side hypothetical test, which means we reject the field as $F j=frac{(n-p-1)r n-p-r i}>F_{{1,n-p-1}}(\alpha)$$ 这是因为我们非常自然的思想学生化残差越大越偏离越应该拒绝因此我们选择单侧假设检验由于$t$分布和$F$分布的关系可以给出等价的检验统计量（双侧的） $$t_j=\left(F_j\right)^{1/2}$$ 拒绝域为 $$\left|t_i\right|\geq t{n-p-1}(\alpha^{}/2).$$

Some notes

The only way we can handle this is by one anomaly. But the number of anomalies in reality is uncertain, and if there are too few, then there are some points that are not suspected and wrongly introduced, and if there are too many assumptions, there are no normal points. The anomaly test is a lead, and there's a lot to follow.

Projections due to variables

The prediction of regression models is a very simple question. Just like we did in Bayesian statistics, put him in the back of the hypothetical test for a simple introduction.

Simple theoretical presentation and point prediction

In theory, we have two predictions. One is the average of projections. $$Ey_0=x_0^{\prime}\beta $$ It's not necessary to consider random errors at this point. The other is the projection. $$\hat{y}_0=x_0^{\prime}\hat{\beta}$$ Unfortunately, we can't know what the error is, and we give the specifics of our predictions based on the principle of non-selectivity, which is the actual application we're using. They're the same form, but they mean different things. We'll come to a conclusion that the projection is the lowest-short-linear no-speculation.

Inter-sector projections

Give theorem. $$\hat{y}_0-y_0\sim N(0,\sigma^2(1+x_0^{\prime}(X^{\prime}X)^{-1}x_0))$$ He's only got a random error subject to a uniform estimate of zero. So export the core amount $$$\frac(hat)0-y_0}{\hat{\sigma}\sqrt{1+x_0^{\prime}(X^{\prime}X)^{-1}x_0}}\sim t{n-p}$$ 所以对于给定的显著性水平可以给出预测区间 $$\left[\hat{y}0-t{n-p}(\frac\alpha2)\hat{\sigma}\sqrt{1+x_0^{\prime}(X^{\prime}X)^{-1}x_0},\hat{y}0+t\n-p\ (\frac\alpha2) \sqrt{1+x 0{${${\(x^)\right]$ It's actually a situation that's estimated in mathematical statistics.

Selecting the regression equation

There should be some sort of problem with the original equation of return.Linear testing of regression models And we're not looking at this in online statistics, and we're defaulting on linear models, just taking the choice of retrogressive variables.

In chapter three, we're going to describe the gradual regression of MSEs in the context of optimizing the convergence of linear variables by removing them, and we're going to be going to tell you how to judge and eliminate the convergence of linear variables, and eventually achieve a gradual return equation, and we're going to need to use an entire chapter to describe the problem, and we need to look at a number of small questions to give us the final picture.Progressive return approach

Calculate all possible returns

For one of them, there is.$p-1$A linear regression model for a variable, a subset of a variable, and a factor.$y$It's a linear regression model, so we actually get it.$2^{p-1}-1$Linear regression model We're looking for a reasonable order and method of calculation, and controlling the increase in the number of calculating indices and controlling errors is what we're looking at in this section.

$S_p$Sequence Method

Let's put$p$A sub-assemble of variables$p$Vector$(u_1,u_2,...,u_p)$To indicate which$u_i=0\text{或}1$ Which means the subset does not contain this variable. Now calculate all possible returns from where they started: Along the edge of the Govt-Step, through every vertex without repetition. I can see that.$S_p$The sequence is characterized by: Each regression subset happens once, and one variable is different from the next. Let's go down to the next one.$S_p$The method of construction of the sequence First of all, we'll define it as follows: $S_p$Sequence $i$This is the introduction of variables$x_i$ $-i$Means the removal of variables$x_i$ That's... $$S_{2}={1,2,-1}$$ Which means that the first calculation is only a variable.$x_1$And then the second time, it was introduced.$x_2$ Third Excerpt$x_1$ Just do it.$x_2$♪ And the return ♪ Here's what we'll do if we start with zero.$S_p$Sequence First definition $T_i$Yes.$S_i$Rewind and change the sequence of the symbol, which is $S_{2}={1,2,-1}$ $T_{2}={1,-2,-1}$ Now we give you the iterative rule. $S_1 ={1}$ $S_2={S_{1}2T_{1}}$ $S 3}S 2}, 3, T 2} Just push it like that.

Matrix to Change

To calculate all possible returns, we need to give an algorithm that works. For the square. $$A_{n^*n}=(a_{ij})$$ Introduce a new Square$B=(b_{ij})$ Remember $$\begin{cases}b_{ii}=1/a_{ii}\b_{ij}=a_{ij}/a_{ii},i\neq j,j=1,...,n\b_{ji}=-a_{ji}/a_{ii},j\neq i,j=1,...,n\b_{kl}=a_{kl}-a_{il}a_{ki}/a_{ii},k\neq i,l\neq i\end{cases}$$ It's called$a_{ii}$It's a central axial change.$B=T_iA$ We give the following nature for the matrix's elimination and transformation.

$T_iT_iA=A$
$T_iT_jA=T_jT_iA$
$A=\begin{bmatrix}A_{11}&A_{12}\A_{21}&A_{22}\end{bmatrix}$ 为$q$阶方阵 $T_{1}T_{2}...T_{q}=\begin{bmatrix}A_{11}^{-1}&A_{11}^{-1}A_{12}\-A_{21}A_{11}^{-1}&A_{22}-A_{21}A_{11}^{-1}A_{12}\end{bmatrix}$

Now we're going to use the matrix's digestive transformation to study linear regression models. $$y=X\beta+e,E(e)=0,Co\nu(e)=c^2I$$ Put$X$- Yes, I do.$$X=(X_q,X_r)\quad\beta=(\varphi_q^\prime,\varphi_r^\prime)$$Remember $B=X'X=\begin{pmatrix}X'q\X'q\end{pmatrix}(X_q,X_r)=\begin{pmatrix}X'X_q&X'X_r\X_r'X_q&X'X_r\end{pmatrix}=\begin{pmatrix}B{11}&B{12}\B_{21}&B_{22}\end{pmatrix}$$ 则有 $$A=\begin{pmatrix}B&X^{\prime}y\y^{\prime}X&y^{\prime}y\end{pmatrix}=\begin{pmatrix}B_{11}&B_{12}&X_q^{\prime}y\B_{21}&B_{22}&X_r^{\prime}y\y^{\prime}X_q&y^{\prime}X_r&y^{\prime}y\end{pmatrix}$$ 那么就有 $$\left.T_{1}T_{2}\cdots T_{q}\boldsymbol{A}=\left(\begin{array}{ccc}{\boldsymbol{B}{11}^{-1}}&{\star}&{{\boldsymbol{B}{11}^{-1}\boldsymbol{X}{q}^{\prime}\boldsymbol{y}}}\{\star}&{\star}&{\star}\{\star}&{\star}&{{\boldsymbol{y}^{\prime}\boldsymbol{y}-\boldsymbol{y}^{\prime}\boldsymbol{X}{q}\boldsymbol{B}{11}^{-1}\boldsymbol{X}{\bord0\shad0\alphaH3D}The \blur0}showing of the \blur0}A new \blur0 \blur0 \blur0 \blur0 \blur0 \blur0 \blur0 \blur0 \blur0 \blur0 \blur0 \blur0 \blur30 \blur0 \blur30 \blur0 \blur0 \blur0 \blur30 \bl30 \bl30 \bl30 \bl30 \bl30 \bl30 \bl30 \bl30 \bl30 \bl30 \bl30 \bl30 \bl30 \bl30 \bl30 \bl30 \bl \ \ \bl30 \bl30 \bl30 \bl30 \bl30 \bl \ \ \ frfr \ \ \ fr \ fr fr \ \ \ fr \ \ I can see that. $B_{11}^{-1}X_{q}^{\prime}\mathbf{y}$ The minimum two-fold estimate for the model selection. $y^{\prime}y-y^{\prime}X_{q}B_{11}^{-1}X_{q}^{\prime}y$ The difference between the squares of the model. So we can see it. Calculates the new minimum 2x2 estimate variable for the matrix elimination conversion problem If you want to introduce new variables, you can do a new matrix to change it. To remove a variable, you just have to do the distillation that you did when you introduced him again. So that each time a new regression model is calculated, it's the result of the last transformation. This iterative method is important for calculating the selection model.

Calculates the best subset returns

The very natural idea is that we should study together with the first two sections. One section has selected all the sub-sets for us, and the other section gives us the best way to assess them. I'll take both.

Return to the curve of the variable subset

For each of the components$q-1$Calculate statistics from a subset of variables$U_{q-1}$Value And what this value is assessing is that under the criteria used, the good and bad of this subset is actually assessing a bunch of the ones that are included in the calculations we've been making.$q-1$A selected model of the variable If$U_{q-1}$ The smaller the better, the smallest subset is called a tatter one. If$U_{q-1}$ The bigger the better, the bigger the one, the bigger the one. Now the Zen reflects the good and bad of this subset under the same criteria as we used. We'll make all the tarts one.$U_{q-1}$ Make a picture of it.$U_{q-1}$ Figure

Practical

The way we actually do it in practice is the way we think about it. It's just a combination of what I've been talking about.$U_{q-1}$ Figure For every evaluation, we can give them our own.$U_{q-1}$ Figure Then we can compare them manually to multiple models.$U_{q-1}$ I'm gonna figure out the best subset I can think of. It's actually a subjective Voting.

Progressive return

It's certainly good to calculate the best subsets back, but when the number of variables is quite large (generally 10 or more), although the matrix is reduced to a large amount of cost, the way backsup is too large to calculate the cost, so we're proposing a way to do not count all the subsets back, and gradual return is the most common of applications. The most excellent sub-regression also cannot address the issue of conjunctive linearity, and gradual regression can address some degree of concentrism, but in a highly linear situation, gradual regression cannot directly affect the high degree of correlation of the variable, and therefore not very well.

The idea of gradual return

Introduce a variable one by one, provided that its partial re-square and (PRSS) are tested to be significant and then the old variable is tested one by one, and the less significant variable is excluded. The effect of the last result is that all selected variables are significant and that variables that are not in the regression equation are not tested significant.

Back to Square

The deviation is the square and measures the degree of interpretation of the variation of the variable from a given variable. In the multiple regression model, the degree of interpretation of variability due to the variable after controlling other variables ' impact For the original model $${y=X_{q+1}\varphi_{q+1}+e}$$ Model after removing a variable $${y=X_q\varphi_q+e}$$ Back to Square$PRSS=RSS_q-RSS_{q+1}=\hat{\beta}_q^2(x_q^{\prime}N_qx_q)$

Remove from Variables

Is there anything to be removed from the existing model? We use the retrogressive coefficients in the hypothetical section. $${H_\text{剔}{ : \beta _ q }=0}$$ Then it's available. $$$$$500{\i}{\hat{\sigma}\sqrt{c\sim n-p}$ Of course he can be used.$F$Form of the test

In fact, the statistical test of this hypothesis is in the form of an equivalent price, which is based on a partial regression and extrapolation. $$\left.F \text{q)=\left(\begin{matrix}n-q-dond{matrix}\right.\right)\frac(hat{beta} (\bardsymbol{x}q^{\prime}N_q\boldsymbol{x}q)}{R\mathrm{SS}I'm sorry, I'm sorry. In general, it's a habit. We'll calculate all the variables for $F.\text{clip} Found the smallest of them, and if the smallest is significant enough (greater than the threshold), then the cut is over and can't continue. Except...

Introduction of variables from the

Or exactly the same idea. Let's do the test, see if we can assume. $$US$US$$ text{ \bardsymbol{\beta}q=\boldsymbol{0}$$ 是否会被拒绝我们还是使用刚才导出的那个检验统计量 $$\left.F\text{quote} =left(\begin{matrix}n-q-1\ \ \matrix}\frac(hat}{\bardsymbol{{\prime}q)}{R\mathrm{SS}I'm sorry, I'm sorry. Or, as we did just now, we're going to calculate all the variables that were not introduced and get a series of tests.

The square and the square must be significantly reduced, otherwise the negatives are not possible.$F$Test The molecule is a squared and form. This time, the biggest of these, if the largest is less than the selected prominence, is that all variables are not significant enough, or else a variable is selected, and the rest of them are redoing the process.

Other gradual thinking

Forward.

I've been introduced to the point where I know that all the amounts are tested.

Backwards

Put all the variables directly into the regression model and then pick them up one by one. Except for self-variant that contributes less to RSS Until there's no amount to be removed.

And then we'll talk about the regression hypothesis and the heretic.

The previous equation of return begins with the assumption that Gauss Markov, where we explain how we got this assumption, is not actually the form of our original scenario of return.

Basic assumptions for the regression equation

A regression problem, we want to predict some variable characteristics from some variables, which we're looking at in the form of.Linear regression base The "Line Regression Model" section of the paper is described.

So we give four basic returns assumptions that if they are not, then the return cannot be achieved.

When x is given, the condition of the disability is zero, which means we ask that the disability is irrelevant to the variable. If the disability is associated with the variable, it means it is not random, that is, it does not capture the characteristics that need to be used for prediction.
Sample i.i.d (time series data generally do not satisfy i.i,d) This is the basic assumption for statistical estimates.
No great anomalies. OLS is not a roust, so be careful with the abnormalities.
No complete junctive We need to decompose the matrix when we solve the OLS, and if we're fully communicative, we can't solve the coefficient.

It can be seen that the assumptions here do not meet the requirements of the Gass Markov hypothesis, and the difference is not equal at this point.

Especially if we go against the first hypothesis, which is internal.Linear regression baseThe various methods of regression analysis described in the report are problematic; we need to consider Broad linear regression "Regression of fixed effects" section Broad linear regression ..to analyze the tool variable returns section

If we go against the assumption of the same difference, Gauss Markov, we need to consider the question of the difference between the same and the different. Section

If we do not comply with the second assumption of the sample i.i.d, it will lead to time-series analysis and the creation of self-regression models.Linear Time Series Analysis Broad linear regression "Return models with disabilities" section

If we break the third hypothesis, we're gonna have to. Broad linear regression "Stand-up Robust Return Technology" section

If we do not comply with the fourth hypothesis, then we will be able to draw up the present "Estimation of regression parameters (under cominglinear)" Section

Same difference and differentiating

I'm giving you a hand. $X$ , About$e$The only assumption for distribution is that the average is zero (the first minimum two-fold scenario) and goes further if the difference in the distribution of the condition is not dependent on$X$, the error is described as the same.

This chapter deals with the theoretical meaning of the difference, the theoretical risk of which we have been assessing in the first place based on the hypothesis of the difference.

If for anything$i=1,2,...,n$I'll give it to you.$X_i$Time$u_i$Difference in the condition distribution $var(u_i|X_i=x)$As Constant And not dependent.$x$, the error item is referred to$u_i$is the same difference; otherwise, the error is described as an alien difference.

The difference is met by the Guass-Markov hypothesis.

OLS estimates are neutral and similar to normal, and this is also true for the heretic model.
The validity of the OIS estimate when the error is equal is the BLUE.
The same difference applies to the difference formula, which is the formula for the difference in the estimated amount. Especially, the standard SE formula we're applying is a staggered and robust equation, so the results of the software are directly trusted, whether or not they are.

Heterogeneity and Application

In practical models, there are few models that can determine the difference; therefore, we use as much as possible the various analytical methods that are robust in the differential.

Unfortunately, the variance analysis is completely unsettled, so we need to be extra careful when we do the differential analysis. Pilot design methodology

The existence of a homogenous margin requires a specific analysis of the problem and, where it is not possible to judge, the use of an exotic model is possible. There are special GLS models when the variance is known. Linear regression base Or we're considering using the above-mentioned standard error of the heterogeneity.

Alien Test

We know that the heretic model is problematic, and we need to know when there is an heretic effect, so we can take note of it.

When retrospect diagnosis is performed, visual graphics are available to observe the problem of the heterogeneity.

The formalization test is: Breush-Pagan (BP) test White test

Reverting the essence of analysis

What's the essence?

All regression analyses want to study the dependency between random variables, and we know that, of course,There's no connection between dependency and causation.The causal inference is a separate course in statistics.

Description by probability distribution, i.e. known$X=x$I'm not sure if you're going to be able to do this.$Y$Probability density of conditions$f_Y|X(y|x)$It describes$X$How much to decide?$Y$But we are more concerned with their expectations than with their probability density:$E( Y| X= x)$ This condition is a random variable.$Y$Yeah.$X$.

Why are we concerned with expectations because of:Because under the principle of average error, the function of the average error is the expectation of condition. Which means... $$g(X)=E(Y|X)\text{ 是 }\underset{g\in\mathbb{F}}{\operatorname*{\arg\min}}E[Y-g(X)]^2\text{ 的最优解}$$

In fact, there are different optimal decomposition functions under different criteria, and they are also widely applied in other regression models: For example, the best solution under the MAE code is the median, and the best solution under the Logit return is the largest entropy rule.$Sigmoid$Functions

So if we use MAE as an indicator to judge the fit, then the essence of regression analysis is that it uses all known parameter models.$\mathbb{F}$Yeah. $E( Y| X)$Approximate. And...We usually use linear functions as$E(Y|X)$And this is the approximate, this is the linear regression model.

Proof of theorem (certification of the best predictor)

The title is set by $X = (X 1, \dots, X p)'$ 为 $p$ 维随机向量，$Y$ 为随机变量。对任意函数 $f: \mathbb{R}^p \to \mathbb{R}$，有： $$E(Y - E(Y|X))^2 \leq E(Y - f(X))^2$$ 即条件期望 $E(Y|X)$ 是 $Y$ is the best predictor (under the mean of average error).

Proof: By expanding squared items and using the nature of the expectations:

Split the margin value to $Y - f(X)$ Split $(Y - E(Y|X)) + (E(Y|X) - f(X))$, and: $$E(Y - f(X))^2 = E\left[(Y - E(Y|X)) + (E(Y|X) - f(X))\right]^2$$ Expand squared and spread formulae according to squared $(a+b)^2 = a^2 + b^2 + 2ab$- It's okay. $$= E(Y - E(Y|X))^2 + E(E(Y|X) - f(X))^2 + 2E\left[(Y - E(Y|X))(E(Y|X) - f(X))\right]$$ Deal with cross-cutting items, use expectations"The Code of Rectitude Expectations"(Law of Iran Projects), extracting the conditions of the cross-section: $$E\left[(Y - E(Y|X))(E(Y|X) - f(X))\right] = E\left[ E\left[(Y - E(Y|X))(E(Y|X) - f(X)) \mid X\right] \right]$$ Because $E(Y|X) - f(X)$ It's about... $X$ function (recognised as $g(X)$) It may be suggested from the expectations of the conditions: $$= E\left[ (E(Y|X) - f(X)) \cdot E\left[Y - E(Y|X) \mid X\right] \right]$$ Imminent conditions for the implementation of the $E(Y|X)$ It's given. $X$ Down $Y$ , and therefore: $$E\left[Y - E(Y|X) \mid X\right] = E(Y|X) - E(Y|X) = 0$$ The cross item after the sub-item becomes: $$E\left[ (E(Y|X) - f(X)) \cdot 0 \right] = 0$$ Merge result. Cross-sections are zero-generation. $$E(Y - f(X))^2 = E(Y - E(Y|X))^2 + E(E(Y|X) - f(X))^2$$ Because $E(E(Y|X) - f(X))^2 \geq 0$(non-negative of squares), so that: $$E(Y - f(X))^2 \geq E(Y - E(Y|X))^2$$ The equals condition is the right and only the right. $E(E(Y|X) - f(X))^2 = 0$ Time, the equal sign is set. Since the square item is not negative, the equivalent is: $$E(Y|X) - f(X) = 0 \quad \text{几乎必然（almost surely）}$$That's... $f(X) = E(Y|X)$

Conclusion Expectations $E(Y|X)$ In the sense of minimizing the average error, yes $Y$ The best predictor. Any other basis $X$ , and then select the projection function for the $f(X)$ It's gonna cause a bigger average error unless... $f(X)$ and $E(Y|X)$ Almost everywhere.

Optimistic linear predictor

Set Random Vector $\begin{pmatrix} Y \ X \end{pmatrix}$ The expectations are: $$E\begin{pmatrix} Y \ X \end{pmatrix} = \begin{pmatrix} \mu_y \ \mu_x \end{pmatrix}$$ The matrix of the agreement is: $$D\begin{matrix}Y \X\end{matrix} =Sigma=\begin{matrix} \Sigma yy} & \Sigma_{yx} \ \Sigma_{xy} & \Sigma_{xx} \end{pmatrix} > $0.00 (Calling)

For Any $\alpha \in \mathbb{R}$、$\beta \in \mathbb{R}^p$, by: $E\left (Y-\left (y-\sigma yx)\Y-(alpha+\bita)^2'- What? That's...Optimistic linear predictorIs: $$\hat{Y} = \alpha^* + \beta^{*\prime}X = \mu_y - \Sigma_{yx}\Sigma_{xx}^{-1}\mu_x + \Sigma_{yx}\Sigma_{xx}^{-1}X$$

If Random Vector $\begin{pmatrix} Y \ X \end{pmatrix}$ Comply with multiple normal distribution: $US$\left (\begin{matrix}\mu end{matrix},\begin{matrix}\sigma y} & \Sigma_{yx} \ \Sigma_{xy} & \Sigma_{xx} \end{pmatrix} \right)$$ 则对任意函数 $f: \mathbb{R}^p \to \mathbb{R}$，有： $$E (Y-(y-\Sigma) ^ (Y-f) ^2 US$2 million under the distribution of multiple normals,The best linear predictor is the best predictor.。

Return to squared and decomposition

In multiple regressions, the total squares can be broken down to: $Y'Y = Y'MY + Y'(I-M)Y = SSR(X)+SSE$$, of which:

$SSR(X) = Y'My$ is the return to squared
$SSE = Y'(I-M)Y$ is the sum of the error squared

Return squared and can be further decomposed to: $$SSR(X) = SSR(X_1) + SSR(X_2|X_1) + \ldots + SSR(X_p|X_1, \ldots, X_{p-1})$$Definition of the condition squared $SSR(X_j|X_1, \ldots, X_{j-1})$ This post is part of our special coverage Syria Protests 2011.

Contained in Model $X_1, \ldots, X_{j-1}$ In the case
Add $X_j$ ♪ And the squared and the increased ♪

This breakdown reflects the contribution levels of the various self-variant variables in the multiple regressions:

$SSR(X_1)$: Independent contribution of the first variable
$SSR(X_2|X_1)$: under control $X_1$ The blogger says:$X_2$ and Add.1 - 2
By such extrapolation, each variable ' s conditional contribution excludes the impact of the preceding variable