Descriptive Statistics and Visualization: Measurement, Distances, and Statistical Graphics

Summary

Statistical analysis and descriptive statistical analysis

Statistical analysis: the process by which the regularity of the data is derived from the performance data

Statistical analysis is the core of the entire statistical research, and is more important and complex than data collection, and most of our course content falls within the scope of statistical analysis.

In the course of statistical analysis, statisticians have been divided about how to organize data. So, two main branches of descriptive statistical analysis and extrapolational statistical analysis were created.

Qualitative statistical analysis studies on how to organize and describe data (e.g., common discrete concentration trends, some visualization)
Instimulating statistical analysis uses samples to reverse the overall situation (e.g., parameter estimates and hypothetical tests)

They're both successful. They can't be lost.

Here we focus on descriptive statistical analysis; as a branch of the study of data shape, descriptive statistical analysis and data analysisExplored data analysisClose.

Some common data classifications

Measure

Depending on the units of the data, we give the following classification.

Interval (quantitative) variables: variables that can change continuously
Order variable: No clear quantitative changes, but grade changes
Classified variables: Variables are expressed in some categories as being equal between classes

Source of data

Depending on the source of the data, there are the following classifications:

Observation data: data collected for observations of the real world
Test data: data collected in experimental human control variables

Time-related

Cross-section data: data collected by objects at the same time (column is characteristic, row is object)
Time series data: data collected from observations of the same object at different times (column is characteristic, row is different times)
Panel data: multiple observations of multiple time (lists of objects and characteristics, rows of different time) with characteristics of the first twoThe regression analysis of panel data will certainly involve highly relevant observations, i.e., non-I.I.D samples, which will seriously affect the analysis.

Distance and Similarity Coefficient

Distance

Definitions

As we have studied in the general analysis, the following definition of distance is required: SeeGeneral analysis of communications and the “Definition of measuring space” section.

In part, we may be able to use this basic definition to determine the distance subjectively if we really need to. We have some of the more common distances down there in the cluster analysis.

Minkowski distance

$$d\left(x,y\right)=\left[\sum_{i=1}^{p}|x_{i}-y_{i}|^{q}\right]^{1/q}$$ Ming's distance is the widest distance used. When?$q=1$And then he was the absolute distance we learned. When?$q=2$And then, he was the O'Shea distance. When?$q=\infty$ ♪ When ♪ $d\left(x,y\right)=\max_{1\leqslant i\leqslant\rho}\mid x_{i}-y_{i}\mid$ It's called Chelby Shev's distance.

♪ With$q$Ming distance is becoming more sensitive to anomalies. Just like we did when we introduced the O'Shea distance.Multi-statistical inferences The "O'S" section.

Ming's distance also requires standardization before operation. The Ming-Zhi distance is applied to the calculation of the distance between orderly variables

Lance and Williams distance

When all the data is right, define the range of the Langham. $$d(x,y)=\sum_{i=1}^{p}\frac{|x_{i}-y_{i}|}{x_{i}+y_{i}}$$ It works well when it's dealing with data that are more skewed and contain abnormalities, and it's not about units.

Ma'am's distance.

Multi-statistical inferences The "Marzie Distance" section. In cluster analysis, we don't usually use marts, because the class is dynamically changing, and it's hard to give a definite matrix of the differences.

Hamming Distance

Hamming Distant It's a measure of two.Long String(or sequence) the measure of the number of elements at the same location. The idea is very simple, with the corresponding character (category) at the same location, and Hamming is one more distance, and it is sufficient to calculate each position of the whole string (or sequence)

Leveshtein Distance

Leveshtein distance, also known as editing distance, is one way to measure the difference between the two strings. It quantifys this difference by calculating the minimum number of operations required to convert a string into another string. He's a Hamming distance extension, which can be used for multiple categories.

Common operations are:

Insert: Inserts a character in a string.
Delete: Remove a character from a string.
Replace: Replace one character in a string with another.

We determine the distance of Leventein based on the number of times the string is changed, and he needs to use dynamic planning to solve it.

VDM Distance (Value Difference Metric)

Value Difference Metric (VDM, value difference measure) It is a similarity measure for classification tasks, which is particularly applicable to processingNominal Properties(i.e., type characteristics) data.

The heart of VDM is throughProbability differenceMeasuring the similarities of two samples in a particular characteristic

For each type characteristic, VDM calculates the probability of conditions under each target category for which the characteristic is different.
The difference in this characteristic between the two samples is determined by the probability differences in the conditions for which they each take the values.

VDM distance measuresCumulative extent of the difference in the distribution of target categories between the two samples with their values obtained in each characteristicI'm sorry. The greater the distance, the more significant the differences in the contribution of the characteristic values of the two samples to the classification results, the less likely they would fall into the same category.

To calculate VDM distance, the next steps are required, and the features we study are discrete, that is, a classification.

calculating the probability of conditions;for features$A$..of the value of the value$a$in the target category$c$The probability of the condition is: $$P(c\mid A=a)=\frac{\text{类别为}c,\text{特征取值为}a\text{的样本数}}{\text{特征取值为}a\text{的样本数}}$$

calculating single feature VDM distance; sample is$x,y$ Only study characteristics $A$ Two samples in the signature.$A$The values are given in the$a_x,a_y$ $C$Total is the target category $ \\mathrm{VDM}A(x,y)=\sum{c=1}^C\left[P(c\mid A=a_x)-P(c\mid A=a_y)\right]^2$$ 计算多特征的两样本 VDM 距离； $$\mathrm{VDM}(x,y)=\sum_{i=1}^m\mathrm{VDM}_{A_i}(x,y)$$

The distance measure of the mixture properties can be achieved by combining VDM distance with OSD distance, commonly known as HVDM; in some cases weighting of VDM characteristics is required to distinguish more important features

VDM is based on the probability of conditions, which may be quite inaccurate when the number of samples is small, and even a problem of zero probability exists.

VDM design targets are classification issues and are not available for return or task. By using the existing cluster centre as a pseudo-label, updating the new cluster results using dynamic cluster methods, recalculating VDM distances, and obtaining the final results over time, the cluster issues can be indirectly used

Similarity factor

Definitions

Unlike distance, the larger the similarity coefficient, the closer the two samples are, the smaller the distance means the closer; we can export the similarity coefficients by distance, but we can also redefine some similar coefficients. The similar coefficients are in general the same.$[0,1]$There are, of course, a few possible breaches.

Clindrum Cosine

Defines the similarity factor as $$\mathrm{cos}\theta_{y}=\frac{\sum_{k=1}^{n}x_{ki}x_{kj}}{\left[\left(\sum_{k=1}^{n}x_{kj}^{2}\right)\left(\sum_{k=1}^{n}x_{kj}^{2}\right)\right]^{1/2}}$$ It's the cosine of the cross between two vectors.At this point, we focus on the vector angle, not on the size of the value, which is very effective for the existence of thin high-dimensional data.

Related coefficient

Draws similarity using two vector-related coefficients $$cor=\frac{\sum_{i=1}^n(a_i-\bar{A})(b_i-\bar{B})}{\sqrt{\sum_{i=1}^n(a_i-\bar{A})^2\sum_{i=1}^n(b_i-\bar{B})^2}}$$ of which$a_i,b_i$It's a sample.$A,B$We've already regulated it.$\bar{A},\bar{B}$ It's vector.$A,B$The average of the sample is the average of the sample rather than the average of the characteristic.

In particular, we are very reluctant to use two vector-related coefficients to study sample similarities except for the following application scene.

Sample characteristics are time series, and at this point we can assume that the distribution is similar within the same sample and therefore the relevant coefficients can be used.
If the characteristics of the sample represent a certain distribution (e.g., the distribution of interest of a user, the pixel histogram of an image) e.g., user A, the interest in five commodity categories min $[5,3,1,0,2]$ User B rating interest in 5 commodity groups $[3,1,2,0,3]$ The correlation coefficient at this time implies similarity in preference.

Jaccard coefficient

Jaccard Index(Yakar Index) is an indicator used to measure the similarities between the two clusters. It's used to calculate the two pools. Similarityand especially for Diagonal characteristics or Pool type of data

For dual feature, we can calculate as follows, and the formula is $$J(A, B)=\frac{|A \cap B|}{|A \cup B|}$$

Intersection: indicates the position of both samples at the same level of feature as 1.
: indicates the position of the characteristic 1 in at least one sample of two samples

For the group characteristics, each sample is a series of random collections, calculating the number of elements that are combined and intersect, comparing, and the formula remains the same. $$J(A, B)=\frac{|A \cap B|}{|A \cup B|}$$

Relevant analysis

Brief description of the analysis

The simple purpose here is to analyze data to see if there's a relationship before several data sets and measure it. Relationship: When one or more variables change, the other variable that corresponds to it, although not certain, changes according to a certain pattern.

The relationship is not causal. The two variables he studied were equally equal.

The correlation between variables is divided into two types of functional and non-confirmity relationships, which obviously are more statistically studied than is the case with the latter, which requires a great deal of knowledge about the intrinsic characteristics of the event.

The vague definition of the relationship shows that he must be in the real world and in our statistics.

From the structure of the data studied, the analysis is divided into numerical data-related analysis, sequence data-related analysis, category data-related analysis, and the analysis of the data-related data of the classification data.

Three relevant coefficients

Related coefficient

The correlation coefficients for two random variables are a linear correlation between them.

The correlation coefficients are linearly relevant, even if there is a function, but non-linear; the correlation coefficient between the two random variables is 0, as if $X\sim N(0,1),Y=cosX,Z=X^2$ then $Corr(X,Y)=0,Corr(X,Z)=0$ We can verify that. When the correlation coefficient is$(0,1)$And sometimes, it proves that there is some degree of linear correlation, but not entirely.

Although the coefficients are only linear, he is still the most widely used indicator, and we have suggested that many other indicators do not replace him, except that they are too complex; and because they are not very complex. The incoherent and independent relevance of the 2D normal distribution is equal.

So, the correlation coefficient is the measure of correlation, not just the measure of linear correlation, when the whole is normal.

Recontribution factor

We want to be able to study the correlation between a random variable and a random vector and to paint it as a value. According to the foregoing, a very simple idea is to use a linear combination of random vectors to describe all his information and then study the relevance of linear combinations and random variables; we call the maximum correlation coefficient between linear combinations and random variables as a compound correlation factor. It's easy. $$00\ \l{\rho^l^rft}& =\frac{Cov^{2}\left(y,l^{\prime}x\right)}{V\left(y\right)\cdot V\left(l^{\prime}x\right)}=\frac{\left(\sigma_{xy}^{\prime}l\right)^{2}}{\sigma_{yy}\cdot l^{\prime}\Sigma_{xx}l} \ &\leqslant\frac{\left(\boldsymbol{\sigma}{xy}^{\prime}\boldsymbol{\Sigma}{xx}^{-1}\boldsymbol{\sigma}{xy}\right)\left(\boldsymbol{l}^{\prime}\boldsymbol{\Sigma}{xx}\boldsymbol{l}\right)}{\boldsymbol{\sigma}{xy}\cdot\boldsymbol{l}^{\prime}\boldsymbol{\Sigma}{xx}\boldsymbol{l}}=\frac{\left(\boldsymbol{\sigma}{xy}^{\prime}\boldsymbol{\Sigma}{xx}^{-1}\boldsymbol{\sigma}{xy}\right)}{\boldsymbol{\sigma}{xy}} \end{aligned}$$ 当时$l=\Sigma_{xx}^{-1}\sigma_{xy}$ 等号成立则有 $$\begin{gathered} \rho_{y}\cdot x =\max_{l\neq0}\rho\left(y,l^{\prime}x\right)=\rho\left(y,\sigma^{\prime}{xy}\boldsymbol{\Sigma}\right) == sync, corrected by elderman == @elder man I'm sorry, I'm sorry. The synthesis can give the conclusion.

When?$p=1$..times when the compound correlation coefficient is natural degradation as normal correlation coefficients.
The compound coefficient is 0, which means it's not relevant.
The compound correlation factor is constant for unit changes
The sum of the individual relative coefficients when the amount of the random vector is independent of each other and the squares of the compound correlation coefficients are equal Under the assumption of multiple normality, the compound coefficient for the sample is (the compound coefficient is much more likely to be estimated) $r y.x=\sqrt{\frac{s{xy}^{\prime}\boldsymbol{S}{xx}^{-1}\boldsymbol{s}{xy}}{s_{yy}}}=\sqrt{\boldsymbol{r}{xy}^{\prime}\hat{\boldsymbol{R}}{xx}^{-1}\boldsymbol{r}_{xy}}.$$

Offset factor

When we look at the most common correlation coefficients, we tend to calculate the Pearson correlation coefficients; in fact, he's affected by indirect correlation, so we usually call it the total correlation coefficients, and now we want to eliminate this indirect influence, and we study the most common correlations. Definitions $$\bardsymbol(Sigma)}{11}\boldsymbol{.}2\boldsymbol{=}\boldsymbol{\Sigma}{11}-\boldsymbol{\Sigma}{12}\boldsymbol{\Sigma}{22}^{-1}\boldsymbol{\Sigma}{21}=(\boldsymbol{\sigma}{i\boldsymbol{j}}\boldsymbol{\cdot}{k+1,\cdots,p})$$ 为$x_2$给定的时候 $x_1$的偏协方差矩阵他的对角线元素称为偏协方差对角线元素称为偏方差定义 $$\rho_{ij+k+1,\cdots,p}=\frac{\sigma_{ij}\cdot_{k+1,\cdots,p}}{\sqrt{\sigma_{ii}\cdot k+1,\cdots,p\sigma_{jj}\cdot_{k+1,\cdots,p}}},\quad1\leqslant i,j\leqslant k$$ 为$x_2$给定的时候 $x_i,x_j$的$(p-k)$阶偏相关系数他剔除了$x_{k+1}$到$x_{p}$的线性影响在多元正态的假定下偏相关系数的极大似然估计（样本偏相关系数为） $$r_{\hat{y}\cdot k+1,\cdots,p}=\frac{s_{\vec{y}}\cdot k+1,\cdots,p}{\sqrt{S_{\hat{p}}\cdot k+1,\cdots,pS_{jj}\cdot k+1,\cdots,p}}$$ 其中 $$S_{11}.{2}=S{11}-S_{12}S_{22}^{-1}S_{21}=\left(s_{ij}._{k+1},\cdots,p\right)$$

Analysis of quantitative data relevance (Pearson correlation factor)

Description and measurement of the relationship

Description of the relationship

Before we do the relationship analysis, we have a very central assumption that we only study linear relevance. We only study linear correlations between variables. We've already emphasized this in many courses in probabilistic theory, mathematical statistics. A quantitative relationship description for only two variables Draw a scattering map and draw a general relationship But we need a more precise quantitative approach.

Relevant relational measures

Here we study and give the probability coefficients directly.$r$ $$r=\frac{\sum\left(x-\overline{x}\right)\left(y-\overline{y}\right)}{\sqrt{\sum\left(x-\overline{x}\right)^2}\cdot\sqrt{\sum\left(y-\overline{y}\right)^2}},$$ And then you can go over the nature of the probabilistic theory that's given you.

The range of values to be taken is$[-1,1]$
Symmetrical
Relevant coefficient size and$x,y$The origin of the event is not a matter of scale.
Only linear relationships (but non-linear transformation of raw data)
There's no guarantee of a causal relationship between the two.

A relevant factor of this kind$r$ Now we call it the Pearson correlation coefficient, which is only a fraction of what counts, and there are plenty of statisticians behind it who contribute to the relevance study.

The general Pearson correlation coefficient is used as a precondition $x,y$ The data of the sample of the variable should be paired and each group of data should be independent of each other, and the sample number should be greater than 30, and the number of samples should be more than 30, and the number of samples should be more than 30, and the number of data in the data should be more than 30, and the number of data in the data should be more than 30, and the number of data in the data should be more than 30, and the number of data in the data should be more than 30, and the number of data in the data should be more than 30, and the data in the data should be more than 30, and the data in the data should be more than 30, and the data in the data should be used in the data form. These conditions are usually assumed to be valid.

The size and relevance of the relevant coefficient

In the traditional sense, The correlation factor is generally judged in accordance with the following rules:

More than 0.8
Medium relevance above 0.5
Weaknesses above 0.3
Not relevant

Assumptions of relevance

The test of the relevancy of the coefficient is generally used.$t$ Test Establishment of assumptions $$\begin{aligned}H_{0}:\rho=0;\H_{1}:\rho\neq0。\end{aligned}$$ Introduce testing statistics $$t=|r|\sqrt{\frac{n-2}{1-r^{2}}}\sim t\left(n-2\right)$$ Execute the two-sided$t$Just test it.

In the case of large samples, basically all the relevant coefficient tests are considered significant, so sometimes the practicalities are omitted, and we don't mention them anymore.

Relevance analysis of sequenced data (Spearman & Kendall & C)

First, we need to be clear about what sort of sequenced data is;

Source: Spearman.

This is essentially a supplement to the Pearson coefficient that requires a logical equidistance of sample data, just the size of the relationship.

That's right here.Changed the analysis of numerical data to qualitative data

Note: The variables involved in the analysis are being qualityd here, even if one of them would have met the Pearson relevance test.

For raw data$x_{i},y_{i}$ Sort them separately in order of small to large, and get the respective serial number of each data set.'},y_{i}^{'}$ 称为原始数据的秩次记秩次的差为$d i}$..so the formula that gives Spearman's coefficient is $$\rho_{s}=1-\frac{6\sum d_{i}^{2}}{n\left(n^{2}-1\right)},$$ The analysis of the relative coefficients and the weak relevance is consistent with Spearman and Pearson.

Generate Kendall's correlation coefficient

The Kendall-related coefficient, also known as the Kendall-related coefficient, is also a thorium-related coefficient, although it targets orderly class variables such as size, age, obese grade (high, moderate, mild, non-obese). It measures the strength of the single-tangular relationship between two orderly variables.The Kendall coefficient uses the concept of pairing to determine the strength and weakness of the coefficient.

Two pairs can be divided into one pair (Concordant) and one pair (Discordant). Consistency refers to the relative relationship between the values obtained by the two variables, which can be understood as having the same symbol as Y2-Y1; differences refer to their relative relationship, with the opposite symbol of X2-X1 and Y2-Y1.Each pair of requirements involves two variables and four elements.

The Kendall coefficient has two formulas, one for Tau-a and the other for Tau-b. The difference is that Tau-b can deal with situations with the same value, i.e., a parallel, which we need to describe separately.

For Tau-a $$\text{Tau-a}=\frac{c-d}{\frac{1}{2}n(n-1)}$$ of which$n$is the number of samples, the denominator measures the total number of possible combinations, which is the number of possible combinations, if not repeated$c+d$ $c$It's a consistent logarithm.$d$It's a divide.

Kendall is related to the nature of the difference between the former, which is a relative improvement of the Pearson, and Spearman, which is based on a study of differences and unanimity, which is more sensitive to issues of their own sort, is not sensitive to abnormal and non-linear relationships and is not covered by linear correlation coefficients.

Where the two data are identical (i.e. duplicate), they are neither considered synergetic nor inconsistent. However, the double value affects the calculation of the denominator (the total logarithm is subject to the deduction of the duplicate logarithm). That's Ta-b. $$\text{Tau-b}=\frac{c-d}{\sqrt{(c+d+t_x)(c+d+t_y)}}$$ of which$t_x,t_y$Yes. $X$ and $Y$ .

Tau-a above, Tau-b only apply to square tables, i.e. The two variables need the same number. For rectangular tables, the maximum value may be less than 1 as calculated by the formula above, for which the improved Tau-c by Tu-1 is introduced $$\text{Tau-c}=\frac{2m(c-d)}{n^2(m-1)}$$ of which$m$is the smaller of the rows and columns,$n$It's a sample volume.

Analysis of the relevance of qualitative data (column analysis)

The matrix (contingency table) is the basis for the analysis of the defined data, which is the frequency tables that are presented when observations are classified according to two or more data.

Definition of the matrix

Assume that the individual in a general is based on two attributes$A,B$I'm sorting out.$r$Zero and$c$We'll take a sample. $f_{ij}$ It's the type of observation frequency that we're looking at so we can build a two-dimensional one.$rc$List of rows It's natural to expand more attribute dimensions, but it's more difficult to visualize and use them more limited. We're just here to study the most basic of the matrix, and we've got more analysis of the matrix than we've ever seen. Multi-statistical analysis and “Responsive analysis” section

Independentness test of the list

We have to introduce the independent test of the list to examine whether the two variables in the matrix are relevant or independent. We've been introduced to this in our analysis. Multi-statistical analysis "Independentness test" section

Correlation factors in the joint table

Adopt$\chi^2$The relevant coefficients (size of measurement of relevance) of the value by means of study of the array are at the heart of this section. We'll introduce three methods and give them the right information. The list is classified and has no concept of size, so the coefficients should be positive and there is no positive or negative correlation, and they are relevant.

$\varphi$Related coefficient

To measure$2\times 2$Level of relevance of the matrix Calculate formulae as $$\varphi=\sqrt{\frac{\chi^{2}}{n}},$$ of which $$\chi^{2}=\sum_{i=1}^{2}\sum_{j=1}^{2}\frac{\left(f_{ij}-e_{ij}\right)^{2}}{e_{ij}}$$ Finally, a simplified result. $$\varphi=\sqrt{\frac{\chi^{2}}{n}}=\frac{ad-bc}{\sqrt{\left(a+b\right)\left(c+d\right)\left(a+c\right)\left(b+d\right)}}$$ $\varphi$The correlation coefficient analysis is meaningless. The relative weight and coefficients are determined by the Pearson coefficient

$C$Related coefficient

- It's for the larger list. - Yes.$\varphi$Theoretical expansion of the relevant coefficients Calculate formulae as $$C=\sqrt{\frac{\chi^{2}}{\chi^{2}+n}},$$ of which $$\chi^{2}=\sum_{i=1}^{1}\sum_{j=1}^{c}\frac{\left(f_{ij}-e_{ij}\right)^{2}}{e_{ij}}$$ The analysis factor means the same size as the Pearson coefficient. I can see it. $C$Linking the relevant coefficients to the rows and columns in the column tables So do not compare the coefficients of multiple arrays. At this point, analysis is meaningless.

$V$Related coefficient

$$V=\sqrt{\frac{\chi^{2}}{n\cdot\min\left[\left(r-1\right),\left(c-1\right)\right]}}=\sqrt{\frac{\chi^{2}}{n\left(m-1\right)}}$$ It is noted that the V-related coefficients derived from the two rows or columns of the rows are not suitable for comparison; When?$min(r,c)=2$And he was.$\varphi$Related coefficient (a form of extension)

Summary of attention

A hyphenation analysis of the amount of causality from the column should be made at the row, because the variable is in the column
When data is divided into two categories, the theoretical frequency should not be less than five.
When data are divided into more categories, the theoretical frequency group of data less than 5 should not exceed 20% of the total data split The matrix analysis should not be inconsistent with these concerns

Typical relevant analysis

Typical relevant analysis is one way to study the correlation between the two sets of variables, which reveals linear correlations between the two sets of variables, and he is very extensive in practical application, and we are promoting the coefficients after they have been extended.

We introduced typical relevant analyses because of two sets of variables (both of which are:$p$and$q$The correlation coefficient between the two is:$pq$The matrix that we often use is not good enough;

Naturally, we will introduce a downscaling of the main ingredients analysis, with as few as possible numbers to map the correlation between the two sets of variables.

Generally relevant

Typical Related Export

Set$x=(x_1,x_2,...,x_p)^{\prime}$and$y=(y_1,y_2,...,y_q)^{\prime}$It's two random sets of variables and $V(x)=\bardsymbol(Sigma}{11}(>0),V(y)$ $= \Sigma{22}\left ( > ♪ Right, Cov\left (x, y\right) ♪ $V(binom xy)=begin{bmatrix}\boldsymbol{\{11}&\boldsymbol{\Sigma}{12}\\boldsymbol{\Sigma}{21}&\boldsymbol{\Sigma}You're not gonna get away with this? We want to use an indicator to flip the line functions that maximize the correlation between the two sets of variables, which is very natural, using two vectors.$u=a^{\prime}x\text{ 和 }\upsilon=b^{\prime}y$ Compress them into single variables. Recalculate$uv$The correlation coefficient between them makes them the largest.

Naturally. $$00\ \oporatorname{Cov}(u,v)& =Cov(a^{\prime}x,b^{\prime}y)=a^{\prime}Cov(x,y)\boldsymbol{b}=\boldsymbol{a}^{\prime}\boldsymbol{\Sigma}{12}\boldsymbol{b} \ V(u)& =V(a^{\prime}x)=a^{\prime}V(x)a=a^{\prime}\boldsymbol{\Sigma}{11}\boldsymbol{a} \ V(v)& =V(\boldsymbol{b}^{\prime}\mathbf{y})=\boldsymbol{b}^{\prime}\boldsymbol{V}(\mathbf{y})\boldsymbol{b}=\boldsymbol{b}^{\prime}\boldsymbol{\Sigma}{22}\boldsymbol{b} \end{aligned}$$ 所以相关系数为 $$\rho(u,v)=\frac{a^{^{\prime}}\boldsymbol{\Sigma}{12}\boldsymbol{b}}{\sqrt{\boldsymbol{a}^{^{\prime}}\boldsymbol{\Sigma}{11}\boldsymbol{a}}\sqrt{\boldsymbol{b}^{^{\prime}}\boldsymbol{\Sigma}{22}\boldsymbol{b}}}$$ 为了避免一些毫无意义的结果重复我们一般要求$uv$都是标准化的变量也就是 $$a^{\prime}\boldsymbol{\Sigma}{11}\boldsymbol{a}=1\mathrm{,}\quad\boldsymbol{b}^{\prime}\boldsymbol{\Sigma}{22}\boldsymbol{b}=1$$ 因此我们希望极大化的相关系数为 $$\rho(u,v)=a^{\prime}\boldsymbol{\Sigma}\bardsymbol{b}$$ We're omitting some unnecessary proofs, giving a way to calculate the coefficient to be so large. Easy to know{11}^{-1}\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21},\Sigma_{21}^{-1}\Sigma_{21}^{-1}\Sigma_{12},\Sigma_{11}^{-1/2}\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}\Sigma_{11}^{-1/2}(\geqslant0)$和$\Sigma_{22}^{-1/2}$ $\boldsymbol{\Sigma}{211}^{-1}\boldsymbol{\Sigma}{12}\boldsymbol{\Sigma}{22}^{1/2}(\geqslant0)$都有着相同的非零特征值，可记为 $\rho_1^2\geq\rho_2^2\geq\cdots\geq\rho_n^2>0$,这里 $m$ 为 $\boldsymbol{\Sigma}The tidbit of $12.00 $a_1,a_2,...,a_m$ Yes.$\Sigma_{11}^{-1}\Sigma_{12}^{-1}\Sigma_{22}^{-1}\Sigma_{21}$Corresponds to$\rho_1^2,\rho_2^2,...,\rho_m^2$. The character vector $b_1,b_2,...,b_m$ Yes$\Sigma_{22}^{-1}\Sigma_{21}\Sigma_{11}^{-1}\Sigma_{12}^{-1}$Corresponds to $\rho_1^2,\rho_2^2,...,\rho_m^2$ . The character vector We'll take it. $a=a_1,b=b_1$When the correlation coefficients are so great, we call it $$u_1=\boldsymbol{a}_1^{\prime}\boldsymbol{x},\quad v_1=\boldsymbol{b}_1^{\prime}\boldsymbol{y}$$ It's the first of the typical variables.$a_1,b_1$It's the first pair of typical correlation coefficients.$p_1$It's the first typical correlation factor. If the amount of information extracted from the first typical variable is not sufficient, we can give the first$i$It's typically relevant.$a_i,b_i,p_i$

Nature of typical variables

The typical variables of the same group are not relevant.

$$u_i=\boldsymbol{a}_i^{\prime}\boldsymbol{x},\quad v_i=\boldsymbol{b}i^{\prime}\boldsymbol{y}$$ 则 $$\begin{aligned}\rho(u_i,u_j)=&\mathrm{Cov}(u_i,u_j)=\boldsymbol{a}^{\prime}\boldsymbol{\Sigma}{11}\boldsymbol{a}j=0,\quad1\leqslant i\neq j\leqslant m\\rho(v_i,v_j)=&\mathrm{Cov}(v_i,v_j)=\boldsymbol{b}^{\prime}\boldsymbol{\Sigma}{22}\boldsymbol{b}_j=0,\quad1\leqslant i\neq j\leqslant m\end{aligned}$$

Relevance between typical vectors

$$\rho(u_{i},v_{i})=\rho_{i},\quad i=1,2,\cdots,m$$ $$\begin{aligned} \rho\left(u_{i},v_{j}\right)& =Cov(u_{i},v_{j})=Cov(a_{i}^{\prime}x,b_{j}^{\prime}y)=a_{i}^{\prime}Cov(x,y)b_{j} \ &=\boldsymbol{\alpha}i^{\prime}\boldsymbol{\Sigma}{11}^{-1/2}\boldsymbol{\Sigma}{12}\boldsymbol{\Sigma}{22}^{-1/2}\boldsymbol{\beta}_j=\rho_j\boldsymbol{\alpha}_i^{\prime}\boldsymbol{\alpha}_j=0,\quad1\leqslant i\neq j\leqslant m \end{aligned}$$

Correlation coefficient between original and typical variables

$$\text{记}A=(a_1,a_2,\cdots,a_m),B=(b_1,b_2,\cdots,b_m),\text{则}\u=A^{\prime}x,\quad v=B^{\prime}y$$ $$\begin{gathered} \operatorname{Cov}(x,u) =Cov(x,A^{\prime}x)=\boldsymbol{\Sigma}{11}A \ \operatorname{Cov}\left(x,\nu\right) =\mathrm{Cov}(x,B^{\prime}y)=\boldsymbol{\Sigma}{12}\boldsymbol{B} \ Cov\left(y,u\right) =\mathrm{Cov}(y,A^{\prime}x)=\boldsymbol{\Sigma}{21}A \ \operatorname{Cov}\left(y,v\right) =\mathrm{Cov}(y,B^{\prime}y)=\boldsymbol{\Sigma}{22}\boldsymbol{B} \end{gathered}$$

Typical and general correlation factors

It's clear from the definition that when$p=q=1$When it happens, it's typically the normal correlation coefficient. When $p=1orq = $1 he's a compound coefficient. So, re-relevance is a typical case, and simple re-relevance is a re-relevance. Their size is linked. The first typical correlation factor is at least Same$x$(or)$y$) any of the weights and$y$(or)$x$The compound correlation coefficient is as large (as can be seen from the first typical relevant definition) Even if all these compound coefficients are small, the first typical correlation coefficient may be significant. The compound coefficient is not less than the relevant coefficient between any of the fractions (as can be seen from the compound definition) Even if all these relevant factors are small, the compound coefficient may be significant.

Typical relevant coefficients after standardization

Sometimes we'll standardize the weights before we'll calculate the typical correlation coefficient. The calculations we've been working on are not standardized, and this is not the same as the factor analysis, the primary component analysis, the classic two techniques of reduction. The standardized matrix is the matrix, and we can calculate the typical correlation coefficients based on the matrix, and calculate them in exactly the same way. The notion of typical correlation coefficients is not static for standardized transformations, but does not mean that our linear combination coefficients are not, which is the result of the promotion of normal correlation and reconnectivity coefficients.

Samples typically relevant

In practical applications, we use sample-related matrices to estimate the overall matrix; using fully consistent computation techniques, we can calculate typical correlation coefficients, typical variables, and the coefficients of their linear combinations. In practice, we usually use standardized and then calculated (using the matrix) so that the coefficient is also analytical. It's typically a purely numerical research method, but it's not worth using it, although it does have a concept of score. The point of understanding is that... How to calculate the coefficients of typical relevant variables, the relevant coefficients, and try to justify them by using the combination coefficients, which is the usual statistical analysis technique after a linear combination

Test of typical correlation coefficients

All the typical overall correlation coefficients are tested as zero.

Consideration of hypothetical tests $$H_0:\rho_1=\rho_2=\cdotp\cdotp\cdotp=\rho_m=0,\quad H_1:\rho_1,\rho_2,\cdotp\cdotp\cdotp,\rho_m\text{ 至少有一个不为零}$$ Create seemingly statistically comparable $$\Lambda_1=\prod_{i=1}^m\text{ (1-}r_i^2)$$ For a full-sized$n$ When the original hypothesis is established, the statistics are available. $$Q_1=-\left[n-\frac12(p+q+3)\right]\text{ln}\Lambda_1$$ Obey freedom.$pq$Yes.$\chi^2$Distribution When the statistical data are too large, the one-sided test rejects the original hypothesis that the correlation between typical variables is significant or otherwise not.

Test of zero for some typical overall correlation coefficient

Our natural hope is to use as few logarithms of typical relevant variables as possible, so we need to test the hypothetical zeros of some of the smaller typical relevant coefficients. Consideration of hypothetical tests $$H_0:\rho_2=\cdotp\cdotp\cdotp=\rho_m=0,\quad H_1:\rho_2,\cdotp\cdotp\cdotp,\rho_m\text{ 至少有一个不为零}$$ If the original hypothesis is accepted, then only the first pair of typical variables is significant, or we think the second is also significant, and we continue to perform the test. $$H_0:\rho_3=\cdotp\cdotp\cdotp=\rho_m=0,\quad H_1:\rho_3,\cdotp\cdotp\cdotp,\rho_m\text{ 至少有一个不为零}$$ So cycle, do sequence check. Test the number of statistics to be $$\Lambda_{k+1}=\prod_{i=k+1}^m\text{ (1-}r_i^2)$$ Of which$k$It's the order of the test we're looking at at at this moment.$k=0$When it's all over the whole thing, then it's before we remove it.$k$A follow-up check. For the big enough.$n$ When the original hypothesis was established, $$Q_{k+1}=-\left[n-k-\frac12(p+q+3)+\sum_{i=1}^kr_i^{-2}\right]\text{ln}\Lambda_{k+1}$$ Obey freedom.$(p-k)(q-k)$Yes.$\chi^2$Distribution. Refusal principle as above

Link analysis

Basic concepts

Basic concepts of linkages

The connection that happens when something happens in nature is called a connection.

The connection is...Two or more variablesIt's a kind of important thing that exists between the values.Some kind of pattern that can be found.

Links can be divided into simple linkages, time series linkages, causal linkages

Simple linkages mean exploring whether there is some statistical link between two or more variables without taking into account time sequences. This association is usually based on the co-existence frequency of variables, i.e. the number of times they occur simultaneously with data centralization.
The time series is about the relationship between variables over time.
Causal linkages are a more in-depth linkage analysis that not only explores the linkages between variables but also attempts to determine whether a variable leads to a change in another variable, i.e. whether there is a causal link, which is part of the causal inference.

Basic concepts of linkage analysis

Linking analysis aims to find hidden linkages between the data entry centralized for the given data record, describing the closeness of the data degrees

There are two types of correlation analysis: the rules of association and the sequence patterns.

The rules of association are used to find relevance to different items that appear in the same event
The sequence pattern is similar, but it seeks temporal correlation between events.

Associated rules

The main target of the linkage rule is transactional data. Library

The rules of association are the knowledge model of the pattern that appears simultaneously between the items in a transaction, and more precisely the rules of association are the impact on the presence of item Y by quantifying the number of items X.

He was created to do a shopping basket analysis, which is to study which customers of goods may buy at the same time as they do at a shopping mall, to help us sell our goods, and to sell them better.

Now, the rules of association can also be used for research into other data, and his core is to study the correlation of a number of qualitative and self-variant variables.

Formalization of rules on association

Only a strict mathematical definition will facilitate further modelling, and here we will use the trade database as the basis for the definition.

The transaction data set that the connection rule excavates is recorded as$D$ Where \\mathrm{D=~~{T_{1},~~T_{2},~~\ldots,~~T_{k},~\ldots,T_{n}}}$ 其中的 $T k$ is called a transaction, and each transaction has a separate number called TID;

All the goods that can be purchased are called "sections." $i_m$ The graphs are not used to describe the text.~~i_{2},~~\ldots,~i_{m}}}$ 是$D$中全体元素的集合所有的$T_k$都 $Subset of I$

Set two items$X,Y$ They all are.$I$=>Y$ expression is called the association rule

Measurement of the associated rules

All the rules of association have expressions. $$X\Rightarrow Y[s,c]$$ That's why the connection has confidence.$c$ Support $s$ He's the measure of our greatest concern for a connection rule.

$s$ It means the probability of both.
$c$ Organisation $X$ In the event of a situation $Y$The probability of a situation.

The method of calculation is by definition easy to give.

It is not enough to use only the rule of supporting confidence to evaluate linkages, because they do not consider the widespread problem of imbalance, so define it.

Expectation of credibility: Description of the associated rule $X = => Y$在没有任何条件影响时，$Y$在所有交易中出现的频率有多大。即没有$X$的作用下，$Y.A.'s own level of support
Improvement: Description$X$It's right.$Y$And what's the impact of that?
Interest:$\frac{\text{置信度}－\text{支持度}} {Max{\text{置信度},\text{支持度}}}$The greater the interest in a rule is greater than zero, the greater the actual value of utilization; the smaller the actual value of utilization is less than zero.

Linkage rule dig

The linkage rule that meets the minimum confidence threshold and the minimum support threshold is strong and meaningful.

The problem with excavating linkage rules is the creation of linkages that are more supportive and more credible than the minimum support threshold and the minimum confidence threshold given by the user, respectively.

Linkage rule dig

Basic concepts

$k$Set of items: Include$k$The collection of items
The frequency of the items is the number of services that contain the items
If the frequency of the collection is greater $\text{最小支持度}\times D\text{中的事务总数}$, the collection is called the Frequent Encyclopedia

So, the question of digging all the connection rules in the transaction database D can be divided into two sub-issues.

Find all frequent items with minimum support
Set) Use frequent set of items to generate desired correlation rules

Apriori algorithm

The Apriori algorithm uses pre-qualification of the mass of the complex, and is being searched in an iterative manner,$k$- The collection is for inspection.$(k+1)$He's gonna need to scan the data once in a while. Library

He's using nature:The non-empty subsets of frequent items are also frequent. Search

His basic implementation model is:

Input: Data set$D$, Support threshold$\alpha$
Output: Maximum frequency$k$Set Steps

Scan the entire data set and get all the data that have emerged as a candidate for a series of frequent entries (see table 2).$k=1$, with zero frequency sets empty)
I've been digging a lot.$k$Set
1. Scan data calculation candidate frequency$k$Support for the set
2. Remove the frequency of the candidate$k$The data set of the items with a concentration of support below the threshold is frequently k-set. If you get it more often,$k$Items are empty and return frequently$k-1$The collection of the items of the collection is the result of the algorithm, which ends. If you get it more often,$k$Only one set is available, and direct returns are frequent$k$The collection of the items of the collection is the result of the algorithm, which ends.
3. Based on frequency$k$Entries, connects to generate frequent candidates$k+1$Entries.
You!$k=k+1,$Step 2

It's the rules of strong association that satisfy minimum support and minimum confidence, and the rules that come from frequent clusters meet the requirements of support.

The Apriori algorithms are very inefficient, but the underlying underlying of the various associated rules algorithms behind them, most of them have made significant improvements in the Apriori algorithms in terms of efficiency.

Multiple rules of association

Simple rules of association: like basketball =>Basketball suits, only items.
Quantification of the rules of association: we changed the rules of association that were initially used only for non-variant types.
Multi-dimensional rules: Sex = “Men” => Purchase = “basketball” involving two dimensions
Cross-layer rules:
- The same level of connection rule: Adidas basketball => Nike's basketball suit.
- Rules of inter-layered association: basketball => Nike's basketball suit.

For the rules on quantitative linkages involving numerical fields, we need to separate the original values to be used in the generation of the linkage rules; this fragmentation can be predefined or produced while the linkage rules are established, which is often more effective, after all, unjustified fragmentation can have negative effects. R Visualization “Prudently processing” section

Digging cross-layer rules requires more advanced algorithms and pre-set settings.

From Association to Analysis

• We need an indicator to measure inter-incident correlation or dependence $A$and$B$Relevance:$corr_{A,B}=\frac P(A\cup B){\mathrm{P(A)P(B)}}=P(B\mid A)/P(B)$ $\cdot$When Set$A$The B-based project is a very important tool for the development of the B-based project.$P(A\cup B)=P(A)P(B)$ That's the corrr.{A,B}=1$,表明$A\text{not B, cur}{A,B}>1$ 表明$A$与B正相关，corr$_{A,B}<1$ 表明$A$ is related to B. Using the relevance indicator for the preceding example, it can be concluded that the relevance of the video and the game will be: $\mathsf{P}({game,video})/(\mathsf{P}({game})\times\mathsf{P}({video}))=0.4/(0.75\times0.6)=0.89$ Conclusion: negative correlation between video and game

Measurement of concentration trends and positions

Classic concepts in mathematical statistics

Mean

Normal mean is defined as $$\overline{x}=\frac{\sum_{i=1}^{n}x_{i}}{n}$$ In part, we use weighted averages, which is the difference in importance between data we have. $$\overline{x}=\frac{\sum_{i=1}^{k}w_{i}x,}{\sum_{i=1}^{k}w_{i}}$$ We can also give the sum of averages. $${\sqrt{x_{1}\cdot x_{2}\cdots x_{n}}}$$ But he rarely appears in traditional statistics.

Medium

To avoid the effects of some extreme anomalies, to increase robustness, we introduced the concept of median ^dd161e.

The fractional study is the relative location of the data, determined by the location of each sample. We can use the median as a measure of concentration, and of course we can use the very poor. Medium Points $$\frac{x_{1}+x_{n}}{2}$$ The data in order can only be studied by the number of numbers and the fractions.

Number

The largest number of data available was only studied in some exceptional cases

The concentration of disaggregated data can only be studied in a numerical manner

Measurement of concentration trends in EDA

The only central purpose of the measurement method in EDA, both here and in the back, is to increase the robustness of the estimate, which is to make the most of the data available to the population.

Statistics are not sensitive to a small amount of large deviation data
Statistics are not sensitive to a large amount of small deviation data.

So we've revised the statistics.

$L$Statistics

We'll take it. $X_{(i)}$ It's the first.$i$Order statistics are intended to increase the robustness of estimates by sequencing statistics, as we do in the median, and this is the relevant paragraph of this paper. Section

$L$The estimated amount is in the form of $$T=\sum_{i=1}^{r}a_{i}X_{(i)}$$ We'll tell you all about the estimates.$L$The estimates, including the averages, the medians, the numbers, the weighting averages, of course.

I'm introducing you.$L$When we do, we can produce a large range of estimates. Ending average That's the calculation of the average after the end of the head.

Percent$X$ Cut the tail, take it off by 1%.$X$
Median average: take 50 per cent of the median
Median: one to two in the middle
Tri-average: three values in quartiles

Assessment of the effects of concentration trend estimates

After the next study, we'll give you the following conclusions.$n$- It's a sample.

$n<$6.00 using median
$n=7$ Two of them are removed from each side's tail.
$n>$8.0 million, 25% on both sides.

Measurement of the discrete trend

Now let's consider the degree of fragmentation of data in a quantitative study.

The classic concept of classical mathematical statistics

Offset

We can think of the worst. $$x_n-x_1$$ Or consider average deviations. $$\sum_{i=1}^{n}(x_{i}-\overline{x})/n$$

Difference and Standard

We've studied this amount more than once in mathematical statistics. $$\sigma^{2}=\frac{\sum_{i=1}^{N}(x_{i}-\mu)^{2}}{N}$$ It's the most common amount of data dissegregation.

It's a standard deviation, and it has the same units as the original data. Root

The sample variance was amended to address the non-selectivity of the sample (and the standard deviation was subsequently amended) $$s^{2}=\frac{\sum_{i=1}^{\pi}(x_{i}-\bar{x})^{2}}{n-1}$$

Measure of relative deviation (variability factor)

The amount given below is ununited.

Defines the variable factor as $$V=\frac{\sigma}{\mu}$$

Isolation measures in EDA

Here we remove the measure method from the front, some of the measures that are of a robust nature.

Medium-digit difference in sample

$${AD}={\frac{1}{n}}\sum_{i=1}^{n}|x_{i}-{M}|$$ of which$M$ is the median of the sample

The median is absolutely different from the median.

$${MAD}=\mathrm{median}{i}{\left|x- \\boldsymbol We calculated all the differences and then we calculated the median. The median absolute difference between the R and the R is about 1.4, so the aim is to estimate the sample variance, which we can basically ignore in the actual calculations.

Four-point difference.

We'll consider the margin of the quarter. $$IQR = d_{F}=F_{U}-F_{L}$$ And what we're doing is we're using it very often in the study of dissegregation and abnormality.Box chart and five-digit summary

Assessment of the effects of several discrete measures

We can get a very simple but important conclusion from looking at the extent to which these statistics are dissegregated in different distributions.

The quartet is the best possible sum of the size of the sample dissipation.

Measurement of the distribution shape

They focus on two characteristics of the sample, whether their centres are concentrated (measures of concentration trends) and whether they are uneven (measures of dispersing trends) are both the shape of the distribution. Their characteristic is that they compare to the normal distribution.

Skewer factor (skewness)

$$g_1=\frac{n}{(n-1)(n-2)s^3}\sum_{i=1}^{n}(x_i-\overline{x})^3=\frac{n^2\mu_3}{(n-1)(n-2)s^3},$$ The symmetrical symmetry factor is 0

Peak coefficient (kurtosis)

$$\begin{array}{rcl}g_2&=&\frac{n(n+1)}{(n-1)(n-2)(n-3)s^4}\sum_{i=1}^n(x_i-\overline{x})^4-3\frac{(n-1)^2}{(n-2)(n-3)}\&=&\\frac{n^2(n+1)\mu}4}(n-1)(n-2)(n-3)s^4}-3\frac{(n-1)^2}(n-2)(n-3)},\end{array} I'm sorry. And then, when you're looking at the peak of distribution, you're going to have a peak greater than zero when you're going to be going to the normal distribution, or you're going to have a small peak of zero.

Descriptive statistical visualization

We don't need to discuss too many visual tools in basic descriptive statistics, and the basic descriptive statistical techniques presented here only involve a part of Excel, which includes

Studying numerical fundamentals:R Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . R Visualization The "Cleveland Plot" section
Basic forms of distribution of research:R Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . R Visualization and the “Regular Nuclear Density Estimates” section of the
Study of basic ratios:R Visualization "The pie-tarts section."
Study the existence of extreme values:R Visualization I'm not sure I'm gonna be able to get a picture of this.

Especially, we're here to add a more non-usual but important graphic to statistics itself.Mathematical statistics Section on the " Empirical Distribution Function "