Clustering Model Evaluation: External, Internal, and Relative Metrics

Performance measures for a polygon model

Clusters are a relatively special type of machine-learning task, and we need to give some slightly different indicators of effectiveness.

What are the goals of clustering? Intuitively, we want to see "integrity" in which the same body sample is as similar as possible, and different groups of samples as possible. In other words, the concentration result is high intra-cluster similarity and low inter-cluster similarity.

See you at the presentation of the cluster algorithm itself.Machine learning progress and unsupervised learning: spectral and graphic Category。

External indicators

By definition, external validation measures assume that the precise or real cluster is known in advance. Real cluster labels (i.e. external information) are used to assess a given cluster category. Usually we don't know the exact grouping; but external measurements can be used to test and validate different grouping methods.

All external measurements need one. $r\times k$ List of rows$N$, the table is based on a grouping$\mathcal{C}$Split with Real Value$T$is defined as follows: $$N(i,j)=n_{ij}=|C_i\cap T_j|$$ In other words, count.$n_{ij}$Representative Division$C_i$and real value split$T_j$Number of points common to all.

Besides, for the sake of clarity,$n_i=|C_i|$Representative Division$C_i$Number of midpoints,$m_j=|T_j|$Representational$T_j$Middle point number. The list can be used to read$T$ and$\mathcal{C}$Yes.$O(n)$Count it in.

Based on a matching measure

purity

The purity quantifys a fraction.$C_{i}$The extent to which only one divided entity is included. In other words, it measures how "purity" each sub-column is. Split$C_i$purity is defined as: $ \\mathrm{purity}i=\frac1{n_i}\max{j=1}^k{n_{ij}}$$ 聚类$C$的纯度定义为所有分簇纯度的带权和： $$\mathrm{purity}=\sum_{i=1}^r\frac{n_i}n\text{purity}i=\frac1n\sumI'm not gonna let you go. Percentage of$\frac{n_i}n$For a breakup$C_i$the percentage of points in the middle.

$C$The greater the purity, the higher the degree of conformity with the true value. The maximum purity value is 1 and means that each cluster is made up of only one point in the division. If$r=k$, the purity value is 1 to indicate a perfect grouping, i.e. the cluster corresponds to the division. But even if it's $r.>k$,纯度也可能为 1(当每个分簇都是一个标准划分的子集时)。若$r<k$, which cannot be pure, because at least one subset contains more than one split point.

Maximum Match

The maximm watching is a map of the selection of the fractions and the dividing, maximizing the number of public points (assuming that one division is given, only one can match it). This is not the case with purity.

Formally, we see the list as a fully-owned part two. Figure$G=(V,E)$Each division and each sub-column is a node, i.e.$V=\mathcal{C}\cup\mathcal{T}$, and there's a side $(C_i,T_j)\in E$and power $w(C_i,T_j)=n_{ij}$,for all$C_i\in\mathcal{C}$and$T_j\in\mathcal{T}$。

Matching in a figure (matching)$M$Yes.$E$A subset that makes$M$The two sides of the equation are not adjacent (i.e. there is no common vertex). Maximum Match Measure is defined as$G$, and then the right match: $$\text{match}=\arg\max_M\left{\frac{w(M)}n\right}$$ One of them matches.$M$The right value is$M$The sum of the weights of all sides, i.e.$w(M)=\sum_e\in Mw(e)$

F Measure

Give a scorer.$C_i$You're...$j_i$Organisation$C_i$The division of the maximum points of the midpoint, i.e.$j_i=\max_j=1^k{n_{ij}}$I'm sorry. A partition.$C_i$The precision (precision) is the same as its purity: $ \\mathrm{prec}i=\frac{1}{n_i}\max{j=1}^k{n_{ij}}=\frac{n_{ij_i}}{n_i}$$

Split$C_i$The recall is defined as:

$$\mathrm{recall}i=\frac{n{ij_i}}{|T_{j_i}|}=\frac{n_{ij_i}}{m_{j_i}}$$

of which$m_{j_i}=|T_{j_i}|$I'm sorry. It measures the division.$T_{j_i}$And the partition.$C_i$Proportion of shared sites.

F-measure is the sum average of the precision and recall values of each fraction. Split$C_i$The F-measure is: $F i=\frac{1mathrm{prec}+\frac{1mathrm{recall}=cdot\mathrm{prec} cdot\mathrm{recall}i}{\mathrm{prec}i+\mathrm{recall}i}=\frac{2n{ij_i}}{n_i+m{j_i}}$$ 聚类$\mathcal{C}$的 F-measure 为各分簇的 F-measure 的均值： $$F=\frac1r\sum{i=1}^rF_i$$

He wants to balance precision with recall.

Measure based on entropy

Conditional entropy

A cluster$C$The term entropy is defined as: $$H(\mathcal{C})=-\sum_{i=1}^rp_{C_i}\log p_{C_i}$$ of which$p_{C_i}=\frac{n_i}n$It's a partition.$C_i$- The probability.

Again, split.$T$The term entropy is defined as: $$H(\mathcal{T})=-\sum_{j=1}^kp_{T_j}\log p_{T_j}$$of which$p_{T_j}=\frac{m_j}n$It's division.$T_j$- The probability.

$T$The split, which is$T$About the partition$C_i$is defined as: $$H(\mathcal{T}|C_i)=-\sum_{j=1}^k\left(\frac{n_{ij}}{n_i}\right)\log\left(\frac{n_{ij}}{n_i}\right)$$

Grouping$C$ Division$T$ The condition is defined as $$00begin{aligned}H\left (T\mathcal{C}right)&=\sum_{i=1}^r\frac{n_i}{n}H(\mathcal{T}|C_i)=-\sum_{i=1}^r\sum_{j=1}^k\frac{n_{ij}}{n}\log\left(\frac{n_{ij}}{n_i}\right)\&♪ I'm not gonna let you go ♪ of which$p_{ij}=\frac{n_{ij}}n$It's a partition.$i$One of the points is also divided.$j$- The probability.

The more the points in a partition spread into different divisions, the larger the conditions. For a perfect group, the value of the condition entropy is 0, while the value of the conditional entropy in the worst case is 0.$\log k$。

Normalize mutual information

Mutual information research grouping$C$and division$T$The amount of information shared between them is defined as: $$I(\mathcal{C},\mathcal{T})=\sum_{i=1}^r\sum_{j=1}^kp_{ij}\log\left(\frac{p_{ij}}{p_{C_i}\cdot p_{T_j}}\right)$$ I've got information.$\mathcal{C}$and$\mathcal{T}$Joint probability$p_{ij}$And the expectation of a combination of probability.$p_{C_i}\cdot p_{T_j}$ Relevance (under independent assumptions).

If$C$and$T$It's independent of each other, then.$p_{ij}=p_{C_i}\cdot p_{T_i}$♪ And so ♪$I(\mathcal{C},T)=0$I'm sorry. However, there is no upper bounds for information.

We can get information from each other. $$I(\mathcal{C},\mathcal{T})=H(\mathcal{T})-H(\mathcal{T}|\mathcal{C})I(\mathcal{C})$$ So we can give a generic information. $$\mathrm{NMI}(\mathcal{C},\mathcal{T})=\sqrt{\frac{I(\mathcal{C},\mathcal{T})}{H(\mathcal{C})}\cdot\frac{I(\mathcal{C},\mathcal{T})}{H(\mathcal{T})}}=\frac{I(\mathcal{C},\mathcal{T})}{\sqrt{H(\mathcal{C})\cdot H(\mathcal{T})}}$$ His range is in. $[0,1]$ In between, close to one means good cluster.

Information discrepancies

This indicator is based on clustering$C$Split with Real Value$T$The information and entropy of these are defined as: $$00begin{aligned}\mathrm{VI} (\mathcal{C},\mathcal{T}&=(H(\mathcal{T})-I(\mathcal{C},\mathcal{T})+(H(\mathcal{C})-I(\mathcal{C},\mathcal{T}))\&=H(\mathcal{T}+H(\mathcal{C}2I(\mathcal{C},\mathcal{T}\end{aligned} $$ Information difference (VI) value 0, current and only$C$and$T$Same. So, the smaller the VI value, the more the grouping$\mathcal{C}$The better.

Pair

Default of $D= {bardsymbol{x}1,\boldsymbol{x}2,\ldots,\boldsymbol{x}m}$, 假定通过聚类给出的簇划分为 $\mathcal{C}={C_1$, $C_2,\ldots,C_k}$, 参考模型给出的簇划分为$C^={C_1^,C_2^,\ldots,C_s^}$.相应地，令$\lambda$ 与$\lambda^$ 分别表示与$C$ 和$C^We'll have the sample paired, define it. $$00begin{gathered} A = \SS, =SS = (\bardsymbol{){i},\boldsymbol{x}{j})\mid\lambda{i}=\lambda_{j},\lambda_{i}^{}=\lambda_{j}^{},i<j)}, \ b= |SD|,SD={(\boldsymbol{x}{i},\boldsymbol{x}{j})\mid\lambda_{i}=\lambda_{j},\lambda_{i}^{}\neq\lambda_{j}^{},i<j)}, \ c= |DS|,DS={(\boldsymbol{x}{i},\boldsymbol{x}{j})\mid\lambda_{i}\neq\lambda_{j},\lambda_{i}^{}=\lambda_{j}^{},i<j)}, \ d= |DD|,~DD={(\boldsymbol{x}{i},\boldsymbol{x}{j})\mid\lambda_{i}\neq\lambda_{j},\lambda_{i}^{}\neq\lambda_{j}^{},i<The blogger says: I'm sorry, I'm sorry. In which SS indicates that both models are sampled in the same clusters to SD, the former in the same clusters, and the latter in different clusters, the DS and DD in the same way.

So we can define it.

Jaccard

Jaccard coefficient (Jaccard Coefficent, JC) $$\mathrm{JC}=\frac{a}{a+b+c}.$$ Perfectly divided Jaccard coefficient is 1.

Rand Index

Rand Index (Rand Index, RI) $$\mathrm{RI}=\frac{2(a+d)}{m(m-1)}.$$ of which$m$It's the total point. Perfectly divided Rand index to 1.

FM Index

FM Index (Fowlkes and Mallows Index, short FMI) $$\mathrm{FMI}=\sqrt{\frac{a}{a+b}\cdot\frac{a}{a+c}}.$$ Perfectly divide FM index to 1.

Link Measures

Definition of Hubert statistics

You!$X$and$Y$Two symmetrys$n\times n$matrix, and$N=\binom n2$I'm sorry. You're the one who's gonna get you.$x,y\in\mathbb{R}^N$- For each other.$X$and Y's upper triangle elements (excluding main diagonal elements) are the vectors obtained by linearization. You're the one who's gonna get you.$\mu_X$Representative$x$, defined as: $$\mu_X=\frac1N\sum_{i=1}^{n-1}\sum_{j=i+1}^nX(i,j)=\frac1Nx^\mathrm{T}x$$ You!$z_x$Centred$x$Vector, defined as: $$z_x=x-1\cdot\mu_X$$ of which$1\in R^N$is the full 1 vector. Again, the order.$\mu_Y$Representative$y$The average of the elements by element,$z_y$Centred$y$vector.

Hubert Statistically defined$X$and$Y$Average element-by-component product: $$\Gamma=\frac1N\sum_{i=1}^{n-1}\sum_{j=i+1}^nX(i,j)\cdot\boldsymbol{Y}(i,j)=\frac1N\boldsymbol{x}^\mathrm{T}\boldsymbol{y}$$

Normalization Hubert Statistically defined$X$and$Y$, and then the following: $$\Gamma_n=\frac{\sum_{i=1}^{n-1}\sum_{j=i+1}^n(\boldsymbol{X}(i,j)-\mu_X)(\boldsymbol{Y}(i,j)-\mu_Y)}{\sqrt{\sum_{i=1}^{n-1}\sum_{j=i+1}^n(\boldsymbol{X}(i,j)-\mu_X)^2\quad\sum_{i=1}^{n-1}\sum_{j=i+1}^n(\boldsymbol{Y}[i]-\mu_Y)^2}}=\frac{\sigma_{XY}}{\sqrt{\sigma_X^2\sigma_Y^2}}$$

Disconnected Hubert Statistics

You!$T$and$C$Yes$n\times n$and the matrix, defined as: $$\left.\bardsymbol{T}(i,j)=\left{begin{array}ll}1&y_i=y_j,:i\neq j\0&\text{others}\right.\right.\quadd\bardsymbol{C}(i,j)=\left{begin{array}{1&\hat{y}_i=\hat{y}_j,:i\neq j\0&\text{Others}\right.$ Meanwhile,$t,c\in\mathbb{R}^N$Other Organiser$T$and$C$and the upper triangulation elements (excluding diagonal elements)$N$& Vector, where$N=\binom n2$Numbers representing different points. Finally, your orders$z_t$and$z_c$Centred$t$Vector and$c$vector.

Dispersed Hubert statistics can use formula (17.14) You're the one who's gonna get you.$x=t,y=c$) Calculated: $$\Gamma=\frac1Nt^\mathrm{T}c=\frac{\mathrm{TP}}N$$

Normalized discrete Hubert statistics

Dispersed Hubert, the uniform version of statistics is$t$and$c$Relevance between $$\Gamma_n=\frac{z_t^\mathrm{T}z_c}{|z_t||z_c|}=\cos\theta $$ Attention.$\mu_T=\frac1Nt^\mathrm{T}t$is the same division ((s)$y_i=y_j$) Point-to-point ratio, regardless of$\hat{y}_i$and$\hat{y}_j$Whether it matches. Thus, it is possible to: $$\mu_T=\frac{t^\mathrm{T}t}N=\frac{\mathrm{TP}+\mathrm{FN}}N$$

Internal indicators

And it's obvious that external indicators are in most cases of no value because we don't have reference models that we can use unless we're known to be real classifications, just to study the performance of the cluster algorithm. Internal indicators often depend on the distance between samples and the approximation, and thereforeMachine learning progress and unsupervised learning: spectral and graphic CategoryClose links, where the integration and modularity can be directly used for performance measurement.

Considering the distance between samples, give the following definition $$00\ \mathrm{avg}& =\frac{2}{|C|(|C|-1)}\sum_{1\leqslant i<j\leqslant|C|}\operatorname{dist}(\boldsymbol{x}{i},\boldsymbol{x}{j}), \ \operatorname{diam}(C)& =\max_{1\leqslant i<j\leqslant|C|}\mathrm{dist}(\boldsymbol{x}{i},\boldsymbol{x}{j}), \ d_{\min}(C_{i},C_{j})& =\min_{\boldsymbol{x}{i}\in C{i},\boldsymbol{x}{j}\in C{j}}\mathrm{dist}(\boldsymbol{x}{i},\boldsymbol{x}{j}), \ d_{\mathrm{cen}}(C_{i},C_{j})& =\mathrm{dist}(\boldsymbol{\mu}{i},\boldsymbol{\mu}The blogger says: I'm sorry, I'm sorry. The four samples are the following: the central distance between the inner samples, the longest distance between the inner samples, the nearest distance between the clusters, the central distance between the clusters.

DB Index

DB Index (Davis-Bouldin Index, short DBI)
$$\mathrm{DBI}={\frac{1}{k}}\sum_{i=1}^{k}\max_{j\neq i}\left({\frac{\mathrm{avg}(C_{i})+\mathrm{avg}(C_{j})}{d_{\mathrm{cen}}(\mu_{i},\mu_{j})}}\right)$$ The smaller the DBI, the better.

Dunn Index

Dunn Index (Dunn Index, DI) $$\mathrm{DI}=\min\limits_{1\leqslant i\leqslant k}\left{\min\limits_{j\neq i}\left(\frac{d_{\min}(C_i,C_j)}{\max_{1\leqslant l\leqslant k}\operatorname{diam}(C_l)}\right)\right}.$$ And the bigger the D, the better.

BetaCV

BetaCV measures the ratio between the intra-clan distance average and the interclan distance average: $$\mathrm{BetaCV}=\frac{avg(C)}{d_{avg}}$$ The smaller the BetaCV value, the better the effect of the cluster because it means that the inner distance is on average less than the interclause distance.

Relative measures

Relative measures compare the group nature of different parameters of the same conglomeration algorithm Yes.

Calinski-Harabasz(CH)

The given data set is $D=x i}The D.R. is a scatter matrix: $S=n\bardsymbol}sum}{j=1}^n(\boldsymbol{x}j-\boldsymbol{\mu})(\boldsymbol{x}^mathrm{ Where \mu=\frac1\sum{j=1}^nx_j$是均值，$\Sigma$是协方差矩阵。散度矩阵可以分解为两个矩阵$S=S_W+S_B$,其中$S_W$是簇内散度矩阵，$S B$ is a cluster-wide dispersion matrix, which is indicated as: $$00\&S{W}=\sum_{i=1}^k\sum_{x_j\in C_i}(x_j-\mu_i)(x_j-\mu_i)^\mathrm{T}\&I'm sorry, I'm sorry. of which$\mu_i=\frac1{n_i}\sum_{x_j\in C_i}x_j$It's a partition.$C_i$average.

For a given$k$Value, Calinski-Harabasz (CH) variance is defined as: $ \begin{aligned}CH(k)&=\frac{\mathrm{tr}(S_B)/(k-1)}{\mathrm{tr}(S_W)/(n-k)}\&=\frac{n-k}{k-1}\cdot\frac{\mathrm{tr}(S_B)}{\mathrm{tr}(S_W)}\end{aligned}$$

of which$(S_W)$and tr$(S_B)$is the trace of the inner and inter-clave-dispersible arrays (i.e. the sum of the diagonal elements).

For a better one.$k$Value, can predict a relatively small dispersion in the cluster, and therefore a higher dispersion is obtained. $CH(k)$ value. On the other hand, we don't want a big one.$k$Value;

Thus, CH values can be mapped and a larger growth area found (and no or only small growth thereafter).

Division stability

The main idea behind the stability of the divide is to be able to$D$The clustering of data sets from the same distributed sample should be similar or “stable”.

The method of partition stability can be used to find the appropriate parameter values for a given cluster algorithm; the book is mainly appropriate for consideration$k$value, the correct number of the fractional clusters.

$D$The joint probability distribution is usually unknown. Thus, for the same distribution of sample data sets, we can use a range of methods, including random disturbances (random perturbation), subsampling or self-help sampling (bootstrap resampling). We'll start with the self-help method:

By From$D$Sampling (replaced, i.e. allowing the same data point to be selected several times, each sample)$D_i$So it's different to generate it.$t$Size$n$The sample. Next, for each sample,$D_i$, with different$k$ Value (from 2 to) $k^\mathrm{max}$) Runs the same group algorithm.

You!$C_k(D_i)$Organisation$k$From Sample$D_i$Get a cluster. Next, the method compares all clusters with a certain group function$C_k(D_i)$and$C_k(D_j)$Distance between. Some external concentration assessment measures can be used as distance measures, e.g., by$C=C_k(D_i),T=C_k(D_j)$And vice versa. Based on these values, we calculate each.$k$The expectations of values are in pairs. Finally, the lowest deviation from the different clusters obtained from the re-sampling data sets$k^*$Yes.$k$The best choice is because it has the highest degree of stability.

Cluster trend

Cluster tendency or clusterability (clusterability) is designed to judge data sets$D$There are meaningful clusters. This is often difficult because it is difficult to define what is a subset in the first place, such as partitioning, hierarchy, density-based, map-based, etc.

Even if you have a sort of cluster, for a given data, Set$D$It remains difficult to define a suitable zero model (null model, i.e., model without any cluster structure). Moreover, even if data are judged to be conglomerate, we still face the problem of determining the number of judgement clusters.

Hopkins statistics are a thin sample test of space randomity. Give a Organisation$n$Data set for points$D$We create$t$A random sample.$R_i$ (Each subsampling contains$m$Point, of which$m\ll n$I'm not sure. Dataspaces of these samples and$D$Same, randomly and evenly generated at each dimension.

Besides, we're going to go straight to...$D$Generating$t$Samples (each inclusive)$m$(Place) (Placed), use unreleased samples. You're the one who's gonna get you.$D_i$Representative's first$i$A direct subsampling. Next, calculate each one.$x_j\in D_i$and$D$Minimum distance between points: $$\delta (\bardsymbol{x}j)=\min{\boldsymbol{x}_i\in D,\boldsymbol{x}_i\neq\boldsymbol{x}_j}{\delta(\boldsymbol{x}_j,\boldsymbol{x}_i)}$$

I'm sorry.$i$- Yes, it's a sample.$R_i$and$D_i$ Hopkins Statistics$d$Definition:

$$\mathrm{HS}i=\frac{\sum{y_j\in\mathbf{R}i}(\delta{\min}(\boldsymbol{y}j))^d}{\sum{y_j\in\mathbf{R}i}(\delta{\min}(\boldsymbol{y}j))^d+\sum{\boldsymbol{x}_j\in\boldsymbol{D}i}(\delta{\min}(\boldsymbol{x}_j))^d}$$

This statistical volume will provide a recent neighbourhood distribution of the data points generated at random and will be distributed over the next few years.$D$Compares the latest neighbourhood distribution of random subsets of the medium data points. If the data are of good fusion, we expect$\delta_{\min}(x_j)$Less than$\delta_{\min}(y_j)\text{,且在这种情况下，HS}_i$ Trends to 1.

If the two closest neighbors are similar, HS$_i$The value is close to 0.5, which means that the data are almost random and not clearly clustered.

And finally, if...$\delta_{\min}(x_j)$Value greater than$\delta_{\min}(y_j)$, HS$_i$A zero, which means a little exclusion, and no cluster.

Based on$t$A different HS.$_i$Value, as judged by the average and variance of the statistical volume$D$Can cluster.