From Euclidean Space to Manifold Topology: Dimensionality Reduction for High-Dimensional Data

This English version was machine-translated from the Chinese original. Technical terms may require verification.

Motives and context of the decline

The questions in this article can also be addressedLinear regression base: linear model, minimum 2x2 estimate and regression diagnosis、Statistical calculations: random number generation, random variable simulation and Monte Carlo methodHow the concept of a relatively close read together is developed in different contexts.

The challenge of high-dimensional data

Modern data science and machine learning often involves processing hundreds of thousands of, if not more, dimensions. The dimensions (i.e. the number of characteristics) are common once they are close to or above the sample amount "Gavy Small Sample" Scenario: The validity, stability and interpretability of traditional reliance on statistical methods that “sample numbers are much greater than their dimensions” are tested.

High-dimensional is not a simple extension of low-dimensional data. It also presents challenges in computing, perception and statistics: rapid increases in the cost of revolving sub-units or complex estimates; the difficulty of humans taking direct control of high-dimensional geometry structures; and the increasing scarcity of limited samples in larger spaces, which destabilizes many classical methods that rely on local neighbourhood or distribution estimates.

Dimensions disaster: from kNN to $p \gg n$

The most representative difficulty is that "Curse of Demension"I'm sorry. As the dimensions grow, the number of samples required to maintain the same sampling density increases dramatically; the distance between the nearest and the most distant neighbours in the high-dimensional zone tends to shrink, leading to a gradual loss of distinction in distance itself.

In the case of k Neighbor (kNN), it selects the closest to the test sample based on distance $k$ A training sample, which is projected by voting or weighted averages; its effectiveness relies on sufficiently intensive samples and distance measures that distinguish the proximity. But when space becomes thinner, nuclear density estimates, and the like are robust in low dimensions, either a near explosive increase in sample volumes or an increase in the range makes it unreliable.

Empirically, using “at least 5 observations per dimension” as a rough threshold for reliable modelling; if the number of features is $p$, corresponding empirical baseline is about $5p$ One observation. But in genetic expression, text mining, etc., it's common. $p \gg n$ In the circumstances, this requirement is often not met.

Response framework for high-level statistics

Rather than simply abandoning the traditional framework, GVS is reorganizing the analysis process through structural assumptions and new theoretical tools. Explored data analysis (EDA) can first explore potential models with relevance analysis, cluster and anomaly testing to provide empirical evidence for subsequent assumptions; and retroversion can offer a moderate deviation in exchange for lower deviations, which will stabilize projections under limited samples.

Structural assumptions such as slurability and low stylity further re-enable variable selection, co-scaling estimates and high-dimensional extrapolation to be identifiable and statistically assured. Under these conditions, the theory of convergence and non-accuracy no longer requires “fixed dimensions, sample sizes are endless”, but allows dimensions to grow in step with sample volumes, providing the theoretical basis for the rare regression, modeling and so on.

And that means that Gaul is not just a curse. It forces us to focus on the balance between computing feasibility, statistical efficiency and interpretability, while also potentially bringing about better linear symmetry and providing space for such ideas as nuclear approaches. The key is not to increase or reduce the dimensions mechanically, but to identify the structures that are actually available in the data.

Why is the decline the key?

In these responses,Demutation This is particularly important: it places high-dimensional data into more compact representations, retaining meaningful geometry, distance or local neighbourhood structures under manageable information losses, thus serving visualization, exploration, computing and subsequent modelling. It can find effective low-dimensional spaces and reveal potential non-linear currents.

Instead of replicating the entire high-dimensional statistical spectrum, the following is a step-by-step discussion of the more common downscaling route and its mathematical intuition, starting with the traditional approach of maintaining global structures. The review and comparison of the PCA, AE etc. that have already been developed in other notes is still dominated by application levels, without replicating the theory.

Traditional approach to maintaining global structure

Early downscaling techniques focus mainly on how to keep data points between data pointsGlobal Geometry(like the distance of the o'clock)

MDS: Faithful reduction of distance

MDS (Multiple Dimensional Scaling) is a classic algorithm for keeping distance. It's a simple hunch:If two dots are far away in high space, they should also be far away in low space.

Assumptions $m$ The distance matrix of a sample in the original space is $D$, the weight is $dist_{ij}$I'm sorry. Our goal is to find low-dimensional mapping. $Z \in \mathbb{R}^{d^* \times m}$♪ And make ♪ $|z_{i}-z_{j}| \approx dist_{ij}$。

Mathically, by building an internal matrix. $B = Z^{\mathrm{T}} Z$, using cosine theorem to extrapolate: $$b_{ij}=-\frac{1}{2}(dist_{ij}^{2}-dist_{i.}^{2}-dist_{.j}^{2}+dist_{..}^{2})$$ of which $dist_{i.}^2$ equal to the average of a row or column. ♪ Against the Matrix ♪ $B$ Decomposition of feature values: $Z = \Lambda }^{1/2} V_{You're not gonna get me out of here. The coordinates are available taking the number of characteristics that corresponds to the number of characteristics.

The main limitation of MDS is that it attempts to keep the distance between all points, and this stringent requirement for European distance (or possibly other distance) can easily be mistaken. When processing non-linear data (e.g., a curly Swiss volume), the O'Schille distance itself may be a wrong measure: two points of space proximity on the curly side may actually be very far in flow.

PA and KPCA: From linear to nuclear techniques

PCA (Main component analysis) There has been a detailed presentation, which is discussed briefly in the macro-level perspective of the backsliding. It needs to be reiterated that the PCA is equal toUsing the MidSpeciesI'm sorry. It's looking for the biggest difference. The data is kept.Global linear structure。

When the linear projection fails,nucleination linear downscaling (KPCA)It's introduced."Nuclear techniques"(Kernel Trick). OtherIntuition.Yes: The indissociable data of low dimensions is mapped into the high (even infinity) space, making it linear.

Yes.FormGo, by solve. $\left(\sum_{i=1}^m z_i z_i^\mathrm{T}\right) W = \lambda W$ And use nuclear functions in calculations $\kappa(x_i, x_j)$ Replaces the direct internalization calculation. Although mapping the data to higher levels seems anti-intuitive, it is introduced for the PAANon-linear processing capacityThis is a bridge between traditional statistical methods and fluent learning.

Fluid learning and probability mapping model (modern downscaling core)

When data is distributed on a curve, yes.Manifold LearningI'm sorry. It is concerned that:Maintain local neighbourhood structures and weaken remote global distances. As to what flow is per se, it is discussed in greater detail in the troplasm.

t-SNE: from distance to probability distribution

t-SNE (t-Distributed Stochastic Neighbor Embedding) It is a data visualization technique that has a high impact on the night before the age of deep learning. It is no longer committed to the rigid constraint of “staying distance”, but instead to “maintaining probability distribution”.

Technical intuition and mathematical forms

Probability of neighbours in the Gave space (Gose distribution): In the high space, instead of using direct distance, ask: $x_j$ Yes, it is. $x_i$ What is the probability of a neighbor? We useGoss distributionTo define the probability of this condition. $p_{j|i}$ $$p_{j|i} = \frac{\exp(-|x_i - x_j|^2 / 2\sigma_i^2)}{\sum_{k \neq i} \exp(-|x_i - x_k|^2 / 2\sigma_i^2)}$$ Watch this one. $\sigma_i$ Calculated separately for each point, by argument Perplexity Decision.

Quarter probability (t-distribution) for low-dimensional space: In low space, we're looking for a spot $y_i, y_j$I'm sorry. To solve it.Crowding Problems,t-SNE uses a freedom of 1 t-distribution (Cossi distribution) $$q_{ij} = \frac{(1 + |y_i - y_j|^2)^{-1}}{\sum_{k \neq l} (1 + |y_k - y_l|^2)^{-1}}$$

Why the t-distribution? Because the t-distribution is..."Rewind"Yeah. The same probability value (similarity) is obtained for the distribution of Goss, t-situation requirement point/point distanceFarrI'm sorry. This forced the data cluster that was originally crowded together in the high space, and was forced to be held in the low space."Blow it up!"And form a clear separation cluster.

Loss function: KL diffusion $$C = KL(P||Q) = \sum_i \sum_j p_{ij} \log \frac{p_{ij}}{q_{ij}}$$ FromGradient AnalysisSee: KL is asymmetrical. If $p_{ij}$ It's big, but... $q_{ij}$ Small (low-dimensional separation) and heavy punishment. On the contrary, if $p_{ij}$ It's small. $q_{ij}$ Very, very small punishment. Eventually.ResultYes: t-SNE is extremely good at it.Retain local structures(Consisting neighbours) but little of the overall structure (the distance between the cluster and the cluster is usually meaningless).

It needs to be explained in advance that the efficiency of t-SNE is not high; the speed bottlenecks in large-scale data are also one of the reasons for the subsequent introduction of UMAP.

Perplexity (disturbation): This is the main parameter of t-SNE, which is commonly taken between 5 and 50, which can be understood as"Estimated number of neighbours"I'm sorry. If set too small, the data will break into millions of small groups to make noise; if set too large, local details will be ignored and the result will become more and more like the PCA.

UMAP: Success in data analysis

UMAP (Uniform Manifold Approximation and Projection) is the current SOTA down-the-dimensional algorithm. It introduces a rigorous Lehman geometry and algebraic purge based on t-SNE, which solves the problem of t-SNE slow and loss of global structures.

Technical intuition: even distribution in flow shape

UMAP is based on one assumption:Data are evenly distributed in a Lehman-like shape. If the data appear to be unevenly distributed in real space (some are secret, others are alien), it's because we use it to observe."Rulers"(Oxley distance) It's wrong.

Measurement of self-adaptation (Liman measure): UMAP for each point $x_i$ Defines a partial Lehman measure. In the data-slipped area, UMAP would"Long"Distance; in a dense area, will"Shorten"Distance. This is achieved through k-neighborhood and a weighted kNN figure is constructed.

Fuzzy Simplifical Complex: By the above measure, the UMAP converts the data into a polo structure (simply double).

Optimizing target: binary Cross-Entropy:t-SNE uses KL only, only focuses on"Close the neighbors."I'm sorry. UMAP uses cross-breathing: $$CE = \sum p_{ij} \log(\frac{p_{ij}}{q_{ij}}) + \sum (1-p_{ij}) \log(\frac{1-p_{ij}}{1-q_{ij}})$$ of whichFirstLike t-SNE, generating gravity, pulling the neighbors. The second one concerns $(1-p_{ij})$It's punishment."Non-neighbors are being drawn in."And the situation, it's a situation.Global repulsion.I'm sorry. Eventually.ResultYes: UMAP keeps the cluster tight and forcesKeep the balance between the clusters right., thus retaining more global structures.

UMAP has more parameters than t-SNE. In addition to controlling local size, it is necessary to control the degree of compactness embedded; it seeks to retain a part of the global structure and can also demonstrate a certain degree of spatial expansion.

n_neighbors: Control the size of local maps. Small values (e.g. 5) capture high frequency details; large values (e.g. 200) capture global profiles (like PCA).

min_dist: Control the compactness of low-dimensional embedded. Small values (0.1) allow overlap and are suitable for cluster analysis; large values (0.8) force points are separated and suitable for displaying a pedestal structure.

Neural network and modern extension

Autoencoder (AE): Non-linear compression

AE and its derivative model are discussed in detail in the chapter on the self-codifier. Put it in the light of the downside, AE isParameterisationThe downside. It's the main one.DifferenceIt's: the flow method is non-parametric, which gives only coordinates and cannot process new data directly; whereas AE learns a function $f(x)$, can process new samples at any time.

Normal AEs usually have less of a de-dimensional effect than a specialized visualization method. For the t-SNE and UMAP scenarios, which often compress space to 2-3D, AE is more suitable to be reduced to several dozen or hundreds of dimensions as a feature pre-treatment first. (b) The features after extraction are then given downstream tasks; for example, VAE (Variational AE) And its variants are still commonly used in generating models (Stable Disfusion) and character decoupling.

Other important models in modern science (notable)

In bioinformatics (especially single-cell sequencing) and computer visual studies, in addition to UMAP, two models are of interest:

PHATE (Potential of Heat-diffusion for Affinity-based Transition Embedding): of whichIntuition.Use"The heat spreads."Process to simulate the probability of a transfer between data points. MainAdvantagesIt is: it is good at retaining data in ** trajectory** and branch structures (e.g. stem cell separation processes) and is better suited than UMAP to demonstrate the continuous evolution of data.

PacMAP (Pairwise Controlled Manifold Approximation): of whichIntuition.It's by designing special features."Medium Distance"Point-to-point, visible balance of local gravity and global repulsive force. CurrentStatusYes: It is considered a strong competitor for t-SNE and UMAP, usually more robust in retaining the global structure than UMAP and less sensitive to parameters.

Metric Learning

Finally, back to the original purpose of the decline. We are often reduced because the distance of the O'Delk is not working in the high space.Measuring learningIt's a reverse thought:Instead of mapping data into low-dimensional space to adapt to the Oxygen distance, you should learn a new distance measurement function. $d(x_i, x_j)$。

Yes.Ma's distance from studyIn this context: Learning a matrix $M$♪ Make the distance ♪ $d(x, y) = \sqrt{(x-y)^T M (x-y)}$ The similarities in data are best reflected.

Yes.Syamese Networks On the other hand, this is the mainstream of modern depth measurement. The mapping of two inputs into the characteristic space through the nervous network directly optimizes the distance between the characteristic vectors (e.g. Triplet Los), bringing the same type of sample closer and different types of sample far away.

A summary of other common dimensions

Beyond the main lines ahead, there are a number of downscaling methods that are common in the project, many of which are not commonly used. Only a brief summary is provided here, highlighting their relationship to the front-line approach.

Linear approach: LDA (linear determination analysis) is the supervised version of the PCA - to find a source when category labels are available"Maximum inter-group dispersion, minimum intra-group dispersion"The projection direction is often used for the downscaling before classification; the factor analysis (FA) assumes that observation variables are generated by a few public factors plus special factors, with greater emphasis on the interpretation of the structure rather than on compression.

Non-linear approach to retaining local structuresLocal linear insulation (LLE) assumes that each point can be reformed by a point linearity within the adjacent area and that the reconstructive weights remain unchanged, while the La Plas characterization (LE) allows the nearest point to be as close as possible through the t-SNE, UMAP, which is the same as the t-SNE, which is described above."Retain local structures"One family, only earlier and less frequent. SNE is the precursor of t-SNE, which also converts similarity to a condition probability, and t-SNE is just using a rear end t distribution to alleviate congestion.

Matrix decomposition perspectiveLSA, NMF and low-level approximation see the down dimension as a matrix decompose. Potential semantic analysis (LSA) is a low approximation of SVD for word-document matrix, often used in text subject space; the non-negative matrix decomposition (NMF) requires both factors to be non-negative, and therefore output is often interpreted as"Partial additions and additions"(e.g. theme, combination of components); low approximation is itself the search for the low matrix closest to the original matrix, with the exception of the PCA/SVD. The objectives of ICA (independent component analysis) are different - not to remove relevance but to find statistically independent elements, often used in blind source separation missions such as signal separation.

Nuclearization and migration learning: nucleination linear downscaling (KPCA, KLDA) introduces nuclear techniques into linear methods, and KPCA has been described earlier, where KLDA is the corresponding monitoring version; and migration learning downscaling (TCA) is oriented towards cross-domain scenarios, mapping source and target domains into the same common subspace, minimizing differences in area distribution.

A reminder.: Degraded, while providing computing and visualization facilities, oftenDamage error analysis and model interpretabilityThe characteristics of the projection are no longer original, the meaning of the business and subsequent interpretation are discounted and the loss is irretrievable in the vast majority of cases. The choice of a model is worth confirming whether these two elements are acceptable.

Summary: How?

Methodology	Core Math Thought	Reservations to global structures	Reservations to local clusters	Suggest scene
PCA	Disaggregation of the synonyms	[Extraordinary] (Excellently)	[Low] (distant)	Data preprocessing, baseline testing, linear data
MDS	Distance Matrix Reconstruct	[Extraordinary]	[Low]	When strictly keeping distance matrix
t-SNE	Probability distribution (KL diffusion)	[Low] (near all)	[Extraordinary] (Excellently)	Explore data analysis, emphasis on cluster separation degrees
UMAP	Popup Pure Double + Cross-Cross	[High] (Good)	[Extraordinary] (Excellently)	Current Preferred, take into account global and local, large data sets
PHATE	Hot-spreading information distance	[Extraordinary]	[High]	Data with continuous evolutionary trajectory (e.g. time series, biodevelopment)
AE	Neural network re-construction error.	[High]	[Low]	When character extractor is required for downstream tasks