Autoencoders and Variational Autoencoders: Reparameterization, KL Divergence, and ELBO

From basic to front, step by step, master the encoder architecture in in-depth learning

The questions in this article can also be addressedIn-depth learning base: neuronet, optimization approach and integration、In-depth learning network architecture: CNN, RNN and Seq2SeqHow the concept of a relatively close read together is developed in different contexts.

Custom Encoder Base

What's a self-encoder?

Core definitions

The cipher is a kind of...Self-supervised learningThe nervous network structure. Its core objective is not to predict labels, but to predict labels.Compression of learning input data, that is, to re-enter the output of the nervous network as far as possible.

Intuitive understanding.

You can imagine the encoder as"Compression-Depressure"Process:

Encoder: conversion of high-resolution images to a short code vector (condensed)
Codec: restore original images from code vector (discharge)

Core thinking

If a network can successfully recover raw data from a compressed code, the compressed code must contain the most central and important features of the data, leaving out noise and redundancy information.

Core structure

The self-codifier consists of three parts:

graph LR
    A[输入 x] --> B[编码器]
    B --> C[隐表征 h/z<br/>瓶颈层（潜在空间）]
    C --> D[解码器]
    D --> E[重构输出 x̂]

Mathistic Symbols Definition

$\mathbf{x}$: Enter data, dimensions are $d$（$\mathbf{x} \in \mathbb{R}^d$）
$\mathbf{h}$ or $\mathbf{z}$: hidden surfaces (potential variables), dimensions are $p$(usually $p) < d$）
$\hat{\mathbf{x}}$: Reconstruct output, dimensions and $\mathbf{x}$ Same
$f_\theta$: Encoding Functions, with parameters $\theta$
$g_\phi$: decode function, parameter is $\phi$

Forward dissemination and training

Encoding Phase $$\mathbf{h} = f_\theta(\mathbf{x}) = \sigma(\mathbf{W}_e \mathbf{x} + \mathbf{b}_e)$$

Decoding Phase $$\hat{\mathbf{x}} = g_\phi(\mathbf{h}) = \sigma'(\mathbf{W}_d \mathbf{h} + \mathbf{b}_d)$$

Training objectives $$\min_{\theta,\phi} \mathbb{E}{\mathbf{x} \sim \mathcal{D}} [\mathcal{L}(\mathbf{x}, g\phi(f_\theta(\mathbf{x})))]$$

Loss Functions

Average error (MSE) For numerical data (e.g. image pixel values): $ \mathcal{L}{\text{MSE}}(\mathbf{x}, \hat{\mathbf{x}}) = \frac{1}{N} \sum{i=1}^{N} \lVert \mathbf{x}_i - \hat{\mathbf{x}}_i \rVert_2^2$$

Diutsil Interpolation (BCE) For binary data or to be consolidated $[0,1]$ Data: $ \mathcal{L}{\text{BCE}}(\mathbf{x}, \hat{\mathbf{x}}) = - \frac{1}{N} \sum{i=1}^{N} \sum_{j=1}^{d} \left[ x_{i,j} \log(\hat{x}{i,j}) + (1 - x{i,j}) \log(1 - \hat{x}_{i,j}) \right]$$

Why do you need it?"Bottlenecks"

Problem

If the depth of the hidden layer $p \geq d$ And without any other constraints, the network can learn to be constant equivalent (in the form of a map)$\mathbf{h}=\mathbf{x}$It doesn't make any sense.

Solutions

Undeveloped self-encoderForce $p < d. Forcing the web to learn the most prominent features of the data, similar to the main components of the data captured by the PCA.

Expert perspective: If the active function is linear and the loss function is MSE, the full-fledged self-encoder is not equal to Main Component Analysis (PCA)I'm sorry. But because the self-encoder uses non-linear activation functions (e. g. ReLU), it learns more powerful than the PCANon-linear flow。

Noise-deductor (DAE)

Core thinking

Add noise to the input data and force the network to learn the roulette character.

Mathistically expressed $$00\ \text{noise input:} &\mathbf}=mathbf{x}+ \varepsilon, \varepsilon\m\mmmatbl{, \mathbf^2\mathbf} \text{training target:} &\mathbf} \\text{loss function:} &\quad \mathcal{L} = \lVert \mathbf{x} - g_\phi(f_\theta(\tilde{\mathbf{x}})) \rVert_2^2 \end{aligned}$$

Meaning: Forced the network to learn the character of a stick, rather than simply copying, and to extrapolate complete information from damaged data.

Rare Encoder (SAE)

Core thinking

Allow hidden dimensions $p > d$, but add in loss functionScreeching constraints。

Loss Functions $$\mathcal{L}{\text{SAE}} = \mathcal{L}{\text{reconstruction}} + \lambda \sum_{i} |h_i|$$

Activate near target thinness using KL-dispersible containment $\rho$： $$\mathcal{L}{\text{SAE}} = \mathcal{L}{\text{reconstruction}} + \beta \cdot D_{\text{KL}}(\rho | \hat{\rho})$$

Meaning: Limiting the same time to only a very small number of neurons activated, simulated the way the bioneurosystem works.

From AE to VAE: The qualitative leap

Although DAE and SAE have improved standards to some extent, they still cannot be solved.Generate new dataThis is the core issue. It's coming. VAE The birth.

VAE

VAE is one of the cornerstones of the deep generation model, which radically changes the generation capacity of the self-codifier by introducing probability distribution.

Why do you need VAE?

Review of the standard self-encoder

Map input to fixed vectors → Potential space is not continuous
Only"Compression"No, I can't."Generate"

VAE Core Insight

VAE does not map input into one"Points", it's a map.Probability distribution(usually in the Gaussian distribution):

I'm not saying."This is the coordinates. $(3, 2)$"
It's about..."This is the average figure. $(3, 2)$ A range around. Internal"

Key strengths

Introduce probability distribution and regularization.
Yes."1"and"7"The results of the decoded data are smooth transitions.
You can generate new data！

Core structure

VAE contains three key components: probability encoder, sample layer, code decoder.

Encoder

Input $\mathbf{x}$, neural network output distribution parameters:

$$\begin{aligned} \boldsymbol{\mu} &= f_\mu(\mathbf{x}) \ \log\boldsymbol{\sigma}^2 &= f_\sigma(\mathbf{x}) \end{aligned}$$

Mean vector $\boldsymbol{\mu}$: Central location of distribution
logarithmic vector $\log\boldsymbol{\sigma}^2$: Dispersion of distribution

Reparatification technique — Core point.

From Distribution $\mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\sigma}^2)$ Medium Sample $\mathbf{z}$ Pass it to the decodor.

Problem: Direct sampling is random,Not transposable. Reverse transmission cannot pass the gradient by random nodes.

Skills: Will randomize"Strip"Come out. $$\mathbf{z} = \boldsymbol{\mu} + \boldsymbol{\sigma} \odot \boldsymbol{\varepsilon}, \quad \boldsymbol{\varepsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$$

Meaning: Now $\boldsymbol{\mu}$ and $\boldsymbol{\sigma}$ Just a certainty parameter calculation, gradients are passed back to the encoder without hindrance, randomity is entered entirely $\boldsymbol{\varepsilon}$ Provision.

Codec

Enter sampler $\mathbf{z}$, output re-constructing $\hat{\mathbf{x}}$： $$\hat{\mathbf{x}} = g_\phi(\mathbf{z})$$

Maths: Deductive extrapolation

Target: Maximum logarithmic

The goal is to make the model generate real data. $\mathbf{x}$ Probability $P(\mathbf{x})$ Maximize: $$P(\mathbf{x}) = \int P(\mathbf{x}|\mathbf{z})P(\mathbf{z}) d\mathbf{z}$$

Because this is an incalculable fraction of a complex neural network, we cannot directly optimize it.

Delineation of the lower boundary (ELBO)

Include approximation distribution $q_\phi(\mathbf{z}|\mathbf{x})$(coding) Approaching real back check. $p(\mathbf{z}|\mathbf{x})$。

Mathically extrapolated (instinct), obtained $\log P(\mathbf{x})$ Bottom boundary: $$\text{ELBO} \underbrace{E}{\mathbf{z} \sim q}[\log p(\mathbf{x}|\mathbf{z})]}- \underbrace{D {text{KL} (\matbf{x}p}mathbf{z}} text{regulated entry}

Loss Functions $$\mathcal{L}{\text{VAE}} = -\text{ELBO} = \mathcal{L}{\text{reconstruction}} + D_{\text{KL}}(q(\mathbf{z}|\mathbf{x}) | p(\mathbf{z}))$$

Intuitive interpretation of the loss function

The VAE Loss is shared by two units."Fight."The power of the world:

Reconstructing loss

Role: Make decoded images as original as possible
Preferative: If you are not bound, you will make a difference $\boldsymbol{\sigma} \to \mathbf{0}$It's degenerated to normal AE.

KL Scatter

Role: Forced encoder output distribution nearing standard normal distribution $\mathcal{N}(\mathbf{0}, \mathbf{I})$
Preferative: If this is only optimized, the encoder will ignore the input and always output the standard noise distribution

BalanceVAE strikes a balance between the two, allowing the code to contain original information and allowing potential space to fit the normal distribution shape, ensuring continuity and generating capacity.

Why VAE generated blurry images

That's VAE's most famous weakness.ReasonVAE uses the Gaussian distribution assumptions and MSE losses. MSE tends to be all possible pixels Remove"Average"... leading to the loss of edge details, similar to over-wrought skin.

The normal AE/VAE is usually trained by MSE Los (the pixel-scale average error). MSE has a big problem, which is not sensitive to high frequency textures or which tends to produce “mixed averages”. But in mathematics, it keeps a lot of pixel-level redundancy that people don't care about.

The VQ-GAN or fine-tuned VAE used by LDM introduced Perceptual Los and Patchgan Discriminator. This change forces the compression model to focus on preserving the semantic structure and texture of the picture and its spatial relationship, while ignoring the meaningless random pixel-level noise.

VAE 's strengths and limitations

Advantages

Training is stable, unlike the way that gan is prone to a pattern collapse.
There are clear probabilistic models and mathematical interpretations
Potential space smooth, consistent, suitable for insertion and exploration
It's a valid reasoning.

Limits

Generating images tends to be vague (compared to PAN)
The potential dimensions of space need experience to choose
Limited ability to express certain complex data distributions

Beta-VAE: Characteristical detoxification

Standard VAE, Potential Vector $\mathbf{z}$ The dimensions are usually...- A entanglement.: Change a value may affect multiple properties (e.g. colour, size, angle) at the same time.

Solutions: adjust KL weights

Modify loss function to weight KL dispersive item $\beta$(usually $beta) > 1$）： $$\mathcal{L}{\beta\text{-VAE}} = \mathcal{L}{\text{reconstruction}} + \beta \cdot D_{\text{KL}}(q_\phi(\mathbf{z}|\mathbf{x}) | p(\mathbf{z}))$$

Rationale

By increasing $\beta$, forced potential distribution $q_\phi(\mathbf{z}|\mathbf{x})$ Strict adherence to standard normal distribution (dependency of dimensions). This kind of constraint forces the model to look for data. MediumBest effective, most independentFactor:

A dimension only controls colour
The other dimension only controls shape.
The third dimension is only an angle.

It's calledDistantReduction。

Cost and trade-offs

$\beta$ Bigger:

Potential space is more entanglement and more interpretable.
Reconstructing images is usually blurry (reconstructing error weights is relatively small)

A balance needs to be found between the level of de-tachment and the quality of the re-construction.

Theoretically.

$\beta$The-VAE is a significant contribution to the theory of DeepMind, which:

It reveals the relationship between potential spatial structures and the solution to the problem.
New ideas for explaining AI
Applications in areas such as intensive learning, robotic control, etc.

Revolutionary breakthrough - VQ-VAE

This is the VAE family.Most revolutionaryOne of the members, proposed by Google DeepMind, broke"Potential space must be continuously distributed by Goss."The dogma.

- It hurts.

Standard VAE assumes that potential variables are continuous, which leads to:

The resulting image edge is blurred
The language, the logical concept of human beings is oftenDisperse(e.g."Cat."、"Dog."(Classification concept)
Could not connect to Transformer and Language Model

Solutions: Codebook mechanism

VQ-VAE IntroductionCodebook The concept.

Forward transmission process

Encoding Encoder Output Continuous Vector $\mathbf{z}_e(\mathbf{x})$

Quantification of vectors Finds the nearest vector in the code book: $ \mathbf{z} (\mathbf{x}) =\text{Codebook}[k^]], \quad k^ = \arg\min_k \lVert \mathbf{z}_e(\mathbf{x}) - \mathbf{e}_k \rVert_2$$

of which $\mathbf{e}_k$ It's the first in the code book. $k$ A code vector.

Decoding The decodor receives a quantified vector $\mathbf{z}_q(\mathbf{x})$, output re-constructing $\hat{\mathbf{x}}$

Direct to estimate

Because"Checklist"and"Quick Search"is not a guideable operation, VQ-VAE is used Straight-Through Estimator：

Forward transmission: Using a quantified vector $\mathbf{z}_q$
Reverse transmission: Copy gradient directly to encoder output $\mathbf{z}_e$

$$\nabla_{\mathbf{z}e} \mathcal{L} = \nabla{\mathbf{z}_q} \mathcal{L}$$

Loss Functions

$$\mathcal{L}=underplace(lVert\mathbf{x} - \\mathbf}\rVert 2^} underbrace{lVert\sec}}(mathbf}) - \mathbf{e}k \rVert_2^2}+ \lderbrace \lVert\mathbf{z} e (mathbf{x} -\text{stop}k) \rVert_2^2}- Next{encoding committee}

Why, VQ-VAE is revolutionary.

It's very clear.

Forced use of discrete, high-quality character codes, leaving vague intermediates behind.

Connect Transformer

This is the most critical innovation. Because potential space becomes discrete (like word Token), images can be transformed into a series of Token sequences, using GPT/Transformer directly to process images!

Practical application

OpenAI DALL-E 1 VQ-VAE based variant
Audio Generation Model MusicLM、AudioLM
Multi-modular Models (e.g., the visual capability component of GPT-4o)

Why VQ-VAE Important

VQ-VAE YesBridges to visual and language：

Disperse continuous images into Token
Make possible a unified language-visual architecture
Laying the foundations for a large multi-modular model

Modern Extension - MAE

Not even in the name."Variational"But MaE is Autoencoder's idea of Transformer TimesThe continuation.

Background: From BERT to visual

BERT, kill the Quartet in the NLP field."Full Fill"Thought. Mae moved this idea to computer vision.

Core approaches

Image Segment Cutting pictures into small pieces, such as $16 \times 16$ Pixels.

Random mask Throw away 75% of the pieces at random.(Masking) - Note the high rate!

Encoding Only the remaining 25 percent is fed to the encoder.

Decoding The decoding device is responsible for completing the 75% of the pieces that were thrown away.

Mathistically expressed

Mask Policy $$\mathbf{M} \in {0, 1}^{N \times N}, \quad \sum_{i,j} M_{i,j} \approx 0.25 \times N \times N$$

Restructure the target. $$\mathcal{L}{\text{MAE}} = \frac{1}{|\mathcal{U}|} \sum{i \in \mathcal{U}} \lVert \mathbf{x}_i - \hat{\mathbf{x}}_i \rVert_2^2$$

of which $\mathcal{U}$ is the index of the masked block.

Why is MaE working?

Force semantic learning If you don't understand,"Dog."The semantics of the dog cannot be filled with the veiled head.

Efficient pre-training

Only 25% of data processed, calculated efficiency High
The masked mission forces models to learn global dependency.

Meaning and impact

MAE proves Autoencoder structure is in progressSelf-supervised learningThe Great Tool:

Without labels, the model can understand the semantics of the image.
Now, many high-performance visual models are trained in this way.
To lay the foundation for the success of Vision Transformer's computer vision.

Integrated application scenario

The self-codifier and its variants are very widely applied in practice. The following are the main areas of application.

Decline and Visualization

Similar to t-SNE or PCA, compress high-dimensional data to 2D or 3D for visualization or reduce the amount of calculation as a pre-processed step.

Unusual detection

Core logic: With a large amount"Normal Data"Training AE. When Input"Unusual Data"This is when the re-engineering error increases significantly.

Decision Formula $$\text{Anomaly}(\mathbf{x}) = \mathbb{I}[\mathcal{L}(\mathbf{x}, \hat{\mathbf{x}}) > \tau]$$

Apply: Credit card fraud detection, early warning of industrial equipment failure

Image to Noise and Fix

Use the denocator idea:

Remove the noise from the old picture.
Complete the masked part of the image

Feature extraction and pre-training

When label data are scarce, a large amount of unlabelled data is used to train the self-codifier. And then keep it.EncoderPartially, access to classification layers fine-tuned.

This method is widely applied in models such as BERT.

Generate new data (VAE)

After training, the encoder is discarded. Directly from $\mathcal{N}(\mathbf{0}, \mathbf{I})$ Sample Random Vector $\mathbf{z}$, feed the decoding device to create an absence of a human face or scene. Mainstream generation models have largely ceased using VAE as the generation structure, but instead used it for compression.

Potential space plug-in (VAE)

Take two figures A and B, coded separately $\mathbf{z}_A$ and $\mathbf{z}_B$。

Calculates the intermediate vector: $$\mathbf{z}_{\text{mid}} = \alpha \mathbf{z}_A + (1-\alpha)\mathbf{z}_B, \quad \alpha \in [0,1]$$

Decoding $\mathbf{z}_{\text{mid}}$, you can see that Figure A smoothly becomes Figure B.

The blog is a blog for the Global Voices community.$\beta$-VAE）

Jean. $\mathbf{z}$ Each dimension controls the independent feature.

For example:

$z_1$ Control hair
$z_2$ Controlling the colour of skin
$z_3$ Control angle

Adjustment $z_1$ , only the color changes, the rest remains unchanged.

Characteristic decoupling

Determines what dimensions of the code represent what information.

For example, in a 100-dimensional vector, the first 50-dimensional represents the sentence content and the second 50-dimensional represents the talking person character.

Disperse the hidden signs.

Forced encoded as a single heat vector (only 1 dimensional and the rest 0), which allows for unsupervised classification.

For example, handwritten digital recognition (0-9) and training self-coding machines to force 10-dimensional code to be a single heat vector. This 10 unique thermal codes may correspond to a single number each, leading to a complete unsupervised classification learning.

Data compression

Encoder output is a low-dimensional vector and can be considered as a direct compression result:

Encoder Executes Compression
Decoding Decoder Executing Decompression
It's a decompression.

Stable Diffusion - Most important application

This is the most important application at this time.I'm sorry. Stable Diffusion, actually, is called"Latent Diffusion Model"：

Not just processing large pictures in pixel space.
First VAE Compress pictures to potential space
In this small space, it is spreading.
Last use. VAE DecodingRevert to "Big Chart"
The self-coding is already the basis of the current mainstream generation model.

Multi-modular Model (VQ-VAE)

DALL-E:VQ-VAE discrete expression +GPT
MusicLM: Dispersion of audio + language model
GPT-4o: Visual ability is based in part on similar discrete expressions