From LLM to VLM: How Language Models Learn Visual Understanding

This English version was machine-translated from the Chinese original. Technical terms may require verification.

The development of large polymodular models is broadly along a technical path: Starts with a pure language LLM, and then CLIP achieves visual-linguistic alignment, then produces VLM, and a more original multi-modular modelling. This paper is a synthesis of this technology and analyses the underlying principles and architecture of the various models.

The questions in this article can also be addressedJoyAI-VL-Interaction: Return from Chat to a continuous interactive visual language model、The difference is good: starting with DeepSeek not using n-gram structuresHow the concept of a relatively close read together is developed in different contexts.

The paper contains three main parts:From plain text to multiple mosaics(LLM CLAIP PLAYING TECHNOLOGY PLAYING)Progress topic(scatter models and primary polymorphs), andAppendix(Basic conceptual supplements, including Cross Attention, Cross-Cyber, Softmax, etc.).

From plain text to multiple mosaics

Multi-modular AI development is not a one-off exercise. More commonly, the route is to train text models and then introduce visual encoders, cross-modular alignment layers and generation interfaces.Each phase is re-energized by a part of the previous phase ' s capacity, while at the same time exposing new interface problems.

Evolution Path:

graph LR
    A[纯语言LLM<br/>基础] -->|扩展视觉编码能力| B[CLIP<br/>视觉-语言对齐]
    B -->|结合LLM的生成能力| C[生成式VLM<br/>多模态理解与生成]

This section will dismantle the technical rationale of these three phases, highlighting the inheritance relationship and evolutionary logic between them.

Phase 1: Pure language model LLM - a single-modular intellectual engine

LLM is the starting point for understanding the subsequent multi-modular architecture.I'm sorry. See how the language model handles token, how to generate text, and then see how CLIP and VLM can access visually.

LLM isGenerating models that process text molluscs only, using Transformer Decoder-only structures, learn language distribution through extensive pre-text training. The common training goal is to improve mission performance by adjusting to human alignment by following instructions for the next Token (Next Token Protection) from the regression prediction.

But the input interface limits for LLM are clear:It can only process text directly and cannot receive multiple mosaics of images, audio, etc.I'm sorry. So multi-modular extension starts with a engineering question:How to get LLM "Yeah."Images? Two technical routes emerged as a result:

1. CLIP routeBuild visual-linguistic alignment to embed images and text into the same vector space
2. VLM route: Load LLM directly"Eyes."Let it see and speak.

Phase 2: CLIP - Building a bridge between visual and linguistic

CLIP (Contrastive Language-Image Pre-training)Aligning visual and languageIt's a reusable embedded problem. It is not a model that is generated, but rather embedded: images and text are mapped into the same vector space and compared by similarity.

Relationship with LLM:

• CLIP.Text EncoderThe Transformer structure that directly inherited LLM.
• CLIP reuses LLM text understanding skills but migrates them to visual-linguistic alignment tasks
• CLIP's training target is from"Forecast the next Token."Change to"Zitru Dock"

Core positioning: visual-linguistic alignment model

CLIP can be understood asEmbedding ModelI'm sorry. It maps images and text into the same low-dimensional vector space (insulated space) and allows for a close text to be compared to the space.

Problems addressed:

• Pure visual models (e.g. ResNet) can only output fixed category labels and cannot understand natural languages
• Pure Language Model Not Available"Yeah."Image
• CLIP provides embedded interfaces between visual and language, and achieves a symmetric alignment across the mosaics

Structure design: Double-Encoder

CLIP uses"Double Tower"The design is straightforward - the visual encoder and the text encoder work independently and do not interfere with each other, but only interact at the end through similarity calculations.

Visual Encoder

Foundation: usually used ViT (Vision Transformer) Series (e.g. ViT-L/14 or ViT-g)

Workflow:

1. Image Segment: Split input images $N$ Small squares, for example $16 \times 16$ Pixels
2. Spreading and Maping: Each Patch is set to a vector by linear mapping
3. Add Location Encoding: Add location information for each Patch
4. Transformer Process: feature extraction and interaction through multilayer Transformer
5. Output Sampling: Take special [CLS] Token The output vector is expressed as a global visual expression

Output: A person who represents the semantics of the imageGlobal Characteristic Vector, the dimensions are as follows: $d=512$

Text Encoder

Foundation: Standard Transformer Encoder()Inheritance from LLM Structure)

Workflow:

1. Interword: text is divided into Token sequences by Tokenizer, if [SOS, A, dog, is, running, EOS, PAD...]
2. Embedded: Convert each Token to a vector
3. Transformer Process: Multilayer Self-Attention allows every Token"Yeah."Context
4. Output Sampling: Take [EOS] Token(End of Security)

Output: a word for the whole sentenceGlobal Text Vector (Text Embeding)

Alignment mechanism

The two towers work independently.There is no complex cross-mode interaction layer.I'm sorry. They only calculate two vectors at the end.Cosine SimilarityInteractive.

$$ \text{Similarity} = \cos(\theta) = \frac{v \cdot t}{|v| \cdot |t|} $$

of which $v$ The video shows the visual embedding of the vector, which is a very important example of the visualization of the vector.$t$ is the text embedded vector (Text Embedding).

Training target: Comparative Learning

The main change in the CLIP is the way in which it is trained:Comparative learningI'm sorry. Its objectives are clear:Let the matching text close to the distance, and the mismatch text push away.。

InfoNCE Los (standard practice of the CLIP)

Basic thinking: In a Batch, the text matching is considered a multi-classification problem.

scene settings: Suppose there's a battling $N$ Yes, it is. $(I_1, T_1), (I_2, T_2), ..., (I_N, T_N)$I'm sorry. of whichPositive sampleIt's a pair of horn lines. $(I_i, T_i)$, i.e. a matching text pair; andNegative sampleIt's for pictures. $I_i$The rest of the same Batch. $N-1$ Text $(T_j, j \neq i)$。

Calculating Process: first calculate the two-cosine similarity of all the text vectors, the composition Construction $N \times N$ is the likeness matrix. For the first $i$ An image that is matched to the correct text by subdivisioning its similarity to all text into a probability distribution through the Softmax function $T_i$ and predict the probability. The loss function uses the classic multi-category cross-brenade loss:

$$ L_i = -\log \frac{\exp(sim(I_i, T_i) / \tau)}{\sum_{j=1}^{N} \exp(sim(I_i, T_j) / \tau)} $$

Symbolic Interpretation:

• $sim(I_i, T_i)$: No. $i$ Chart and $i$ Similarity of paragraph text
• $\tau$ (Tau):Temperature coefficient (Temperature)Controlling distribution"Pointy"
  • $\tau$ Small: fine differences in similarity are magnified and models focus on the most difficult negative samples to distinguish
  • $\tau$ Large: distributions become smoother
• $\exp(\cdot) / \sum \exp(\cdot)$: Softmax function, converts a similar value to probability

Basic logic: InfoNCE forced the model to $N$ options"Pick out"Right about that. For positive samples, the gradient brings the two in close proximity; for negative samples, the gradient keeps the two away.

Limitations: InfoNCE relies on Batch Size. The bigger the Batch, the more negative the sample, the harder the mission, the better the characteristics the model learned. If the bat is small, the model is easy to guess randomly.

SigLIP (Sigmaid Los improved)

SigLIP is a Google-driven improvement that is based on the idea of translating the issue of multi-classification into one of the most important ways to improve the situation of the population. $N \times N$ (b) A separate issue of two classifications. Unlike InfoNCE, SigLIP no longer relies on Softmax for global integration, thus avoiding the calculation of the denominator and the resulting communication costs in distributed training.

Algorithmic principles: For each element in the likeness matrix $(i, j)$, separate target labels depending on whether they are positive samples. When? $i=j$ ♪ When the positive sample is right, the label ♪ $y_{ij}=1$, expect Sigmoid output close to 1; $i \neq j$ ♪ When negative samples are right, label ♪ $y_{ij}=0$Sigmoid output is expected to be close to zero. Loss function defined as:

$$ L = - \frac{1}{N} \sum_{i}\sum_{j} \left[ y_{ij} \log \sigma(z_{ij}) + (1-y_{ij}) \log (1-\sigma(z_{ij})) \right] $$

Symbolic Interpretation：

• $N$: Number of graphics in the Batch
• $y_{ij}$: Second classification label,$y_{ij}=1$ Which means positive samples are correct. $(i=j)$，$y_{ij}=0$ The negative sample is right. $(i \neq j)$
• $z_{ij}$: Section $i$ Chart and No. $j$ Similarity fractions of paragraph text
• $\sigma(\cdot)$: Sigmoid function, replacing multi-classical Softmax

This approach has two direct advantages: first, because the total sum of the synchronized denominator between GPUs is not required to support super-large-scale Batch training (e.g. 32k scale); and second, because in the same size of the model, SigLIP is generally better performing than InfoNCE in the zero sample classification task.

CLIP embedded space alignment and capability boundaries

CLIP has acquired two types of common-purpose capability by building a unified visual-linguistic embedded space:Zero sample classificationandCross-modular Search。

In the case of zero sample classification tasks, CLIP does not need to be trained for specific categories, but simply converts the category name into a natural language description (see also the table below). Like"A photo of the {class}") to calculate the similarity of the image to the description of the text in the various categories, and to complete the classification. It's a very low cost of cross-model access, either."Search the text."Or is it?"Search the map in writing.", the CLIP can complete vector point count at milliseconds.

However, this embedded space also has a cost. CLIP encoders tend to remain under a comparative learning driveGlobal Semantics, and loses a portion of the details (e.g. OCR text, number of objects, precise space location). Training objectives are already met by a broadly matched text.

Another limitation is that the CLIP is only encoder, not coder, so it's not possible to execute it."Forecast the next word". It can judge."Do you think so?"、"Isn't that right?"But not directly."Say it."Image content and lack of expertise in generating complex scene descriptions. VLM addresses this interface gap: after the visual features are aligned, they are sent to the language model, which generates text.

Phase 3: Generating VLM - Putting eyes on LLM

CLIP's settled."Visual-Language Alignment"Problem, but it can't."Talk."I'm sorry. The generation interface is supplemented by the generation VLM (Visual Language Model):Give LLM access visual features so that it can generate text answers around images。

Representative models include LLAVA, InstractBLIP, Qwen-VL, MiniGPT-4, which usually extract features from CLIP visual capabilities and then align them through layers"Translation"LLM is given the final opportunity to produce text using LLM ' s expression skills, thus achieving detailed description, reasoning and question and answer questions on the content of the image.

Succession to LLM and CLIP:

• VLM's.LLM directly to the bottom of the language(e.g., Vicuna, Qwen-7B) inherits its powerful linguistic expression skills
• VLM's.Visual encoder directly reuses CLIP(e.g. CLIP VIT-L/14) to inherit the visual manifestations they have learned
• VLM's innovation is thatAlign Layer, the visual features of the CLIP"Translation"To understand the LLM.

Structure design: three-part structure

Generating VLM"Visual encoder - Language Model - Modular Aligning Layer"The three-part structure with clear division of labour and synergies between components.

Visual EncoderThe design philosophy of the image characterization is:Reuse pre-trained CLIP/SigLIP(Freezing parameters). CLIP has mastered excellent visual manifestations through comparative learning, and direct re-use can save the process of starting training while lowering training costs and preventing catastrophic oblivion. Early on, Qwen-VL adopted CLIPVIT-based structures, and the InstractBLIP used ViT-g/14, VLM directly inherited CLIP's visual encoder without having to retrain visual representation.

Language Model Bottom (LLM Backbone)The philosophy of design is the same.Reuse Plain Language LLM(e.g., Vicuna, Qwen-7B). Pure LLM has acquired powerful linguistic expressions, logic and world knowledge, and direct reuse can quickly acquire multimodular capabilities and maintain original pure text capability. LLM receives visual features (as special"Visual Token") and text Prompt, self-regression generation to predict the next Token. The use of Vicuna (a fine-tuned Llama), the use of Qwen-7B by Qwen-VL, and the use of Qwen-7B by LaVA are also used by Vicuna, where VLM directly re-references LLM pre-training weights, which inherits its linguistic understanding and earning capacity.

Model Aligning Layer (Adapter/Projector) is an important interface in the VLM architecture, as"Translator"It's for visual features."Translation"The language that you can understand is the language of the LLM. Two mainstream design concepts currently exist, representing direct mapping and fine extraction of technical routes.

Option I: LLAVA (Linear/MLP Production) — Simple and efficient

LLAVA uses the most direct two-tier MLP structure (Linear →Gelu →Linear), the underlying assumption being that visual features and language are embedded in a linear space to be matched.

• Architecture:

  • Visual encoder: CLIP Vit-L/14 (freezing parameters)
  • Align: two floors MLP
  • LLM: Vicuna/Llama (blip, Full Fine-tanning or LoRA)

• In the process of deduction:

  1. Picture input ViT, output all Patch feature sequences$H_v$(if$576 \times 1024$)
  2. $H_v$After alignment, the dimensions are mapd from 1024 to 4096 in LLM.
  3. Output vector $H'v$ considered"Visual Token"
  4. Text Prompt was converted to Embeedding by Tokenize$H_t$
  5. Will $H'v$和$H_t$拼接：$[\text{Token}{\text{Visual}}, \text{Token}_{\text{Text}}]$
  6. Throw it to LLM for Next Token Prevention

• Reverse dissemination mechanisms:

  1. Calculating Cross-Entropy Loses
  2. Gradient from Los →LLM →ScopeMLP →VIT
  3. Gradient to output end stopped due to the ViT freeze
  4. Only update MLP and LLM parameters

Option II: InstractBLIP (Q-Former) - fine extraction pie

InstractBLIP considers that throwing all the Patches directly to LLM is too redundant and slow to calculate, and therefore uses Q-Former (Querying Transformer) structures to achieve compression and dynamic extraction of information.

• Architecture:

  • Visual encoder: ViT-g/14 (freezing)
  • Aligning layer: Q-Former (lightweight BERT structure)
  • LLM: Vicuna/Flan-T5 (freezing or LoRA)

• Working mechanisms of Q-Former:

  Q-Former's core is32 Query Vectors for Learning, the characterization of the command sensor is obtained through two layers of Attention:

  Step A: Self-Attention (mixed text with Query)

  • Enter: [\\text{Query}{\text{Learned}}, \text{Token}TTt =$Close
  • Learned Queries interacts with Text Tokens, adjusts themselves to the text
  • For example: the text is"Find the dog.", the Query vector becomes"Looking for dog-like features."shape

  Step B: Cross-Attention (extracting information from images)

  • Q(Query): Queries that integrate text command information
  • K(Key): frozen ViT image feature, map by linear layer
  • V(Value): frozen ViT image feature, map by linear layer

  The mathematical process of Cross-Attention is: $$ \text{Attention}(Q, K, V) = \text{Softmax}\left(\frac{Q \cdot K^T}{\sqrt{d_k}}\right) \cdot V $$

  32 Query vectors are calculated to be similar to 257 Image Patches, as Q has been integrated"Find the dog.", on behalf of"Dog."Image Patches received high marks, and eventually Query vectors"Suck it away."The most relevant visual information is given, and irrelevant background is ignored.

Training strategy: multi-phase training

The training of the VLM is multistaged and gradually upgraded.Pre-training alignment phaseIt's about letting the LLM"Read it."Visual features, which at this stage freeze visual encoders and LLM, are trained only in the alignment layer, using large-scale graphic matching (e.g. Concaptual Captions, COCO) to match the task with the graphic, so that the alignment layer learns to map visual features into embedded spaces that LLM understands.

Command fine-tuning phaseThe modeling community answers the questions by drawing pictures. At this point, LLM (or part of the layer) and alignment layers were unfrozen using mission data such as VQA, Caption, Dialogue and so forth, for training in cross-blanc losses (Next Token Protection). Through multitasking data, modeling institutions combine visual understanding and language generation to develop a full VQA capability.

Balance mechanismIn order to prevent LLM from forgetting language knowledge or hallucinogenic, it is common to mix pure text data in training data to ensure that models retain their original language understanding while acquiring multi-modular capabilities.

Capacity and constraints

The generating VLM has a strong multimodular understanding: it supports visual questions and answers (VQA), image descriptions, OCR text recognition, and can be reasoned on the basis of images, such as writing codes, creating poetry, explaining scientific concepts, etc.

However, VLM also has significant limitations:

• High cost of reasoning: needs to run both ViT and LLM, with a visible demand usually above 16GB
• The problem of hallucinations is very strong.: The model may describe what does not exist in the image because the pattern is not fully aligned - visual encoder features may be missing and the alignment transmission signal is weakened, resulting in LLM cause"I can't see."And rely on language a priori to guess, for example, to describe a white cat as"White cat in red hat."
• Limited capacity to understand details: The loss of information from the CLIP encoder makes it difficult to identify fine words, count and understand small details

Evolving Summary: Succession and Innovation of the Three Level Models

And we can put the contents together, and the relationship between LLM, CLIP, VLM can be organized into this table below.

Comparative table:

Contrast Dimensions	Plain Language LLM	CLIP type model	Generating VLM
Enter a mollusc	Text only	Image + text (independent two-channel)	Image/Video+Text (Mixed Series)
Basic structure	Single Tower Transformer Decoder	Double Tower Encoder (ViT + Text Encoder)	Dota Integration (View Encoder+ Alignment Layer+LLM Decoder)
Training objectives	Forecast the next Token.	Textbooks Compare Learning	Forecast the next Token (input visual features)
Scope of competence	Text Task	Match, classify, retrieve	Text understanding, description, question and answer
The reasoning.	Generate Word by Word	One-time calculation vector point, extremely fast	Visual encoding (slow) + Word by Word generation, highest delay
Typical application	Translation, summary, logical reasoning	Zero sample classification, graphic search	VQA, image description, OCR

These three are not substitutes for each other, but...Reuse and CompletionRelationship:

CLIP inherits the text capability of LLM:

• CLIP.Text EncoderThe same Transformer structure as LLM.
• CLIP has reverted to the LLM's text understanding and presentation capability.
• CLIP is innovative in extending text capability to visual-linguistic alignment tasks

2. VLM succession to both LLM and CLIP:

• From LLMVLMDirectly re-enact LLM from the bottom of the language.It inherited its language expression, its logic and its world knowledge.
• From CLIPVLMVisual encoders go straight to pre-trained CLIP.And it inherited the visual expression that it had learned.
• VLM Innovation: In designAlign Layer, the visual features of the CLIP"Translation"To understand the LLM.

3. Commonality of the three:

• Structure Same Source: Main structures are from Transformer
• The expression is similar.: Consisting discrete information (pixels, words) into high-dimensional and dense semantic vectors (Embeding)
• Data driver: Self-monitoring/semi-monitoring pre-training, which relies on large-scale Internet data

The topic of progress - proliferation models and original multimodules

Based on an understanding of CLIP and VLM, this section discusses two progressive topics: how CLIP in the proliferation model leads to image generation, and from"Suture."The structure goes to the original multi-modular technological evolution.

CLIP in the proliferation model: How does the text guide the image generation?

The core of text understanding of image generation models such as Stable Diffusion remains CLIP Text Encoder。

Text Encoding

User inputtips like"A cyberpunk city", CLIP Text Encoder processes text and output features. Unlike VLM,This isn't just the last EOS vector.I'm sorry. Stable Diffusion uses CLIP text code DeviceFull Token Sequence Output on the Last Level, output shape is$77 \times 768$(Assuming maximum length 77, dimension 768) this retains a separate semantic information for each word.

Injection U-Net (Injection)

The heart of Stable Diffusion is one.U-NetIt's responsible for predicting noise and denocating. U-Net's full.Cross-Attention Layer。

• Q (Query)FromImage feature for the U-Net current layer(Noise Chart Generating)
• K (Key) & V (Value)FromCLIP text feature sequence($77 \times 768$)

Physical meaning of the generation process

When U-Net processes a pixel area of the image, it asks as Query:"What should I draw here?"It compares it with 77 tokens of CLIP, if there is one in the text."city"This word, and the corresponding Key matches the current Query area, then"city"The corresponding Value will be weighted. Eventually, U-Net, based on semantic maps provided by CLIP, took random noises one by one"Sculpt"becomes an image that matches the description of the text.

Role: CLIP for generating models"Navigation maps", U-Net gradually adjusts noise to the semantic direction of CLIP Embeding.

From"Suture."To original multiform

The LLAVA, InstractBLIP that was discussed earlier is owned by"Suture."(Glue approach): Take ready visual and linguistic models and stick them in a paired layer. There are two obvious limitations to such a programme.

"Suture."Issues:

- Yes.Information is compromised.I'm sorry. CLIP, which is designed for comparative learning, tends to retain the global semantic and discard details (e.g. OCR text, number of objects, spatial location). For example, VLM has difficulty in seeing small words because ViT has been able to compress this information as early as the coding phase.

Two.Modular GapI'm sorry. LLM is not direct."Yeah."Images, which are translated mathematical vectors, have natural modulation boundaries.

Native multiple.(e.g. GPT-4o, Gemini, Chameleon)End-to-End Early FusionThe concept of (end-to-end early integration) addresses these issues. The core approach is no longer using CLIP, but trainingVisual Tokenizer(e.g. VQ-VAE), slice images and convert them intoSeparated Token ID(e.g. Token #482 for a texture) then it will be $[\text{Token}{\text{Text}}, \text{Token}As a hybrid sequence, train a huge Transformer.

The advantages of this structure are that the model can output images token and generate images directly (no external connection to Stable Diffusion); that understanding is more detailed and no longer subject to pre-training objectives of the CLIP; and that supportIntersect Input Output(Texts in mix)

VQ-VAE: Paranormal multimodular visual Tokenization

The original multimodulars seek to turn images into similar text.Discrete Tokens (dispersed Token)。

Core objectives

One.$256 \times 256$And the image becomes a series of integers:[382, 10, 998, ...]I'm sorry. So that the LLM can predict the next word, like the next word."Image Block"。

VQ-VAE Structure

VQ-VAE (Vector Quantized - Varial AutoEncoder) has three parts:

Encoder (coding):

• CNN compresses images into low-resolution feature maps grids (e.g.$32 \times 32$vector)

Codebook -- key:

• I saved it.$K$Learning vectors (e.g. 8192) as$e_1, e_2, ..., e_K$
• This is a..."Dictionary"

Quantification (Quantification-Check Dictionary):

• For each vector on the feature map, found in CodebookMost likeThe vector.$e_k$
• Core Operations: Directly$e_k$Replace Encoder Output
• Index to Record$k$- That's it.Visual Token

Decoder (Code decoder):

• Reassemble features using the vectors in Codebook Figure
• Reverting to pixel images by inverse volume

The problem of reverse transmission:

Problem:VAE 's quantitative operation (recent neighboring argmin) isNot transposable- It's impossible to calculate."Take Index"The gradient of this operation.

Solutions: Same way VAE solved this problem Straight-Through Estimator (STE)

• When forward transmission: quantify, replace encoder with a Codebook vector
• When it's being transmitted backwards:Fraud Gradient- The gradient that sent Deoder back jumped over the quantitative layer and copied it to the encoder output unmoved
• Logical: Although the middle is broken, the gradient passes directly through the codebook vector and encoder output, assuming it's close enough.

Combined with LLM

Once you've trained VQ-VAE:

1. Image past Encoder → Quantification → Get Token Sequence
2. Image Tokens and Text Tokens.
3. Train Transformer predictor sequences
4. When GeneratingLLM predicts Image Token ID → to check the vector in Codebook → to throw to Decoder → to generate pixels Figure

This process allows for multi-modular understanding and generation into the same end-to-end training process.

Concluding remarks

From LLM to CLIP, to the generated VLM, the change occurs mainly in two locations: how input is expressed and how the different modes are aligned.LLMThe text generation base is provided.CLIPThe video is also available in the following video:VLMThe visual feature access language generation; the diffusion model uses a text encoder to inject language tips into the image generation process, and the original multimodular route further attempts to convert images into a Token sequence that can be processed by Transformer.

These routes are not entirely for who they are. CLIP is suitable for retrieval and matching, generating VLM is suitable for generating answers around images, spreading models are suitable for text to image generation, and original multimodulars attempt to place understanding and generation in the same end-to-end training process. When doing the system, the choice of route depends on whether the mission requires a search, question and answer, generation or a more finer typologies interaction.

Understanding the rationale, structure differences and applicable boundaries of these models will help to judge whether a new paper is changing the visual coding, aligning interfaces, generating methods, or adjusting training objectives.

Appendix: Detailed Core Foundation Concepts

The core underlying concepts in the contents of this appendix are included for further learning.

Cross Attention

Core definitions

Cross Attention is a focus mechanism that allows models to refer to and integrate information from another sequence when dealing with one sequence. It's mathematically expressed as:

$$ \text{Attention}(Q, K, V) = \text{Softmax}\left(\frac{Q \cdot K^T}{\sqrt{d_k}}\right) \cdot V $$

Of which Query comes from one source, Key and Value.

Distinction from Self-Attention

Self-Attention: Q, K, V are all from the same input to capture the dependency relationships within the sequence.

Cross-Attention: Q comes from one sequence, K and V from another sequence, which is used to integrate information across the sequence.

Apply scene

• Machine translation: Decode query Encoder's hidden status through Cross Attention
• Multimodular Generation: Image features of U-Net in the Sttable Diffusion merge CLIP text features through Cross Attention
• Visual questions and answers: Question Question Question Query image features to get the answer thread

Cross Entropy

From information theory to loss function

Cross-breathing is derived from the theory of information and is used to measure the difference in the two probability distributions. Understanding its application in depth learning requires retroactivity of its sources of informatics.

Information Content: For probability of occurrence $p(x)$ Events $x$, the amount of information is defined as: $$ I(x) = -\log(p(x)) $$

Intuitive understanding: when the probability of an event is lower, we feel the more."Surprise.", the greater the amount of information obtained.

Entropy: Random Variables $X$ The entropy is the expectation of its amount of information: $H(P)=\matbb{E}{P}[I(X)] = - \sum{x} p(x) \log(p(x)) $$

Entropy means the distribution using the best code $P$ The average bits used to encode are the lower bounds of the amount of information.

Cross Entropy: Use Based Distribution $Q$ Encoding to encode from distribution $P$ Data: $$ H(P, Q) = - \sum_{x} p(x) \log(q(x)) $$

Category II scenario: dual cross-cracker

For the issue of the sub-classification, labels $y \in {0, 1}$, the probability of the model predicts is $\hat{y} = \sigma(z)$, of which $\sigma$ It's the Sigmoid function.

Probability distribution means:

• Real Tab $y=1$: $P = [1-p, p]$ of which $p$ It's a positive probability.
• Real Tab $y=0$: $P = [1, 0]$
• Model prediction: $Q = [1-\hat{y}, \hat{y}]$

Diutsil cross-cracker losses: $$ L = - [y \log(\hat{y}) + (1-y) \log(1-\hat{y})] $$

This formula allows for two situations to be dealt with in a uniform manner:

• When? $y=1$: $L = -\log(\hat{y})$ (only for probabilities of positive classes)
• When? $y=0$: $L = -\log(1-\hat{y})$ (Concerning only the probability of predicting negative classes)

Cooperation with Sigmoid:

The Sigmoid function output models $z$ Map to $(0, 1)$, meets the probabilities of entropy. $$ \hat{y} = \sigma(z) = \frac{1}{1 + e^{-z}} $$

Multi-Category scene: classification cross-crear

Yeah. $K$ Classification issues, labels coded with One-hot $y \in {0, 1}^K$, the model output gets the probability distribution through Softmax $\hat{y} \in [0, 1]^K$。

Categorized cross-brenade losses: $$ L = - \sum_{k=1}^{K} y_k \log(\hat{y}_k) $$

Because $y$ One-hot code, just... $y_{target} = 1$The remainder is 0, so the simplification is: $$ L = - \log(\hat{y}_{target}) $$

The Softmax.:

Softmax will logits $z = [z_1, ..., z_K]$ Convert to probability distribution: $$$2 millioni = \frac{e^{z_i}}{\sum{j=1}^{K} e^{z_j}} $$

Comparison with average error (MSE)

MSE Loss Functions: $$ L_{MSE} = \frac{1}{2}(y - \hat{y})^2 $$

For forecast $\hat{y}$ Guide: $$ \frac{\partial L_{MSE}}{\partial \hat{y}} = \hat{y} - y $$

Key issues: When used in conjunction with Sigmoid/Softmax, the active function conductor in the gradient chain law will cause the gradient to disappear.

In the case of Sigmoid, the full gradient is: $$ \frac{\partial L_{MSE}}{\partial z} = \frac{\partial L_{MSE}}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial z} = (\hat{y} - y) \cdot \hat{y}(1 - \hat{y}) $$

When the prediction is almost completely wrong ($\hat{y} \approx 0$, $y=1$)

• Error $(\hat{y} - y) \approx -1$
• Sigmoid Wizard $\hat{y}(1-\hat{y}) \approx 0$
• Total gradient $\approx 0$, leading to the disappearance of the gradient, slowness in the re-engineering of parameters

The elegant design of cross-breathing entropy.:

As noted earlier, the cross-bridge, in conjunction with Sigmoid/Softmax, is perfectly offset by the number of lines that activate the function, which eventually translates into: $$ \frac{\partial L_{CE}}{\partial z} = \hat{y} - y $$

Even when the projections are completely incorrect, the gradient remains maximum to ensure rapid contraction.

Softmax (incorporated index function)

Core definitions

Softmax converts any real vector to a probabilistic distribution, which is a standard activated function in a multi-classic question. For Vector $z = [z_1, z_2, ..., z_K]$:

$$ S_i = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}} $$

Mathistics

• Non-negative: $e^z$ Always be right. Make sure it's not negative.
• Integrative: All outputs combined are 1 and constitute a legal probability distribution
• Mono-Telephone: Keep the relative size of the input fraction
• Zoom in.: The index function magnifies the difference between fractions, making models more predictive"Decisive."

Why? "Soft" max?

• Hard Max: Direct Output [1, 0, 0]It's not a guide.
• Softmax: Output smooth probability distribution as [0.66, 0.24, 0.10], which allows the relative size information to be enabled and kept

Compared to Sigmoid

• Sigmoid: For a second or multiple-label classification, each category is independent and the probability does not necessarily sum up to 1
• Softmax: For cross-classification, all categories compete, and the probability and probability is mandatory 1

Transformer's attention to Q, K, V, O.

Symbolic description

• $d_{model}$: total embedded dimensions of the model (e.g. BERT-base 768)
• $h$: Number of heads (e.g. 12)
• $d_k$: the dimension of each head ($d_k = d_{model} / h$)
• $X$: Enter the matrix, shape $[\text{Size}{\text{Batch}}, \text{Length}{\text{Sequence}}, d_{\text{model}}]$

Q, K, V Matrix

In the multi-direction, each head. $i$ There are separate weight matrices. $W_i^Q, W_i^K, W_i^V$:

(1) Q (Query) Matrix:

• Enter projection to"Query Subspace"
• Effect: Generate query vectors for matching other vectors
• Multiple meaning: Each head is concerned with different characteristics of input Dimensions

(2) K (Key) Matrix:

• Enter projection to"Keyspace"
• Role: Generate a Query Matched Feature Vector

(3) V (Value) Matrix:

• Enter projection to"Value Space"
• Effect: Store actual content information, extract the corresponding V after matching Q and K

O (Output) Matrix

• Role: Information integration and integration
• Process: close all head outputs, pass $W^O$ Interact at full dimensions
• Meaning: Integration of the diverse features of different head extractions

Design rationale

The multihead mechanism allows the model to follow information in different subspaces in parallel with different locations, each learning different patterns of attention. The ultimate integration of these diverse features through the O matrix indicates that the complex dependency relationships in the sequence are more abundant than the single-headed focus.