Self-Attention and Transformer Architecture: BERT, GPT, and Multi-Head Attention

This English version was machine-translated from the Chinese original. Technical terms may require verification.

Let's do it again.Sequence input and neural network memoryThis topic, but apart from the classic neural network that we've discussed earlier, we're here to talk about something new -- self-care mechanisms and Transformer structures and their basic self-supervisory learning, which was only introduced in 2016, but which is now a remarkable invention.

Self-supervised learning

Self-supervised Learning (Self-Supervised Learning, SSL) is an unannotated learning approach, assuming we have an article$x$ So self-supervised learning can divide it into model inputs.$x^{\prime}$ Label with Model$x^{\prime\prime}$ A model for monitoring learning can then normally be trained without manual marking.

Here we have two typical self-supervised learning models: Bert and GPT, both of which now have quite a good presence in the language model. It's also the central reason why self-monitoring learning is now valued.

Two-way encoder (BERT) from Transformers

The BERT model is a classic model of self-supervised learning, the BERT is an EncoderOnly structure for Transformer, and the structure of BERT is identical to that of Transformer's encoder, which has a lot of self-consciousness and disability connections, homogenization, etc. BERT can enter a row of vectors, output another line of vector, output the same length as the input.

BERT is usually used in natural language processing, and BERT's input is a text. Special symbols are usually used to replace words in sentences, and the word "MASK" is used to denote a special symbol, which can be seen as a new Chinese word, which is not in the dictionary and which means a mask.

After the mask, a sequence is entered into the BERT, and the corresponding output of the BERT is the other sequence. Use linear variations to this vector (the linear variations mean input vectors multiplied by a matrix) and then do softmax and output a distribution. Our goal of training is to output a character as close as possible to the real answer. We need to work together to train Bert and linear models and try to predict the coded words.

After such training, Bert learned to fill the gap. But then it could be used for a variety of interesting downstream missions. These tasks may not necessarily relate to filling, but they may be completely different. When Bert learns to do these tasks, there's still a need for some marked data.

Some marked data for BERT, which can learn tasks, divide BERTs and use them as fine-tuning processes that produce this BERT before fine-tuning. So the process of BERTs is self-supervised learning, or pre-training.

Usage of BERT

In previous studies, we have only trained a model that fills in empty space, which is clearly not enough for each application scene, and now we need to fine-tune the scenario to fit it.

Assume that the downstream task is to enter a sequence and output a category ... This is a classification problem, only input is a sequence ... For example, emotional analysis is such a problem, we need to add CLS characters at the front end of the original text sequence, that is, a special word classification symbol.

Now BERT will export a long sequence, and we're concerned with the output of the CLS synonyms, multiplying it with a linear transformation of the vector used for classification through Softmax. So we connect a sequence-to-sequence model and a classification problem. (In fact, CLS is also used in BERT's training to deal with chronology or the next chronology)

BERT does not have a way to solve emotional analysis problems from scratch, and it still needs some labeling data, many sentences and their positive or negative labels to train the BERT model. In the course of the training, BERT was put together with this linear transformation, called the complete emotional analysis model. In the training, linear variations and BERT models used gradients to update parameters. Linear transformation parameters are randomly initializedand BERT's initial parameter is from the BERT that learned to fill the blanks.I don't know. In fact, the backbone of BERT is a huge Transformer encoder that uses pre-trained BERTs to do better than random BERTs.

BERT is theoretically not limited in length, but since the BERT model is a Transformer encoder, where the self-care mechanism requires a great deal of effort, it is a very important question how to modify the attention mechanism to obtain better computational performance.

In essence, using BERT to deal with practical issues is a combination of fine-tuning and pre-training. Because of the high cost of pre-training work, for most ordinary developers it is necessary to study fine-tuning models to fit their tasks.

Validity of BERT

Why is Bert working? The most common explanation is that, when entering a string of text, each text has a corresponding vector called embedded. BERT can consider the context, depending on the context, and he can export different vectors, and calculate the cosine similarities between these vectors, and we find that BERT learned the meaning of every word in filling out. . . . . Maybe it really understands Chinese, for which it is no longer relevant. . . . . . . because it understands the Chinese meaning, it can do better in the next task.

One technique is word embedding, where there is a technique called Continuous Bag Of Words, CBOW. The continuous wordbag model does exactly the same thing as BERT, digs the middle, predicts the content of the blanks. The continuous wordbag model can give each term a vector, representing the meaning of the word. The continuum bag model is a very simple model, using two variations. In fact, BERT is now embedded. BERT can also create different words from the same vocabulary that are embedded in the context of words (contextualized word embedding)

And it's very absurd that Bert, who's trained in words, can be used to classify proteins, DNA and music. If the DNA sequence is pre-processed into a meaningless sequence, what's the purpose of BERT? BERT can analyze the semantics of a valid sentence. How can you give it an incomprehensible sentence? But actually, it's better than random initialization in text than it is in a lot of issues, and maybe it's just a set of better initialization parameters.

Generating Preparatory Training (GPT)

In self-supervisory learning, in addition to the model of the Bert series, there is also a very well-known model - the model of the — GPT series. Bert does fill in blanks, and GPT does the task of modeling when monitoring learning. GPT does the task of predicting the next word. GPT uses the DecorOnly structure of Transformer.

Enter a term for GPT, BOS (begining of dialogue), GPT will output an embedding. Then use this embedding to predict the next word that should appear. And then we continue to input the next BOS and the next word into the decoding structure of the sentence, just like the decoder structure in Transformer. GPT training is training this Transformer encoder and decodor structure.

The GPT model is based on Transformer's decoding device, and because of the masking mechanism, GPT can predict the next word, so it has the capacity to generate, and it can constantly predict, the next word to produce a full article. Even if GPT's decryptor thinks this sentence is over, we can enter the whole paragraph into the encoder and continue to predict.

The GPT series can complete a sentence, and how can it be used in downstream tasks? For example, how can GPT be used for questions and answers or other tasks related to natural language processing? GPT can do the same thing as Bert, or take GPT out to a simple sorter, which will work, but not in the GPT paper.

Assuming GPT translates, we start with "Translate English to French," the sentence represents the description of the problem. Then give it a few examples, then we can enter English and expect him to give it.

There is no gradient decline at all in this learning process, which is called in-text learning, with better output from the context of the model alone.

Self-supervised learning can be used not only for words but also for voice and computer vision (Com-puter Vision, CV). The philosophy of the training is exactly the same: BERT trains speech filling, GPT trains voice forecasting. The self-monitoring model of the voice has worked well.

The GPT model uses a pure Decoder structure, the Bert model uses a pure Encoder structure, and the Decoder structure has proved to be more effective in its current use.

Self-care mechanisms

Vector Series Input

Our goal is the same, and now we need models that can accept a vector sequence that can change the length as input. Actually, we are.Cyclical Neural Network in Basic Network Structure RNNThe behaviour required.

And there we introduced it. Unholy coding in basic network architecture Help us make a vector sequence of sentences. Of course it's based on learning. Word embedding The glossary can also be expressed as a vector.

Compared to the one-dimensional dimension of the One-Hot code, the word embedding mechanism ensures that the dimension of the vector is not excessive and that the distance between vectors is meaningful.

Vector sequence input can take different forms of output, and we're thinking about the next few, and we have a simple introduction in RNN.

• Inputs are the same as outputs, word type labels, network nodes are the same type of problem
• Output label, the classic classification and regression problem, although input is a sequence, output is still a single label
• Sequence-to-sequence, we don't know the length of the output sequence, we need to learn for ourselves, machine translation, LLM dialogue is sequence-to-sequence technology.

The rationale for a self-care mechanism

We start with the first type: input is as many as output, for example word type, and sequence is marked with a label for each vector in the sequence.

In order for the full-connected network to take into account information before and after the sequence, we need to enter the entire sequence into the network, for example, in audio recognition, we usually enter a window containing the current audio frame and the five audio frames (each audio frame is obtained by processing the original audio file according to certain audio processing rules, usually a window of 25 ms). The operation of entering a window for the network would allow us to make good use of the information in front and back, but it would be very difficult to select the size of the window, which would require the use of a self-care mechanism.

The self-care model "eats" the data of the entire sequence and enters several vectors, and it produces several vectors. At this point, the output vector takes into account the data of the entire sequence, and then we can add him to the whole network and get further output, and then we'll use the focus model to handle the whole sequence. If needed, we can also stack the self-care model with the whole network model. The article that built on the focus model was published by Google Lab.

The self-care mechanism works according to the figure below, and a sequence of vectors can be entered.$x$, or an output of a hidden layer, so use$a$, he outputs a vector after processing the entire input sequence.$b$ Here's the down vector. $b^1$ The process by which the vector is understood$b$The generation. First of all, we need to be grounded. $a^1$ Find out what's inside the input sequence. $a^1$ Other vectors. The purpose of attention is to consider the whole sequence, but it is not desirable to wrap all the information in one window. So introduce a focus mechanism, based on vectors. $a^1$ Find out which parts of a long sequence are important. $a^1$ The degree of correlation can be quantified. $\alpha$Come on.

The attention mechanism model requires the input of two vectors to return one of the markers, which is generally achieved using point accumulation. Multiply input vectors by matrix$W^q,W^k$Get Vector$q,k$, and then will be measured$q,k$Point size is given, and this is multiplied by the matrix to introduce some new linearity on the one hand, and to adjust the vector size on the other. This method is also the focus calculation used by Transformer later described.

The self-focus model generally uses the Query-Key-Value, QKV model. Calculated separately $a^1$ and $a^2$、$a^3$、$a^4$ It's connected. First of all...$a^1$Multiply$W^q$Get Query$q$, then $a^2$、$a^3$、$a^4$ Multiply$W^k$Got keys$k$, will be asked$q$and key$k$The calculation of internality allows for attention points to measure relevance. Usually we do.$a^1$It's relevant to itself. Softmax processing of all correlations gets a set of correlations.$\alpha^{\prime}$ $$\alpha_{1,i}^{\prime}=\exp\left(\alpha_{1,i}\right)/\sum_j\exp\left(\alpha_{1,j}\right)$$ When we get relevance, we can count down.$b^1$ $$\boldsymbol{b}^1=\sum_i\alpha_{1,i}^{\prime}\boldsymbol{v}^i$$ of which$a^i$ and $W^v$ It's Jack.$v^i$ , for elements of higher relevance,$v^i$ You have a higher weight, the final output.$b^1$That's how attention works.

The whole self-care module is actually a stack of very simple algebra modules, but it's this simple matrix operation that eventually produces a powerful intelligence.

And then we're going to go back to the self-absorbed operation from the perspective of the matrix multiplication.$a^i$Reset to Column Matrix$I$, the matrix$I$Separate and Matrix$W^q,W^k,W^v$Multiplication allows you to get the matrix$Q,K,V$ Every column of them is ours.$q,k,v$

Will$k$Reset to Column Matrix$K$ $q$ Reset to Column Matrix$Q$ In fact, the calculations...$K^{T}Q$ Get the matrix.$A$ Each column in it is the focus of our research.$\alpha$ Softmax calculates.$\alpha^{\prime}$ As$A^{\prime}$

Reconstruct$v$As Column Matrix$V$, calculate$VA^{\prime}$ You can get the matrix.$O$ Each column is a self-directed output.$b$ For the entire matrix, only the matrix.$W^q,W^k,W^v$We need to learn in models. Of course, it's just a case of one-sided attention, and when we look at multiple attention, the training parameter matrix increases.

Other self-care mechanisms

Attention.

There's a progressive version of attention - multi-head self-attension. The use of multiple attention is very widespread and leads to better results in many missions. As for the size of the head, this is another parametric.

The original self-absorption mechanism.$q$ Find it. $k$ It's called a head, and now we're going to copy this structure and we're going to consider a group.$W^q,W^k,W^v$ Get a set.$q,k,v$ Now let's consider two different sets.$W^q,W^k,W^v$ Another correlation is measured. In practice, we get it straight ahead.$q,k,v$ Multiply two arrays and get different heads.

We're in one of the original ones with multiple attention mechanisms.$b^i$There will be more than one position.$b$We usually pass him through a transformation, and then we pass a sequence vector down. This matrix of change, we call it. $W^O$ That means there will be four training matrices in the multi-headed matrix on the attention mechanism. $W^q,W^k,W^v,W^O$

Location encoding

So far, one information that may be important has been missing from the attention level, that is, information on location. For a self-directional layer, each input appears at the front or the end of the sequence, and it has no such information at all. Information about the location is ignored in the current model, which is not acceptable when dealing with serial problems.

When you need to think about location information, you need to use the location code, and we define a vector for each location in the sequence. There's a different location. $e$Put $e$ Add $a^i$ It's over. In the original "Attention Is All You Need" paper, the location vector was generated through a sine function and a cosine function, avoiding the awkwardness of artificially fixed vector lengths. What's a better location vector is an outstanding thing, and the sine function is just a habit, not a rule.

The location code is also learning in part of the structure, but this is generally only in technical reports on pre-training techniques, and fine-tuning does not generally consider the location code.

Cutting attention.

The use of attention is extensive, and in the area of natural language processing, NLP, besides Transformer, there is also BERT. But not only can he be used in the NLP area, but he needs a little change.

In calculating the attention matrix, complexity is the square of length. Assuming the length of the matrix is $L$Calculating the attention matrix needs to be done. $L \times L$Second-inventory, if $L$ The value is very large, the calculated amount is significant, and a large memory is required to keep the matrix.

Cutting off attention (truncated self-attension) addresses the excessive length of vector sequences. To cut off attention and not look at a single sentence when you're paying attention, just look at a small range that is set by humans.

Attention to CNN and RNN.

Attention to CNN

Attention can also be used on images. So far, when it comes to attention, the extent to which it applies is when it is entered into a set of vectors. An image can also be seen as a vector sequence. An image with a resolution of 5 x 10 can be expressed as a mass of 5 x 10 x 3. The pixels of each position can be considered a three-dimensional vector and the whole image is 5 x 10 vectors. This would make it possible to focus on one image.

If we compare a condensed neural network to a self-observation, a condensed neural network can be seen as a simplified form of self-observation, because when we do a condensed neural network, we think of only feeling the information in the field. And when you look at yourself, you think about the whole image. In a congested neural network, we're going to define the feeling field. Each neuron only considers feeling the information in the field, and feeling the size of the field is human. And it's like feeling wild is automatically learned, and the network itself determines the shape, location, size of feeling wild.

By setting the right parameters, you can do exactly the same thing as a congested neural network. Since a condensed neural network is a subset of self-consciousness, it shows greater flexibility. More data are needed for more flexible models. If the data are not sufficient, it is possible to overcompile. Instead, a more limited model, which is suitable for use at times when data are scarce, may be less than comparable. If the restrictions were good, they would also have good results.

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale puts your attention on the image, tearing an image into 16 x 16 image blocks (pact), which imagines each image block as a word (word). It is then handled by a self-directional mechanism. When the amount of data is larger, this method can produce better results than a convoluted neural network.

Attention to RNN

Let's compare our attention to circulatory neural networks. Currently, much of the role of the circular neural network can be replaced by attention. Because they're all created to process sequences.

There's a clear difference between the attention and the circulation neural network, where each vector takes into account the entire input sequence, and each vector of the cycling neural network takes into account only the vector already entered on the left, which does not take into account the right vector.

Even considering a two-way circular neural network, with the length of the sequence increasing, RNN is more likely to forget what was entered long before, because its memory does not necessarily last so long. But out of focus, out of a query, out of a key, as long as they match (mATCH), "the world is closer." The ease with which attention can extract information from very far-off vectors of the entire sequence is clearly better for long text.

Another more important difference between self-direction and circulatory neural networks is that the circulatory neural network does not have the means to be synchronized when it handles input and output in a sequence. For example, calculating the vector for the second output requires not only the vector for the second input, but also the output vector at the previous point of time. When the input is a set of vectors and the output is another set of vectors, the circulation neural network cannot process all the inputs in parallel, but is self-conscious.

Transformer

Transformer is a self-conscious sequence-to-sequence model that, unlike a sequence-to-sequence model based on a circular neural network, can be calculated in parallel.

This chapter presents Transformer in two ways, on the one hand, the structure of Transformer, i.e. the encoder and decoder-decoder attention, and, on the other hand, the training process of Transformer and the training techniques of serial to sequence models.

The sequence-to-sequence model input and output are one series, and the relationship between the input and the output sequence length is two scenarios. In the first case, the input is the same length as the output; in the second case, the machine determines the length of the output. This includes voice recognition, machine translation, chat robots (question-and-answer robots), voice synthesis, multi-label classification (which cannot be processed using traditional classification models) and the use of sequence-to-series mechanisms.

GeneralSequence-to-sequence models split into encoders and coders., the encoder handles the input sequence, and then " throws" the processed result to the decoder, which determines the sequence to be exported. This is also the choice of Transformer structure.

Transformer Encoder

This is followed by the Transformer encoder, which enters a row and outputs another row. Transformer's encoder uses self-consciousness, enters a row of vectors and outputs another vector of the same length.

Encoders are divided into blocks (blocks), each of which enters a row and output a row. Enter a row to the first block, the first block to output another row, and so on, the last block to output the final vector sequence.

Each block of Transformer's encoder is not a layer of a nervous network, and within each block, you enter a row of vectors and do your own attention, consider the information of the entire sequence and output another row of vectors. And the next thing you can do is throw it in the whole network, and you get another row, and that's the block.

The real Transformer is more complicated than that, with the design of the residual connection. Vector $b$ Enter (multiple head) the vector after attention level $a$, output vector $a$ Add its input vector $b$ Get a new output. After such a residual connection, the layering is consolidated. The result of layering is entered into a full-connected neural network and the output of the input and full-connected network becomes a new residual connection, which is then consolidated into a layer, which gives a complete block structure.

In particular, in order to maintain location information in the network structure, we also need to embed the location information at the input end of each block. Repeating$N$After a piece, we got one layer of Transformer, which is also the encoder structure of Transformer.

Transformer decoder

This is followed by a code decoder, which is more commonly referred to as a self-returning (autogressive) decoder.

The decryptor reads the output of the encoder first.I don't know. In order for the decoder to produce an output, it first needs to be given a special symbol, BOS, which represents the beginning, which is a special word (token). And we've developed a verb (vocabulary) code, adding a special symbol BOS to the text that might have been created by the decodor.

After accepting BOS, the decryptor spits out a vector based on the input of the encoder obtained, which was processed by Softmx, which is the probability distribution of the words in a vocabulary, and it's one, and now we can find the highest score against this vector, which is the first output of the declinator.

Next, the output of the decoder is used as a new input for the decoder. According to two inputs (BOS and just output words), the decoding device will output another vector, corresponding to the word in a vocabulary, and repeat this step, which means that the decoding device input is the output at the previous point in time, and it will treat its output as the next input.

Because of the structure of one word and one word output, there may be a problem of error transmission, with all errors occurring after one word is exported.

The decoding device in Transformer uses a structure similar to that of the encoder and includes a multilayered stacking structure with a multi-directional + disability connection + layer integration, full front-to-back neural network + disability connection + layer integration. The decoder ends up making another softmax, making its output a probability.

In addition, the decoding device uses masked self-obtention, which can prevent each location from choosing the input information behind it through a mask (mask). He essentially prohibits the original self-directional mechanism from visiting its output, the sequence.$a_1,a_2,a_3,...$ Calculating$b_2$♪ Sometimes, only use ♪$a_1,a_2$ Do Not Use$a_3$and subsequent inputs, including qk queries and v calculations.

Why do you need to put a mask in your attention? $a_1$ Again. $a_2$Again. $a_3$Again.$a_4$I don't know. It's not the same as the original focus. $a_1$ Call. $a_4$ It's the whole thing in the model. The decoder's output is one that's produced, so it's the left thing, it's not the right thing.

To stop the decoder, a special EOS symbol is required. The output of the model after the decoder output EOS is terminated.

The model structure described above is a self-repatriated decodator and the self-regression output takes place one word at a time, which means that he is a sequenced structure that cannot run in parallel, resulting in the overall speed of its operation not being too high.

There is also a non-self-returning decoding device, and the non-self-returning decoding device may be "eating" by a whole row of BOS words, producing a row of words at a time. The exact number of BOS entries is not conclusive, some models are determined by additional taxologues, others by many BOS entries and the post-EOS portion of the output is discarded.

Non-self-returning decodors have many advantages. The first advantage is parallelization. Another advantage is that non-self-return decodors are more able to control the length of its output. But they are often less effective than self-regressor decoders.

Encoder - Decoding Attention

The description of the decodors that was presented earlier is actually missing, and we have not been able to explain in detail how the output of the decodors was used by the decodors, nor has the structure on the map fully explained the problem. This is the encoder-decoder attention mechanism.

Or does the encoder enter a row and output a row?$a_1,a_2,a_3,...$ I don't know. The decryptors will then "eat" BOS and get a vector by hiding their attention. Then multiply this vector with a matrix, make a transformation, get a query. $q$ At this time$a_1,a_2,a_3,...$Add a matrix generating key$k_1,k_2,...$ And then we calculate the attention score, we do the Softmax, and we calculate the weighted output.$v$ Here.$v$ You're going to hand it over to the whole network, get the output. So you get the output vector of the block.

This step. $q$ It's from the decoder.$k$ Call. $v$ It comes from the encoder, and it's called the encoder -- the decoder's attention, so the decoder is created by one. $q$Get the information from the encoder. come out as the input of the entire network of the next decoder. Finally, the output of the block is obtained through a full-connected network.

In the follow-up block, the mask captures the output of the front block and then gets a new output to calculate according to the required length limit of the mask at this time$q$ Then turn to the encoder -- the decoder.

When there are multiple outputs, the decoding device eats BOS and other outputs, and still produces equal vectors, and then multiples.$q$, and then turn over the encoder-decoder attention and full-connected network to the next block.

Training on models

The basic idea is to calculate the intersection between the standard answer (Ground Truth) and the distribution, after all, our output is a probability distribution after Softmax, a very natural classification question.

In training, we usually give him the right output of the first few words, let him predict a word, then calculate the crossbow of the sentence, which is called teacher coercion.

After the previous line of thought, the criterion for our assessment is BLEU (BiLingual Evaluation Unit- STUDY) scores. Although BLEU was first used to evaluate the results of machine translation, it is now widely used to evaluate the quality of many application output sequences. The decoding device produces a full sentence, then compares it with the correct answer and compares it to the BLEU score. But when training takes place, each term is considered separately, and the smallest is cross-paramerium, which does not necessarily maximize the BLEU score. But when it's done, it's not the lowest cross-cream model, but the highest BLEU score model.

In fact, the word-by-word calculation of cross-breathing is based on the fact that BLEU cannot be calibrated and therefore not optimized. However, intensive learning training can be used to address problems that cannot be optimized. Specifically, loss functions that cannot be optimized are treated as incentives for enhanced learning and code decoders as intelligent. This is currently a very common method and has been a great success in the training of reasoning models.

Bert and GPT embedded

Both Bert and GPT, the model's reading of original sentences and predictions are based on token, which means we need to convert the original sentence into token and then input it into the model. The way to convert words into token vectors is called Embedding.

All words in the original statement are coded using Onehot, which is a high-dimensional thin vector. Converts each word to a one-dimensional vector by creating a word vector table. In particular, the English vocabulary will do more finer cuttings, such as playing or cutting into play and #ing.

Token Embeddings Layer will convert every word to $D$ Vector. Use linear transformations below $$v_{n}= E x_n$$ Vector for each One-hot code$x_n$ The dimension is$K$, is the dimension of the One-hot coding space, which is usually large.$v_n$Is word vector embedded, size$D$ 768 D for Bert.$E$It's embedded in a matrix and learning through reverse transmission.

In order to make sure that the model also learns the sequence, we need to add Postings to the embedded layer. He created one at every location.$D$Vee vectors, we'll follow up on this vector and the front token embeddings, get the final token and input it into the follow-up model.