Deep Learning Basics: Neural Networks, Optimization, and Normalization

This English version was machine-translated from the Chinese original. Technical terms may require verification.

Before we start.

There is a difference between deep learning in the way we think about things and traditional machine learning, and here we need to come to an understanding of how deep learning thinks about our problems.

A case and our thoughts on him.

A basic process

The number of video hits is projected as an example of the operation of machine learning.

Our goal is to find a function whose input is information from the back and the output is the total number of views that this channel will have the next day.

Step 1, we need to find a model, which is essentially a function that contains unknown parameters, such as the most basic linear function. $$y=b+wx_1$$

Step 2 is the definition of loss (loss), which is also a function. This functionInput is the parameter in the model, the output represents whether the model given by a group of parameters is sufficient to fit our reality.

In general, loss should be defined as a difference between the projected and the real value, such as MSE MAE, if, of course,$y,\hat{y}$ They're all probabilities. We also choose cross entropy.

Next step into machine learning is step 3: solve a perfect problem. Finds a value for an unknown parameter to see which value to use to minimize the value of lost L. The most rapid method of decreasing gradients is that of optimizing the introduction.

The loss function appears to be a function consisting of predictions and real values, which are in fact influenced by model parameters and training samples, so he is a function of model parameters. So there's room for the gradient.

Think about Sigmod approaching.

Of course, purely linear models are approaching, which may not be good in a complex real world, and the usual method is to use them.A set of Sigmod functions and any linear segment functions to approach, and segment linear functions can almost approach any function at this timeSigmoid's number is also a hyperparameter

The Sigmod function is $$y=c\frac1{1+e^{-(b+wx_1)}}$$ of which$c$ $b~w$ Unknown parameter

And we can put it in brief. $$y=c\sigma(b+wx_1)$$ of which$\sigma$ For the Sigmoid function

Specially, we can get the Sigmoid function to have more variables, or characteristics of the field of deep learning, to get the Sigmoid function below $$y=c\sigma(b+w_1x_1+w_2x_2+w_3x_3+...)$$

Usually, we use it. $\theta$ It refers to all the parameters that need to be optimized, in which the number of Sigmoids is used as an excess rather than an optimisation, at which point our loss function is recorded as $L(\theta)$ It's a multi-purpose function, which can still be used to solve the optimal problem by decreasing the gradient.

A little detail.

There's a problem with the details, when the gradient actually goes down. The N data is randomly divided into one batch (batch). Each batch contains B-data (known as "batch size".

All of the data would have been included as a loss, and only the data contained in one batch would have been included as a loss. We'll update the parameters once every Batch.

All batches were examined once, called a round (epoch), and each update parameter was called an update.

ReLU

In fact, we could have done more deformation of the model, and we were thinking of using the Sigmod function to get closer to the real situation, and in fact he could have been seen as the sum of two modified linear units. $$c*\max(0,b+wx_1)$$

In machine learning, Sigmoid or ReLU is called the Activation function. There are, of course, other commonly activated functions, but Sigmoid and ReLU are the most commonly activated functions.

In-depth learning

Just like we already are. Machine Learning Introduction and Monitoring of the Neuronet in LearningIt's the same structure that we usually call Sigmoid or ReLU.NeuronThe network structure of many neurons is calledNeural network Each row is called one.Hidden player A lot of hidden layers are "deep."In-depth learning

Practice methodology

This is the problem we encounter when we really deal with a problem and some ways to deal with it.

Model deviation

The problems represented by the real world are extremely complex, the model ' s structure is too simple and too flexible, leading to a proposed comparison of effects Bad

Solutions: Increased complexity and flexibility of models, selection of more complex traditional models or deeper learning models, introduction of more features

Optimization issues

Gradient drops are common, but there are many problems that may be stuck to the lowest local value, and the resulting under-optimization will lead to poor alignment.

Model deviations and under-optimization can lead to poor alignment, and it often takes a lot of experience to distinguish the difference between the two, and if small models are to achieve better results than complex models, it is likely that the optimization has not been achieved. Need to improve optimization methods

Compromise

Good alignment and poor generalization often mean overcompatibility, often because models are too flexible and training data are not sufficiently covered

I've dealt with the following lines.

• Increase training data, or rely onData enhancementTo get more training data.
• Limit the flexibility of models, with the focus on reducing parameters that need to be optimized
• Reduced features
• Early Stopping, Regularization Dropout

Do not match

The mismatch is due to a serious error in prediction due to different patterns of distribution of training and testing data in nature

Normally, this can be overcome by gathering more data, but mismatches mean that the training set is not distributed in the same way as the test set.

Whether or not a mismatch is encountered depends on the researcher ' s understanding of the data itself and the way the training set is produced and the test set is produced to determine whether there is a mismatch.

In-depth learning base

This chapter introduces the common concept of in-depth learning, which will be the foundation for the various in-depth studies that follow.

Generation of neuronet models and in-depth learning

Machine Learning Introduction and Monitoring of the Neuronet in Learning

The world's smallest and very small.

The question of global and local minimum values is a very common part of the issue of optimization;All algorithms based on gradients can't achieve the lowest level of direct search.

And searching on the basis of a drop in gradient is normal for most neural networks, so we've studied some remedial strategies.

• Here.Multiple sets of different parameter values to initialize multiple neural networks, with the least error resolution (i.e. the most common method to optimize) is used as the final parameter when trained in standard methods.
• Use 'Simulate a retreat." The simulation of a retreat at every step accepts the result that is worse than the current one, which is not the optimal solution, and helps us to get out of the region.
• UseRandom gradient down It increases the randomity factor for the gradient drop, even if it's in the smallest part.
• Genetic algorithm

Small local value and saddle point

The question of local miniscule values and saddlepoints responds to a common question that we have raised in the previous “practical approach”:As the parameters continue to be updated, the loss of training will not decline, but we are still not satisfied with this loss.

Critical points and their types

In the case of previous optimizations, our basic guess is to optimize the moment when the parameters are subdivided into zero losses, at a time when the algorithm based on the decline in gradients cannot continue to optimize the parameters to reduce losses, and training is over.

For a gradient of zero, the most common position is a local minimum and a local maximum, but due to our downward orientation, general in-depth learning is reduced to a local minimum.

In fact, the loss is not just zero in a local very small gradient, but other points that might make the gradient zero, for example.Saddle pointI don't know. A classic example is the centre of the saddle face.

When the saddle point is constricted, the gradient drop algorithm does not help us to continue to optimize the loss, but there is clearly a lower point of loss around it, which can make the loss even lower, and we have no good way of reducing it when we get to a local small point.

Methodology for determining the type of threshold

It is necessary to know the shape of the loss function to determine whether a critical point is a local very small value or a saddle point. But how do you know the shape of the loss function? The network itself is complex, and the loss function calculated with a complex network is clearly complex.

But we can think of doing something about the loss function with a local, approximate loss function for Taylor, in the parameter group.$\theta^{\prime}$ There's a spread around. $$L(\boldsymbol{\theta}) \approx L\left(\boldsymbol{\theta}^{\prime}\right)+\left(\boldsymbol{\theta}-\boldsymbol{\theta}^{\prime}\right)^{\mathrm{T}} \boldsymbol{g}+\frac{1}{2}\left(\boldsymbol{\theta}-\boldsymbol{\theta}^{\prime}\right)^{\mathrm{T}} \boldsymbol{H}\left(\boldsymbol{\theta}-\boldsymbol{\theta}^{\prime}\right) .$$ of which$g$and$H$Gradients and Hessian matrix store first and second steps differentials, respectively

At all points with a gradient of 0, the approximate result becomes $$L(\boldsymbol{\theta}) \approx L\left(\boldsymbol{\theta}^{\prime}\right)+\frac{1}{2}\left(\boldsymbol{\theta}-\boldsymbol{\theta}^{\prime}\right)^{\mathrm{T}} \boldsymbol{H}\left(\boldsymbol{\theta}-\boldsymbol{\theta}^{\prime}\right) .$$ We can build on the rest.$\left(\boldsymbol{\theta}-\boldsymbol{\theta}^{\prime}\right)^{\mathrm{T}} \boldsymbol{H}\left(\boldsymbol{\theta}-\boldsymbol{\theta}^{\prime}\right)$ To judge.$\theta^{\prime}$The shape of the nearby error surface (error surface) determines whether this is a local small value, a local great value or a saddle point for the loss function.

We'll use it.$v$Replace$\theta - \theta^{\prime}$ Here's the conclusion.

• If for all $v$ Both $v^Hv > It's a small local value.
• If for all $v$ Both $v^Hv < It's local.
• If it's now and then, > 0$ 时而 $v^{T}Hv < It's a saddle.

In fact, this conclusion can be streamlined directly.$H$. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

• $H$Qualitative values are positive
• $H$The feature value is all negative
• $H$Characteristic values are positive and negative

The way out of the saddle.

Let's start with a simple situation. We need an optimised function that's very simple and can calculate a clear matrix directly.$H$ So all we need is the characterization of the plan matrix to determine whether it's at the saddle point. At this time $H$ It also indicates the direction of the parameters that can be updated, the matrix.$H$direction of negative characteristic vectors

From this point of view, the saddle dots don't seem so scary. But actually, we're hardly going to really figure out the Heisen Matrix, because the Heisen Matrix needs to count a double calibration, it's very large, and it's going to have to find its characterization values and its characterization vectors, so there's hardly any way to escape the saddle spots. There are other ways to get out of the saddle spots that are much smaller than the Heisen Matrix.

Now we have a rule based on experience:Small local values are less common when training a large neural network of parameters. Most of the time, we train to a very small gradient, and the parameters are no longer updated, and often just meet the saddles.

In other words, the minimum values often don't exist. We don't have to run away from the saddles.

Batch and momentum

As we said earlier, the calculation of losses in practice does not always use all the data. To achieve this, the bulk idea referred to above is still followed: the data are randomly divided into one watch, each of which calculates the loss and updates the parameters once; all of which are read once, called an epoch.

In fact, before each round begins, we are redrawn, that is to say, the volume of each round is different. To enhance the effectiveness of the training to avoid possible oversizing and accelerated training using parallel calculations

Impact of batch size on gradient reduction

Let's look at the two most extreme situations.

• No batch size (batch size) is the size of the training data, and this method of updating parameters using full load data is the Batch Gradient Decline (BGD). At this point, the model had to read all 20 training data in order to calculate the loss and gradient and update the parameters once.
• Batch size equals 1 and the method used at this time is the random gradient reduction method (Stochastic Gra-dient Decent, SSD), also known as the incremental gradient reduction method. The loss calculated with a single data is relatively more noiseful, so the direction of its renewal is generally convoluted.

The drop in the volume gradient is more stable and accurate than the decrease in the random gradient. However, random noise has been introduced on the gradient of the drop in the random gradient, which makes it easier to escape the local minimum than the drop in the volume gradient in the non-comb optimization problem.

While we have just said that a volume gradient decline would require a larger iterative calculation, it takes into account the accelerated calculations currently provided by GPU. Large quantities do not necessarily take longer than small quantities.

Actually... When the bat size is smaller, it takes more time to run through a round.I don't know. Assuming that the training data are only 60000, mass size 1 and 60000 updates are required to "run" a round; if the mass size is equal to 1,000, 60 updates to "run" a round, the time to calculate the gradient is almost equal. However, the time gap between 60,000 updates and 60 updates is very large.

In fact, small batches also help with testing. It is assumed that some methods (e.g. large bulk learning rates) can train large volumes as well as small volumes. The results of the experiment show thatSmall quantities will be better at testing. The current explanation is that the randomity of small quantities has helped us to jump out of some of the lost canyons that have been dramatically transformed into a lost basin.

Dynamic Method

Momentum method is another way to counter a saddle point or a local minimum.

Our thinking is that, in the traditional gradient reduction method, we decide on the direction of the next optimisation according to the gradient of the current point, whereas the kinetic method does not move the parameters only in the reverse direction of the gradient, but in the direction of the reverse direction of the gradient plus the direction of the previous step. We believe that the inertia of this movement allows us to take off the saddlepoints and the minimal local effects and enter the lost basin.

We'll use it.$m$Other Organiser $g$is the gradient that gives you the drop structure of the kinetic method, where $\mu$ It's learning.$\lambda$ is the weight parameter for the previous direction $ \begin{array} \bardsymbol{{0}=0 \ \boldsymbol{m}{1}=-\eta \boldsymbol{g}{0} \ \boldsymbol{m}{2}=-\lambda \eta \boldsymbol{g}{0}-\eta \boldsymbol{g}{1} \ ... \end{array}$$

Learning rate for adaptation

On adaptive learning rates

The threshold is not necessarily the biggest obstacle to training a network.

Sometimes we find that the gradient of the loss function (measured in its model) is still large, but the loss does not continue to decline, and we are not stuck at some point with a gradient of zero, but we cannot continue to update the parameters to reduce the loss. It's often because we're stuck at some point because of the high rate of learning. It's a parameter that continues to circulate at both ends of a valley, but cannot reach the bottom.

This is not in line with the general training situation, as the parameters of the in-depth learning model are so many that they tend to be reduced when the gradient is still large, and we do not need a small gradient to stop training.

We can try to lower the learning rate, which determines the pace at which the parameters are updated, and the learning rate is too high to slip slowly into the valley. However, it is not always valleys, and too little learning is usually not allowed to move on.

In conclusion, in the gradient decline, all parameters are subject to the same learning rate, which is clearly not sufficient and should be customized for each parameter, i.e. introducedAdaptive learning rate (adaptative learning rate) The method gives each parameter a different learning rate.

If the gradient is very small and flat in one direction, we would like the learning rate to increase; if it is very steep and steep in one direction, we would like the learning rate to be smaller.

AdaGrad

AdaGrad (Adaptive Gradient) is a typical self-adaptation learning rate method that allows for automatic adjustment of the learning rate to the size of the gradient. When AdaGrad can achieve a higher gradient, the learning rate decreases, and the learning rate increases when the gradient is smaller.

This is our traditional way of updating our parameters. $$\bardsymbol(theta}{t+1}^{i}\leftarrow\boldsymbol{\theta}{t}^{i}-\eta\boldsymbol{g}_{t}^{i}$$

Now there's a learning rate that customizes with parameters, i.e., the original learning rate.$\eta$♪ Turn into ♪$\frac\eta{\sigma_t^i}$ $$\theta_{t+1}^i\leftarrow\theta_t^i-\frac\eta{\sigma_t^i}\boldsymbol{g}_t^i$$ This learning rate goes with the parameters.$i$Number of times$t$Now our learning rate becomes parameter-related.

A common type associated with parameters is the mean root of the gradient. The process of updating the parameters is: $$\theta_1^i\leftarrow\theta_0^i-\frac\eta{\sigma_0^i}\boldsymbol{g}_0^i$$ of which$\boldsymbol{\theta}_0^i$is the initialization parameter. And...$\sigma_0^i$..the calculation process is $$\sigma_0^i=\sqrt{\left(\boldsymbol{g}_0^i\right)^2}=\begin{vmatrix}\boldsymbol{g}_0^i\end{vmatrix}$$ of which$g_0^i$It's a gradient. Will$\sigma_0^i$the value of which can be replaced by an updated formula$\frac{g_0^i}{\sigma_0^i}$+1 or 1. First time updating parameters, from$\boldsymbol{\theta}_0^i$Update to$\boldsymbol{\theta}_1^i$Sometimes, you add$\eta$Or cut it off.$\eta$,It's not about the size of the gradient. This is the first step.

The second update of parameters is as follows: $$\bardsymbol(theta}{2}^{i}\leftarrow\boldsymbol{\theta}{1}^{i}-\frac{\eta}{\sigma_{1}^{i}}\boldsymbol{g}_{1}^{i}$$ 其中 $$\sigma 1^sqrt{frac)^left[\left(\boldsymbol{g}^i\right}^left(\boldsymbol{g}^i\right)^ $ The average root of the gradient is used to limit learning rates$\mu$ That's how it came to be.

The same thing can be repeated. $$\boldsymbol\theta}i\leftarrow\boldsymbol{t^i-\frac\eta}\boldsymbol{g}t^i\quad\sigma_t^i=\sqrt{\frac1{t+1}\sum{i=0}^t\left(\boldsymbol{g}_t^i\right)^2}$$

This algorithm works:Adapting to lower learning rates when the gradient is larger and increasing learning rates when the gradient is smaller ensures that the overall optimization is limited in length

RMSProp

The learning rate required by the same parameter will change over time. RMSProp (Root Mean Squared Protection) was designed to address this issue. He made the gradients from time to time very important.

RMSprop does not have a dissertation, and Geoffrey Hinton has an in-depth study on Coursera, where he talks about RMSprop, which, if quoted, needs to refer to the link to the corresponding video.

RMSprop's first step is the same as Adagrad's, which is $$\sigma_0^i=\sqrt{\left(\boldsymbol{g}_0^i\right)^2}=\begin{vmatrix}\boldsymbol{g}_0^i\end{vmatrix}$$

The second update process is as follows: $$\boldsymbol{\theta}_2^i\leftarrow\boldsymbol{\theta}_1^i-\frac\eta{\sigma_1^i}\boldsymbol{g}_1^i\quad\sigma_1^i=\sqrt{\alpha\left(\sigma_0^i\right)^2+\left(1-\alpha\right)\left(\boldsymbol{g}_1^i\right)^2}$$ of which$\alpha$ is a hyperparameter between 0-1. In RMSprop, you can adjust yourself to the importance of the new calculation gradient.$\alpha$ The smaller the newly calculated gradient, the more important.

The next steps are: $$\boldsymbol(t+mathbf) loftarrow\boldsymbol(t\frac\eta}\boldsymbol{g}t^i\quad\sigma_t^i=\sqrt{\alpha\left(\sigma{t-1}^i\right)^2+\left(1-\alpha\right)\left(\boldsymbol{g}_t^i\right)^2}$$

RMSprop can quickly "stamp the brakes." Adjusting the learning rate as soon as possible based on the latest gradientWhether it's bigger or abbreviated

Adam

The most commonly used optimizer strategy or optimizer (optimizer) is Adam Adam, who can be considered RMSprop plus momentum, using kinetics to update the direction of the parameters and be able to adapt to the learning rate.

PyTorch has already written Adam Optimizer, which contains some super-parameters that need to be determined by humans, but it's always enough with the PyTorch pre-set parameters.

Movement of learning rates

None of the study rate adjustment algorithms we have in front of us avoid a problem, and the learning rates in all directions are affected by several gradients. Although the RMSProb algorithm reinforces the current gradient, it still takes time to adjust the learning rate to the right level, and these calculations and the time spent on computing resources are actually wasted.

Pass.Learning rate scheduling It can be solved. Previous study rate adjustment$\eta$It's a fixed value, and the learning rate is moving. Medium$\eta$It's about time, as shown below. $$\boldsymbol{\theta}_{t+1}^i\leftarrow\boldsymbol{\theta}_t^i-\frac{\eta_t}{\sigma_t^i}\boldsymbol{g}_t^i$$ The most common strategy in the program is learning rate decline, also known asLearning rate retreatingI don't know. As the parameters update, let$\eta$It's getting smaller and smaller, and it helps us get to target accuracy faster in the finely optimized part of the back.

In addition to the decline in the learning rate, there is another classic way of moving the learning rate - preheating. The method of preheating is to make learning grow bigger and smaller, and how much, how much, how much, how much, how much, how much, how much, how much. Preheat is used in training for the disability network, Bert and Transformer.

The core of preheat When we use Adam, RMSprop or AdaGrad, we need to calculate$\sigma$I don't know. And...$\sigma$It's a statistical result. From$\sigma$Know the steepness of a particular direction. The results of the statistics require enough data to be accurate, starting with the results$\sigma$It's not accurate. At first, the low learning rate was used to explore and gather information on the surface of errors.$\sigma$Statistics, etc.$\sigma$When statistics are more accurate, learning rates slowly climb.

Preheat can be consulted in Adam's advanced version - - R Adam.

Optimization of summary

After the previous subsections, we have finally become quite clear in introducing the issue of optimization, and we have evolved from the most primitive gradient to this version. $$\bardsymbol(theta}{t+1}^{i}\leftarrow\boldsymbol{\theta}{t}^{i}-\frac{\eta_{t}}{\sigma_{t}^{i}}\boldsymbol{m}_{t}^{i}$$

There's momentum in this version.$m_{t}^{i}$ The previous section, “Bulk and kinetic”, does not update the parameters in the direction of the gradient calculated at a given time, but uses all past calculations of the gradient as a weighted sum as the direction of the update.

We then use the self-adaptation learning rate methodology to help us optimize and add to the learning rate schedule.

This is the complete version of the current optimization, which, in addition to Adam, has various variations. But they're all fixing the drive in different ways, adapting to the learning rate, and doing the learning rate movement.

In particular, both the kinetic method and the self-adaptation learning rate take into account past gradients, but the kinetic method also takes into account their direction, while the self-adaptation learning rate focuses more on its size.

Batch Harmonization

Normalization.

If the error surface is rough, it is harder to train. Can you just change the surface of the error and make it better to train by "shaping the mountains"? Batch Normalization BN is one of the ideas of “plaining the mountains”.

Let us now reflect on a matter of the nature of the error that is difficult to train. The rugged error surface is essentially a small parameter disturbance leading to a large error, and we deal with this problem in traditional statistics --Normalization Largely variable-differentiated regression factors are generated in online regression models. That's what we're talking about. Characterization

Consider in-depth learning

Since the Depth Learning Model is a layered web structure, his integration naturally raises a number of other issues.

Although we've got the initial data on training.$x$It's been consolidated, but it's going through a network.$W_1$Later, we get it.$z$No regularization, which leads to training next level$W_2$There are some difficulties.

So we're right.$z$Re-incorporate (the choice of the form of integration depends on the activation function), where the common habit is to put integration before the activation function, and then transmit the result of the activation function to the bottom of the network.

Special: The training is conducted on a catch, so the integration is also done on a catch-wide basis, which means we need a slightly larger watch size to ensure the approximation of distribution.

In the case of batch integration, the following operations are often carried out: $$\hat{\boldsymbol{z}}^i=\gamma\odot\tilde{\boldsymbol{z}}^i+\beta $$ Of which,$\odot$represents the multiplication of elements by element.$\beta,\boldsymbol{\gamma}$ It can be conceived as a network parameter that needs to be learned again.

Why?$\boldsymbol{\beta}$Call.$\boldsymbol{\gamma}$And?

If you do this, you'll have to do it.$\tilde{\boldsymbol{z}}$The average must be zero, and if the average is zero, this will limit the network, which may have a negative impact, so we need to put$\beta,\boldsymbol{\gamma}$Add it back so that the output average of the network's hidden layer is not zero. Let the Internet learn.$\boldsymbol{\beta},\boldsymbol{\gamma}$To adjust the distribution of the output, to adjust it.$\hat{\boldsymbol{z}}$the distribution.

Batch integration is designed to make each dimension the same, if$\gamma$Call.$\beta$It's not like it's all the same.

It's possible, but actually in training,$\boldsymbol{\gamma}$The initial values are set at one, so...$\boldsymbol{\gamma}$All values are 1 vectors.$\beta$is all zero vectors, or zero vectors. So the distribution of each dimension of the network is closer at the beginning of the training, and perhaps it's been trained long enough to find a better surface of error, to go to a better place, and then...$\gamma,\beta$I put it in slowly, so I added it.$\gamma,\beta$Batch aggregation is often helpful for training.

Batch Harmonization at Test

These are all parts of the training, and the tests are sometimes called extrapolations. What's wrong with mass integration when it's tested? At the time of testing, we will get all the test data at once, and it is not appropriate to continue with the batch.

In fact, mass integration does not require any special treatment at the time of testing, and PyTorch has already done it. When you're training, if you're doing batch integration, it's done every batch.$\mu,\sigma$ , which is used to calculate the moving average. Assuming there's now a lot of calculations.$\boldsymbol{\mu}^1,\boldsymbol{\mu}^2,\boldsymbol{\mu}^3,\cdots\cdots,\boldsymbol{\mu}^t$, the moving average is calculated $$\bar{\boldsymbol{\mu}}\leftarrow p\bar{\boldsymbol{\mu}}+(1-p)\boldsymbol{\mu}^t$$ Of which,$\bar{\boldsymbol{\mu}}$Yes.$\boldsymbol{\mu}$It's an average.$p$It's a factor. It's also a constant. It's also a super-parameter and one that needs to be adjusted.

In PyTorch,$p$Set 0.1. Calculating an average slide to update$\mu$average. At the end of the test, you don't have to count what's inside.$\mu$Call.$\sigma$Got it. Because at the time of the test, there was no batch in the real application, so you could just take it.$\bar{\mu}$Call.$\bar{\boldsymbol{\sigma}}$ Which means...$\boldsymbol{\mu},\boldsymbol{\sigma}$During training, the average move received replaced the original$\boldsymbol{\mu}$Call.$\boldsymbol{\sigma}$, and that's how batch integration works when it's tested.

The role of bulk consolidation

There's a paper to support the core of the presentation. Batch consolidation can change the surface of the error and make it more smoother. So we can choose a higher rate of learning, thereby increasing the efficiency of training.

While the literature considers the final training to be in the same location, the increased efficiency of training can save resources and time for training, and therefore there is merit in bulk consolidation.

Of course, mass subordination is not the only way to normalize, and there are many ways to regularize them, which are the result of dealing with the appearance of error, although there are significant differences in specific thinking, most of which are by chance found by the author of the article.

Neural network, multilayer sensor MLP, deep learning.

Neutron model

Neural networks are a broad, parallel network of simple adaptive units whose organization can simulate the interaction of bioneurological systems with real world objects.This is our definition of neural network.

In fact, the neural network is not a branch of machine learning, it is much earlier in biology than machine learning; the neural network that we are introducing here is the product of the interlocking of machine learning and neural network learning, and the basic theory of deep learning before it comes into being.

The most basic component of the nervous network is the neuron model; it accepts data from multiple connected neurons and gives output; The basic neurons model is the MP neurons model. The neurons receive input signals from several other neurons, which are transmitted through the connection of weights, the total input values received by the neurons are compared to the threshold values of the neurons, and are then processed through the Activation function to produce neurons output.

Connecting many of these neurons to a certain level of structure gives them a neural network, which in mathematics is a mathematical model with very many parameters, receiving input, giving output.

Sensor and multilayer network

Perceptron consists of two layers of neurons, which receive external input signals and pass them to the output layer, which is a neuron. Select the appropriate active function.

It should be noted that the sensor is only active with an output-level neurons, i.e. with only one layer of functional neurons, with very limited learning skills, which in fact proves mathematically that the sensor cannot handle any non-linear problem.

To solve the problem of non-linear subdivisions, consideration needs to be given to using multi-layer functional neurons to add an intermediate layer to the input neurons and output neurons, and it also has an active function to find the right weights in training.

Each of the most classic multi-layer nervous network structures is fully interconnected with the lower neurons, and there is no inter-layer connection between the neurons and no inter-layer connection.Multi-layer front feed neural network." (multi-layer feedforward neuralnetworks, MLP) is also one of our most basic nervous network structures.

Error/dissemination algorithm (BP)

Multilayered networks are much better at learning than single-layer sensory machines, and training them is a problem.

In reality, most of the neuronets are trained by BP algorithms, which can be used not only for the multilayer frontal feed neural network that we've introduced, but also for a lot of neural network training, but the BP neural network usually refers specifically to the multilayeral feeder neural network (MLP) that is trained by BP algorithms.

Now, let's talk about the BP algorithm, the training data set.$D$ Each sample includes$d$Inputs and$l$So, we built a possession.$d$ It's an input neurons. $l$An output neuron.$q$The multilayer front feed network structure of a hidden neurons Activate all functions selected as Sigmod functions

For any training example, the average error caused by the network is $$E_k=\frac{1}{2}\sum_{j=1}^{l}(\hat{y}_j^k-y_j^k)^2.$$ We need to learn the parameters of input to the hidden layer.$d\times q$ A weight to hide the layer to the output layer.$q\times l$ Right $q+l$The threshold of a neuron, it's obviously not realistic to optimise so many parameters at a time, and BP is an iterative learning algorithm that updates the parameters in a broad sense learning rule for each round of it. $$v\leftarrow v+\Delta v.$$

BP algorithm is based on the Gradient Decline policy, adjusting parameters in the negative gradient direction of the target. gave error$E_k$ Learning rate$\eta$ Yes. $$\Delta w_{hj}=-\eta\frac{\partial E_{k}}{\partial w_{hj}}$$ Depending on the link of influence, we can give you That's a good idea.{j}^{k}}\cdot\frac{\partial\hat{y}{j}^{k}}{\partial\beta_{j}}\cdot\frac{\partial\beta_{j}}{\partial w_{hj}}$$ 因此 有 $$\frac{\partial\beta_{j}}{\partial w_{hj}}=b_{h}.$$ $$\begin{aligned} g_{j}& =-\frac{\partial E_{k}}{\partial\hat{y}{j}^{k}}\cdot\frac{\partial\hat{y}{j}^{k}}{\partial\beta_{j}} \ &=-(\hat{y}{j}^{k}-y{j}^{k})f^{\prime}(\beta_{j}-\theta_{j}) \ &=\hat{y}{j}^{k}(1-\hat{y}{j}^{k})(y_{j}^{k}-\hat{y}{j}^{k}). \end{aligned}$$ 我们就能给出BP算法权重的更新公式为 $$\Delta w== sync, corrected by elderman == Similarly, we can give updated formulas for other parameters;

The learning rate controls the pace at which algorithms are updated, too much can produce oscillations, too much can lead to too slow a contraction, and the choice of the appropriate learning rate is an issue worth considering in training.

The algorithm that follows is based on a model of minimization of MSEs, and if we read all the training data at once, we can optimize the entire MSE, but this slows down the pace of training, especially when the training data sets are too large, and in fact we usually use batch training for NN-based models.

There's a mathematical theory: a multi-layer frontal feed network can approach a continuous function with any degree of precision. However, how to set the number of hidden neurons remains an open question and is usually applied by a “test-in” method. Special-by-error) adjustments

Because of its powerful representational ability, the BP nervous network is often colluded; either we divide the training and test sets and terminate the training without continuing to reduce the concentration error, or we add a regular item to the target function to punish the network complexity. $$E=\lambda\frac1m\sum_{k=1}^{m}E_{k}+(1-\lambda)\sum_{i}w_{i}^{2},$$

In-depth learning

In theory, the more complex models the more parameters, the greater the capacity, which means that it can perform more complex learning tasks. In general, however, complex models are inefficiently trained, prone to oversatisfaction and therefore difficult to favour.

And with cloud computing, big data age,A significant increase in computing capacity mitigates training inefficiencies and a significant increase in training data reduces the risk of overloading, therefore, with "Deep Learning""(Deep learning) The complex model represented is beginning to get attention.

The typical deep learning model is the deep neural network; In terms of increasing the complexity of models,The increase in the number of hidden spheres is clearly more effective than the increase in the number of hidden neurons., because increasing the number of hidden layers increases not only the number of neurons with active functions, but also the number of layers that activate the embedded functions.

We'll use two subsections to briefly discuss two small issues in the field of in-depth learning, which will be the basis for our detailed presentation on in-depth learning.

FFN before full connection.

The whole connection means that the parameters are densely connected, and the so-called FeedForward network, FFN, consists of an active layer in the middle of two linear transformers.

FeedForward extracts deeper features from linear transformations and non-linear activation functions, which first map data into space at high latitudes and then into space at low latitudes; and by introducing non-linear transformations to activate functions, enhance model alignment to complex models.

While the structure and multilayer sensor MLP are the same logic of formation as the entire pre-connected neural feed network FPN, the FPN places greater emphasis on Module in a large network, which is Module for enhancing the ability to develop the most complex models, and MLP emphasizes that we use a separate network.

In the Transformer model, the FNN layer occupies the vast majority of parameters. Instead, the attention-level QKVO matrix does not occupy that much of the parameters, although they are at the heart of the attention mechanism.

Disabled connection

ResNet is a landmark model in the history of in-depth learning, and the models of in-depth learning prior to ResNet are generally between 20 and 30 floors, but after the emergence of ResNet, the number of in-depth learning models is raised to over 100 floors, even to 1,000 floors.

ResNet primarily addresses degradation of deep networksIt's not to solve the problem of alignment, it's not to solve the problem of the disappearance of gradients and the explosion of gradients, which can already be solved by bulk consolidation, but it's not as good as the subsurface network when your network level is further deepened, which is called degradation.

The key features of in-depth learning, compared to traditional machine learning, are deeper network layers, non-linear conversion (activated), automatic characterization extraction and characterization. Non-linear conversion is a key objective, which maps data to high latitudes to facilitate better completion of “data disaggregation”. As the network deepens, it is introduced.激活函数And more and more, the data are being mapped into more discrete spaces, and it is difficult to get the data back to where they came from.

The core idea of the disability network is that each additional layer should more easily include the original function as one of its elements. If we can map the new layer with constant training, then there will be no degradation, while the new layer can bring new identification possibilities, and thus better results.

Disable connections are very simple in terms of implementation.$x$As input, after multiple layers of full connection and activation (e.g. through a FPN), output$F(x)$So the reverse transmission process behind us is to match this function.$F$.

After using the residual connection, our output became$F(x)+x$ In order to ensure the dimensions of the residual connection, we need to introduce some layers of a 1 width. In debris (one of the smallest Modules used for disability connections), input can be disseminated more rapidly forward via cross-layer data lines. It brings about better performance.

Questions about deep learning Essence

In-depth learning of DLs is parametrical, that is, Over-Parametrization, but still successful, contrary to some theoretical assumptions about the original machine, and in the absence of a new theoretical explanation.

The optimization of in-depth learning is an obvious non-optimization problem, but he uses a simple optimisation method (Adam, SSD, etc., using gradients) that tends to yield good results in practice.

This is done with the current initial technology of randomization (avoiding local non-ejectability), parametrical neural network over-theatre (super-high space avoids local excellence), gradients disappear (reLU, Adam, Resnet, Batch Normal, Layer Normal).

Categorization and cross-radon losses

Classification and return are the two most common types of issues for in-depth learning. There has been only sporadic cross-plymerization, starting with negative logarithmic losses, which have been associated with subclassifications, multiclassifications and the realization of works.

Start with negative logarithmic losses

Instead of looking at the model structure for the time being, it would be a black box. Assuming that the model has given the right type of probability. $p$, zero of which<p<1$。我们希望 $p. The larger the loss, the smaller the loss; the most direct option is negative logarithmic loss:

$$L=-\log p$$

The code is simple:

import math

loss = -math.log(p)

Start Category 2

The issue is now classified as a category II. Labels only 0 and 1, use $y$ Show real labels, use $p$ It means the model thinks $y=1$ The probability, then. $y=0$ The probability is... $1-p$I don't know. Write two scenarios into the same pattern:

$$L=-[y\log p+(1-y)\log(1-p)]$$

This is the source of binary Cross Entropy, BCE.

Training usually takes place in a mini-batch. Calculates the loss for each sample, and then fixes or averages the values for the frame; the common default settings are averages. To look straight at the formula, write it in a loop:

import math

def binary_cross_entropy(ys, ps):
    total_loss = 0.0
for y, p in zip(ys, ps):
    loss = -(y * math.log(p)
             + (1 - y) * math.log(1 - p))

    total_loss += loss

return total_loss / len(ys)

The actual project will be quantified using a NumPy or a stretch frame without hand Write for Loop. The visualization up there is assuming everything. $p$ It's all strictly located. $(0,1)$ I'll meet you there. $\log 0$。

The binary label can be considered as subject to Bernoulli distribution. The negative logarithmic is as follows:

$$\mathrm{NLL}=-[y\log p+(1-y)\log(1-p)]$$

So, minimizing BCE is the equivalent of minimizing negative logarithm, which is maximizing. This idea remains valid when it is later extended to multi-classification and soft target.

From output logit to probability

The previous section always assumed that the model had given probabilities. $p$, but the last linear layer of the neural network usually produces a real number, i.e. logit, which is not limited in scope. In the second category, it can be written as follows:

$$z=w^{T}x+b$$

Sigmoid can map any actual number. $(0,1)$, so this output can be interpreted as probability:

$$p=\sigma(z)=\frac{1}{1+e^{-z}}$$

import math

def sigmoid(z):
    return 1 / (1 + math.exp(-z))

A single sample of BCE can be written in visual terms:

import math

def sigmoid(z):
    return 1 / (1 + math.exp(-z))
def binary_cross_entropy_from_logit(z, y):
    p = sigmoid(z)
    return -(y * math.log(p) + (1 - y) * math.log(1 - p))

This code corresponds to a mathematical definition, but it is not suitable for direct training: extreme logit will allow $p$ Round to 0 or 1, then trigger $\log 0$I don't know. In Sigmoid, logit is also a probability logarithmic:

$$z=\log\frac{p}{1-p}$$

BCEWithLogits

Extreme logit gives rise to numerical instability that has nothing to do with the mathematical definition, but that actually affects the calculation. Combining Sigmoid with the BCE, you can bypass the problem of probability before logarithm.

Start with the following formula:

$$L=-[y\log\sigma(z)+(1-y)\log(1-\sigma(z))]$$

Take Sigmoid and simplify it.

$$L=y\log(1+e^{-z})+(1-y)\log(1+e^z)$$

Further organized as:

$$L=\log(1+e^z)-yz$$

The BCEWithLogits format can eventually be changed to a numerically stable form:

$$L=\max(z,0)-zy+\log(1+e^{-|z|})$$

The code is as follows:

import math

def bce_with_logits(z, y):
    return (
        max(z, 0)
        - z * y
        + math.log1p(math.exp(-abs(z)))
    )

That's why PyTorch is used. torch.nn.BCEWithLogitsLoss , instead of first doing Sigmoid. Multi-Category Use torch.nn.CrossEntropyLoss It also receives logits directly, but its internal combinations are LogSoftmax and NLLLOSs, not mixed.

For a single sample requesting a gradient, the result happens to be:

$$\frac{\partial L}{\partial z}=\frac{1}{1+e^{-z}}-y=p-y$$

The code is direct:

def grad_bce_logit(z, y):
    p = sigmoid(z)
    return p - y

If the damage is averaged, the number of samples will also need to be divided. Compare Sigmoid to MSE, this gradient will not add up Let's go. $p(1-p)$As a result, it is usually easier to optimise in the Sigmoid saturation area; this does not mean that the entire network does not have gradients that disappear.

Softmax and multiple categories

There's only one logit that fits the second category. For multiple categories, the last linear layer will output a vector with dimensions equal to the number of categories. Take three categories, for example:

$$[x_1,x_2,x_3]$$

If Sigmoid is done separately for each value, the probability of each category is independent of each other and does not have to be 1 in sum, which is more appropriate for multi-label classification. Multiple classifications require competition between categories, and Softmax translates the whole group of logits into a combined probability distribution of 1:

$$p_i=\frac{\exp(x_i)}{\sum_j\exp(x_j)}$$

At this probability distribution, the multi-classical cross-tape is:

$$L=-\sum_i y_i\log p_i$$

If the target is one-hot vector, only the one that corresponds to the correct category will be retained. Look at a visual realization:

import math

def softmax(logits):
    exp_values = [math.exp(z) for z in logits]
    total = sum(exp_values)
return [v / total for v in exp_values]

target 只需要接受一个目标类别，不需要全部 label
def cross_entropy(logits, target):
    probs = softmax(logits)
    return -math.log(probs[target])

This version is close to the formula, but the larger logit will spill over the index. When calculating Softmax, you can subtract the maximum value first; when calculating crossbow, you can use LogSumExp:

import numpy as np

def softmax(logits):
    logits = np.asarray(logits, dtype=float)
    shifted = logits - np.max(logits)
    exp_values = np.exp(shifted)
    return exp_values / np.sum(exp_values)
def cross_entropy(logits, target):
    logits = np.asarray(logits, dtype=float)
    m = np.max(logits)
    logsumexp = m + np.log(np.sum(np.exp(logits - m)))
    return logsumexp - logits[target]

Two categories Softmax can also be transformed into two loguit differences Sigmoid. For example, the second probability is that $\sigma(x_2-x_1)$So the two are very close in mathematics and engineering.

Crossing entropy. $k$ The gradient of a logit is also simple:

$$\frac{\partial L}{\partial x_k}=p_k-y_k$$

Other isolated topics

Multi-label classification usually calculates BCE for each label separately, rather than using Softmax. Each label is highly probable because it is not mutually exclusive and does not need to add up to equal 1.

LLM's next token projection is essentially to calculate multi-classic cross-paramerics on the vocabulary. Training usually calculates loss only for valid target token and takes averages on all valid token of a bat; other options can be used for different frames.

Padding should not be involved in loss calculations, usually by ignore_index Or los mask excludes.

Crossing entropy is also acceptable, soft target is retained at this time $-\sum_i y_i\log p_i$ All items in it. As long as the target is the combined probability distribution of 1 the gradient remains $p-y$I don't know. Knowledge distillation and label smoothing are used in this form, in which the distillation of knowledge can be used to transmit more information on the distribution of categories given by the teacher model.

Temperature parameters will replace logit in Softmax with $x_i/T$I don't know. The following is given for a value stabilization:

import math

def softmax(logits, temperature=1.0):
    if temperature <= 0:
        raise ValueError("temperature must be positive")
scaled = [z / temperature for z in logits]
maximum = max(scaled)
exp_values = [math.exp(z - maximum) for z in scaled]
total = sum(exp_values)
return [v / total for v in exp_values]

When $T>1$ 时，概率分布会更平滑；当 $0<T<The distribution will be sharper at $1.00.