Deep Learning Network Architectures: CNNs, RNNs, and Seq2Seq

This English version was machine-translated from the Chinese original. Technical terms may require verification.

After the introduction of the basics of in-depth learning, we can formally build the networks that we like, and here we can present some of the simple, but very practical, network structures that have evolved in the early stages of in-depth learning, which are very helpful in helping us to understand the depth of learning.

Volatile Neural Network CNN

We start with a condensed neural network, looking at its architecture. The web is a very typical network structure.Regularly used for tasks like image classificationI don't know. By building up a network of neurons, we can see how the network structure is designed and why a reasonable network structure can optimize its performance.

This section will be the first section of our research network architecture.

In particular, our first research network is structured in the following way:Machine Learning Introduction and Monitoring of the Neuronet in LearningThe network structure we're looking at is the most basic.Connect all networks

Volume can also be one-dimensional, for time series or other sequence problems, or for scaling order size

Organisation

First, we need to know how to input images as models. We usually use RGB format to describe a digital picture, using a three-dimensional vector.ScaleWe can describe this picture. One dimension represents the breadth of the image, the other dimension represents the height of the image and one dimension represents the image.Channel The RGB model has three channels, red and green.

Network input is often a vector, and therefore a three-dimensional dimension representing the image needs to be “strengthened” before it is “discarded” into the network. Suppose we use a picture of 100 pixels both tall and wide. An image is made up of 100 x 100 x 3 numbers, and a row of these numbers is a huge vector.

In the area of image processing, in order to facilitate the design of the network, there is a general need to ensure that images are of the same size and reinvested in depth learning Medium

The input vectors in the image processing field are often relatively long, treating the vectors as input for a fully connected network, resulting in a large number of weights to be optimized on the first level. More parameters bring greater flexibility and capability to the model, but also increase the risk of overloading. Normally, we don't choose a full-connected network, and as to how to do it, it's people's observations and experiences in the area of image identification, and we'll stay and introduce them.

The central point of the image classification issue is classification, and we need to code the classification indicators using a single thermal vector and to optimize losses using cross-breathing, which can be cited.Classification component of the in-depth learning baseThere's not much to add.

You don't need the whole image for the test mode.

Local testing of ideas

Suppose our mission is to make the network recognize the image of the animal. For neurons in an image-identified nervous network, what it does is to detect the presence of particularly important models (pattern), which represent an object. For example, three neurons see a bird's mouth, an eye, a bird's claw, three patterns, which means a bird is seen in a neural network.

People often capture the most important features when they judge an object. When these features are seen, an object is intuitively seen. For machines, this may be an effective way to judge objects in images. But assuming a neuron is used to determine whether a pattern is present, it may not be necessary for every neuron to see a complete image. Because you don't have to look at the whole picture to judge whether or not important patterns appear.

This means:Only a small fraction of the images will be used as input to enable them to detect the emergence of particular critical patterns.

Feel the introduction of wildness.

With this idea, we could consider starting the first simplification of the entire network.

A congested neural network will set a zone, that is,Feel freeEvery neuron only cares about what happens in the wild, and it's our decision. For example, we could set a size to$3\times 3$Then there's only one neuron.$3\times3\times3=27$A single input, even with a bias, will not exceed 30 parameters.

Feel wild can overlap with each other, and we can get multiple neurons to prepare for one.

Feel wild can be big or small, because patterns are smaller or larger. Some models may be detected within 3x3 and others may be detected within 11 x 11.

Feel wild can only think of certain corridors. At present, the feeling is that all three RGB corridors are considered, but there may be patterns only in red or blue, that is, some neurons can consider only one channel. Then, when it comes to network compression, it comes to the architecture of such networks.

The feeling of the wild is usually connected, but there are some issues that we may really need to feel separate from each other to capture some of the features, some of which may indeed be scattered at different locations of the image.

Feel wild can be not only square, as in the example just given, 3 x 3, 11 x 11, but also rectangular, and can be designed to feel wild in the understanding of the problem. Here's some basic design ideas that apply to most questions.

Feel the basic design of wildness.

Generally, when image recognition is done, it may not be felt that some patterns appear only in a particular channel, so they look at the entire channel. Now that you're looking at all the tunnels, when you describe a feeling field, you just talk about its height and width, not its depth, because its depth is equal to the number of tunnels, and the combination of height and width is calledNuclear Size。

The common feeling field setting is 3x3 nuclear size. Seven by seven, nine by nine is a pretty big nuclear size, not our first choice. There's usually a group of neurons in the same sensory field to keep this range, for example. 64 or 128 neurons to keep a field of feeling.。

We move the feeling from the upper left corner to the right, and we create a new range, the new feeling field. The amount moved is calledStep (stride)I don't know. The scale is a super-parameter that requires artificial adjustments. Because there is an overlap between feeling wild and feeling wild to ensure that the characteristics of feeling wild borders are correctly identified, the scale is often not too large, usually set at 1 or 2.

If you do not “show” the field beyond the image, there are no neurons to detect the pattern of presence at the boundary, which leaves behind the image boundary, so that the general boundary will also be considered. Just do it outside the range.Filling, fill is a supplement, usually with zero fill, plus 0 if you feel a part of wild beyond the image, then consider that the value inside is zero.

In addition to moving horizontally, there will also be moving vertically, with the same vertical range and horizontally.

And in this way, we've cleaned the whole image, so every inch of the whole image is covered by a feeling field. That is, there's a bunch of neurons in every location in the image that's testing that place.

The same pattern may appear in different areas of the image

The idea of pattern repetition

The same pattern may appear in different areas of the image. For example, it could come out. Now the upper left corner of the image may also appear in the middle of the image, and the same pattern is not too problematic in different positions of the image. Because we use multiple neurons to prepare the same sensory field, the pattern has a matching neurons wherever it appears.

But the neurons that test birds' mouths do the same thing, but they do it in a different range. In that case, there's really no need for a neurons to test the mouth of a bird in every range. If there is a neuron to detect the mouth of a bird in a different range, there will be too many parameters, and a corresponding simplification is required.

Share parameters

Allowing neurons to share parameters, i.e., to share parameters (parameter sharing), is that both neurons have exactly the same weight.

Parameter sharing generally targets neurons that are prepared for different senses, and they use exactly the same parameters, but their output will not always be the same because their input is different and their range of care is different.

The manner of sharing is entirely up to you, but we also have some customary principles in the area of image processing. Very naturally, we said that every field of feeling has several neurons to identify patterns.Neurons with the same number should be shared between different feeling fields. Because they're identifying the same features.

So in fact, all the feeling fields used the same set of parameters, which are calledFilter This significantly reduces the size of the parameters we need to optimize.

Volume Layer

Feel wild and share the parameters.Volume layerIt's called using the network on the roll.Volatile neural network。

Volkswagen neural networks have a greater deviation. However, large model deviations are not necessarily bad, as it is more difficult to match when models are highly biased and less flexible.

Volkswagen layers are specially designed for images, and feeling fields and parameters are shared for images. Although the model of a condensed neural network is highly skewed, its use on the image is not a problem. If it is applied to tasks other than images, it is necessary to consider whether these tasks have image-specific features.

Multilayer Neural Network

After a contours map, we can imagine what we've got, assuming we're looking at an image of a size of six by six, using 64 filters (neurons), with a sense field set at 3 by three, and a scale of one.

So each filter will give us a set of 4 x 4 numbers, 64 of them calledFeature Map The feature map can be seen as a new image, except that the channel of the image is not three channels, but 64 channels, each of which corresponds to a filter.

Volume layers can be stacked over many layers, and we can continue to set filter sizes to 3 x 3, but turn heights to 64 to match the number of channels.

The deep learning effect of the deep neural network is achieved by stacking multiple layers of contours.

Special: The size of the filter is always set at 3 x 3 and the network will not be able to see a wider pattern. As the number of layers increases, the network becomes more visible, thus observing a more holistic pattern, which is why nuclear sizes vary too much, and we make up for it in depth.

The bottom sample doesn't affect the mode test.

Sample Down

Make a bigger image.Downsampling, remove all the columns of the even number of images, remove all the lines of the odd number and turn the image into a quarter, without affecting what's inside. The pattern of the images is unchanged.

Gather

There are no parameters, so it is not a layer, it has no weight, it has nothing to learn. It's an operator, and its behavior is fixed and it doesn't have to be based on data.

Each filter produces a set of numbers, which, when it comes to gathering them, can be grouped into two groups of two, three, three, four, four, or, you know, we decided on that.

There are many different versions of the convergence.Maximum concentration (max popling) In each group, one representative is chosen, the largest. In addition to the biggest gathering,Average convergence, the average aggregation is the average of each group.

Convergence and pool are different translations of the same meaning, as can be seen from the English terminology interpretation.

What we did together was to reduce the image to a fraction of the original size of hundreds of pixels. After the assembly, the path of the image remains the same, but the width and heights are rapidly shrinking.

A general network of curly nerves.

Generally, in practice, volumes are used interchangeably with aggregates, which can be done several times in volume and at a single assembly. For example, two volumes, one gathering. However, convergence may cause a little harm to the performance of models. Assuming it's a very fine thing to test, to do random sampling, it's probably a little less effective.

So in recent years, the design of the network of images has often started to throw out the convergence, making this full-volume neural network, which is a whole network of volumes, without any need to gather. The most important effect of this convergence is to reduce the volume of the operation by reducing the image by sampling it below.

As computing capacity has grown in recent years, many network architectures are often designed not to converge, but to use a whole volume, volume through volume, to see if they can be done better, if the computing resources are sufficient to support it.

The general structure is a build-up of clusters, which can only be achieved, and many may choose not to. If you're done with a few volumes and aggregates, do the output.Flatten, and then throw this vector into the entire connector layer and eventually pass a softmax to get the image recognition. This is a classic image recognition network.

A flattening is the straight-up of the matrix into a vector that allows the results of the matrixing to be entered into structures such as the full connectivity layer.

Application of a rolling neural network: chess and closing remarks

In addition to image recognition, another of the most common applications of the congested nervous network is to play chess.

Chess is actually a sort of problem, and web input is the location of blacks and whites on the board, and the output is the position where the next step should be. The network input is a vector with 19 x 19 positions on the chessboard that can be expressed as a 19 x 19-D vector. In this vector, if there is a black item at a certain location, this position is filled with one, if there is white, with one, with zero.

By presenting the board as a vector, the network will know the momentum on the board. Throw this vector into a network, and chess can be seen as a classification problem, and then predict the next step through the network. Response The best position for the drop, the following is a classification question with 19 x 19 categories, and the network will export the best of 19 x 19 categories, thereby choosing the next drop. The problem could be solved by a fully connected network, but with a more effective condensed neural network.

Special: In the original paper of AlphaGo, the location of each board, that is pixels on each board, is described in 48 channels, that is, each location on the board is described in 48 numbers. This number was given by chess experts.

The condensed neural network, actually, isn't easy to use. It's designed for images. If a problem doesn't share the characteristics of the image, then you shouldn't be using a convoluted neural network. Since chess can build a neural network with a roll, this means that it has common features with images. For example, the local model and its repetition, of course he does not apply to the pool, which is quite natural.

In fact, a condensed neural network cannot deal with zooming down or spinning. Assuming that all the objects in the images are smaller, when a contours neural network learns to identify the images on a certain size, we zoom in on the objects, and its performance is much lower.

As a result, data enhancement is often done in image recognition. The so-called data enhancement is to amplify a small piece of each of the training data in order to show the condensed neural network a pattern of different sizes; to rotate the image and show it what it looks like when an object rotates, so that the condensed neural network will achieve good results.

Circulatory Neural Network RNN

The circular neural network is a very classical network structure in the field of in-depth learning and has a wide range of applications in real life.

Take the example of slot filling. Assuming that the booking system hears the user saying, "I want to arrive in Shanghai on 1 June." The system has some slots (slot): the destination and the time of arrival, the system automatically needs to know which one of each words on this side belongs to, for example, Shanghai belongs to the destination slot, and June 1 belongs to the arrival slot.

This problem can be solved by using a former feedforward neural network, which enters word by word into the neural network so that the network tells us which word should be placed.

Naturally, we need to present some basic NLP-related questions here, of course, which all researchers need to know.

Unhot Encoding

Only thermal coding is developed in the field of statistics, and we've introduced multiple coding methods in the broad linear regression between the variable and the virtual variable, and here we use one-hot encoding in the broad linear regression that is most basic.

Of course, since the words may not have been seen, we are used to one dimension representing the other.

Assuming that the word is a vector, the vector is thrown into the front feed neural network, where the output is a probability distribution, which represents the probability that the input word belongs to each cell.

But there are problems with the feed network. Assumes user 1 says: "Arrival in Shanghai on June 1st." User 2 said: "On June 1st, when Shanghai left Shanghai, Shanghai became the departure point. But for the neural network, input the same thing, output should be the same thing. In the case of “Shanghai”, the output either gives the highest probability of destination or the highest probability of departure.

In this case,If the neural network has memory,It remembers that it saw “arrive”, before it saw “Shanghai”; or it remembers that it had seen “depart”, before it saw “Shanghai”. Through memory, it produces different outputs according to context. The problem of the front-line neural network was solved.This introduces the circular neural network that this chapter is about to introduce.

RNN Base

What's RNN?

In RNN, every neurons in the hidden layer produce an output that will be stored.Memoory cell, the neurons will consider not only input, but also values stored in memory cells.

The memory is short asCell, the value of the memory is also known asHidden state

When you do a circular neural network, it takes into account the order of the sequence and enters a different output after the sequence has been changed.

Note that RNN network structure input is a sequence, which is a model for sequence input, and what is a sequence that can refer to the previous example as a uniquely thermal coding part

Because the current state of concealment uses the same definition as the previous state of concealment, the calculation of the hidden state is circular (recurrent), and the hidden neural network based on circulation is known as the circular neural network. This is RNN's name.

RNN Structure

We're also dealing with the example of the previous slot filling, where the user says, "I want to get to Shanghai on June 1st, and "arrive" becomes a vector "throw" into the nervous network, where the hidden layer is output. $a_1$，$a_1$ Generate the probability that "arrive" will be filled in each slot $y_1$。

Next $a_1$ It'll be stored in memory, and Shanghai will be converted into input, and this hidden layer will consider both Shanghai and the memory. $a_1$♪ Got ♪ $a_2$I don't know. Based on $a_2$ Got it. $y_2$，$y_2$ is the probability of each tank filling.

We use the same neural network, which uses the same elements of a sequence several times at different time points, and the network has the same weight.

When you have a memory, you enter the same word, and you want to output different problems, you can solve them. Because there are different values in the memory, the output of the hidden layer will be different, so the final output will be different.

The architecture of a circular neural network can be designed at random, for example, from the single hidden layer just presented to the output layer, and we can add multiple layers, and the hidden layer can be set up with different memory elements, thus creating different types of RNN.

Elman and Jordan networks

What we're talking about is simple circulation networks (Simplement Recurrent Network, SSN, also known as Elman), which store the value of the hidden layer and read it at the next point of time. Each neuron value handles the hidden value of its own input into the previous series.

And there's another one called Jordan Network, where Jordan has the value of the entire network output, which reads in at the next point in time and puts it in memory. Add him to the next element of the sequence.

Elman's network has no target, and it's hard to control what it learns about hidden layers (what it learns to put in memory), but Jordan's network has a goal, and it's better known about memory storage.

Two-way circular neural network

Circulatory neural networks can also be two-way. RNN just entered a sentence that reads from the beginning to the end. But it can be read in the opposite direction.

We can train a positive circular neural network at the same time, and we can train a reverse recycle neural network, and then take out the hidden layers of both. $y_t$

The advantage of a two-way circular neural network (Bi-RNN) is that when neurons produce output, they see a wider range. If there's only a positive network, produce $y_t$、$y_{t+1}$ The neurons only saw it. $x_1$ Present. $x_{t+1}$ .

But a two-way circular neural network is created. $y_{t+1}$ It's not just the Internet. $x_1$Here. $x_{t+1}$ All the input. It looked at the end of the sentence. $x_{t+1}$ . The network is the entire input sequence.

Assuming that a sink is considered to fill the network, it would be equal to looking at the entire sentence before deciding on the slot for each word, which would be better performance than looking at half the sentence.

Long-term and short-term Memory Network

LSTM Foundation

The memory element mentioned earlier is the simplest one that can be stored in memory at any time or read out. But the most common memory isLong Short-term Memory NetworkLong, short-term memories.

LSTM is complicated. One LSTM memory, three.GateWhen the output of an outside neuron wants to be written in memory, it has to be passed.Input gate, the entry door is open to write value into memory. If you lock this up, there's no way to put it in. As for entering doors, the nervous network learns by itself when to open and close the doors.

There's one out there.Output door (output gate)The output door will determine whether the rest of the neurons can read the value out of this memory. When the output door is closed, there is no way to read the value, and when the output door is open, the value can be read. This memory is the source of the output. As with the entry door, when it opens and when it closes, the network learns.

The third door is calledForget gateForgetting the door determines when memory must forget what it remembers. When will this forgotten door forget the value of the memory object, and when will the value of the memory be preserved?

The whole LSTM can be seen as having 4 inputs, 1 output. Of these four inputs, one is the value of the memory object that wants to be stored, but not necessarily stored, and there is a signal that manipulates the entry door, a signal that manipulates the output door, and a signal that manipulates the forgotten door, with four entries, but it only gets one output.

The last circular neural network, its memory is washed away at every point in time, and every point of time is washed away, so the short term is very short.

But if it is a short-term memory for a long time, it will be longer and its value will be saved as long as the door is not forgotten. It's a long, short-term memory of LSTM.It's a long time to forget the door, and it's a short time to remember only the latest.

The mathematical structure of a LTSM memory is as follows: $c'=g(z)f\left(z_i\right)+cf\left(z_f\right)$$

Assuming the input to be stored in the cell is called$z$, manipulates the incoming signal.$z_i$The signal of the forgotten door is controlled.$z_f$Control the output door as$z_o$ When you combine these things, you get an output.$a$。

Assuming the four inputs were in the cell, they were stored.$c$I don't know. Put$z$Got by Activating Function$g(z)$,$z_i$Getting through another active function$f(z_i)$ (Activating functions usually selects the sigmoid function) because its value is between 0 and 1, and the value between 0 and 1 represents the extent to which the door has been opened. (if$f$the output is 1, which is the open state, and the reverse represents the closed door).

Next, put$g(z)$Multiply$f(z_i)$Got it.$g(z)f(z_i)$For forgetting the door.$z_f$, also obtained by sigmoid function$f(z_f)$And then we'll put the value in the memory.$c$Multiply$f(z_f)$Received c$f(z_f)$Plus$c^\prime=g(z)f(z_i)+cf(z_f)$Well...$c^{\prime}$It's the value that's stored in memory.

In the light of the current calculations, this...$f(z_i)$Control this.$g(z)$I don't know. Assume input$f(z_{i})= 0$ There.$g(z)f(z_i)$It's equal to zero. It's like there's no input.$f(z_{i})= 1$ It's like...$g(z)$As input.

Well...$f(z_f)$Whether or not to wash away the value of the memory element.$f(z_{f}) = 1$ ♪ Forget when the door opens ♪$c$It'll pass directly, and it'll be worth it before. If$f(z_{f})= 0$ ♪ Forget the door close ♪ $cf(z_{f}) = 0$

And then add up the two values.$(c^{^{\prime}}=g(z)f(z_i)+cf(z_f))$Write it in memory.$c^{\prime}$I don't know. The switch to the forgotten door is the opposite of the instinct, and when the forgotten door opens it represents the memory, and when closed it represents the forgotten.

Well...$c^\prime$Pass.$h(c^{\prime})$, will$h(c^\prime)$Multiply$f(z_o)$Got it.$a$I don't know. Output door.$f(z_o)$And by control,$f(z_o)$If it's equal to one, it means$h(c^{^{\prime}})$I can pass.$f(z_o)$is equal to 0, which means that the values that exist in the memory are not read through the output door.

The order of execution of the entire LSTM neurons is:Enter, enter the door, forget the door, output door One of the most complex door-state operations is possible in parallel.

LSTM rationale

There will be a lot of neurons in the original nervous network, and we'll multiply input by different weights as input to different neurons, each of which is a function, enter a value and output a value. Their interior is a linear function plus bias. Finally, by adding the output of the neurons to an active function to the non-linear, you can act as input for the lower neurons.

The basic RNN network structure that we are introducing adds memory elements, adding the results of the last memory to the input of this neuron, adding a parameter to the memory of the network.

In fact, the LSTM membrane structure that we have described before is basically in the form of simple neurons. We use the four separate linear combinations (with bias) of the original input as input, entering door checks, forgetting door checks and output door checks. Based on the results of the last checks, an output is given, which may be zero of the output or a number stored in memory.

From this perspective, if it's LSTM, just think of LSTM as a neuron. So using a LSTM neuron, it's actually a simple neuron.

In summary, LSTM has four inputs and one output. Assuming that the number of neurons used is the same as the LSTM, LSTM needs four times as many parameters as the general neural network. But there is no more structural innovation in the architecture of networks.

The LSTM actually doesn't just look like this, but the real LSTM will connect the output of the last time as the input of the next time, the value of which is not just the input of the time. $x_{t+1}$ And look at the output of the previous point. $h_t$ I don't know. Adds a peephole connection and pulls the value in the memory. That means it's both considered and considered. $x_{t+1}, h_t, c_t$

Gate Control Cycle Unit (Gated Recurent Unit, GRU) It's a slightly simplified version of LSTM, which has only two doors. Although one door is missing, its performance is similar to that of LSTM, with one third of its parameters missing, and it is more difficult to match.

The spirit of GRU is: no old, no new. It connects the input door with the forgotten door, which means that when the input door opens, the forgotten door closes automatically (formulates the value in the memory), and when the forgotten door does not have to format the value in it, the entered door closes. That's to erase the value in the memory to bring in the new value.

RNN Learning Mode

If you want to learn, you need to define a loss function to assess the good or bad of the model and select a parameter to minimize the loss. CNN is a classification task, with output being a well-coded classification vector, using cross-breathing to measure loss.

In the case of RNN, the issue of slot filling that we have just described is actually the target of the vector used for classification, so we should calculate the loss in the same way.

With the loss function, the model calculates the gradient by the classic reverse dissemination algorithm, which is then updated by the optimizer. The specific loss function is to be determined according to the type of task.

Within the circular neural network, an evolutionary version of reverse transmission is presented for ease of calculation, i.e. reverse transmission over time (BackPropagation Through Time, BPTT). The BPT is actually very similar to reverse transmission, but the circular neural network operates on time series, so BPT takes time information into account.

RNN training is difficult and, generally speaking, we hope that as the number of rounds increases, as the parameters are constantly updated, losses will slowly decline and eventually recede.

Unfortunately, when training a circular neural network, the learning curve is very vibrating and then shaking to a certain place, not because the program has Bug, but rather because of RNN.

RNN appears to have a very steep or rugged variation in total losses. The surface of the error is very flat, and some are very steep. This could lead us to use the gradient decline to find the right parameters, and because of the relatively large learning rate set and the sudden increase in the gradient, the loss could suddenly be prolonged by skipping the lost cliff. In this case,Clipping solves this problem, when the gradient is greater than a threshold, do not let it exceed that threshold. Promise not to fly too far.

When ReLU activates the function,Gradient disappearance (vanishing gradient) Source from the Sigmoid function. But RNN will have a smooth surface of error that does not come from the disappearance of the gradient. Changing the Sigmoid function to ReLU would not help.

It's hard for RNN to train.The core reason is its network structure for processing sequences., all parameters are shared throughout the sequence and repeated repeatedly.

In order to calculate the gradient of the earlier time-long steps, it is necessary to multiply the gradient of the subsequent time-long steps through the weight matrix. If the feature value of the weight matrix is less than 1, these continuous multipliers result in a rapid decrease in the gradient value. On the contrary, if the characteristic value of the weight matrix is greater than 1, the continuous product during the reverse dissemination will result in a rapid increase in the gradient value.

Disappearing or explosion of the gradient to resolve RNN

What techniques are there to solve this problem? The technique widely used is LSTM, LSTM can make the surface of error less rugged. It'll remove the flat places and solve the problem of the gradient.

LSTM will not solve the problem of the gradient explosion. Some areas are still very rugged, others are still very dynamic, but not particularly flat. If most of the LSTM is very dynamic, so when you're doing LSTM, you can set up a little bit of the learning rate, and you can guarantee training at a very low level. The use of clipping gradients is, of course, a very practical common method.

LSTM is effective in processing the disappearance of the gradient because he controls the flow of information, and in LSTM, once it affects the memory, the influence is kept unless the memory is washed away by the forgotten door. Otherwise, if the memory thing changes, it just adds something new, doesn't wash away the original value, so it doesn't have to disappear.

Forgetting the door may wash away the value of the memory. In fact, the first version of LSTM was designed to solve the problem of the disappearance of the gradient, so it didn't forget the door, which was added later.

RNN Other Applications

In the case of slot fillings, it is assumed that the amount of input is the same as the amount of output, that is to say, the number of words we enter, we give each word slot label, and RNN can do more complicated things.

More pair.

Emotional analysis is typical of multiple-to-one sequences.

The machine can learn a circular neural network, enter a character sequence, and the neural network reads it over. At the last point in time, the hidden layer is removed and, through several variations, the final emotional analysis is available.

Emotional analysis is a classification problem, but since input is a sequence, it is handled with RNN. Similarly, we can use RNN as a key term extraction.

Multiple pairs of sequences

RNN can also deal with multiple-to-multiple issues, such as input and output being sequenced, but output sequences are shorter than input series. Voice recognition is a classic example of multi-RNN.

Serial to Sequence

Another RNN application is sequence-to-sequence (Sequence-to-Sequence, Seq2Seq), where RNN input and output are sequences (but the length of both is different).

The input was longer, the output was shorter. And here we're thinking about not knowing who's older and who's shorter. Machine translation, for example, enters the English word series into the Chinese character series. The length of the English and Chinese series is unknown.

This is just a preliminary study of the RNN network, which is not well-developed, and which requires more papers to help us understand the different details of different algorithms.RNN's core is sequence and memory, which is widespread in modern society, so his improvements can be described as very broad.