Generative and Diffusion Models: GANs, Conditional Generation, and Diffusion Training

This English version was machine-translated from the Chinese original. Technical terms may require verification.

Generate Models

This chapter describes the generation model. So far, the network we learn is essentially a function, which is to provide an input. $x$, the network can output one result. $y$I don't know. of which output $y$ It can be either a number, a category or a sequence. Currently, these networks can cover most of the problems we encounter on a daily basis. This chapter will introduce another type of model, whose role is to generate a component.

Generate an anti-network

Generator

The network in the generation model is used as a generator (generator). Specifically, a random variable is used in model input $z$ With original input $x$ Enter the variable from the random distribution. When entering, you can combine it in a vector. $x$ and $z$ enter together, or $x$、$z$ when the length is the same, add both as input.

This variable $z$ In particular, it's not fixed, that every time we use the network, we get a new sample from a random distribution. $z$I don't know. Normally, our requirements for this random distribution are simple enough to allow easy sampling or to write directly the functions of the random distribution, such as the Gaussian distribution, the flat distribution, etc. So every time there's an input, $x$ At the same time, we all sampled from random distribution. $z$, to get the final output $y$I don't know. With the sampling. $z$ Different, we get output and y will be different. By the same token, for a network, its output is no longer fixed, but a complex distribution, and we call it a generator for a complex distribution.

First of all, we need to explain why the research generator is being developed, and now we have a background, video frame technology, and we want to see the next frame from the previous frame, which means that the web input is a few frames and the output is the next frame. If we train this model using the supervisory learning approach described earlier at this point in time, we will find many places where there are debris.

This is due to the fact that training data in monitoring learning have both left-to-left and right-to-right output for the same corner storage. When we're training, the Internet is instructed to learn to turn left for a training data. Similarly, for a right-to-right training data, the network is instructed to learn the right-to-right output. But in fact, both data may be trained at the same time, so the network learns to be “honor both”. That's what left it.

So how should we solve this problem? The answer is to give the network the chance to output all possible results, or to export a probability distribution, rather than the original single output. When we give the network a random distribution, the network input will add one. $z$, then the output becomes a non-fixed distribution that includes the possibility of a left and a right. This problem of non-fixing answers was resolved at this time.

Back in the generator discussion, what do we need for this type of generation model? The answer is when our mission needs to be "creative" output, or we want to know how many possible models can be produced, and these outputs are the right ones. In the generation model, what's very well known is the generation anti-network, which we usually call PAN.

First of all, there's unlimited generation, which means we don't need original input. $x$I don't know. All they have to do is get the original input. $x$ . For unlimited GAN, we just need to output random vectors.$z$, the model produces images (usually images) based on the results that have been learned.

Organisation

In GAN, in addition to the generator, we have to train an additional discriminator, which is usually a neural network. The sorter will enter a picture, output a scale, and the larger it is, the larger it is, the larger it is, the more it is, the better it is.

The sorter is, in essence, a neural network, like the generator, designed by ourselves, which can be either a condensed neural network or a Transformer, as long as it produces the input we want.

In model training, the parameters of the first generation generator are almost entirely random, so it doesn't know how to draw our target, so what it's drawing is some weird noise. The goal of the separator study is to identify the picture of the successful generator output. As the judge is divided, the generator evolves and tries to deceive the judge. And, of course, at the same time, the diagnosing device will be progressive and increasingly demanding, and the images produced by the “coercing” generator will become more and more responsive. So the generator and the judge have always interacted with each other, promoted relationships, as we call the "inner volume". In the end, the generator learns to draw images that fit the needs, and the judge learns to distinguish between pictures that are true and false, which is the training process of the GAN.

Training process for generator and identifier

Here's how the generator and the judge work. Generator and sentence It's two networks, and we have to start initializing the parameters separately before training.

The first step of the training is the stationary generator, which is trained only in the grader. Because the generator's initial parameters are randomly initialized, it learns nothing, enters a series of vectors from which it is given, and its output must be random, confusing images.

At the same time, we will have a database of images that meets our needs, which can be obtained by reptiles, etc. We'll take some pictures from this library to compare them with the results produced by the generator and to train the judge.The training objective of the determination device is to distinguish between real images and images generated by the generator. Specifically, we've marked all the real pictures with one, all the images from the generator with zero. And then, for the judge, it's a sorting or regression problem.

After we've finished training the judge, the second step, fix the judge, train the generator. The purpose of the training generator is to allow the generator to find a way to deceive the judge, since in the first step the judge has learned to distinguish the difference between the true and the false. If the generator can fool the judge, the image produced by the generator may be false. The specific operation is as follows: the generator first enters a vector that can be derived from sampling data from the Goss distribution that we described earlier and produces a picture. And then we put this picture into the judge, and the judge will give it a score. It's fixed here. It just needs to give a higher score to a more "real" picture.The goal of the generator training is to make the picture more real, that is, to increase the score.

There are many layers of neural network for both the generator and the judge in the real scene, and we usually view them together as a larger network, but they do not adjust the model parameters for the component. The training methodology is largely consistent with the web-based training approach described in the previous chapters.

Summarizing, the two steps of the GAN algorithm. Step one, stationary generator training determinationer; step two, fixator training generator. This is followed by the repetition of the above training, the training of the training generator for the fixator. After training the generator, the generator will be used to produce more new images before training the adjudicator. Train the generator after the grader has been completed, so over and over again. When one of them is trained, the other is fixed, and it is expected that they will all achieve the best in their own goals.

GAN application case

According to the previous thought training model, the quality of the images will increase as the number of training rounds increases, and only one image database will need to be used throughout the training process without re-marking. After tens of thousands of rounds of training, essentially generated images can be false.

Likewise, we can use GAN to create faces we've never seen before. The generator in the Gan that we used to introduce was to enter a vector and output a picture. In addition, we can make an internal difference in the vector input, and in the output part we can see a continuous change between the two images.

For example, we enter a vector to produce a man who looks very serious, while another vector to produce a smiling woman through a Gan. So when we enter the numerical vector between these two vectors, we can see that this man is laughing.

Another example is the input of a vector to produce a person looking to the left, while the input of a vector to produce a person looking to the right, where we make an internal difference, and the machine does not simply fold two pictures together, but creates a face. It's amazing that we don't actually enter a positive face when we're training, but the machine can learn to make a difference between these two left and right faces, and it should get a face that looks positive.

But if we're not restrained, GAN can produce some strange pictures that are completely out of human mind, but it can fool a judge of a complex neural network.

GAN's Theory Introduction

This subsection will present GAN at the theoretical level, which is why the interaction between the generator and the judge can produce human face pictures. The goal of PAN training is to minimize loss functions by adjusting model parameters.

The generator input is a series of vectors sampled from the distribution, and the generator produces a more complex distribution, as we call it. $P_G$ And we also have a series of data, and the raw data themselves form another distribution, which we call $P_{data}$I don't know. The effect of training is hope.$P_G$ and$P_{data}$ As much as possible. Our goal now is to train network parameters in a group of generator models that can be generated.$P_G$and $P_{data}$The smaller the difference, the better it is. $G^∗$。

Current$P_G$and $P_{data}$They're very complex structures, and it's difficult to measure their differences directly. Get in gan$P_G$and $P_{data}$The sampling is not difficult, the former can be generated by the generator, while the latter can be obtained by sampling in the diagram. So the spirit of kan is to estimate the difference through these two samplings without a formula to measure the difference, which is the introduction of the grader.

There it is.$P_G$and $P_{data}$After the sample is sampled, we're going to train a sorter whose goal is to give it a higher score when you see the real data and a lower score when you see the resulting data. We can consider it an optimisation. The problem is easy to resolve. It's actually the construction of a sort of sorter. The most basic loss function of PAN is closely related to JS diffusion.

As for the selection of specific indicators, we are not here to discuss them, and it is enough for the GAN to understand its ideas.

There's a lot of problems with the spread of JS, because we don't know.$P_G$and $P_{data}$The specific distribution, with only samples available, is actually very small owing to the number of samples taken or their own problems. First, for both non-overlapping distributions, the JS diffusion values are Log2 regardless of the specific distribution. Also, for two overlapping distributions, the JS diffusion value does not necessarily reflect the difference between the two distributions well. Because the JS diffusion value is capped, when the overlapping parts of the two distributions are large, the JS dispersion does not distinguish between the different distributions. So we need a better measure of two distributional differences.

Since it's a question of JS dispersion, someone must have asked if it could be solved in a different way to measure the similarity of the two distributions. Yeah, so there was Wasserstein, or used Wasserstein distance. Wasserstein's distance, also known as the bulldozer's distance (Earth Mover's Disturbance, EMD), considered the distance between the samples between the two distributions, that is, all possibilities. He can better measure the deviation between the two distributions, much better than the JS diffusion.

So WIAN actually replaces JS distance with Wasserstein distance, and this Gan is called WIAN.

GAN Performance Assessment Method

In order to assess the quality of the content generated by a GAN, one way is to train the classification system for an image, and then enter the picture produced by the GAN into the classification system of the image, to see what results it produces. The input of the image classification system is a picture, the output is a probability distribution, the probability distribution representative of the picture is the probability of a cat, the probability of a dog, the probability of a zebra, etc. The more this probability is distributed, the better the picture that is now generated. The image recognition system would be very confused if the resulting picture was a different one, and the probability distribution would be very evenly distributed.

This is a possible approach, but it is not enough. It's a problem called mode collapse. The pattern collapse is a situation that happens in the course of training GAN, and we're going to find the model that's generated and the images that it's exporting are coming and going, and it's probably just as good as it seems to you, but it's just a few more, and that's all that's going to come out, and that's just the pattern collapse.

The reason for the pattern collapse is the intuitive understanding that this place is a blind spot for the judge, and when the generator learns to produce the image, it can always cheat the judge, who can't see it as false.

But the problem of a model collapse, at least we know it's a problem that can be seen, and when the generator always produces this face, you don't say your generator is a good generator.

But there are some questions you don't know, and it's harder to detect, that you don't know if the images from the generator are really diverse. The problem is called model loss, which means that GAN is well placed to generate centralized training data, but it is difficult to generate untrained data, “lack of imagination”. The data that you produce, which is only a part of the real data, and the data that you simply see, may feel good, and the diversity of distribution is enough, but you do not know that the diversity of the real data is actually larger.

Although there are problems with model collapses, model loss, etc., we need to measure whether the images produced by the generator are sufficiently diverse. One approach has been to use the image classification that we have described earlier to throw a series of pictures into the image classification machine to see which category it is judged to be, as shown in figure 8.27. Each picture gives us a distribution, and we'll put it all evenly, and then look at the average distribution. If the average distribution is very centralized, it means that diversity is not enough now, and if the average distribution is very flat, it means that diversity is enough now.

When we use this image sorter to make assessments, there seems to be some mutuality in the diversity and quality of results. Because when we were talking about quality, we said that the greater the distribution, the higher the quality, the more even the distribution of diversity. But the more evenly distributed, the lower the quality because the more evenly distributed, the less representative the picture is, the lower the quality. It should be emphasized here that quality and diversity are not assessed in the same way, but in the same way as in the case of a single picture, the distribution is very concentrated when it is thrown into a sorter. And diversity looks at the average of a bunch of images, and the more even the image sorter outputs in a bunch of images, the greater the diversity now.

There used to be a very often used score, called the Incaption Score. It is by definition an assessment using the Inception Network, which measures quality and diversity. If the quality is high and the diversity is high, the Incaption scores are higher. At present, researchers usually use another assessment called Fréchet Information Division (FID). Specifically, the image of the human face generated by the generator is thrown into the InceptionNet to allow the Inception network to output its categories. What we need here is not the final category, but the output vector of the hidden layer before entering Softmax, the dimensions of which are thousands of dimensions, representing this picture, as figure 8.28 shows. All the red dots in the figure represent the vector that comes out after throwing the real picture into the Inception network. This vector is actually very high in dimensions, even thousands of dimensions, and we'll take it down to two dimensions. The blue dot is a picture produced by the GAN generator that is thrown into the Inception Network and later into the vector before Softmax. And we're going to assume that both the real picture and the resulting picture follow Goss distribution, and then we're going to calculate the distance between the two distributions. The smaller the distance between the two distributions, the smaller the distance, the closer the picture sets, the higher the quality. There are a few more details here. First of all, is it okay to assume that Goss distribution is okay? Another problem is that, if the distribution of the network is to be accurate, it will require the production of a large number of samples, which will require a little operation and will be an unavoidable problem for FID.

There is also a situation. Assuming that GAN produces a picture that looks exactly like the real one, then the FID will be zero, because the two are identical. If you don't know what the real data looks like, the output of this generator might feel great, but the FID must be very small, but if it produces the same picture as the training data in the database, it's not better to take some images directly from the training data set, so there's no need to train the generator. Our training generator actually wants it to produce a new picture, a human face that doesn't exist in the training set.

This is not an ordinary measure that can be detected. How do we solve this? There are some ways, for example, to use a sorter, which is to judge whether the picture is real or not, or whether it comes from your training set. The catalog input is a picture, the output is a probability, which says if it's from your training set. If the probability is one, then the representative says that the picture is from your training set, and if the probability is zero, then the representative says that the picture is not from your training set. But another problem, assuming that the generator learned to reverse all the pictures in the training data, was nothing. But your sorter would say that this picture is from your training set, because it's from your training set, and it's just a right and left reverse. There is no comparison between classification and similarity. So GAN's assessment is very difficult, and even how to assess a generator's poor performance is a researchable subject.

Conditional Generation

We're going to introduce conditional generation. The generator in the Gan that we talked about earlier, it didn't enter any conditions, it just entered a random distribution, and then produced a picture. Now we want to go further and hope to manipulate the output of the generator, and we'll give it a condition x to make an output by following the condition x and input z. For example, the creation of a text to a picture is a matter of condition. We now use conditions (such as text restraints) to encode them and then put them in the generator.

In conditional GAN, we need the ability to process conditions, we need text and image pairs of data to train them, so conditional GAN, the general training is this pair of labels. So when you see these real pairs of data, you give it a point, you see red eyes, but the text is black hair, you give it a point, you see black hair, but the text is red eyes, you give it a point, so you can train the judge.

In practice, it's not always good enough to train the differencer with a negative and positive pair. There is often a need to add a bad picture: a good picture has been produced, but the text is not adequate. So we usually get our training data out, then deliberately mix the text with the picture, or make some mistake, and then tell the judge to see the situation, and the output doesn't match. That's the way to train the judge. And then the generator and the judge were trained over and over again, and that was the conditioned GAN.

Spread Model

Diffusion model (diffusion model) is a production model using the idea of physical thermal diffusion. The proliferation model has many different variants, and this chapter mainly presents the most well-known de-noise diffusion probability model (Denoising Diffusion Production Model, DDPM). Now the more successful systems of image generation using proliferation models, such as DALL-E, Imagen in Google, Stable Diffusion, are basically using similar methods as their proliferation models.

This is followed by a presentation of how the proliferation model works. We came to see how it generated a picture. In the first step of creating the picture, we're going to sample a picture of noise, a vector from the Goss distribution. There are numbers in this vector, and the dimensions of this vector are exactly the size of the picture to be generated.

Then there's a noise module, the network. Enter a picture of the noise, and the output filters out a little bit of the noise, and it's possible to see a cat's shape, and then do it, and the cat's shape comes out. The more noise you're going to do, the more you're expecting to see a clear picture, the more noise you're going to get. Every noise usually has a number, and the closer to the finished image, the smaller.

And then there's this noise model. This is where the same noise model is used over and over again, but because each noise process is actually of different intensity, the bigger the noise, the stronger the number, the extra number of the noise module, the size of the current noise intensity.

There's a noise predictor in the noise model that predicts the noise in the picture. Tell us what the noise in this picture looks like. And when you subtract it from the noise, you get the result of the noise, so the noise model here is not the input of a noiseful picture. The output is a post-noise image, which actually produces the noise of a picture that you enter, and then pulls off the images that you enter to achieve the noise effect. We rarely choose to use end-to-end models here because of training difficulties.

The next question is how to train this noise predictor? A denocation model is based on the ID of a noise image and the number of denocations. We need a standard answer to noise, so we can train the noise predictor. That means we need pairs of training data.

How can the data of the Noise Predictor's training be created by man? We need to take a picture from the data set, randomly sample a noise set from the Goss distribution and add it to it, produce a little noiseful image, and then get a more noiseful picture again, and so on, the whole picture doesn't look like it.

The noise-added process is referred to as the forward process, or the diffusion process. After this proliferation process, there's training data for the noise predictor. For the Noise Predictor, the training data is this image that's finished with the noise that's now the first time it's done, it's the input of the network, and the noise that's added is the output that the network should predict, the standard answer to the Internet output. So after this proliferation process, there's training data.

It just came out of a noise and didn't consider words. If you want to train an image-generated model, he's "eating" text to produce a picture, but still needs data that match the picture to the text. With this data, we just need to encode the text and add the noise predictor.

In order to obtain sufficient data to be studied in proliferation models, ImageNet was slightly inadequate, after all, he lacked a word label. Current mainstream dissemination model data are generally derived from LAION, which has 58.50 billion pictures and has more comprehensive labels and searches for Demo, even in multiple languages.

For context, compare the related generative media essay and the transformer architecture notes.