{"id":8512,"date":"2022-05-20T13:52:02","date_gmt":"2022-05-20T13:52:02","guid":{"rendered":"https:\/\/wealthrevelation.com\/data-science\/2022\/05\/20\/generative-adversarial-networks\/"},"modified":"2022-05-20T13:52:02","modified_gmt":"2022-05-20T13:52:02","slug":"generative-adversarial-networks","status":"publish","type":"post","link":"https:\/\/wealthrevelation.com\/data-science\/2022\/05\/20\/generative-adversarial-networks\/","title":{"rendered":"Generative Adversarial Networks"},"content":{"rendered":"
\n

After\u00a0Deep Autoregressive Models<\/a>, Deep Generative Modelling<\/a> and Variational Autoencoders<\/a> we now continue the discussion with G<\/strong>enerative A<\/strong>dversarial N<\/strong>etworks (GANs).<\/p>\n

Introduction<\/strong><\/h3>\n

So far, in the series of deep generative modellings (DGMs [Yad22a]), we have covered autoregressive modelling [?], which estimates the exact log likelihood defined by the model and variational autoencoders [?], which was variational approximations for lower bound optimization. Both of these modelling techniques were explicitly defining density functions and optimizing the likelihood of the training data. However, in this blog, we are going to discuss generative adversarial networks (GANs), which are likelihood-free models and do not define density functions explicitly. GANs follow a game-theoretic approach and learn to generate from the training distribution through a set up of a two-player game.<\/p>\n

\"A<\/p>\n

A two player model of GAN along with the generator and discriminators.<\/p>\n<\/div>\n

GAN tries to learn the distribution of high dimensional training data and generates high-quality synthetic data which has a similar distribution to training data. However, learning the training distribution is a highly complex task therefore GAN utilizes a two-player game approach to overcome the high dimensional complexity problem. GAN has two different neural networks (as shown in Figure ??) the generator and the discriminator. The generator takes a random input \"zsim and produces a sample that has a similar distribution as \"p_d\". To train this network efficiently, there is the other network that is utilized as the second player and known as the discriminator. The generator network (player one) tries to fool the discriminator by generating real looking images. Moreover, the discriminator network tries to distinguish between real (training data \"xsim) and fake images effectively. Our main aim is to have an efficiently trained discriminator to be able to distinguish between real and fake images (the generator\u2019s output) and on the other hand, we would like to have a generator, which can easily fool the discriminator by generating real-looking images.<\/p>\n

Objective function and training<\/h2>\n

Objective function<\/h3>\n

Simultaneous training of these two networks is one of the main challenges in GANs and a minimax loss function is defined for this purpose. To understand this minimax function, firstly, we would like to discuss the concept of two sample testing by Aditya grover [Gro20]. Two sample testing is a method to compute the discrepancy between the training data distribution and the generated data distribution:<\/p>\n

(1) <\/span> \u00a0 <\/span>\"begin{equation*}<\/p>\n

\nwhere \"p_{theta_g}\" and \"p_d\" are the distribution functions of generated and training data respectively. The term \"F\" is a set of functions. The textit{max} part is computing the discrepancies between two distribution using a function \"D_{theta_d} and this part is very similar to the term \"d\" (discrepancy measure) from our first article cite{?} and KL-divergence is applied to compute this measure in second cite{?} and third articles cite{?}. However, in GANs, for a given set of functions \"F\", we would like compute the distribution \"p_{theta_g}\", which minimizes the overall discrepancy even for a worse function \"D_{theta_d}in. The above mentioned objective function does not use any likelihood function and utilizing two different data samples from training and generated data respectively.<\/p>\n

By combining Figure ?? and Equation 1<\/a>, the first term \"mathbb{E}_{xsim corresponds to the discriminator, which has direct access to the training data and the second term \"mathbb{E}_{xsim represents the generator part as it relies only on the latent space and produces synthetic data. Therefore, Equation 1<\/a> can be rewritten in the form of GAN\u2019s two players as:<\/p>\n

(2) <\/span> \u00a0 <\/span>\"begin{equation*}<\/p>\n

\nThe above equation can be rearranged in the form of log loss:
\n<\/a><\/p>\n

(3) <\/span> \u00a0 <\/span>\"begin{equation*}<\/p>\n

In the above equation, the arguments are modified from \"p_{theta_g}\" and \"D_{theta_d} to \"theta_g\" and\u00a0 \"theta_d\" respectively as we would like to approximate the network parameters, which are represented by \"theta_g\" and \"theta_d\" for the both generator and discriminator respectively. The discriminator wants to maximize the above objective for \"theta_d\" such that \"D_{theta_d}(x), which indicates that the outcome is close to the real data. Furthermore, \"D_{theta_d}(G_{theta_g}(z))\" should be close to zero as it is fake data, therefore, the maximization of the above objective function for \"theta_d\" will ensure that the discriminator is performing efficiently in terms of separating real and fake data. From the generator point of view, we would like to minimize this objective function for \"theta_g\" such that \"D_{theta_d}(G_{theta_g}(z)). If the minimization of the objective function happens effectively for \"theta_g\" then the discriminator will classify a fake data into a real data that means that the generator is producing almost real-looking samples.<\/p>\n

Training<\/h3>\n

The training procedure of GAN can be explained by using the following visualization from Goodfellow et al. [GPAM+14]. In Figure 2(a), \"z\" is a random input vector to the generator to produce a synthetic outcome \"xsim (green curve). The generated data distribution is not close to the original data distribution \"p_d\" (dotted black curve). Therefore, the discriminator classifies this image as a fake image and forces generator to learn the training data distribution (Figure 2(b) and (c)). Finally, the generator produces the image which could not detected as a fake data by discriminator(Figure 2(d)).<\/p>\n

\"GAN\u2019s<\/p>\n

GAN\u2019s training visualization: the dotted black, solid green lines represents pd and p\u03b8
respectively. The discriminator distribution is shown in dotted blue. This image taken from Goodfellow
et al. [GPAM+14].<\/p>\n<\/div>\n

The optimization of the objective function mentioned in Equation 3<\/a> is performed in th following two steps repeatedly:
begin{enumerate}
item Firstly, the gradient ascent is utilized to maximize the objective function for \"theta_d\" for discriminator.
<\/a><\/p>\n

(4) <\/span> \u00a0 <\/span>\"begin{equation*}<\/p>\n

\nitem In the second step, the following function is minimized for the generator using gradient descent.<\/p>\n

(5) <\/span> \u00a0 <\/span>\"begin{equation*}<\/p>\n

\nend{enumerate}<\/p>\n

However, in practice the minimization for the generator does now work well because when \"D_{theta_d}(G_{theta_g}(z) then the term \"log has the dominant gradient and vice versa.<\/p>\n

However, we would like to have the gradient behaviour completely opposite because \"D_{theta_d}(G_{theta_g}(z) means the generator is well trained and does not require dominant gradient values. However, in case of \"D_{theta_d}(G_{theta_g}(z), the generator is not well trained and producing low quality outputs therefore, it requires a dominant gradient for an efficient training. To fix this problem, the gradient ascent method is applied to maximize the modified generator\u2019s objective:
In the second step, the following function is minimized for the generator using gradient descent alternatively.
<\/a><\/p>\n

(6) <\/span> \u00a0 <\/span>\"begin{equation*}<\/p>\n

\ntherefore, during the training, Equation 4<\/a> and 6<\/a> will be maximized using the gradient ascent algorithm until the convergence.<\/p>\n

Results<\/h2>\n

The quality of the generated images using GANs depends on several factors. Firstly, the joint training of GANs is not a stable procedure and that could severely decrease the quality of the outcome. Furthermore, the different neural network architecture will modify the quality of images based on the sophistication of the used network. For example, the vanilla GAN [GPAM+14] uses a fully connected deep neural network and generates a quite decent result. Furthermore, DCGAN [RMC15] utilized deep convolutional networks and enhanced the quality of outcome significantly. Furthermore, different types of loss functions are applied to stabilize the training procedure of GAN and to produce high-quality outcomes. As shown in Figure 3, StyleGAN [KLA19] utilized Wasserstein metric [Yad22b] to generate high-resolution face images. As it can be seen from Figure 3, the quality of the generated images are enhancing with time by applying more sophisticated training techniques and network architectures.<\/p>\n

\"GAN<\/p>\n

GAN timeline with different variations in terms of network architecture and loss functions.<\/p>\n<\/div>\n

Summary<\/h2>\n

This article covered the basics and mathematical concepts of GANs. However, the training of two different networks simultaneously could be complex and unstable. Therefore, researchers are continuously working to create a better and more stable version of GANs, for example, WGAN. Furthermore, different types of network architectures are introduced to improve the quality of outcomes. We will discuss this further in the upcoming blog about these variations.<\/p>\n

References<\/h2>\n

[GPAM+14]<\/strong> Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, DavidWarde-Farley, Sherjil
Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in
neural information processing systems, 27, 2014.<\/p>\n

[Gro20]<\/strong> Aditya Grover. Generative adversarial networks.
https:\/\/deepgenerativemodels.github.io\/notes\/gan\/, 2020.<\/p>\n

[KLA19]<\/strong> Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for
generative adversarial networks. In Proceedings of the IEEE\/CVF conference on computer
vision and pattern recognition, pages 4401\u20134410, 2019.<\/p>\n

[RMC15]<\/strong> Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation
learning with deep convolutional generative adversarial networks. arXiv preprint
arXiv:1511.06434, 2015.<\/p>\n

[Yad22a]<\/strong> Sunil Yadav. Deep generative modelling. https:\/\/data-scienceblog.
com\/blog\/2022\/02\/19\/deep-generative-modelling\/, 2022.<\/p>\n

[Yad22b]<\/strong> Sunil Yadav. Necessary probability concepts for deep learning: Part 2.
https:\/\/medium.com\/@sunil7545\/kl-divergence-js-divergence-and-wasserstein-metricin-
deep-learning-995560752a53, 2022.<\/p>\n

\n

Sunil Yadav<\/a><\/h3>\n
<\/div>\n

<\/a><\/p>\n

Sunil Yadav is an experienced researcher with a keen focus on applying academic research to solve real-world problems. He believes a research paper has more value if it can be used for the welfare of society in general and the wellness of people in particular. He finished his PhD in mathematics and computer science and has a focus on computer vision, 3D data modelling, and medical imaging. His research interests revolve around understanding the visual data and producing meaningful output using the different areas of mathematics, including Deep learning, Machine learning, and computer vision.<\/p>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

https:\/\/data-science-blog.com\/en\/blog\/2022\/05\/20\/generative-adversarial-networks\/<\/p>\n","protected":false},"author":0,"featured_media":8513,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[2],"tags":[],"_links":{"self":[{"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/posts\/8512"}],"collection":[{"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/comments?post=8512"}],"version-history":[{"count":0,"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/posts\/8512\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/media\/8513"}],"wp:attachment":[{"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/media?parent=8512"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/categories?post=8512"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/tags?post=8512"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}