{"id":553,"date":"2020-08-21T11:48:17","date_gmt":"2020-08-21T11:48:17","guid":{"rendered":"https:\/\/data-science.gotoauthority.com\/2020\/08\/21\/understanding-lstm-forward-propagation-in-two-ways\/"},"modified":"2020-08-21T11:48:17","modified_gmt":"2020-08-21T11:48:17","slug":"understanding-lstm-forward-propagation-in-two-ways","status":"publish","type":"post","link":"https:\/\/wealthrevelation.com\/data-science\/2020\/08\/21\/understanding-lstm-forward-propagation-in-two-ways\/","title":{"rendered":"Understanding LSTM forward propagation in two ways"},"content":{"rendered":"
\n

*This article is only for the sake of understanding the equations in the second page of the paper named \u201cLSTM: A Search Space Odyssey\u201d.<\/a> If you have no trouble understanding the equations of LSTM forward propagation, I recommend you to skip this article and go the the next article.<\/p>\n

1. Preface<\/strong><\/h3>\n

I\u00a0 heard that in Western culture, smart people write textbooks so that other normal people can understand difficult stuff, and that is why textbooks in Western countries tend to be bulky, but also they are not so difficult as they look. On the other hand in Asian culture, smart people write puzzling texts on esoteric topics, and normal people have to struggle to understand what noble people wanted to say. Publishers also require the authors to keep the texts as short as possible, so even though the textbooks are thin, usually students have to repeat reading the textbooks several times because usually they are too abstract.<\/p>\n

Both styles have cons and pros, and usually I prefer Japanese textbooks because they are concise, and sometimes it is annoying to read Western style long texts with concrete straightforward examples to reach one conclusion. But a problem is that when it comes to explaining LSTM, almost all the text books are like Asian style ones. Every study material seems to skip the proper steps necessary for \u201cnormal people\u201d to understand its algorithms. But after actually making concrete slides on mathematics on LSTM, I understood why: if you write down all the equations on LSTM forward\/back propagation, that is going to be massive, and actually I had to make 100-page PowerPoint animated slides to make it understandable to people like me.<\/p>\n

I already had a feeling that \u201cDoes it help to understand only LSTM with this precision? I should do more practical codings.\u201d For example Fran\u00e7ois Chollet, the developer of Keras, in his book, said as below.<\/span>\"\"<\/p>\n

\u00a0<\/p>\n

For me that sounds like \u201cWe have already implemented RNNs for you, so just shut up and use Tensorflow\/Keras.\u201d Indeed, I have never cared about the architecture of my Mac Book Air, but I just use it every day, so I think he is to the point. To make matters worse, for me, a promising algorithm called Transformer <\/strong><\/em>seems to be replacing the position of LSTM in natural language processing<\/strong><\/em>. But in this article series and in my PowerPoint slides, I tried to explain as much as possible, contrary to his advice.<\/p>\n

But I think, or rather hope,\u00a0 it is still meaningful to understand this 23-year-old algorithm, which is as old as me. I think LSTM did build a generation of algorithms for sequence data, and actually Sepp Hochreiter, the inventor of LSTM, has received Neural Network Pioneer Award 2021 for his work<\/a>.<\/p>\n

I hope those who study sequence data processing in the future would come to this article series, and study basics of RNN just as I also study classical machine learning algorithms.<\/p>\n

\u00a0*In this article \u201cDensely Connected Layers\u201d is written as \u201cDCL,\u201d and \u201cConvolutional Neural Network\u201d as \u201cCNN.\u201d<\/span><\/p>\n

2. Why LSTM?<\/strong><\/h3>\n

First of all, let\u2019s take a brief look at what I said about the structures of RNNs,\u00a0 in the first<\/a> and the second article<\/a>. A simple RNN is basically densely connected network with a few layers. But the RNN gets an input every time step, and it gives out an output at the time step. Part of information in the middle layer are succeeded to the next time step, and in the next time step, the RNN also gets an input and gives out an output. Therefore, virtually a simple RNN behaves almost the same way as densely connected layers with many layers during forward\/back propagation if you focus on its recurrent connections.<\/p>\n

\"\"<\/p>\n

That is why simple RNNs suffer from vanishing\/exploding gradient problems, where the information exponentially vanishes or explodes when its gradients are multiplied many times through many layers during back propagation. To be exact, I think you need to consider this problem precisely like you can see in this paper.<\/a> But for now, please at least keep it in mind that when you calculate a gradient of an error function with respect to parameters of simple neural networks, you have to multiply parameters many times like below, and this type of calculation usually leads to vanishing\/exploding gradient problem. \"\"<\/p>\n

LSTM was invented as a way to tackle such problems as I mentioned in the last article<\/a>.<\/p>\n

3. How to display LSTM<\/strong><\/h3>\n

I would like you to just go to image search on Google, Bing, or Yahoo!, and type in \u201cLSTM.\u201d I think you will find many figures, but basically LSTM charts are roughly classified into two types: in this article I call them \u201cSpace Odyssey type\u201d and \u201celectronic circuit type\u201d, and in conclusion, I highly recommend you to understand LSTM as the \u201celectronic circuit type.\u201d<\/p>\n

*I just randomly came up with the terms \u201cSpace Odyssey type\u201d and \u201celectronic circuit type\u201d because the former one is used in the paper I mentioned, and the latter one looks like an electronic circuit to me. You do not have to take how I call them seriously.<\/em><\/p>\n

However, not that all the well-made explanations on LSTM use the \u201celectronic circuit type,\u201d and I am sure you sometimes have to understand LSTM as the \u201cspace odyssey type.\u201d And the paper \u201cLSTM: A Search Space Odyssey<\/a>,\u201d which I learned a lot about LSTM from,\u00a0 also adopts the \u201cSpace Odyssey type.\u201d<\/p>\n

\"LSTM<\/a><\/p>\n

The main reason why I recommend the \u201celectronic circuit type\u201d is that its behaviors look closer to that of simple RNNs, which you would have seen if you read my former articles.<\/p>\n

*Behaviors of both of them look different, but of course they are doing the same things.<\/p>\n

If you have some understanding on DCL, I think it was not so hard to understand how simple RNNs work because simple RNNs\u00a0 are mainly composed of linear connections of neurons and weights, whose structures are the same almost everywhere. And basically they had only straightforward linear connections as you can see below.<\/p>\n

\"\"<\/a><\/p>\n

But from now on, I would like you to give up the ideas that LSTM is composed of connections of neurons like the head image of this article series. If you do that, I think that would be chaotic and I do not want to make a figure of it on Power Point. In short, sooner or later you have to understand equations of LSTM.<\/p>\n

4. Forward propagation of LSTM in \u201celectronic circuit type\u201d<\/strong><\/h3>\n

*For further understanding of mathematics of LSTM forward\/back propagation, I recommend you to download my slides<\/a>.<\/p>\n

The behaviors of an LSTM block is quite similar to that of a simple RNN block: an RNN block gets an input every time step and gets information from the RNN block of the last time step, via recurrent connections. And the block succeeds information to the next block.<\/p>\n

\"\"<\/a><\/p>\n

Let\u2019s look at the simplified architecture of\u00a0 an LSTM block. First of all, you should keep it in mind that LSTM have two streams of information: the one going through all the gates<\/strong><\/em>, and the one going through cell connections<\/strong><\/em>, the \u201chighway\u201d of LSTM block. For simplicity, we will see the architecture of an LSTM block without peephole connections<\/em><\/strong>, the lines in blue. The flow of information through cell connections is relatively uninterrupted. This helps LSTMs to retain information for a long time.<\/p>\n

\"\"<\/a><\/strong><\/p>\n

In a LSTM block, the input and the output of the former time step separately go through sections named \u201cgates\u201d: input gate, forget gate, output gate, and block input. The outputs of the forget gate, the input gate, and the block input join the highway of cell connections to renew the value of the cell.<\/p>\n

*The small two dots on the cell connections are the \u201con-ramp\u201d of cell conection highway. <\/em><\/p>\n

*You would see the terms \u201cinput gate,\u201d \u201cforget gate,\u201d \u201coutput gate\u201d almost everywhere, but how to call the \u201cblock gate\u201d depends on textbooks.<\/em><\/p>\n

Let\u2019s look at the structure of an LSTM block a bit more concretely. An LSTM block at the time step \"(t)\" gets \"boldsymbol{y}^{(t-1)}\", the output at the last time step,\u00a0 and \"boldsymbol{c}^{(t-1)}\", the information of the cell at the time step \"(t-1)\", via recurrent connections. The block at time step \"(t)\" gets the input \"boldsymbol{x}^{(t)}\", and it separately goes through each gate, together with \"boldsymbol{y}^{(t-1)}\". After some calculations and activation, each gate gives out an output. The outputs of the forget gate, the input gate, the block input, and the output gate are respectively \"boldsymbol{f}^{(t)},. The outputs of the gates are mixed with \"boldsymbol{c}^{(t-1)}\" and the LSTM block gives out an output \"boldsymbol{y}^{(t)}\", and gives \"boldsymbol{y}^{(t)}\" and \"boldsymbol{c}^{(t)}\" to the next LSTM block via recurrent connections.<\/p>\n

\"\"<\/a><\/p>\n

You calculate \"boldsymbol{f}^{(t)}, as below.<\/p>\n

*You have to keep it in mind that the equations above do not include peephole connections, which I am going to show with blue lines in the end.<\/p>\n

The equations above are quite straightforward if you understand forward propagation of simple neural networks. You add linear products of \"boldsymbol{y}^{(t)}\" and \"boldsymbol{c}^{(t)}\" with different weights in each gate. What makes LSTMs different from simple RNNs is how to mix the outputs of the gates with the cell connections. In order to explain that, I need to introduce a mathematical operator called Hadamard product<\/strong><\/em>, which you denote as \"odot\". This is a very simple operator. This operator produces an elementwise product of two vectors or matrices with identical shape.<\/p>\n

\"\"<\/a><\/p>\n

With this Hadamar product operator, the renewed cell and the output are calculated as below.<\/p>\n