{"id":1109,"date":"2020-09-07T14:16:00","date_gmt":"2020-09-07T14:16:00","guid":{"rendered":"https:\/\/data-science.gotoauthority.com\/2020\/09\/07\/lstm-back-propagation-following-the-flows-of-variables\/"},"modified":"2020-09-07T14:16:00","modified_gmt":"2020-09-07T14:16:00","slug":"lstm-back-propagation-following-the-flows-of-variables","status":"publish","type":"post","link":"https:\/\/wealthrevelation.com\/data-science\/2020\/09\/07\/lstm-back-propagation-following-the-flows-of-variables\/","title":{"rendered":"LSTM back propagation: following the flows of variables"},"content":{"rendered":"
\n

First of all, the summary of this article is: please just download my Power Point slides which I made<\/a> and be patient, following the equations.<\/em><\/p>\n

I am not supposed to use so many mathematics when I write articles on Data Science Blog. However using little mathematics when I talk about LSTM backprop is like writing German, never caring about \u201cder,\u201d \u201cdie,\u201d \u201cdas,\u201d or speaking little English in English classes (which most high school English teachers in Japan do) or writing Japanese without using any Chinese characters (which looks like a terrible handwriting by a drug addict). In short, that is ridiculous. And all the precise equations of LSTM backprop, written on a Blog is not a comfortable thing to see. So basically the whole of this article is an advertisement on my PowerPoint slides<\/a>, sponsored by DATANOMIQ<\/a>, and I can just give you some tips to get ready for the most tiresome part of understanding LSTM here.<\/p>\n

*This article is the fifth article of \u201cA gentle introduction to the tiresome part of understanding RNN<\/a>.\u201d<\/p>\n

\u00a0*In this article \u201cDensely Connected Layers\u201d is written as \u201cDCL,\u201d and \u201cConvolutional Neural Network\u201d as \u201cCNN.\u201d<\/span><\/p>\n

1. Chain rules<\/h3>\n

This article is virtually an article on chain rules of differentiation. Even if you have clear understandings on chain rules, I recommend you to take a look at this section. If you have written down all the equations of back propagation of DCL, you would have seen what chain rules are. Even simple chain rules for backprop of normal DCL can be difficult to some people, but when it comes to backprop of LSTM, it is a pure torture.\u00a0 I think using graphical models would help you understand what chain rules are like. Graphical models are basically used to describe the relations of variables and functions in probabilistic models, so to be exact I am going to use \u201csomething like graphical models\u201d in this article. Not that this is a common way to explain chain rules.<\/p>\n

First, let\u2019s think about the simplest type of chain rule. Assume that you have a function \"f=f(x)=f(x(y))\", and relations of the functions are displayed as the graphical model at the left side of the figure below. Variables are a type of function, so you should think that every node in graphical models denotes a function. Arrows in purple in the right side of the chart show how information propagate in differentiation.<\/p>\n

\"\"<\/p>\n

Next, if you a function \"f\" , which has two variances\u00a0 \"x_1\" and \"x_2\". And both of the variances also share two variances\u00a0 \"y_1\" and \"y_2\". When you take partial differentiation of \"f\" with respect to \"y_1\" or \"y_2\", the formula is a little tricky. Let\u2019s think about how to calculate \"frac{partial. The variance \"y_1\" propagates to \"f\" via \"x_1\" and \"x_2\". In this case the partial differentiation has two terms as below.<\/p>\n

\"\"<\/p>\n

In chain rules, you have to think about all the routes where a variance can propagate through. If you generalize chain rules, that is like below, and you need to understand chain rules in this way to understanding any types of back propagation.<\/p>\n

\"\"<\/p>\n

The figure above shows that if you calculate partial differentiation of \"f\" with respect to \"y_i\", the partial differentiation has \"n\" terms in total because \"y_i\" propagates to \"f\" via \"n\" variances. In order to understand backprop of LSTM, you constantly have to care about the flow of variances, which I showed as arrows in purple above.<\/p>\n

2. Chain rules in LSTM<\/h3>\n

I would like you to remember the figure below, which I used in the second article<\/a> to show how errors propagate backward during backprop of simple RNNs. After forward propagation, first of all, you need to calculate \"frac{partial, gradients of the error function with respect to parameters, at every time step. But you have to be careful that even though these gradients depend on time steps, the parameters \"boldsymbol{theta}\" do not depend on time steps.<\/p>\n

*As I mentioned in the second article<\/a> I personally think \"frac{partial should be rather denoted as \"(frac{partial because parameters themselves do not depend on time. The textbook by MIT press<\/a> also partly use the former notation. And you are likely to encounter this type of notation, so I think it is not bad to get ready for both.<\/p>\n

The errors at time step \"(t)\" propagate backward to all the \"boldsymbol{h}. Conversely, in order to calculate \"frac{partial errors flowing from \"J^{(s)},. In the chart you need arrows of errors in purple for the gradient in a purple frame, orange arrows for gradients in orange frame, red arrows for gradients in red frame. And you need to sum up \"frac{partial to calculate \"frac{partial, and you need this gradient \"frac{partial to renew parameters, one time.<\/p>\n

\"\"<\/p>\n

At an RNN block level, the flows of errors and how to renew parameters are the same in LSTM backprop, but the flow of errors inside each block is much more complicated in LSTM backprop. And in this article and my PowerPoint slides<\/a>, I use a special notation to denote errors: \"delta<\/p>\n

* Again, please be careful of what \"delta means. Neurons depend on time steps, but parameters do not depend on time steps. So if \"star\" are neurons,\u00a0 \"delta, but when \"star\" are parameters, \"delta should be rather denoted like \"delta. In the Space Odyssey paper<\/a>,\u00a0 \"boldsymbol{star}\" are not used as parameters, but in my PowerPoint slides<\/a> and some other materials, \"boldsymbol{star}\" are used also as parameteres.<\/p>\n

\"\"<\/p>\n

As I wrote in the last article, you calculate \"boldsymbol{f}^{(t)}, as below. Unlike the last article, I also added the terms of peephole connections in the equations below, and I also added the variances \"bar{boldsymbol{f}^{(t)}}, for convenience.<\/p>\n

With\u00a0 Hadamar product operator, the renewed cell and the output are calculated as below.<\/p>\n