# The Recurrent Neural Network

## Learning objectives

1. Understand the principles behind the creation of the recurrent neural network
2. Obtain intuition about difficulties training RNNs, namely: vanishing/exploding gradients and long-term dependencies
3. Obtain intuition about mechanics of backpropagation through time BPTT
4. Develop a Long Short-Term memory implementation in Keras
5. Learn about the uses and limitations of RNNs from a cognitive science perspective

## Historical and theoretical background

The poet Delmore Schwartz once wrote: **"...time is the fire in which we burn"**. We can't escape time. Time is embedded in every human thought and action. Yet, so far, we have been oblivious to the role of time in neural network modeling. Indeed, in all models we have examined so far we have implicitly assumed that **data is "perceived" all at once**, although there are countless examples where time is a critical consideration: movement, speech production, planning, decision-making, etc. We also have implicitly assumed that **past-states have no influence in future-states**. This is, the input pattern at time-step $t-1$ does not influence the output of time-step $t-0$, or $t+1$, or any subsequent outcome for that matter. In probabilistic jargon, this equals to assume that each sample is drawn independently from each other. We know in many scenarios this is simply not true: when giving a talk, my next utterance will depend upon my past utterances; when running, my last stride will condition my next stride, and so on. You can imagine endless examples.

Multilayer Perceptrons and Convolutional Networks, in principle, can be used to approach problems where time and sequences are a consideration (for instance [Cui et al, 2016](https://arxiv.org/pdf/1603.06995.pdf)). Nevertheless, introducing time considerations in such architectures is cumbersome, and better architectures have been envisioned. In particular, **Recurrent Neural Networks (RNNs)** are the modern standard to deal with **time-dependent** and/or **sequence-dependent** problems. This type of network is "recurrent" in the sense that they can **revisit or reuse past states as inputs to predict the next or future states**. To put it plainly, they have **memory**. Indeed, memory is what allows us to incorporate our past thoughts and behaviors into our future thoughts and behaviors.

### Hopfield Network

One of the earliest examples of networks incorporating "recurrences" was the so-called **Hopfield Network**, introduced in 1982 by [John Hopfield](https://en.wikipedia.org/wiki/John_Hopfield), at the time, a physicist at Caltech. Hopfield networks were important as they helped to reignite the interest in neural networks in the early '80s. In his 1982 paper "*Neural networks and physical systems with emergent collective computational abilities*", Hopfield wanted to address the fundamental question of **emergence** in cognitive systems: Can relatively stable cognitive phenomena, like memories, emerge from the collective action of large numbers of simple neurons? After all, such behavior was observed in other physical systems like vortex patterns in fluid flow. Brains seemed like another promising candidate.

Hopfield networks are known as a type of **energy-based** (instead of error-based) network because their properties derive from a global energy-function (Raj, 2020). In resemblance to the McCulloch-Pitts neuron, Hopfield neurons are binary threshold units but with recurrent instead of feed-forward connections, where each unit is **bi-directionally connected** to each other, as shown in **Figure 1**. This means that each unit *receives* inputs and *sends* inputs to every other connected unit. A consequence of this architecture is that **weights values are symmetric**, such that weights *coming into* a unit are the same as the ones *coming out* of a unit. The value of each unit is determined by a linear function wrapped into a threshold function $T$, as $y_i = T(\sum w_{ji}y_j + b_i)$.

 Figure 1: Hopfield Network

Hopfield network's idea is that each configuration of binary-values $C$ in the network is associated with a **global energy value $-E$**. Here is a simplified picture of the training process: imagine you have a network with five neurons with a configuration of $C_1=(0, 1, 0, 1, 0)$. Now, imagine $C_1$ yields a global energy-value $E_1= 2$ (following the energy function formula). Your goal is to *minimize* $E$ by changing one element of the network $c_i$ at a time. By using the weight updating rule $\Delta w$, you can subsequently get a new configuration like $C_2=(1, 1, 0, 1, 0)$, as new weights will cause a change in the activation values $(0,1)$. If $C_2$ yields a *lower value of $E$*, let's say, $1.5$, you are moving in the right direction. If you keep iterating with new configurations the network will eventually "settle" into a **global energy minimum** (conditioned to the initial state of the network).

A fascinating aspect of Hopfield networks, besides the introduction of recurrence, is that is closely based in neuroscience research about learning and memory, particularly Hebbian learning (Hebb, 1949). In fact, Hopfield (1982) proposed this model as a way to capture **memory formation and retrieval**. The idea is that the energy-minima of the network could represent the **formation of a memory**, which further gives rise to a property known as **[content-addressable memory (CAM)](https://en.wikipedia.org/wiki/Content-addressable_memory)**. Here is the idea with a computer analogy: when you access information stored in the random access memory of your computer (RAM), you give the "address" where the "memory" is located to retrieve it. CAM works the other way around: you give information about the **content** you are searching for, and the computer should retrieve the "memory". This is great because this works even when you have **partial or corrupted** information about the content, which is a much more **realistic depiction of how human memory works**. It is similar to doing a google search. Just think in how many times you have searched for lyrics with partial information, like "song with the beeeee bop ba bodda bope!".

It is important to highlight that the sequential adjustment of Hopfield networks is **not driven by error correction**: there isn't a "target" as in supervised-based neural networks. Hopfield networks are systems that "evolve" until they find a stable low-energy state. If you "perturb" such a system, the system will "re-evolve" towards its previous stable-state, similar to how those inflatable "Bop Bags" toys get back to their initial position no matter how hard you punch them. It is almost like the system "remembers" its previous stable-state (isn't?). This ability to "return" to a previous stable-state after the perturbation is why they serve as models of memory.

### Elman Network

Although Hopfield networks where innovative and fascinating models, the first successful example of a recurrent network trained with backpropagation was introduced by [Jeffrey Elman](https://en.wikipedia.org/wiki/Jeffrey_Elman), the so-called **Elman Network** (Elman, 1990). Elman was a cognitive scientist at UC San Diego at the time, part of the group of researchers that published the famous PDP book.

In 1990, Elman published "Finding Structure in Time", a highly influential work for in cognitive science. Elman was concerned with the problem of representing "time" or "sequences" in neural networks. In his view, you could take either an "explicit" approach or an "implicit" approach. The **explicit** approach represents time **spacially**. Consider a vector $x = [x_1,x_2 \cdots, x_n]$, where element $x_1$ represents the first value of a sequence, $x_2$ the second element, and $x_n$ the last element. Hence, the spacial location in $\bf{x}$ is indicating the temporal location of each element. You can think about elements of $\bf{x}$ as sequences of words or actions, one after the other, for instance: $x^1=[Sound, of, the, funky, drummer]$ is a sequence of length five. Elman saw **several drawbacks** to this approach. First, although $\bf{x}$ is a sequence, the network still needs to represent the sequence all at once as an input, this is, a network would need five input neurons to process $x^1$. Second, it imposes a rigid limit on the duration of pattern, in other words, the network needs a fixed number of elements for every input vector $\bf{x}$: a network with five input units, can't accommodate a sequence of length six. True, you could start with a six input network, but then shorter sequences would be misrepresented since mismatched units would receive zero input. This is a problem for most domains where sequences have a variable duration. Finally, it can't easily distinguish **relative** temporal position from **absolute** temporal position. Consider the sequence $s = [1, 1]$ and a vector input length of four bits. Such a sequence can be presented in at least three variations:

$$
x_1 = [0, 1, 1, 0]\\
x_2 = [0, 0, 1, 1]\\
x_3 = [1, 1, 0, 0]
$$

Here, $\bf{x_1}$, $\bf{x_2}$, and $\bf{x_3}$ are instances of $\bf{s}$ but spacially displaced in the input vector. Geometrically, those three vectors are very different from each other (you can compute similarity measures to put a number on that), although representing the same instance. Even though you can train a neural net to learn those three patterns are associated with the same target, their inherent dissimilarity probably will hinder the network's ability to generalize the learned association.

The **implicit** approach represents time by **its effect in intermediate computations**. To do this, Elman added a **context unit** to save past computations and incorporate those in future computations. In short, **memory**. Elman based his approach in the work of [Michael I. Jordan](https://people.eecs.berkeley.edu/~jordan/) on serial processing (1986). Jordan's network implements recurrent connections from the network output $\hat{y}$ to its hidden units $h$, via a "memory unit" $\mu$ (equivalent to Elman's "context unit") as depicted in **Figure 2**. In short, the memory unit keeps a running average of **all past outputs**: this is how the past history is implicitly accounted for on each new computation. There is no learning in the memory unit, which means the weights are fixed to $1$.

 Figure 2: Jordan Network

**Note**: Jordan's network diagrams exemplifies the two ways in which recurrent nets are usually represented. On the left, the **compact format** depicts the network structure as a circuit. On the right, the **unfolded representation** incorporates the notion of time-steps calculations. The unfolded representation also illustrates how a recurrent network can be constructed in a pure feed-forward fashion, with as many layers as time-steps in your sequence. One key consideration is that the weights will be identical on each time-step (or layer). Keep this unfolded representation in mind as will become important later.

Elman's innovation was twofold: **recurrent connections between hidden units and memory** (context) units, and **trainable parameters from the memory units to the hidden units**. Memory units now have to "remember" the past state of hidden units, which means that instead of keeping a running average, they "clone" the value at the previous time-step $t-1$. Memory units also have to learn useful representations (weights) for encoding temporal properties of the sequential input.  **Figure 3** summarizes Elman's network in compact and unfolded fashion.

 Figure 3: Elman Network

**Note**: there is something curious about Elman's architecture. What it is the point of "cloning" $h$ into $c$ at each time-step? You could bypass $c$ altogether by sending the value of $h_t$ straight into $h_{t+1}$, wich yield mathematically identical results. The most likely explanation for this was that Elman's starting point was Jordan's network, which had a separated memory unit. Regardless, keep in mind we don't need $c$ units to design a functionally identical network.

Elman performed multiple experiments with this architecture demonstrating it was capable to solve multiple problems with a sequential structure: a temporal version of the XOR problem; learning the structure (i.e., vowels and consonants sequential order) in sequences of letters; discovering the notion of "word", and even learning complex lexical classes like word order in short sentences. Let's briefly explore the temporal XOR solution as an exemplar. **Table 1** shows the XOR problem:

**Table 1**: Truth Table For XOR Function

| $x_1$ | $x_2$ | $y$ |
|---|---|--------|
| 0 | 0 | 0      |
| 0 | 1 | 1      |
| 1 | 0 | 1      |
| 1 | 1 | 0      |

Here is a way to transform the XOR problem into a sequence. Consider the following vector:

$$
s= [1, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1, 0, 1,...]
$$

In $\bf{s}$, the first and second elements, $s_1$ and $s_2$, represent $x_1$ and $x_2$ inputs of **Table 1**, whereas the third element, $s_3$, represents the corresponding output $y$. This pattern repeats until the end of the sequence $s$ as shown in **Figure 4**.

 Figure 4: Temporal XOR

Elman trained his network with a 3,000 elements sequence for 600 iterations over the entire dataset, on the task of predicting the next item $s_{t+1}$ of the sequence $s$, meaning that he fed inputs to the network **one by one**. He showed that **error pattern** followed a predictable trend: the mean squared error was **lower every 3 outputs**, and higher in between, meaning the network learned to predict the third element in the sequence, as shown in **Chart 1** (the numbers are made up, but the pattern is the same found by Elman (1990)).

```python
import numpy as np
import pandas as pd
import altair as alt
```

```python
s = pd.DataFrame({"MSE": [0.35, 0.15, 0.30, 0.27, 0.14, 0.40, 0.35, 0.12, 0.36, 0.31, 0.15, 0.32],
                  "cycle": np.arange(1, 13)})
alt.Chart(s).mark_line().encode(x="cycle", y="MSE").properties(title='Chart 1')
```

An immediate advantage of this approach is the network can take **inputs of any length**, without having to alter the network architecture at all.

In the same paper, Elman showed that the **internal (hidden) representations** learned by the network grouped into meaningful categories, this is, **semantically similar words group together** when analyzed with [hierarchical clustering](https://en.wikipedia.org/wiki/Hierarchical_clustering). This was remarkable as demonstrated the utility of RNNs as a model of cognition in sequence-based problems.

### Interlude: vanishing and exploding gradients in RNNs

Turns out, training recurrent neural networks is hard. Considerably harder than multilayer-perceptrons. When faced with the task of training **very deep networks**, like RNNs, the gradients have the impolite tendency of either (1) **vanishing**, or (2) **exploding** (Bengio et al, 1994; Pascanu et al, 2012). Recall that RNNs can be unfolded so that recurrent connections follow pure feed-forward computations. This unrolled RNN will have as many layers as elements in the sequence. Thus, a sequence of 50 words will be unrolled as an RNN of 50 layers (taking "word" as a unit).

Concretely, the **vanishing gradient problem** will make close to impossible to learn **long-term dependencies** in sequences. Let's say you have a collection of poems, where the last sentence refers to the first one. Such a dependency will be hard to learn for a deep RNN where gradients vanish as we move backward in the network. The **exploding gradient problem** will completely derail the learning process. In very deep networks this is often a problem because more layers amplify the effect of large gradients, compounding into very large updates to the network weights, to the point values completely blow up.

Here is the intuition for the **mechanics of gradient vanishing**: when gradients *begin small*, as you move backward through the network computing gradients, they will get even smaller as you get closer to the input layer. Consequently, when doing the weight update based on such gradients, the weights closer to the output layer will obtain larger updates than weights closer to the input layer. This means that the weights closer to the input layer will hardly change at all, whereas the weights closer to the output layer will change a lot. This is a serious problem when **earlier layers matter for prediction**: they will keep propagating more or less the same signal forward because no learning (i.e., weight updates) will happen, which may significantly hinder the network performance.

Here is the intuition for the **mechanics of gradient explosion**: when gradients *begin large*, as you move backward through the network computing gradients, they will get even larger as you get closer to the input layer. Consequently, when doing the weight update based on such gradients, the weights closer to the input layer will obtain larger updates than weights closer to the output layer. Learning can go wrong really fast. Recall that the signal propagated by each layer is the outcome of taking the product between the previous hidden-state and the current hidden-state. If the weights in earlier layers get really large, they will forward-propagate larger and larger signals on each iteration, and the predicted output values will spiral-up out of control, making the error $y-\hat{y}$ so large that the network will be unable to learn at all. In fact, your computer will "overflow" quickly as it would unable to represent numbers that big. Very dramatic.

The mathematics of gradient vanishing and explosion gets complicated quickly. If you want to delve into the mathematics see [Bengio et all (1994)](http://ai.dinfo.unifi.it/paolo/ps/tnn-94-gradient.pdf), [Pascanu et all (2012)](https://arxiv.org/abs/1211.5063), and [Philipp et all (2017)](https://arxiv.org/abs/1712.05577).

For our purposes, I'll give you a simplified numerical example for intuition. Consider the task of predicting a vector $y = \begin{bmatrix} 1 & 1 \end{bmatrix}$, from inputs $x = \begin{bmatrix} 1 & 1 \end{bmatrix}$, with a multilayer-perceptron with 5 hidden layers and tanh activation functions. We have two cases:

- the weight matrix $W_l$ is initialized to large values $w_{ij} = 2$
- the weight matrix $W_s$ is initialized to small values $w_{ij} = 0.02$

Now, let's compute a single forward-propagation pass:

```python
import numpy as np
```

```python
x = np.array([[1],[1]])
W_l = np.array([[2, 2],
                [2, 2]])

h1 = np.tanh(W_l @ x)
h2 = np.tanh(W_l @ h1)
h3 = np.tanh(W_l @ h2)
h4 = np.tanh(W_l @ h3)
h5 = np.tanh(W_l @ h4)
y_hat = (W_l @ h5)
print(f'output for large initial weights: \n {y_hat}')
```

    output for large initial weights:
     [[3.99730269]
     [3.99730269]]

```python
x = np.array([[1],[1]])
W_s = np.array([[0.02, 0.02],
                [0.02, 0.02]])

h1 = np.tanh(W_s @ x)
h2 = np.tanh(W_s @ h1)
h3 = np.tanh(W_s @ h2)
h4 = np.tanh(W_s @ h3)
h5 = np.tanh(W_s @ h4)
y_hat = (W_s @ h5)
print(f'output for small initial weights: \n {y_hat}')
```

    output for small initial weights:
     [[4.09381337e-09]
     [4.09381337e-09]]

We see that for $W_l$ the output $\hat{y}\approx4$, whereas for $W_s$ the output $\hat{y} \approx 0$. Why does this matter? We haven't done the gradient computation but you can probably anticipate what it's going to happen: for the $W_l$ case, the gradient update is going to be very large, and for the $W_s$ very small. If you keep cycling through forward and backward passes these problems will become worse, leading to gradient explosion and vanishing respectively.

### Long Short-Term Memory Network

Several challenges difficulted progress in RNN in the early '90s (Hochreiter & Schmidhuber, 1997; Pascanu et al, 2012). In addition to vanishing and exploding gradients, we have the fact that the **forward computation is slow**, as RNNs can't compute in parallel: to preserve the time-dependencies through the layers, each layer has to be computed sequentially, which naturally takes more time. Elman networks proved to be effective at solving relatively simple problems, but as the sequences scaled in size and complexity, this type of network struggle.

Several approaches were proposed in the '90s to address the aforementioned issues like time-delay neural networks (Lang et al, 1990), simulated annealing (Bengio et al., 1994), and others. The architecture that really moved the field forward was the so-called **Long Short-Term Memory (LSTM) Network**, introduced by [Sepp Hochreiter](https://en.wikipedia.org/wiki/Sepp_Hochreiter) and [Jurgen Schmidhuber](https://en.wikipedia.org/wiki/J%C3%BCrgen_Schmidhuber) in 1997. As the name suggests, the defining characteristic of LSTMs is the addition of units combining both short-memory and long-memory capabilities.

In LSTMs, instead of having a simple memory unit "cloning" values from the hidden unit as in Elman networks, we have a (1) **cell unit** (a.k.a., memory unit) which effectively acts as long-term memory storage, and (2) a **hidden-state** which acts as a memory controller. These two elements are integrated as a circuit of logic gates controlling the flow of information at each time-step. Understanding the notation is crucial here, which is depicted in **Figure 5**.

 Figure 5: LSTM architecture

In LSTMs $x_t$, $h_t$, and $c_t$ represent vectors of values. Lightish-pink circles represent element-wise operations, and darkish-pink boxes are fully-connected layers with trainable weights. The top part of the diagram acts as a **memory storage**, whereas the bottom part has a double role: (1) passing the hidden-state information from the previous time-step $t-1$ to the next time step $t$, and (2) to regulate the **influx** of information from $x_t$ and $h_{t-1}$ **into** the memory storage, and the **outflux** of information **from** the memory storage into the next hidden state $h-t$. The second role is the core idea behind LSTM. You can think about it as making **three decisions** at each time-step:

1. **Is the *old information* $c_{t-1}$ worth to keep in my memory storage $c_t$?** If so, let the information pass, otherwise, "forget" such information. This is controlled by the *forget gate*.
2. **Is this *new information* (inputs) worth to be saved into my memory storage $c_t$?** If so, let information flow into $c_t$. This is controlled by the *input gate* and the *candidate memory cell*.
3. **What elements of the information saved in my memory storage $c_t$ are relevant for the computation of the next hidden-state $h_t$?** Select them from $c_t$, combine them new hidden-state output, and let them pass into the next hidden-state $h_t$. This is controlled by the *output gate* and the *tanh* function.

Decisions 1 and 2 will determine the information that keeps flowing through the memory storage at the top. Decision 3 will determine the information that flows to the next hidden-state at the bottom. The conjunction of these decisions sometimes is called "memory block". Now, keep in mind that this *sequence of decision* is just a convenient *interpretation* of LSTM mechanics. In practice, the weights are the ones determining what each function ends up doing, which may or may not fit well with human intuitions or design objectives.

 Figure 6: LSTM as a sequence of decisions

To put LSTMs in context, imagine the following simplified scenerio: we are trying to **predict the next word in a sequence**. Let's say, squences are about sports. From past sequences, we saved in the memory block the type of sport: "soccer". For the current sequence, we receive a phrase like "A basketball player...". In such a case, we first want to "forget" the previous type of sport "soccer" (*decision 1*) by multplying $c_{t-1} \odot f_t$. Next, we want to "update" memory with the new type of sport, "basketball" (*decision 2*), by adding $c_t = (c_{t-1} \odot f_t) + (i_t \odot \tilde{c_t})$. Finally, we want to output (*decision 3*) a verb relevant for "A basketball player...", like "shoot" or "dunk" by $\hat{y_t} = softmax(W_{hz}h_t + b_z)$.

LSTMs long-term memory capabilities make them good at capturing long-term dependencies. The memory cell effectively counteracts the vanishing gradient problem at preserving information as long the forget gate does not "erase" past information (Graves, 2012). All the above make LSTMs very useful for a wide variety of applications (see [here for a list](https://en.wikipedia.org/wiki/Long_short-term_memory#Applications)). For instance, when you use [Google's Voice Transcription](https://ai.googleblog.com/2015/08/the-neural-networks-behind-google-voice.html) services an RNN is doing the hard work of recognizing your voice.

### RNNs and cognition

As with Convolutional Neural Networks, researchers utilizing RNN for approaching sequential problems like natural language processing (NLP) or time-series prediction, do not *necessarily* care about (although some might) how good of a model of cognition and brain-activity are RNNs. What they really care is about solving problems like translation, speech recognition, and stock market prediction, and many advances in the field come from pursuing such goals. Still, RNN has many **desirable traits as a model of neuro-cognitive activity**, and have been prolifically used to **model several aspects of human cognition and behavior**: child behavior in an object permanence tasks (Munakata et al, 1997); knowledge-intensive text-comprehension (St. John, 1992); processing in quasi-regular domains, like English word reading (Plaut et al., 1996); human performance in processing recursive language structures (Christiansen & Chater, 1999); human sequential action (Botvinick & Plaut, 2004); movement patterns in typical and atypical developing children (Muñoz-Organero et al., 2019). And many others. Neuroscientists have used RNNs to model a wide variety of aspects as well (for reviews see Barak, 2017, Güçlü & van Gerven, 2017, Jarne & Laje, 2019). Overall, RNN has demonstrated to be a productive tool for modeling cognitive and brain function, in distributed representations paradigm.

## Mathematical formalization

There are two mathematically complex issues with RNNs: (1) computing hidden-states, and (2) backpropagation. The rest are common operations found in multilayer-perceptrons. LSTMs and its many variants are the facto standards when modeling any kind of sequential problem. Elman networks can be seen as a simplified version of an LSTM, so I'll focus my attention on LSTMs for the most part. My exposition is based on a combination of sources that you may want to review for extended explanations (Bengio et al., 1994; Hochreiter & Schmidhuber, 1997; Graves, 2012; Chen, 2016; Zhang et al., 2020).

The LSTM architecture can be desribed by:

**Forward pass**:
- non-linear forget function
- non-linear input function
- non-linear candidate-memory function
- non-linear output function
- memory cell function
- non-linear hidden-state function
- softmax function (output)

**Backward pass**:
- Cost-function
- Learning procedure (backpropagation)

Following the indices for each function requires some definitions. I'll assume we have $h$ hidden units, training sequences of size $n$, and $d$ input units.

$$
\begin{align}
\text{input-units} &= x_i \in \mathbb{R}^d \\
\text{training-sequence} &= s_i \in \mathbb{R}^n \\
\text{output-class} &= y_i \in \mathbb{R}^k \\
\text{Input-layer} &= X_t \in \mathbb{R}^{n\times d} \\
\text{hidden-layer} &= H_t \in \mathbb{R}^{n\times h}
\end{align}
$$

### Forget function

The **forget function** is a sigmoidal mapping combining three elements: input vector $x_t$, past hidden-state $h_{t-1}$, and a bias term $b_f$. We didn't mentioned the bias before, but it is the same bias that all neural networks incorporate, one for each unit in $f$. More formally:

$$
f_t = \sigma(W_{xf}x_t + W_{hf}h_{t-1} + b_f)
$$

Each matrix $W$ has dimensionality equal to (number of incoming units, number for connected units). For example, $W_{xf}$ refers to $W_{input-units, forget-units}$. Keep this in mind to read the indices of the $W$ matrices for subsequent definitions.

Here is an important insight: What would it happen if $f_t = 0$? If you look at the diagram in **Figure 6**, $f_t$ performs an elementwise multiplication of each element in $c_{t-1}$, meaning that every value would be reduced to $0$. In short, the network would completely "forget" past states. Naturally, if $f_t = 1$, the network would keep its memory intact.

### Input function and Candidate memory function

The **input function** is a sigmoidal mapping combining three elements: input vector $x_t$, past hidden-state $h_{t-1}$, and a bias term $b_f$. It's defined as:

$$
i_t = \sigma(W_{xi}x_t + W_{hi}h_{t-1} + b_i)
$$

The **candidate memory function** is an hyperbolic tanget function combining the same elements that $i_t$. It's defined as:

$$
\tilde{c}_t = tanh(W_{xc}x_t + W_{hc}h_{t-1} + b_c)
$$

Both functions are combined to update the memory cell.

### Output function

The **output function** is a sigmoidal mapping combining three elements: input vector $x_t$, past hidden-state $h_{t-1}$, and a bias term $b_f$. Is defined as:

$$
o_t = \sigma(W_{xo}x_t + W_{ho}h_{t-1} + b_o)
$$

### Memory cell function

The **memory cell function** (what I've been calling "memory storage" for conceptual clarity), combines the effect of the forget function, input function, and candidate memory function. It's defined as:

$$
c_t = (c_{t-1} \odot f_t) + (i_t \odot \tilde{c_t})
$$

Where $\odot$ implies an elementwise multiplication (instead of the usual dot product). This expands to:

$$
c_t = (c_{t-1} \odot \sigma(W_{xf}x_t + W_{hf}h_{t-1} + b_f)) + (\sigma(W_{xi}x_t + W_{hi}h_{t-1} + b_i) \odot tanh(W_{xc}x_t + W_{hc}h_{t-1} + b_c))
$$

### Hidden-state function

The next **hidden-state function** combines the effect of the output function and the contents of the memory cell scaled by a tanh function. It is defined as:

$$
h_t = O_t \odot tanh(c_t)
$$

### Output function

The output function will depend upon the problem to be approached. For our our purposes, we will assume a multi-class problem, for which the **softmax function** is appropiated. For this, we first pass the hidden-state by a linear function, and then the softmax as:

$$
\begin{align}
z_t &= (W_{hz}h_t + b_z)\\
\hat{y}_t &= softmax(z_t) = \frac{e^{z_t}}{\sum_{j=1}^K e^{z_j}}
\end{align}
$$

The softmax computes the exponent for each $z_t$ and then normalized by dividing by the sum of every output value exponentiated. In this manner, the output of the softmax can be interpreted as the likelihood value $p$.

### Cost function

As with the output function, the cost function will depend upon the problem. For regression problems, the Mean-Squared Error can be used. For our purposes (classification), the cross-entropy function is appropriated. It's defined as:

$$
E_i = - \sum_t y_ilog(p_i)
$$

Where $y_i$ is the true label for the $ith$ output unit, and $log(p_i)$ is the log of the softmax value for the $ith$ output unit. The summation indicates we need to aggregate the cost at each time-step.

### Learning procedure: Backpropagation Through Time (BPTT)

Originally, Hochreiter and Schmidhuber (1997) trained LSTMs with a combination of approximate gradient descent computed with a combination of real-time recurrent learning and backpropagation through time (BPTT). Nevertheless, LSTM can be trained with pure backpropagation. Following Graves (2012), I'll only describe BTT because is more accurate,  easier to debug and to describe.

**Note**: we call it **backpropagation through time** because of the sequential time-dependent structure of RNNs. Recall that each layer represents a time-step, and forward propagation happens in sequence, one layer computed after the other. Hence, when we backpropagate, we do the same but backward (i.e., "through time").

 Figure 7: Three-layer simplified RNN

I reviewed backpropagation for a simple multilayer perceptron [here](https://com-cog-book.github.io/com-cog-book/features/multilayer-perceptron.html#Backpropagation-algorithm). Nevertheless, I'll sketch BPTT for the simplest case as shown in **Figure 7**, this is, with a generic non-linear hidden-layer similar to Elman network without "context units" (some like to call it "vanilla" RNN, which I avoid because I believe is derogatory against vanilla!). This exercise will allow us to review backpropagation and to understand how it differs from BPTT. We begin by defining a simplified RNN as:

$$
\begin{align}
z_t &= W_{hz}h_t + b_z\\
h_t &= \sigma(W_{hh}h_{t-1} + W_{xh}x_t+b_h)
\end{align}
$$

Where $h_t$ and $z_t$ indicates a hidden-state (or layer) and  the output respectively. Therefore, **we have to compute gradients w.r.t. five sets of weights**: $\{W_{hz}, W_{hh}, W_{xh}, b_z, b_h\}$.

First, consider the error derivatives w.r.t. $W_{hz}$ at time $t$, the weight matrix for the linear function at the output layer. Recall that $W_{hz}$ is shared across all time-steps, hence, we can compute the gradients at each time step and then take the sum as:

$$
\frac{\partial{E}}{\partial{W_{hz}}} = \sum_t\frac{\partial{E}}{\partial{z_t}} \frac{\partial{z_t}}{\partial{W_{hz}}}
$$

Same for the bias term:

$$
\frac{\partial{E}}{\partial{b_z}} = \sum_t\frac{\partial{E}}{\partial{z_t}} \frac{\partial{z_t}}{\partial{b_z}}
$$

That part is straightforward. The issue arises when we try to compute the gradients w.r.t. the wights $W_{hh}$ in the hidden layer. Consider a three layer RNN (i.e., unfolded over three time-steps). In such a case, we have:

- $E_3$ depens on $z_3$
- $z_3$ depends on $h_3$
- $h_3$ depens on $h_2$
- $h_2$ depens on $h_1$
- $h_1$ depens on $h_0$, where $h_0$ is a random starting state.

Now, we have that $E_3$ w.r.t to $h_3$ becomes:

$$
\frac{\partial{E_3}}{\partial{W_{hh}}} =
\frac{\partial{E_3}}{\partial{z_3}}
\frac{\partial{z_3}}{\partial{h_3}}
\frac{\partial{h_3}}{\partial{W_{hh}}}
$$

The issue here is that $h_3$ depends on $h_2$, since according to our definition, the $W_{hh}$ is multiplied by $h_{t-1}$, meaning **we can't compute $\frac{\partial{h_3}}{\partial{W_{hh}}}$ directly**. Othewise, we would be treating $h_2$ as a constant, which is incorrect: is a function. What we need to do is to **compute the gradients separately**: the direct contribution of ${W_{hh}}$ on $E$ and the indirect contribution via $h_2$. Following the rules of calculus in multiple variables, we compute them independently and add them up together as:

$$
\frac{\partial{E_3}}{\partial{W_{hh}}} =
\frac{\partial{E_3}}{\partial{z_3}}
\frac{\partial{z_3}}{\partial{h_3}}
\frac{\partial{h_3}}{\partial{W_{hh}}}+
\frac{\partial{E_3}}{\partial{z_3}}
\frac{\partial{z_3}}{\partial{h_3}}
\frac{\partial{h_3}}{\partial{h_2}}
\frac{\partial{h_2}}{\partial{W_{hh}}}
$$

Again, we have that we can't compute $\frac{\partial{h_2}}{\partial{W_{hh}}}$ directly. Following the same procedure, we have that our full expression becomes:

$$
\frac{\partial{E_3}}{\partial{W_{hh}}} =
\frac{\partial{E_3}}{\partial{z_3}}
\frac{\partial{z_3}}{\partial{h_3}}
\frac{\partial{h_3}}{\partial{W_{hh}}}+
\frac{\partial{E_3}}{\partial{z_3}}
\frac{\partial{z_3}}{\partial{h_3}}
\frac{\partial{h_3}}{\partial{h_2}}
\frac{\partial{h_2}}{\partial{W_{hh}}}+
\frac{\partial{E_3}}{\partial{z_3}}
\frac{\partial{z_3}}{\partial{h_3}}
\frac{\partial{h_3}}{\partial{h_2}}
\frac{\partial{h_2}}{\partial{h_1}}
\frac{\partial{h_1}}{\partial{W_{hh}}}
$$

Essentially, this means that we compute and add the contribution of $W_{hh}$ to $E$ at each time-step. The expression for $b_h$ is the same:

$$
\frac{\partial{E_3}}{\partial{b_h}} =
\frac{\partial{E_3}}{\partial{z_3}}
\frac{\partial{z_3}}{\partial{h_3}}
\frac{\partial{h_3}}{\partial{b_h}}+
\frac{\partial{E_3}}{\partial{z_3}}
\frac{\partial{z_3}}{\partial{h_3}}
\frac{\partial{h_3}}{\partial{h_2}}
\frac{\partial{h_2}}{\partial{b_h}}+
\frac{\partial{E_3}}{\partial{z_3}}
\frac{\partial{z_3}}{\partial{h_3}}
\frac{\partial{h_3}}{\partial{h_2}}
\frac{\partial{h_2}}{\partial{h_1}}
\frac{\partial{h_1}}{\partial{b_h}}
$$

Finally, we need to compute the gradients w.r.t. $W_{xh}$. Here, again, we have to add the contributions of $W_{xh}$ via $h_3$, $h_2$, and $h_1$:

$$
\frac{\partial{E_3}}{\partial{W_{xh}}} =
\frac{\partial{E_3}}{\partial{z_3}}
\frac{\partial{z_3}}{\partial{h_3}}
\frac{\partial{h_3}}{\partial{W_{xh}}}+
\frac{\partial{E_3}}{\partial{z_3}}
\frac{\partial{z_3}}{\partial{h_3}}
\frac{\partial{h_3}}{\partial{h_2}}
\frac{\partial{h_2}}{\partial{W_{xh}}}+
\frac{\partial{E_3}}{\partial{z_3}}
\frac{\partial{z_3}}{\partial{h_3}}
\frac{\partial{h_3}}{\partial{h_2}}
\frac{\partial{h_2}}{\partial{h_1}}
\frac{\partial{h_1}}{\partial{W_{xh}}}
$$

That's for BPTT for a simple RNN. The math reviewed here generalizes with minimal changes to more complex architectures as LSTMs. Actually, the only difference regarding LSTMs, is that we have more weights to differentiate for. Instead of a single generic $W_{hh}$, we have $W$ for all the gates: forget, input, output, and candidate cell. The rest remains the same. For a detailed derivation of BPTT for the LSTM see [Graves (2012)](https://www.cs.toronto.edu/~graves/preprint.pdf) and [Chen (2016)](https://arxiv.org/abs/1610.02583).

## Interlude: Sequence-data representation

Working with sequence-data, like text or time-series, requires to pre-process it in a manner that is "digestible" for RNNs. As with any neural network, RNN can't take raw text as an input, we need to **parse** text sequences and then **"map"** them into vectors of numbers. Here I'll briefly review these issues to provide enough context for our example applications. For an extended revision please refer to [Jurafsky and Martin (2019)](https://web.stanford.edu/~jurafsky/slp3/), [Goldberg (2015)](http://u.cs.biu.ac.il/~yogo/nnlp.pdf), [Chollet (2017)](https://www.manning.com/books/deep-learning-with-python), and [Zhang et al (2020)](https://d2l.ai/chapter_recurrent-neural-networks/index.html).

Parsing can be done in multiple manners, the most common being:

- Using **word** as a unit, which each word represented as a vector
- Using **character** as a unit, with each character represented as a vector
- Using **n-grams** of words or characters as a unit, with each n-gram represented as a vector. N-grams are sets of words or characters of size "N" or less.

The process of parsing text into smaller units is called "tokenization", and each resulting unit is called a "token", the top pane in **Figure 8** displays a sketch of the tokenization process.

 Figure 8: Tokenization

Once a corpus of text has been parsed into tokens, we have to map such tokens into numerical vectors. Two common ways to do this are **one-hot encoding** approach and the **word embeddings** approach, as depicted in the bottom pane of **Figure 8**. We used one-hot encodings to transform the MNIST class-labels into vectors of numbers for classification in the [CovNets chapter](https://com-cog-book.github.io/com-cog-book/features/cov-net.html). In a one-hot encoding vector, each token is mapped into a *unique* vector of zeros and ones. The vector size is determined by the vocabullary size. For instance, for the set $x= \{"cat", "dog", "ferret"\}$, we could use a 3-dimensional one-hot encoding as:

$$
\text{cat}=
\begin{bmatrix}
1 \\
0 \\
0
\end{bmatrix},
\text{dog}=
\begin{bmatrix}
0 \\
1 \\
0
\end{bmatrix},
\text{ferret}=
\begin{bmatrix}
0 \\
0 \\
1
\end{bmatrix}
$$

One-hot encodings have the advantages of being straightforward to implement and to provide a unique identifier for each token. Its main disadvantage is that tends to create really **sparse** and **high-dimensional** representations for a large corpus of texts. For instance, if you tried a one-hot encoding for 50,000 tokens, you'd end up with a 50,000x50,000-dimensional matrix, which may be unpractical for most tasks.

**Word embeddings** represent text by mapping tokens into vectors of real-valued numbers instead of only zeros and ones. This significantly increments the representational capacity of vectors, reducing the required dimensionality for a given corpus of text compared to one-hot encodings. For instance, 50,000 tokens could be represented by as little as 2 or 3 vectors (although the representation may not be very good). Taking the same set $x$ as before, we could have a 2-dimensional word embedding like:

$$
\text{cat}=
\begin{bmatrix}
0.1 \\
0.8
\end{bmatrix},
\text{dog}=
\begin{bmatrix}
0.2 \\
1
\end{bmatrix},
\text{ferret}=
\begin{bmatrix}
0.6 \\
0.2
\end{bmatrix}
$$

You may be wondering why to bother with one-hot encodings when word embeddings are much more space-efficient. The main issue with word-embedding is that **there isn't an obvious way to map tokens into vectors** as with one-hot encodings. For instance, you could assign tokens to vectors at random (assuming every token is assigned to a *unique* vector). The problem with such approach is that the semantic structure in the corpus is broken. Ideally, you want words of similar meaning mapped into similar vectors. We can preserve the semantic structure of a text corpus in the same manner as everything else in machine learning: by **learning from data**. There are two ways to do this:

- Learning the word embeddings **at the same time** you train the RNN.
- Utilizing **pretrained** word embeddings, this is, embeddings learned in a different task. This is a form of "transfer learning".

Learning word embeddings for your task is advisable as semantic relationships among words tend to be **context dependent**. For instance, "exploitation" in the context of mining is related to resource "extraction", hence relative neutral. But, "exploitation" in the context of labor rights is related to the idea of "abuse", hence a negative connotation. This is more critical when we are dealing with different languages. Nevertheless, learning embeddings for every task sometimes is impractical, either because your corpus is too "small" (i.e., not enough data to extract semantic relationships), or too "large" (i.e., you don't have enough time and/or resources to learn the embeddings). Examples of freely accessible pretrained word embeddings are Google's [Word2vec](https://code.google.com/archive/p/word2vec/) and the [Global Vectors for Word Representation](https://nlp.stanford.edu/projects/glove/) (GloVe).

## Code implementation

As in previous chapters, I'll use Keras to implement both (a modified version of) the Elman Network for the XOR problem and an LSTM for review prediction based on text-sequences.

### Elman Network

By now, it may be clear to you that Elman networks are a simple RNN with two neurons, one for each input pattern, in the hidden-state. Originally, Elman trained his architecture with a truncated version of BPTT, meaning that only considered two time-steps for computing the gradients, $t$ and $t-1$. We will implement a modified version of Elman's architecture bypassing the "context" unit (which does not alter the result at all) and utilizing BPTT instead of its truncated version.

### Generating data

Nowadays, we don't need to generate the 3,000 bits sequence that Elman used in his original work. We can simply generate a single pair of training and testing sets for the XOR problem as in **Table 1**, and pass the training sequence (length two) as the inputs, and the expected outputs as the target. This is very much alike any classification task. An important caveat is that simpleRNN layers in Keras expect an input tensor of shape (number-samples, timesteps, number-input-features). In our case, this has to be: *number-samples*= 4,  *timesteps*=1, *number-input-features*=2. No separate encoding is necessary here because we are manually setting the input and output values to binary vector representations. Finally, we won't worry about training and testing sets for this example, which is way to simple for that (we will do that for the next example).

```python
# Libraries for this section

from keras.layers import Dense, SimpleRNN
from keras.models import Sequential
import altair as alt
import pandas as pd
import numpy as np
```

    Using TensorFlow backend.

```python
# features
X = np.array([[[0, 0, 1, 1]],
              [[0, 1, 0, 1]]]).T

# expected values
y = np.array([[0, 1, 1, 0]]).T

print(f'training data shape: {X.shape}')
print(f'targets data shape: {y.shape}')
```

    training data shape: (4, 1, 2)
    targets data shape: (4, 1)

### Elman network architecture in Keras

Defining a (modified) in Keras is extremely simple as shown below.

```python
# Define a network as a linear stack of layers
model = Sequential()

# Add a recurrent layer with 2 units
model.add(SimpleRNN(2, input_shape=(1, 2)))

# Add the output layer with a sigmoid activation
model.add(Dense(1, activation='tanh'))

```

The model summary shows that our architecture yields 13 trainable parameters.

```python
model.summary()
```

    Model: "sequential_1"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    simple_rnn_1 (SimpleRNN)     (None, 2)                 10
    _________________________________________________________________
    dense_1 (Dense)              (None, 1)                 3
    =================================================================
    Total params: 13
    Trainable params: 13
    Non-trainable params: 0
    _________________________________________________________________

## Elman network Application: XOR classification

Next, we compile and fit our model. I'll utilize [Adadelta](https://github.com/keras-team/keras/blob/master/keras/optimizers.py#L376) (to avoid manually adjusting the learning rate) as the optimizer, and the Mean-Squared Error (as in Elman original work). I'll train the model for 15,000 epochs over the 4 samples dataset.

```python
model.compile(optimizer='Adadelta',
              loss='mean_squared_error',
              metrics=['acc'])
history = model.fit(X, y,
                    epochs=5000,
                    verbose=0)
```

**Chart 2** shows the error curve (red, right axis), and the accuracy curve (blue, left axis) for each epoch.

```python
alt.data_transformers.disable_max_rows()

loss = history.history['loss']
accuracy = history.history['acc']

df = pd.DataFrame({"accuracy":accuracy, "loss":loss, "time-step": np.arange(0, len(accuracy))})

base = alt.Chart(df).mark_line(color="blue").encode(x="time-step", y="accuracy")
loss = alt.Chart(df).mark_line(color="red").encode(x="time-step", y="loss")
(base  + loss).properties(title='Chart 2').resolve_scale(y='independent')
```

We see that accuracy goes to 100% in around 1,000 epochs (note that different runs may slightly change the results).

### LSTM

In a strict sense, LSTM is a type of layer instead of a type of network. What I've calling LSTM networks is basically any RNN composed of LSTM layers. Most RNNs you'll find in the wild (i.e., the internet) use either LSTMs or [Gated Recurrent Units (GRU)](https://en.wikipedia.org/wiki/Gated_recurrent_unit). We don't cover GRU here since they are very similar to LSTMs and this chapter is dense enough as it is. If you want to learn more about GRU see [Cho et al (2014)](https://arxiv.org/abs/1406.1078) and [Chapter 9.1 from Zhang (2020)](https://d2l.ai/chapter_recurrent-modern/gru.html).

For this section, I'll base the code in the example provided by Chollet (2017) in chapter 6. As a side note, if you are interested in learning Keras in-depth, [Chollet's book](https://www.amazon.com/Deep-Learning-Python-Francois-Chollet/dp/1617294438) is probably the best source since he is the creator of Keras library.

### Reading and preprocessing data from Keras

If you are like me,  you like to check the [IMDB](https://www.imdb.com/) reviews before watching a movie. For this example, we will make use of the [IMDB dataset](https://www.imdb.com/interfaces/), and Lucky us, Keras comes pre-packaged with it. The IMDB dataset comprises 50,000 movie reviews, 50% positive and 50% negative. Keras give access to a numerically encoded version of the dataset where each word is mapped to sequences of integers. We can download the dataset by running the following:

```python
# Libraries for this section

import tensorflow as tf
from tensorflow.keras.layers import Dense, LSTM, Embedding
from tensorflow.keras.models import Sequential

from keras.datasets import imdb
from keras.preprocessing.sequence import pad_sequences
import altair as alt
import pandas as pd
import numpy as np
```

**Note**: This time I also imported Tensorflow, and from there Keras layers and models. Keras happens to be integrated with Tensorflow, as a high-level interface, so nothing important changes when doing this. Yet, there are some implementation issues with the optimizer that require importing from Tensorflow to work.

```python
maxlen = 5000
(train_data, train_labels), (test_data, test_labels) = imdb.load_data(num_words=maxlen)
```

The parameter `num_words=5000` restrict the dataset to the top 5,000 most frequent words. We do this to avoid highly infrequent words. Often, infrequent words are either typos or words for which we don't have enough statistical information to learn useful representations.

Data is downloaded as a (25000,) tuples of integers.

```python
print(f'train-data shape: {train_data.shape}, train-labels shape: {train_labels.shape}')
print(f'test-data shape: {test_data.shape}, test-labels shape: {test_labels.shape} \n')
print(f'first 10 words of first sequence: {train_data[0][0:10]},\nfirst sequence label: {train_labels[0]}')
```

    train-data shape: (25000,), train-labels shape: (25000,)
    test-data shape: (25000,), test-labels shape: (25000,)

    first 10 words of first sequence: [1, 14, 22, 16, 43, 530, 973, 1622, 1385, 65],
    first sequence label: 1

If you are curious about the review contents, the code snippet below decodes the first review into words.

```python
word_index = imdb.get_word_index()
reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])
decoded_review = ' '.join([reverse_word_index.get(i - 3, '?') for i in train_data[0]])
print(f'first 100 characters of first sequence:\n{decoded_review[0:100]}....')
```

    first 100 characters of first sequence:
    ? this film was just brilliant casting location scenery story direction everyone's really suited the....

Next, we need to **"pad" each sequence with zeros** such that all sequences are of the same length. We do this because Keras layers expect same-length vectors as input sequences. Given that we are considering only the 5,000 more frequent words, we have max length of any sequence is 5,000. Hence, we have to pad every sequence to have length 5,000.

```python
X_train = pad_sequences(train_data, maxlen=maxlen)
X_test = pad_sequences(test_data, maxlen=maxlen)
```

As a result, we go from a list of list (samples= 25000,), to a matrix of shape (samples=25000, maxleng=5000)

```python
print(f'train-data shape: {X_train.shape}, train-labels shape: {X_test.shape} \n')
print(X_train[0],X_test[0])
```

    train-data shape: (25000, 5000), train-labels shape: (25000, 5000)

    [  0   0   0 ...  19 178  32] [  0   0   0 ...  14   6 717]

Finally, we will take only the first 5,000 training and testing examples. We do this because training RNNs is computationally expensive, and we don't have access to enough hardware resources to train a large model here.

```python
training_size=5000
training_sentences = X_train[0:training_size]
testing_sentences =  X_test[0:training_size]
training_labels = train_labels[0:training_size]
testing_labels = test_labels[0:training_size]
```

```python
print(f'train-data shape: {training_sentences.shape}, train-labels shape: {training_labels.shape}')
print(f'test-data shape: {testing_sentences.shape}, test-labels shape: {testing_labels.shape}')
```

    train-data shape: (5000, 5000), train-labels shape: (5000,)
    test-data shape: (5000, 5000), test-labels shape: (5000,)

Let's compute the percentage of positive reviews samples on training and testing as a sanity check. We want this to be close to 50% so the sample is balanced.

```python
print(f'percentage of positive reviews in training: {training_labels.sum()/training_size}')
print(f'percentage of positive reviews in testing: {testing_labels.sum()/training_size}')
```

    percentage of positive reviews in training: 0.5092
    percentage of positive reviews in testing: 0.4858

### Word embeddings with Keras

We will use word embeddings instead of one-hot encodings this time. Again, Keras provides convenience functions (or layer) to learn word embeddings along with RNNs training. An embedding in Keras is a layer that takes two inputs as a minimum: the **max length of a sequence** (i.e., the max number of tokens), and the **desired dimensionality of the embedding** (i.e., in how many vectors you want to represent the tokens). For instance, for an embedding with 5,000 tokens and 32 embedding vectors we just define `model.add(Embedding(5,000, 32))`. We will do this when defining the network architecture.

### LSTM architecture in Keras

Defining RNN with LSTM layers is remarkably simple with Keras (considering how complex LSTMs are as mathematical objects). I'll define a relatively "shallow" network with just 1 hidden LSTM layer.

```python
# Define a network as a linear stack of layers
model = Sequential()

# Add embedding layer with:
    # - Max number of tokens: 10,000
    # - Number of embeddings vectors: 32
model.add(Embedding(maxlen, 32))

# Add LSTM layer with 32 units (sequence length)
model.add(LSTM(32))

# Add output layer with sigmoid activation unit
model.add(Dense(1, activation='sigmoid'))
```

## LSTM Application: IMDB review prediction

It's time to train and test our RNN. I'll run just five epochs, again, because we don't have enough computational resources and for a demo is more than enough. If you run this, it may take around 5-15 minutes in a CPU.  For instance, my Intel i7-8550U took ~10 min to run five epochs.

```python
model.compile(optimizer=tf.keras.optimizers.RMSprop(),
              loss=tf.keras.losses.BinaryCrossentropy(),
              metrics=['acc'])

history = model.fit(training_sentences, training_labels,
                    epochs=5,
                    batch_size=128, # update gradients every 128 sequences
                    validation_split=0.2, # validation subsample
                    verbose=0)
```

**Note**: a "validation split" is different from the testing set: It's a sub-sample from the training set. For instance, with a training sample of 5,000, the `validation_split = 0.2` will split the data in a 4,000 effective training set and a 1,000 validation set. The network is trained only in the training set, whereas the validation set is used as a real-time(ish) way to help with hyper-parameter tunning, by synchronously evaluating the network in such a sub-sample. To learn more about this see the [Wikipedia article on the topic](https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets).

```python
print(f"final training accuracy:{history.history['acc'][-1]}")
print(f"final validation accuracy:{history.history['val_acc'][-1]}")
```

    final training accuracy:0.8964999914169312
    final validation accuracy:0.843999981880188

We obtained a **training accuracy of ~90%** and **validation accuracy of ~84%** (note that different runs may slightly change the results). The top-pane in **Chart 3** shows the training and validation curves for accuracy, whereas the bottom-pane shows the same for the loss. It is clear that the network overfitting the data by the 3rd epoch. This is expected as our architecture is shallow, the training set relatively small, and no regularization method was used.

```python
loss = history.history['loss']
val_loss = history.history['val_loss']
accuracy = history.history['acc']
val_acc = history.history['val_acc']

df = pd.DataFrame({"accuracy":accuracy,
                   "val_accuracy": val_acc,
                   "loss":loss,
                   "val_loss": val_loss,
                   "time-step": np.arange(1, len(accuracy)+1)})

accu = alt.Chart(df).mark_line(color="#0202d6").encode(x="time-step", y="accuracy")
val_accu = alt.Chart(df).mark_line(color="#7272a1").encode(x="time-step", y="val_accuracy")

loss = alt.Chart(df).mark_line(color="#d60202").encode(x="time-step", y="loss")
val_loss = alt.Chart(df).mark_line(color="#cc6e6e").encode(x="time-step", y="val_loss")

((accu  + val_accu)&(loss + val_loss)).properties(title='Chart 3')
```

Finally, the model obtains a **test set accuracy of ~84%** echoing the results from the validation set. All things considered, this is a very respectable result!

```python
score = model.evaluate(testing_sentences, testing_labels, verbose=0)
print(f'Test loss score: {score[0]}')
print(f'Test accuracy score:{ score[1]}')
```

    Test loss score: 0.3852371459245682
    Test accuracy score:0.8384000062942505

## Limitations

### Training RNNs is hard and costly

As I mentioned in previous sections, there are three well-known issues that make training RNNs really hard: (1) vanishing gradients, (2) exploding gradients, (3) and its sequential nature, which make them computationally expensive as parallelization is difficult. I won't discuss again these issues. Many techniques have been developed to address all these issues, from architectures like LSTM,  GRU, and ResNets, to techniques like gradient clipping and regularization (Pascanu et al (2012); for an up to date (i.e., 2020) review of this issues see [Chapter 9 of Zhang et al book](https://d2l.ai/chapter_recurrent-modern/index.html).).

The quest for solutions to RNNs deficiencies has prompt the development of new architectures like Encoder-Decoder networks with "attention" mechanisms (Bahdanau et al, 2014; Vaswani et al, 2017). This new type of architecture seems to be outperforming RNNs in tasks like machine translation and text generation, in addition to overcoming some RNN deficiencies.

### Do RNNs "really" understand...anything?

Critics like Gary Marcus have pointed out the apparent inability of neural-networks based models to "really" understand their outputs (Marcus, 2018). This is prominent for RNNs since they have been used profusely used in the context of language generation and understanding. For instance, even state-of-the-art models like [OpenAI GPT-2](https://openai.com/blog/better-language-models/) sometimes produce incoherent sentences. Marcus gives the [following example](https://thegradient.pub/gpt2-and-the-nature-of-intelligence/):

> (**Marcus**) Suppose for example that I ask the system what happens when I put two trophies a table and another: *I put two trophies on a table, and then add another, the total number is...*

> (**GPT-2 answer**) *...is five trophies and I'm like, 'Well, I can live with that, right?*

From Marcus' perspective, this lack of coherence is an exemplar of GPT-2 incapacity to understand language (note that GPT-2 is not an RNN as I've described so far, but it serves to show the point since its performance is even better than LSTMs or GRUs).

Yet, I'll argue two things. First, this is an unfairly underspecified question: **What do we mean by understanding?** From a cognitive science perspective, this is a fundamental yet strikingly hard question to answer. If you ask five cognitive science what does it "really" mean to understand something you are likely to get five different answers. What do we need is a falsifiable way to decide when a system "really" understands language. Is lack of coherence enough? I produce incoherent phrases all the time, and I know lots of people that do the same. In any case, it is important to question whether human-level understanding of language (however you want to define it) is necessary to show that a computational model of any cognitive process is a good model or not. We have several great models of many natural phenomena, yet not a single one gets all the aspects of the phenomena perfectly. For instance, Marcus has said that the fact that GPT-2 sometimes produces incoherent sentences is somehow a proof that human "thoughts" (i.e., internal representations) can't possibly be represented as vectors (like neural nets do), which is a non-sequitur.

Second, **Why should we expect that a network trained for a narrow task like language production should understand what language "really" is?** The exercise of comparing computational models of "cognitive processes" with "full-blown" human cognition, makes as much sense as comparing a model of bipedal locomotion with the entire motor control system of an animal. A model of bipedal locomotion is just that: **a model of a sub-system or sub-process within a larger system, not a reproduction of the entire system**. The fact that a model of bipedal locomotion does not capture well the mechanics of "jumping", does not undermine it's veracity or utility, in the same manner, that the inability of a model of language production to "understand" all aspects of language does not undermine its plausibility as a model of...languague production.

## Conclusions

Recurrent neural networks have been prolific models in cognitive science (Munakata et al, 1997; St. John, 1992; Plaut et al., 1996; Christiansen & Chater, 1999; Botvinick & Plaut, 2004; Muñoz-Organero et al., 2019), bringing together intuitions about how cognitive systems work in time-dependent domains, and how neural networks may accommodate such processes.

Nevertheless, problems like vanishing gradients, exploding gradients, and computational inefficiency (i.e., lack of parallelization) have difficulted RNN use in many domains. Although new architectures (without recursive structures) have been developed to improve RNN results and overcome its limitations, they remain relevant from a cognitive science perspective.

## References

- Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. ArXiv Preprint ArXiv:1409.0473.
- Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2), 157–166.
- Botvinick, M., & Plaut, D. C. (2004). Doing without schema hierarchies: A recurrent connectionist approach to normal and impaired routine sequential action. Psychological Review, 111(2), 395.
- Barak, O. (2017). Recurrent neural networks as versatile tools of neuroscience research. Current Opinion in Neurobiology, 46, 1–6. https://doi.org/10.1016/j.conb.2017.06.003
- Chen, G. (2016). A gentle tutorial of recurrent neural network with error backpropagation. arXiv preprint arXiv:1610.02583.
- Elman, J. L. (1990). Finding Structure in Time. Cognitive Science, 14(2), 179–211. https://doi.org/10.1207/s15516709cog1402_1
- François, C. (2017). 6. Deep Learning for text and sequences. Deep learning with Python. Manning.
- Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078.
- Christiansen, M. H., & Chater, N. (1999). Toward a connectionist model of recursion in human linguistic performance. Cognitive Science, 23(2), 157–205.
- Goodfellow, I., Bengio, Y., & Courville, A. (2016). 10. Sequence Modeling: Recurrent and Recursive Nets. In Deep Learning. MIT Press. https://www.deeplearningbook.org/contents/mlp.html
- Hebb, D. O. (1949). The organization of behavior: A neuropsychological theory. Psychology Press.
- Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780.
- Hopfield, J. J. (1982). Neural networks and physical systems with emergent collective computational abilities. *Proceedings of the National Academy of Sciences*, *79*(8), 2554–2558.
- Güçlü, U., & van Gerven, M. A. (2017). Modeling the dynamics of human brain activity with recurrent neural networks. Frontiers in Computational Neuroscience, 11, 7.
- Graves, A. (2012). Supervised sequence labelling. In Supervised sequence labelling with recurrent neural networks (pp. 5-13). Springer, Berlin, Heidelberg.
- Jarne, C., & Laje, R. (2019). A detailed study of recurrent neural networks used to model tasks in the cerebral cortex. ArXiv Preprint ArXiv:1906.01094.
- John, M. F. (1992). The story gestalt: A model of knowledge-intensive processes in text comprehension. Cognitive Science, 16(2), 271–306.
- Jordan, M. I. (1986). Serial order: A parallel distributed processing approach, ICS Report 8604. *Institute for Cognitive Science, UCSD, La Jolla*.
- Marcus, G. (2018). Deep learning: A critical appraisal. ArXiv Preprint ArXiv:1801.00631.
- Munakata, Y., McClelland, J. L., Johnson, M. H., & Siegler, R. S. (1997). Rethinking infant knowledge: Toward an adaptive process account of successes and failures in object permanence tasks. Psychological Review, 104(4), 686.
- Muñoz-Organero, M., Powell, L., Heller, B., Harpin, V., & Parker, J. (2019). Using Recurrent Neural Networks to Compare Movement Patterns in ADHD and Normally Developing Children Based on Acceleration Signals from the Wrist and Ankle. Sensors (Basel, Switzerland), 19(13). https://doi.org/10.3390/s19132935
- K. J. Lang, A. H. Waibel, and G. E. Hinton. A Time-delay Neural Network Architecture for Isolated Word Recognition. Neural Networks, 3(1):23-43, 1990
- Pascanu, R., Mikolov, T., & Bengio, Y. (2013). On the difficulty of training recurrent neural networks. International Conference on Machine Learning, 1310–1318.
- Philipp, G., Song, D., & Carbonell, J. G. (2017). The exploding gradient problem demystified-definition, prevalence, impact, origin, tradeoffs, and solutions. ArXiv Preprint ArXiv:1712.05577.
- Plaut, D. C., McClelland, J. L., Seidenberg, M. S., & Patterson, K. (1996). Understanding normal and impaired word reading: Computational principles in quasi-regular domains. Psychological Review, 103(1), 56.
- Raj, B. (2020, Spring). Neural Networks: Hopfield Nets and Auto Associators [Lecture]. http://deeplearning.cs.cmu.edu/document/slides/lec17.hopfield.pdf
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, \Lukasz, & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 5998–6008.
- Zhang, A., Lipton, Z. C., Li, M., & Smola, A. J. (2020). 8. Recurrent Neural Networks. In Dive into Deep Learning. https://d2l.ai/chapter_convolutional-neural-networks/index.html

## Useful online resources

Here a list of my favorite online resources to learn more about Recurrent Neural Networks:

- Stanford Lectures: Natural Language Processing with Deep Learning, Winter 2020. [Coruse webpage](http://web.stanford.edu/class/cs224n/index.html#schedule).
- Bhiksha Raj's Deep Learning Lectures 13, 14, and 15 at CMU. [Course webpage](http://deeplearning.cs.cmu.edu/)
- Geoffrey Hinton's Neural Network Lectures 7 and 8. [YouTube Videos](https://www.youtube.com/watch?v=2fRnHVVLf1Y&list=PLiPvV5TNogxKKwvKb1RKwkq2hm7ZvpHz0)


```python
from IPython.core.display import display_html
from urllib.request import urlopen

link = 'https://gist.github.com/robblack007/eca03fa9f7586860235d/raw/ef05a2f29febc94a9c9f99ca20fd0b65e74ed451/custom.css'
display_html(urlopen(link).read(), raw=True)
```

<link href='http://fonts.googleapis.com/css?family=EB+Garamond' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Lato:100,100italic' rel='stylesheet' type='text/css'>
<link rel="stylesheet" href="//cdn.jsdelivr.net/font-hack/2.013/css/hack-extended.min.css">

<style>
	@font-face{
   		font-family: "Computer Modern";
        	src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');}

        div.cell{
	        max-width:    800px;
	        margin-left:  auto;
	        margin-right: auto;}

	h1 {
        	font-family: 'Lato', sans-serif;}

	h2 {
        	font-family: 'Lato', serif;}

	h3{
		font-family:   'Lato', serif;
        	margin-top:    12px;
		margin-bottom: 3px;}

        h4{
		font-family: 'Lato', serif;}

	h5 {
        	font-family: 'Lato', serif;}

	div.text_cell_render{
        	font-family:  'EB Garamond', 'Computer Modern', 'Helvetica Neue', Arial, Helvetica, Geneva, sans-serif;
	        line-height:  140%;
	        font-size:    14pt;
		max-width:    650px;
		margin-left:  auto;
        	margin-right: auto;}

	.CodeMirror{
        	font-family: 'Inconsolata', monospace;}

	.text_cell_render h1 {
	        font-weight:   100;
	        font-size:     48pt;
		line-height:   100%;
	        color:         #CD2305;
	        margin-bottom: 0.5em;
	        margin-top:    0.5em;
	        display:       block;}

	.text_cell_render h2 {
        	font-weight:   100;
        	font-size:     36pt;}

	.text_cell_render h3 {
        	font-weight:   100;
        	font-size:     21pt;}

	.text_cell_render h4 {
        	font-weight:   100;
        	font-size:     18pt;}

	.text_cell_render h5 {
	        font-weight:   100;
	        font-size:     16pt;
	        color:         #CD2305;
	        font-style:    italic;
	        margin-bottom: 0.5em;
	        margin-top:    0.5em;
	        display:       block;}

	.text_cell_render h6 {
	        font-weight:   100;
	        font-size:     14pt;
	        color:         #CD2305;
	        font-style:    italic;
	        margin-bottom: 0.5em;
	        margin-top:    0.5em;
	        display:       block;}

	.warning{
	    	background-color: #FFFFE0;
	    	border-color: #FFCC99;
	    	border-left: 5px solid #FFCC99;
	    	padding: 0.5em;}

	.error{
	    	background-color: #fcf2f2;
	    	border-color: #dFb5b4;
	    	border-left: 5px solid #dfb5b4;
	    	padding: 0.5em;}

</style>

# Dinámica de un robot manipulador planar

```python
from sympy import var, sin, cos, Matrix, Integer, eye, Function, Rational, exp, Symbol, I, solve, pi, trigsimp, dsolve, sinh, cosh, simplify
from sympy.physics.mechanics import mechanics_printing
mechanics_printing()
```

```python
var("m1 m2 J1 J2 l1 l2 t g")
```

```python
q1 = Function("q1")(t)
q2 = Function("q2")(t)
```

```python
x1 = l1*cos(q1)
y1 = l1*sin(q1)
v1 = x1.diff("t")**2 + y1.diff("t")**2
v1.trigsimp()
```

```python
x2 = l1*cos(q1) + l2*cos(q1 + q2)
y2 = l1*sin(q1) + l2*sin(q1 + q2)
v2 = x2.diff("t")**2 + y2.diff("t")**2
v2.trigsimp()
```

```python
K1 = Rational(1, 2)*m1*v1
K1
```

```python
K2 = Rational(1, 2)*m1*v2
K2
```

```python
U1 = m1*g*y1
U1
```

```python
U2 = m2*g*y2
U2
```

```python
K = K1 + K2
K
```

```python
U = U1 + U2
U
```

```python
L = (K - U).expand().simplify()
L
```

```python
τ1 = (L.diff(q1.diff(t)).diff(t) - L.diff(q1)).simplify().expand().collect(q1.diff(t).diff(t)).collect(q2.diff(t).diff(t))
```

```python
τ2 = (L.diff(q2.diff(t)).diff(t) - L.diff(q2)).simplify().expand().collect(q1.diff(t).diff(t)).collect(q2.diff(t).diff(t))
```

```python
τ1
```

```python
τ2
```

```python
from scipy.integrate import odeint
from numpy import linspace
```

```python
def pendulo_doble(estado, tiempo):
    # Se importan funciones necesarias
    from numpy import sin, cos, matrix
    # Se desenvuelven variables del estado y tiempo
    q1, q2, q̇1, q̇2 = estado
    t = tiempo

    # Se declaran constantes del sistema
    m1, m2 = 1, 1
    l1, l2 = 1, 1
    g = 9.81

    # Se declaran constantes del control
    kp1, kp2 = -30, -60
    kv1, kv2 = -20, -20

    # Señales de control nulas
    #tau1, tau2 = 0, 0

    # Posiciones a alcanzar
    qd1, qd2 = 1, 1

    # Se declaran señales de control del sistema
    tau1 = kp1*(q1 - qd1) + kv1*q̇1
    tau2 = kp2*(q2 - qd2) + kv2*q̇2

    # Se calculan algunos terminos comunes
    ϕ1 = m1*l1**2
    ϕ2 = m1*l1*l2
    ϕ3 = m1*l2**2

    # Se calculan las matrices de masas, Coriolis,
    # y vectores de gravedad, control, posicion y velocidad
    M = matrix([[2*ϕ1 + 2*ϕ2*cos(q2) + ϕ3, ϕ2*cos(q2) + ϕ3],
                [ϕ2*cos(q2) + ϕ3, ϕ3]])
    C = matrix([[-2*ϕ2*sin(q2)*q̇2, -ϕ2*sin(q2)*q̇2], [ϕ2*sin(q2)*q̇1, 0]])
    G = matrix([[m1*l1*cos(q1) + m2*l1*cos(q1) + m2*l2*cos(q1 + q2)], [m2*l2*cos(q1 + q2)]])
    Tau = matrix([[tau1], [tau2]])
    q = matrix([[q1], [q2]])
    q̇ = matrix([[q̇1], [q̇2]])

    # Se calcula la derivada del estado del sistema
    qp1 = q̇1
    qp2 = q̇2

    qpp = M.I*(Tau - C*q̇ - G)
    qpp1, qpp2 = qpp.tolist()

    return [qp1, qp2, qpp1[0], qpp2[0]]
```

```python
t = linspace(0, 30, 500)
estados_simulados = odeint(func = pendulo_doble, y0  = [0, 0, 0, 0], t = t)
```

```python
q1, q2, q̇1, q̇2 = list(zip(*estados_simulados.tolist()))
```

```python
%matplotlib notebook
from matplotlib.pyplot import plot, style, figure
from mpl_toolkits.mplot3d import Axes3D
style.use("ggplot")
```

```python
fig1 = figure(figsize=(8, 8))

ax1 = fig1.gca()

p1, = ax1.plot(t, q1)
p2, = ax1.plot(t, q2)
ax1.legend([p1, p2],[r"$q_1$", r"$q_2$"])
ax1.set_ylim(-4, 4)
ax1.set_xlim(-0.1, 10);
```

    <IPython.core.display.Javascript object>

```python
%matplotlib inline
from matplotlib import animation
from numpy import sin, cos, arange
```

```python
l1, l2 = 1, 1
# Se define el tamaño de la figura
fig = figure(figsize=(12.6, 6.6))

# Se define una sola grafica en la figura y se dan los limites de los ejes x y y
axi = fig.add_subplot(111, autoscale_on=False, xlim=(-2.1, 2.1), ylim=(-2.1, 2.1))
axi.set_xticklabels([])
axi.set_yticklabels([])
axi.axes.get_xaxis().set_visible(False)
axi.axes.get_yaxis().set_visible(False)

# Se utilizan graficas de linea para el eslabon del pendulo
linea, = axi.plot([], [], "-o", lw=2, color='gray')

def init():
    # Esta funcion se ejecuta una sola vez y sirve para inicializar el sistema
    linea.set_data([], [])
    return linea

def animate(i):
    # Esta funcion se ejecuta para cada cuadro del GIF

    # Se obtienen las coordenadas x y y para el eslabon
    xs, ys = [[0, l1*cos(q1[i]), l1*cos(q1[i]) + l2*cos(q1[i]+q2[i])],
              [0, l1*sin(q1[i]), l1*sin(q1[i]) + l2*sin(q1[i]+q2[i])]]
    linea.set_data(xs, ys)

    return linea

# Se hace la animacion dandole el nombre de la figura definida al principio, la funcion que
# se debe ejecutar para cada cuadro, el numero de cuadros que se debe de hacer, el periodo
# de cada cuadro y la funcion inicial
ani = animation.FuncAnimation(fig, animate, arange(1, len(q1)),
                              interval=25, init_func=init)

# Se guarda el GIF en el archivo indicado
ani.save('./imagenes/simulacion-doble.gif', writer='imagemagick');
```

```python

```


```python
from IPython.display import Image
from IPython.core.display import HTML
Image(url= "https://i.imgur.com/6TgxQrm.png")
```

```python
from sympy import *; x,h,t,y,z   = symbols("x h t y z", real=True)
f, g, h = symbols('f g h', cls=Function)
f = -x**2+100
g = 5*(floor(sqrt(f)))

for i in range(0,200,1):
    i = i
    if i == 0:
        print("""for  f(x) = -x**2+100 red line    and     g(x) = 5*(floor(sqrt(f))  blue line      dF = green line

     """)

    if i == 0:
        print(" f +",i,"and g +",i,"         Current Iteration:",i)
        p0 = plot((f+i),(g+i), diff(f+i),show = False,xlim = (1,10.5),size = (9,5),legend = True)
        p0[0].line_color = 'r'
        p0[2].line_color = 'g'
        p0.show()
    if i == 2:
        print("f *",i,"and g *",i,"          Current Iteration:",i)
        p1 = plot((f*i),(g*i),show = False,xlim = (1,10.5),size = (9,5),legend = "hello")
        p1[0].line_color = 'r'
        p1.show()
    if i == 20:
        print(" f root of",i,"and g root of",i,"     ex. f**(1/i)            Current Iteration:",i)
        p1 = plot((f**(1/i)),(g**(1/i)),show = False,ylim = (0,1.6),xlim = (1,10.5),size = (9,5),legend = True)
        p1[0].line_color = 'r'
        p1.show()

```

```python

```


### Instructions

When running the notebook the first time, make sure to run all cells before making changes in the notebook. Hit Shift + Enter to run the selected cell or, in the top menu, click on: `Kernel` > `Restart Kernel and Run All Cells...` to rerun the whole notebook. If you make any changes in a cell, rerun that cell.

If you make any changes in a coding cell, rerun the notebook by `Run` > `Run Selected Cell and All Below`

```python
# Import dependencies
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)      # Ignore warnings
warnings.simplefilter(action='ignore', category=UserWarning)        # Ignore warnings
import sys
import numpy as np
import matplotlib.pyplot as plt
from scipy.interpolate import InterpolatedUnivariateSpline
from IPython.display import Javascript, display, clear_output
import sys
sys.path.append('python/')                # Define path for libraries
import downloadSPARCdata as dsd           # Import library to download data files
```

# SPARC Activity

The following notebook will walk you through rotation curve plotting using an online database. As the main activity in this module, select from a list of 175 galaxies to plot their rotation curves and calculate the amount of Dark Matter for all of them!

You will download rotation curve data of galaxies from the Spitzer Photometry & Accurate Rotation Curves (SPARC) database and plot them with and without a Dark Matter component to understand how the "missing mass" affects the rotation of stars.

You may either download and unzip the data files yourself or have Python do the work.

#### Option 1. Download and unzip the data files yourself:

1. Go to http://astroweb.cwru.edu/SPARC/ and under "BASIC SPARC DATA", download the `Rotmod_LTG.zip` file for "Newtonian Mass Models".
2. Open (extract/unzip) the zip file to preferably the same location as where your Python notebook is located.
3. Choose a galaxy (any file) of your choice. Do not rename the file. Put the name of the chosen galaxy in the variable in the cell below and run it.
4. Make a note of the directory (file location) of the SPARC file of your galaxy **with respect to this location of this Python notebook**. For example, if your file is located in the same location as this code, leave the following cell as is. But if it is, say, in the next folder "up" from this one, use the extension `../`. As an example, if the SPARC file is located two folders up then one folder "down" (into a different folder named, say, 'otherfolder'), you would write:
`SPARC_file_directory='../../otherfolder/'` in the cell below and run it.

```python
SPARC_file_directory='data/sparc/'                 #note that '' means the string variable is blank
```

#### Option 2. Let Python download and unzip the data files

1. By clicking the YES button, you can download and unzip SPARC data files to your computer.

```python
#NBVAL_IGNORE_OUTPUT
#Because the button doesn't save to the repository correctly.
print("Would you like to download and unzip SPARC data files to your computer?")
dsd.displaybuttons
```

    Would you like to download and unzip SPARC data files to your computer?

    VBox(children=(HBox(children=(Button(button_style='success', description='Yes', icon='check', style=ButtonStyl…

### Choose a galaxy

Select any galaxy from the dropdown menu.

```python
#NBVAL_IGNORE_OUTPUT
#Because the dropdown doesn't save to the repository correctly.
galaxylist = ['NGC5005']                 # default list of galaxies
def on_change(change):                   # add selected galaxynames to the list
    if change['type'] == 'change' and change['name'] == 'value':
        galaxylist.append(change['new'])
        display(Javascript('IPython.notebook.execute_cell_range(IPython.notebook.get_selected_index()+1, IPython.notebook.ncells())'))
dsd.dropdownmenu.observe(on_change)
display(dsd.galaxyoptions)
```

    Box(children=(Box(children=(Label(value='Galaxy: '), Dropdown(index=79, options=('CamB', 'D512-2', 'D564-8', '…

Once you selected a galaxy, click on the cell below; , then select `Run` → `Run Selected Cell and All Below` to reveal the rotation curve and the image of the chosen galaxy in the following cells.

```python
# Writing the chosen galaxy name in a text file would allow us to use the selection in libraries outside of this notebook
chosengalaxy = galaxylist[-1]              # define the last element of the list as the selected galaxy
textfile = open("python/chosengalaxy.txt", "w")
textfile.write(chosengalaxy)
textfile.close()
```

The downloaded files are named as "[galaxy name]\_rotmod.dat". In the following code cell, we have defined the file path for you to import the data for your chosen galaxy.

```python
# Define file path for .dat files
SPARC_file = SPARC_file_directory + chosengalaxy + '_rotmod.dat'

# Load the galaxy data
galaxydata = np.loadtxt(SPARC_file)
```

Now, lets print the header to see what variable each column corresponds to.

```python
# Print header
for lines in open(SPARC_file, 'r'):
    if lines.startswith('#'):
        print(lines)
```

    # Distance = 16.9 Mpc

    # Rad	Vobs	errV	Vgas	Vdisk	Vbul	SBdisk	SBbul

    # kpc	km/s	km/s	km/s	km/s	km/s	L/pc^2	L/pc^2

__Header key__:

>_Rad_: radius or distance from the center of galaxy (in kiloparsec)

>_Vobs_: observed velocity/measured datapoints (in km/s)

>_errV_: uncertainty in the observed velocity (in km/s)

>_Vgas_: velocity of the gas component (in km/s)

>_Vdisk_: velocity of the disk component (in km/s)

>_Vbul_: velocity of the bulge component (in km/s)

>_SBdisk_: surface brightness of the disk component (in Luminosity/$ \rm parsec^2$)

>_SBbul_: surface brightness of the bulge component (in Luminosity/$ \rm parsec^2$)

Split columns into arrays and name them according to the header displayed in the cell above.

```python
# Split columns into arrays
Rad,Vobs,errV,Vgas,Vdisk,Vbul,SBdisk,SBbul = galaxydata.T
```

The distance to the galaxy is given in the data file in Megaparsecs (1 Mpc equals about 3.26 million light years). You can define this value in your Python notebook for your reference.

```python
# Define distance in Mpc
firstline = open(SPARC_file).readline()         # Open the data file
firstline = firstline.split()                   # Split the first line into separate strings
distance = float(firstline[3])                  # Take the 4th value of the first line (counts: 0,1,2,3...)
print("The distance to {} galaxy is {} Mpc.".format(chosengalaxy,distance))
```

    The distance to NGC5005 galaxy is 16.9 Mpc.

Due to the low number of datapoints, the rotation curve might look choppy. In order to smooth it, you can define a polynomial using Scipy's Interpolated Univariate Spline function, then choose your range and number of datapoints (sampling radii) to define the new, more smooth curve of each component.

```python
# Spline function
def interpd(x,y):
    return InterpolatedUnivariateSpline(x,y,k=3)      # Degree of the smoothing spline: 3

# Bulge
def bulge(r,bpref):
    polynomial = interpd(Rad,bpref*Vbul)              # bpref is the bulge prefactor added to the bulge
    return polynomial(r)

# Disk
def disk(r,dpref):
    polynomial = interpd(Rad,dpref*Vdisk)             # dpref is the disk prefactor added to the disk
    return polynomial(r)

# Gas
def gas(r):
    polynomial = interpd(Rad,Vgas)                    # Note that the gas doesn't have a prefactor
    return polynomial(r)
```

### Observed velocity of the chosen galaxy

Plot the observed velocity (measured datapoints) and the error bars on each measurement of the chosen galaxy as a function of radius.

```python
plt.figure(figsize=(8,6))                                                # size of the galaxy
plt.errorbar(Rad,Vobs,yerr=errV, marker='o', markersize=8, \
             ecolor='gray',color='gray', linestyle='none', linewidth=2)  # plot datapoints with errorbars
plt.xlabel('radius (kpc)',size=12)                                       # label x-axis
plt.ylabel('velocity (km/s)',size=12)                                    # label y-axis
plt.title(str('Observed Velocity of ' + chosengalaxy), size=14)          # title of the plot
plt.xlim(0,np.max(Rad)+0.2)                                              # range of the x-axis
plt.ylim(0,np.max(Vobs)+100)                                             # range of the y-axis
plt.show()                                                               # show the plot
```

A prefactor or mass-to-light ratio ($\Upsilon$) is added to the disk and the bulge which will be useful when fitting the curve of each component. These prefactors help scaling the curve up and down. You can either change these values manually and see the magnitude of the relevant curve change in the cells below or import the fitting parameters from a python library.

>pref_bulge: bulge prefactor

>pref_disk: disk prefactor

Note that the gas prefactor is fixed. The mass of the gas was calculated assuming a factor of 1.33 to account for the contribution of helium.

```python
import widget_SPARC as fit        # Import widget library for using the fitting parameters
import importlib
importlib.reload(fit)             # Reload widget library so the changes take effect
clear_output()
```

```python
# Prefactors
pref_bulge = fit.best_bpref
pref_disk = fit.best_dpref

# Radius for plotting
radius = np.linspace(np.min(Rad),np.max(Rad),1000) # starting at the same radius where the given datapoints start,
                                                   # ending at the same radius where the given datapoints end,
                                                   # with 1000 datapoints
```

### Rotational velocity of each luminous component

Plot the rotation velocity of each component.

```python
plt.figure(figsize=(8,6))
plt.plot(radius, gas(radius),              label="Gas velocity")
plt.plot(radius, bulge(radius,pref_bulge), label="Bulge velocity")
plt.plot(radius, disk(radius,pref_disk),   label="Disk velocity")
plt.xlabel('radius (kpc)',  size=12)
plt.ylabel('velocity (km/s)', size=12)
plt.title(str('Velocity of Luminous Components of ' + chosengalaxy), size=14)
plt.xlim(0,np.max(Rad + 0.1))
plt.ylim(0,np.max(Vobs + 80))
plt.legend(bbox_to_anchor=(1,1), loc="upper left")
plt.show()
```

Each component was calculated using the following model:

>Bulge: residual luminosity profile

>Disk: observed [3.6] surface brightness profile

>Gas: H1 surface density profiles or mass models

However, these calculations are beyond the scope of this workshop.

### Total velocity curve of luminous matter

Now that we have all of the luminous components or light matter: bulge, disk and gas, we can add them in quadrature with equivalent prefactors to get the total velocity of the luminous matter.

>__Total velocity of luminous matter__:

    \begin{equation}
    v_{total,light}(r) = \sqrt{\lvert v_{gas}\rvert v_{gas} + \Upsilon _{bulge} \lvert v_{bulge}\rvert v_{bulge} + \Upsilon _{disk} \lvert v_{disk}\rvert v_{disk}}
    \end{equation}

```python
def total_velocity_light(r,pref_bulge,pref_disk):
    return np.sqrt(gas(r)**2 + bulge(r,pref_bulge)**2 + disk(r,pref_disk)**2)
```

Plot the luminous components and the total velocity.

```python
plt.figure(figsize=(9,6))

plt.plot(radius, gas(radius),               label="Gas velocity",   color='#1f77b4')
plt.plot(radius, bulge(radius,pref_bulge),  label="Bulge velocity", color='#ff7f0e')
plt.plot(radius, disk(radius,pref_disk),    label="Disk velocity",  color='#2ca02c')

plt.plot(radius,total_velocity_light(radius,pref_bulge,pref_disk), label="Total velocity of luminous matter", color='#d62728', linewidth=3)
plt.errorbar(Rad,Vobs,yerr=errV, marker='o', markersize=8, \
             ecolor='gray',color='gray', linestyle='none', linewidth=2, label = "Observed velocity")
plt.xlabel('radius (kpc)',size=12)
plt.ylabel('velocity (km/s)',size=12)
plt.title(str('Rotation Curve of ' + chosengalaxy + ' - Luminous Matter'), size=14)
plt.xlim(0,np.max(Rad + 0.2))
plt.ylim(0,np.max(Vobs) + 100)
plt.legend(bbox_to_anchor=(1,1), loc="upper left")
plt.show()
```

Notice that the total velocity does not align with the measured velocity. Not even if you scale the other components up and down with the prefactor. Let's add the "missing matter".

### Add Dark Matter

Dark Matter can be characterized by two parameters: the central mass density ($\rho_0$) and the core radius ($r_c$).

***Central mass density***: The central mass density is the density at the center of the galaxy, changing this value changes the magnitude of the Dark Matter curve.

***Core radius***: The core radius (also called "cutoff radius" or "scale radius") indicates where the density falls off by a factor of $e$ (~2.7). Adjusting this factor changes where the "bump" of the curve is located.

>__Velocity__:

    \begin{equation}
    v_{DM}(r) = \sqrt{4 \pi G \rho_{0} r_c^2 \big( 1- \frac{r_c}{r} \arctan{\frac{r}{r_c}}\big)}
    \end{equation}

    where:

        $G$ = gravitational constant

        $\rho_0$ = central mass density (in solar mass/$\rm kpc^3$)

        $r_c$ = core radius (in kpc)

Set parameters or import the fitting parameters from our Python library. The fitting parameters were calculated using the Python library: lmfit.

```python
# Dark matter halo parameters
rho0 = fit.best_rho0     # Central mass density (in solar mass/kpc^3)
rc = fit.best_rc         # Core radius (in kpc)

# Constant parameters
G = 4.300e-6             # Gravitational constant (kpc/solar mass*(km/s)^2)

# Print values
print("The fitted best values for the {} galaxy: ".format(chosengalaxy))
print("    Central mass density of the dark matter halo = {:.3e} solar mass/kpc^3".format(rho0))
print("    Core radius of the Dark Matter halo = {:.3f} kpc".format(rc))
```

    The fitted best values for the NGC5005 galaxy:
        Central mass density of the dark matter halo = 1.693e+07 solar mass/kpc^3
        Core radius of the Dark Matter halo = 9.917 kpc

```python
# Equation for dark matter halo velocity
def halo(r,rho0,rc):
    v = np.sqrt(4*np.pi*G*rho0*rc**2*(1 - rc/r * np.arctan(r/rc)))
    return v
```

Now, calculate the total velocity with the Dark Matter component included.

```python
def total_velocity_withDM(r,pref_bulge,pref_disk,rho0,rc):
    return np.sqrt(gas(r)**2 + bulge(r,pref_bulge)**2 + disk(r,pref_disk)**2 + halo(r,rho0,rc)**2)
```

Plot all components and the total velocity with the Dark Matter.

```python
plt.figure(figsize=(12,8))

plt.plot(radius, gas(radius),              label="Gas velocity",   color='#1f77b4')
plt.plot(radius, bulge(radius,pref_bulge), label="Bulge velocity", color='#ff7f0e')
plt.plot(radius, disk(radius,pref_disk),   label="Disk velocity",  color='#2ca02c')
plt.plot(radius, halo(radius,rho0,rc),     label="Dark matter halo velocity",  color='#bcbd22')

plt.plot(radius,total_velocity_light(radius,pref_bulge,pref_disk), label="Total velocity of luminous matter", color='#d62728', linewidth=3)
plt.plot(radius,total_velocity_withDM(radius,pref_bulge,pref_disk,rho0,rc), label="Total velocity with dark matter", color='k', linewidth=3)

plt.errorbar(Rad,Vobs,yerr=errV, marker='o', markersize=8, \
             ecolor='gray',color='gray', linestyle='none', linewidth=2, label = "Observed velocity")
plt.xlabel('radius (kpc)',size=12)
plt.ylabel('velocity (km/s)',size=12)
plt.suptitle(str('Rotation Curve of ' + chosengalaxy), size=14)
plt.title(str('Distance = {} Mpc'.format(distance)), size=12)
plt.xlim(0,np.max(Rad + 0.2))
plt.ylim(0,np.max(Vobs) + 100)
plt.legend(bbox_to_anchor=(1,1), loc="upper left")
plt.show()
```

From the plot, we can see that the total velocity curve with dark matter matches the observed data points.

### References

>Jimenez, Raul, Licia Verde, and S. Peng Oh. **Dark halo properties from rotation curves.** _Monthly Notices of the Royal Astronomical Society_ 339, no. 1 (2003): 243-259. https://doi.org/10.1046/j.1365-8711.2003.06165.x.

>Lelli, F., McGaugh, S. S., & Schombert, J. M. (2016). **SPARC: Mass models for 175 disk galaxies with Spitzer photometry and accurate rotation curves.** _The Astronomical Journal_, 152(6), 157. https://doi.org/10.3847/0004-6256/152/6/157

>Matt Newville, Renee Otten, Andrew Nelson, Antonino Ingargiola, Till Stensitzki, Dan Allan, Austin Fox, Faustin Carter, Michał, Ray Osborn, Dima Pustakhod, lneuhaus, Sebastian Weigand, Glenn, Christoph Deil, Mark, Allan L. R. Hansen, Gustavo Pasquevich, Leon Foks, … Arun Persaud. (2021). __lmfit/lmfit-py: 1.0.3 (1.0.3).__ Zenodo. https://doi.org/10.5281/zenodo.5570790.

>“Megaparsec: Cosmos.” __Megaparsec__ | _COSMOS_. Swinburne University of Technology. Accessed November 12, 2021. https://astronomy.swin.edu.au/cosmos/m/megaparsec.
***