$$
\newcommand{\mat}[1]{\boldsymbol {#1}}
\newcommand{\mattr}[1]{\boldsymbol {#1}^\top}
\newcommand{\matinv}[1]{\boldsymbol {#1}^{-1}}
\newcommand{\vec}[1]{\boldsymbol {#1}}
\newcommand{\vectr}[1]{\boldsymbol {#1}^\top}
\newcommand{\rvar}[1]{\mathrm {#1}}
\newcommand{\rvec}[1]{\boldsymbol{\mathrm{#1}}}
\newcommand{\diag}{\mathop{\mathrm {diag}}}
\newcommand{\set}[1]{\mathbb {#1}}
\newcommand{\norm}[1]{\left\lVert#1\right\rVert}
\newcommand{\pderiv}[2]{\frac{\partial #1}{\partial #2}}
\newcommand{\bb}[1]{\boldsymbol{#1}}
$$
.

# Tutorial 4: Sequence Models

## Todays Agenda
---
- Reminders and recaps
- classical approach for NLP tasks
- What RNNs are and how they work
- memory RNN (LSTM, GRU)
- B-directional RNN
- BPTT
- Gradient cliping
- Embedding
- impplementation of basic RNN
- Sentiment analysis for movie reviews

if we have time:
- Stock prediction using GRU

## imports
---

```python
# Setup
%matplotlib inline
import numpy as np
import os
import sys
import time
import matplotlib.pyplot as plt
import warnings
warnings.simplefilter("ignore")

# pytorch
import torch
import torch.nn as nn
import torchtext
import torchtext.data as data
import torchtext.datasets as datasets
import torch.nn.functional as f
```

```python
plt.rcParams['font.size'] = 20
data_dir = os.path.expanduser('~/.pytorch-datasets')
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
```

```python
device
```

    device(type='cpu')

## Reminders and recaps
---

Last tutorial we learned about CNNs- Conv layers to perform feature extranction, and FC layers for classification.

- the convolution layers output does is not relay on the filter size, but on the input size.
- the FC layers depend only on the input and output size.

[image from](https://www.researchgate.net/figure/A-vanilla-Convolutional-Neural-Network-CNN-representation_fig2_339447623)

However, we only talked about stationary data, not about **persistent state**.

what do we do when there is another dimention?

Sometimes, we want to look at previus inputs and outputs

## Classical NLP
---

### Forms of tasks we can preform:

* One-to-one: from fixed-sized input to fixed-sized output (e.g. image classification).
* One-to-many: Sequence output (e.g. image captioning takes an image and outputs a sentence of words).
* Many-to-one: Sequence input (e.g. sentiment analysis)
* Many-to-many: Sequence input and sequence output (e.g. Machine Translation).
* Many-to-many: Synced sequence input and output (e.g. video classification where we wish to label each frame of the video).

as a learning problem, we can use classical machine learning algorithms.

for the sake of convenience we will talk about naiva base.

A quick review of the Naive Bayes (NB) method for text classification:input vector $\mathbf{x} = \langle x_1, x_2, \ldots, x_m \rangle$ of feature values, and a set of classes $\{ c_1, c_2, \ldots, c_k \}$ the text should be classified as.

We will predict the class $c_i$ that maximizes $Pr(c_i \mid \mathbf{x})$.

By Bayes rule (this is the "Bayes" part in the name "Naive Bayes"),

$$ argmax_i Pr(c_i | x) = argmax_i \frac{Pr(\mathbf{x} | c_i) \cdot Pr(c_i)}{Pr(\mathbf{x})} = argmax_i Pr(\mathbf{x} | c_i) \cdot Pr(c_i) $$

without going too much into details, NB called like that because it used the base rule, and because it's navie...assuming I.I.D, i.e:
$$  Pr(\mathbf{x} | c_i) =  \prod_{j=1}^m Pr(x_j | c_i) $$

Great, now we just need to think how we calculate the probabilities....
$$ Pr(x_j | c_i) \approx \frac{\# (x_j, c_i)}{\sum_k \# (x_k, c_i)} $$

the basic approch. is called **Bag Of Words (BOW)**.
in this approch, we count all the words in the document and use each one as a feature.
in general, expressing features as raw words is very easy to use with naive base.

in this case, you probably see the problem right away.

the strongest features are now very expresive.. 'the', ',' and 'very' doesn't say much about this review.

this words exist both in good and bad reviews.

that's why we need to express, not only the document frequncies, but also the general word frequncies.

something like that is called **TF-IDF** as **erm frequency–inverse document frequency**

where **TF** is the simplest cound we just did
and **IDF** usually define as $log(\frac{N}{Df_x})$ where N is the total documents and Df is the number of documents containing the term x.

in total we get $$\# (x_j, c_i) log(\frac{\sum_i \# (c_i)}{\sum_i \# (c_i,x_j)}) $$

and we solved one problem, however, order of words still matter and naive base, ever with smarter representation, is still naive in the notion of corrolation and order.

another NLP basic task is text generation, or prediction of the next word in a sequnce.

for that, we're going to see a model called **N-gram**.

for that matter we want to learn
$$
Pr(x_n | x_1, x_2, ..., x_{n-1}) \approx \frac{\#{x_1, x_2, ..., x_n}}{\#{x_1, x_2, ..., x_{n-1}}}  = \frac{\#{x_1, x_2, ..., x_n}}{\sum_{x'} \#{x_1, x_2, ..., x_{n-1}, x'}}
$$

were the lenght of the sequnce that we depand on is the **N**.

now, this course is in deep learning and not NLP, but you're welcome to read more about more advanced technics like smoothing n-grams and much more.

here is an example for 1,2 and 3-grams input:

Like always, we want to express a parametric strong model to forcast a prediction of a sequnce that catch long dependencies and does not relay on naive approch.

So let's dive back to deep learning...

## Recurrent layers
---

The idea is to produce output, like simple FC layer, but also produce a state, that we will denote by $\vec{h}_t$ (hidden state)

An RNN has two inputs:

- Current sample, $\vec{x}_t \in\set{R}^{d_{i}}$.
- Previous **state**, $\vec{h}_{t-1}\in\set{R}^{d_{h}}$.

and it produces two outputs which depend on both inputs:
- Current layer output, $\vec{y}_t\in\set{R}^{d_o}$.
- Current **state**, $\vec{h}_{t}\in\set{R}^{d_{h}}$.

as such
$$
\begin{align}
\forall t \geq 0:\\
\vec{h}_t &= \varphi_h\left( \mat{W}_{hh} \vec{h}_{t-1} + \mat{W}_{xh} \vec{x}_t + \vec{b}_h\right) \\
\vec{y}_t &= \varphi_y\left(\mat{W}_{hy}\vec{h}_t + \vec{b}_y \right)
\end{align}
$$

where,
- $\vec{x}_t \in\set{R}^{d_{i}}$ is the input at time $t$.
- $\vec{h}_{t-1}\in\set{R}^{d_{h}}$ is the **hidden state** of a fixed dimension.
- $\vec{y}_t\in\set{R}^{d_o}$ is the output at time $t$.
- $\mat{W}_{hh}\in\set{R}^{d_h\times d_h}$, $\mat{W}_{xh}\in\set{R}^{d_h\times d_i}$, $\mat{W}_{hy}\in\set{R}^{d_o\times d_h}$, $\vec{b}_h\in\set{R}^{d_h}$ and $\vec{b}_y\in\set{R}^{d_o}$ are the model weights and biases.
- $\varphi_h$ and $\varphi_y$ are some non-linear functions. In many cases $\varphi_y$ is not used.

:

note that:
- The layer does not dependent on time
- The same layer is applied at successive time steps, propagating the hidden state.

### Modeling time-dependence

If we imagine **unrolling** a single RNN layer through time,

RNNs layers can be stacked to build a deep RNN model.

We can see how late outputs can now be influenced by early inputs, through the hidden state.

Forward pass seems pretty easy...

What about **backpropagation** ?

## Backprop from Loss for RNN
---

**Backpropagation through time**, or BPTT

1. Loss is a sum of losses from all outputs

$$ L =\sum_{k=1}^{t}L_k $$

2. Calculate Gradient of each loss w.r.t. each parameter at each timestep

$$
\pderiv{L_t}{\mat{W}} = \sum_{k=1}^{t}
\pderiv{L_t}{\hat y_t} \cdot
\pderiv{\hat y_t}{\vec{h}_t} \cdot
\pderiv{\vec{h}_t}{\vec{h}_k} \cdot
\pderiv{\vec{h}_k}{\mat{W}}
$$

3. For each parameter, accumulate gradients from all timesteps

$$
\pderiv{L}{\mat{W_i}} = \sum_{k=1}^{t}
\pderiv{L_t}{\hat {W_i}_t}
$$

As you can see, the chain rule here can create a really long chain...
Remember the Vanishing gradients problem?

one naive way to limit the vanishing gradients problem is to trancate the gradients with a fixed lenght.
**Truncated backpropagation through time** or **TBPTT**

## Gradients Cliping
---

A popular method to avoid exploading gradients, is to use `gradients cliping`

the idea is super simple, yet efficient only for exploading and not for vanishing gradients

the idea is as such:
look at a layer $\matrix{L}$ with gradient matrix $\matrix{G}$

we would simply look at the norm of the gradients and when they exceed some threshold, we would clip it down.

i.e: if $\norm{\matrix{G}} \ge c$
$$
\matrix{G} = C \frac{\matrix{G}}{\norm{\matrix{G}}}
$$

and the process would look sort of like this:

you can read the [theoretical justification](https://openreview.net/pdf?id=BJgnXpVYwS) that actually came in 2020

### RNN Implementation

Based on the above equations, let's create a simple RNN layer  with PyTorch.

```python
import torch.nn as nn

class RNNLayer(nn.Module):
    def __init__(self, in_dim, h_dim, out_dim, phi_h=torch.tanh, phi_y=torch.sigmoid):
        super().__init__()
        self.phi_h, self.phi_y = phi_h, phi_y

        self.fc_xh = nn.Linear(in_dim, h_dim, bias=False)
        self.fc_hh = nn.Linear(h_dim, h_dim, bias=True)
        self.fc_hy = nn.Linear(h_dim, out_dim, bias=True)

    def forward(self, xt, h_prev=None):
        if h_prev is None:
            h_prev = torch.zeros(xt.shape[0], self.fc_hh.in_features)

        ht = self.phi_h(self.fc_xh(xt) + self.fc_hh(h_prev))

        yt = self.fc_hy(ht)

        if self.phi_y is not None:
            yt = self.phi_y(yt)

        return yt, ht

```

We'll instantiate our model

```python
N = 3 # batch size
in_dim, h_dim, out_dim = 1024, 10, 1

rnn = RNNLayer(in_dim, h_dim, out_dim)
rnn
```

    RNNLayer(
      (fc_xh): Linear(in_features=1024, out_features=10, bias=False)
      (fc_hh): Linear(in_features=10, out_features=10, bias=True)
      (fc_hy): Linear(in_features=10, out_features=1, bias=True)
    )

And manually "run" a few time steps

```python
# t=1
x1 = torch.randn(N, in_dim, requires_grad=True) # requiring grad just for torchviz
y1, h1 = rnn(x1)
print(f'y1 ({tuple(y1.shape)}):\n{y1}')
print(f'h1 ({tuple(h1.shape)}):\n{h1}\n')

# t=2
x2 = torch.randn(N, in_dim, requires_grad=True)
y2, h2 = rnn(x2, h1)
print(f'y2 ({tuple(y2.shape)}):\n{y2}')
print(f'h2 ({tuple(h2.shape)}):\n{h2}\n')
```

    y1 ((3, 1)):
    tensor([[0.5718],
            [0.5578],
            [0.5016]], grad_fn=<SigmoidBackward>)
    h1 ((3, 10)):
    tensor([[-0.4299,  0.8944,  0.3502,  0.6787,  0.4357,  0.0061,  0.1597,  0.1902,
             -0.1483,  0.0930],
            [-0.4613,  0.7198,  0.6111,  0.8106, -0.3908, -0.2790, -0.5295, -0.1814,
             -0.3756, -0.0510],
            [ 0.1095, -0.4184, -0.8313,  0.4733, -0.6570, -0.6746, -0.6946,  0.7163,
             -0.7147,  0.3134]], grad_fn=<TanhBackward>)

    y2 ((3, 1)):
    tensor([[0.5669],
            [0.6768],
            [0.5615]], grad_fn=<SigmoidBackward>)
    h2 ((3, 10)):
    tensor([[-0.7498,  0.2989,  0.6249,  0.6237,  0.2206,  0.2341,  0.4152, -0.4381,
             -0.4282,  0.0050],
            [ 0.0861,  0.7062, -0.6788, -0.0679, -0.9191,  0.8050,  0.6100,  0.0448,
             -0.9095,  0.5696],
            [ 0.8221,  0.5921, -0.7072, -0.2782,  0.1083, -0.7948, -0.3569,  0.2418,
             -0.4331, -0.4357]], grad_fn=<TanhBackward>)

As usual, let's visualize the computation graph and see what happened when we used the same RNN block twice, by looking at the graph from both $y_1$ and $y_2$.

```python
import torchviz

torchviz.make_dot(
    y2, # Note: Change here to y2 to see the fullly unrolled graph!
    params=dict(list(rnn.named_parameters()) + [('x1', x1), ('x2', x2)])
)
```

## LSTM and GRU
---

We talked about the vanishing gradients problem with RNN, however, we didn't mention how to deal with those.
Luckyly, Gradient based methods invented long time ago...
Today we will talk about **Long Short term Memory (LSTM)** and **Gated Recurrent Unit (GRU)**.

both are a type of recurrent cell that tries to preserve long term information. The idea of LSTM was presented back in 1997, but flourished in the age of deep learning.

The cell have `memory` or `context` that's base on the `state` of basic RNN and on 3 main gates:

- Input gate: decides when to read data into the cell.
- Output gate: outputs the entries from the cell.
- Forget gate: a mechanism to reset the content of the cell.
These gates learn which information is relevant to forget or remember during the training process. The gates contain a non-linear activision function (sigmoid).

* we denote the input  $X_t \in \mathbb{R}^{n\times d}$ at timestemp $t$ and the hidden state of the previous time step is $H_{t−1}\in \mathbb{R}^{n\times h}$.

- **Forget gate**: $$ F_t = \sigma(X_tW_{xf} +H_{t-1}W_{hf} +b_f) \in \mathbb{R}^{n\times f}, $$
- **Input gate**: $$ I_t = \sigma(X_tW_{xi} +H_{t-1}W_{hi} +b_i) \in \mathbb{R}^{n\times h},$$
- **Output gate**: $$ O_t = \sigma(X_tW_{xo} +H_{t-1}W_{ho} +b_o) \in \mathbb{R}^{n\times o}, $$

- **candidate memory**:$$ \tilde{C}_t = \text{tanh}(X_tW_{xc} +H_{t-1}W_{hc} +b_c) \in \mathbb{R}^{n\times c}$$

- **memory / context**: $$ C_t = F_t \odot C_{t-1} + I_{t} \odot \tilde{C}_t$$

- **hidden state**: $$ H_t = O_t \odot \text{tanh}(C_t)$$

as you can see, we save 2 diffrent states, `hidden` and `context` (and of course it's not mentioned, but we have the output as well).

the idea really help with vanishing gradients, yet a big set of weights is added to the simple RNN.

Now we will see that pytorch implementation can also take the hole seqance and deal with it at once

```python
lstm = nn.LSTM(10, 20, 2) #LSTM is not an encoder decoder, thus we can project the output size, but default we use the hidden size
input = torch.randn(5, 3, 10) #5 batch size, seq of 3, input dim 10

h0 = torch.randn(2, 3, 20) #2 for 2 layers of the LSTM
c0 = torch.randn(2, 3, 20)
output, (hn, cn) = lstm(input, (h0, c0))

print('input shape =',input.shape)
print('hidden shape =',h0.shape)
print('output shape =',output.shape)
```

    input shape = torch.Size([5, 3, 10])
    hidden shape = torch.Size([2, 3, 20])
    output shape = torch.Size([5, 3, 20])

We now have 8 sets of weights to learn (i.e., the U and W for each of the 4
gates within each unit), whereas with simple recurrent units we only had 2. Training
these additional parameters imposes a much significantly higher training cost.

**Gated Recurrent Unit** or **GRU** ease this burden by dispensing with the
use of a separate context vector, and by reducing the number of gates to 2 — a reset gate, r and an update gate, z.

- **reset gate**:
$$ R_t = \sigma(X_tW_{xr} + H_{t-1}W_{hr} +b_r) \in \mathbb{R}^{n \times h}$$
- **update gate**:
$$ Z_t = \sigma(X_tW_{xz} + H_{t-1}W_{hz} +b_z) \in \mathbb{R}^{n \times h} $$

- **candidate hidden state**:
 $$ \tilde{H}_{t} = \text{tanh}\left(X_t W_{xh} + (R_t \odot H_{t-1})W_{hh} \right) + b_h$$
- **hidden state**:
$$ H_t = Z_t \odot H_{t-1} +(1-Z_t) \odot \tilde{H}_t. $$

* Whenever the update gate $Z_t$ is close to 1, we simply retain the old state. In this case the information from $X_t$ is essentially ignored, effectively skipping time step $t$ in the dependency chain.
* In contrast, whenever $Z_t$ is close to 0, the new latent state $H_t$ approaches the candidate latent state $\tilde{H}_t$.
* These designs can help us cope with the vanishing gradient problem in RNNs and better capture dependencies for sequences with large time step distances.

Here we have only 2 sets of weights to avoid the vanishing gradients problem, and it words pretty well,
note that both cells still have the output weights, so **RNN** has `2` sets of parameters, **LSTM** has `9` sets of parameters and **GRU** has `3` sets of parameters.

```python
def num_params(layer):
    return sum([p.numel() for p in layer.parameters()])

rnn = nn.RNN(10, 20, 2)
lstm = nn.LSTM(10, 20, 2)
gru = nn.GRU(10,20,2)
print(f'RNN params: {num_params(rnn)}')
print(f'LSTM params: {num_params(lstm)}')
print(f'GRU params: {num_params(gru)}')
```

    RNN params: 1480
    LSTM params: 5920
    GRU params: 4440

## BI-DIRECTIONAL RNN
---
This units solve to some degree the vanishing gradients. please note that non of them have resiliance to exploding gradients.

However, we didn't talk about the problem of future context.

NLP has very complex tasks that we can't cover in this tutorial and deserve a course of itself.

However, we can think of problems such as part-of speech (POS) taging or translation. and figure that for some tasks, future input is require in order to forcast the current prediction.

"Where are you meeting him? here by the rocks, and his favorite food is roasted fox!"

"איפה אתה פוגש אותו? כאן ליד הסלעים, והאוכל האהוב עליו הוא שועל צלוי!"

roasted fox start with adjective and then noun, while in hebrew it's more common to have the noun first and then the adjective. how can we translate the sentence if the hidden state does not contain the future noun?

For that we use Bi-directional RNN!

## Embedding
---

What is a **word embedding**? How do we get one?

Embeddings encode tokens as tensors in a way that maintain some **semantic** meaning for our task.

brief about ways to embed:

1. Embedding Layer: We train an embeding layer with our desired size, take a long time and require a lot of data, but learn the specific relations in our corpus

2. Word2Vec: statistical method for efficiently learning a standalone word embedding from a text corpus. it's based on CBOW (continues bag of words) and C-SKIP-GRAM (continues skip-gram).

3. GloVe (Global Vectors for Word Representation):extension to the word2vec method for efficiently learning word vectors. using matrix factorization techniques such as Latent Semantic Analysis (LSA). if Word2Vec use a window to determine the context, GloVe used the statistics of a word to accure in the hole text.

Here we will demostrate basic Embedding

```python
embedding_layer = nn.Embedding(num_embeddings=5, embedding_dim=8)

token_idx = torch.randint(low=0, high=5, size=(6,))
print(token_idx)
embedding_layer(token_idx)
```

    tensor([1, 4, 4, 0, 2, 0])

    tensor([[ 0.3224, -0.2507,  0.9565, -0.2492,  0.5129, -0.6653,  0.2410,  0.4221],
            [-0.4857,  1.8846,  1.0196, -1.3601,  1.5018,  0.0842,  0.0420,  0.2448],
            [-0.4857,  1.8846,  1.0196, -1.3601,  1.5018,  0.0842,  0.0420,  0.2448],
            [-0.7727,  0.9137,  0.3285,  1.0202, -0.3199, -2.1010,  1.7539, -0.1325],
            [-0.8582,  0.3344, -0.6477, -0.4448,  0.0501,  0.5932,  1.4060, -1.1410],
            [-0.7727,  0.9137,  0.3285,  1.0202, -0.3199, -2.1010,  1.7539, -0.1325]],
           grad_fn=<EmbeddingBackward>)

## Sentiment analysis for movie reviews
---

The task: Given a review about a movie written by some user, decide whether it's **positive**, **negative** or **neutral**.

Classically this is considered a challenging task if approached based on keywords alone.

Consider:

     "This movie was actually neither that funny, nor super witty."

To comprehend such a sentence, it's intuitive to see that some "state" must be kept when "reading" it.

So far we only worked with images or tabular data.
all the optimization process we lean on works with neumerical parameters, so what can we do to work with numbers again?

we will use the simple folowing steps:
* Load the dataset
* Decide on tokens and tokenize our dataset (words/ chars)
* Build a vocabulary from all existing tokens
* Encode each text/sentence into sequences of numerical numerical

For the sentiment analysis task, we will use `torchtext` to accelerate the preperation process.

We're goint to use IMDB dataset: this dataset contains movie reviews which are labeled as positive and negative.

If you want to load other datasets or load a custom dataset: https://torchtext.readthedocs.io/en/latest/datasets.html

Denote some Special tokens:
* \<sos> - start of sentence
* \<pad> - use to pad sentences so we can load batch of sentences with diffrent lenghts.
* \<eos> - end of a sentence.
* \<unk> - for workd that doesn't come in the train set at all or if the model thinks that no word in the vocabulary is good as next word

let's check what is the encoding assigned for `else` and what word belong to the token `10`

```python
review_parser = torchtext.data.Field(
    sequential=True, use_vocab=True, lower=True,
    init_token='<sos>', eos_token='<eos>', dtype=torch.long,
    tokenize='spacy', tokenizer_language='en_core_web_sm'
)

# This Field object converts the text labels into numeric values (0,1,2)
label_parser = torchtext.data.Field(
    is_target=True, sequential=False, unk_token=None, use_vocab=True
)
```

```python
# Load SST, tokenize the samples and labels
# ds_X are Dataset objects which will use the parsers to return tensors
ds_train, ds_valid, ds_test = torchtext.datasets.SST.splits(
    review_parser, label_parser, root=data_dir
)

n_train = len(ds_train)
print(f'Number of training samples: {n_train}')
print(f'Number of test     samples: {len(ds_test)}')
```

    Number of training samples: 8544
    Number of test     samples: 2210

```python
for i in ([111, 4321, 7777, 0]):
    example = ds_train[i]
    label = example.label
    review = str.join(" ", example.text)
    print(f'sample {i:04d} [{label}]:\nreview: {review}\n')
```

    sample 0111 [positive]:
    review: the film aims to be funny , uplifting and moving , sometimes all at once .

    sample 4321 [neutral]:
    review: the most anti - human big studio picture since 3000 miles to graceland .

    sample 7777 [negative]:
    review: an ugly , revolting movie .

    sample 0000 [positive]:
    review: the rock is destined to be the 21st century 's new ` ` conan '' and that he 's going to make a splash even greater than arnold schwarzenegger , jean - claud van damme or steven segal .

```python
hist = [ds_train[i].label for i in range(len(ds_train))]
plt.hist(hist);
```

```python
review_parser.build_vocab(ds_train)
label_parser.build_vocab(ds_train)
vocab = review_parser.vocab
print(f"Number of tokens in training samples: {len(review_parser.vocab)}")
print(f"Number of tokens in training labels: {len(label_parser.vocab)}")
```

    Number of tokens in training samples: 15482
    Number of tokens in training labels: 3

```python
print(f'first 20 tokens:\n', review_parser.vocab.itos[:20], end='\n\n')
```

    first 20 tokens:
     ['<unk>', '<pad>', '<sos>', '<eos>', '.', 'the', ',', 'a', 'and', 'of', 'to', '-', 'is', "'s", 'it', 'that', 'in', 'as', 'but', 'film']

```python
# Show that some words exist in the vocab
for w in ['film', 'actor', 'schwarzenegger', 'spielberg']:
    print(f'word={w:15s} index={review_parser.vocab.stoi[w]}')

for i in [19,492,3404,715]:
    print(f'index={str(i):14} is the word ={review_parser.vocab.itos[i]}')

```

    word=film            index=19
    word=actor           index=492
    word=schwarzenegger  index=3404
    word=spielberg       index=715
    index=19             is the word =film
    index=492            is the word =actor
    index=3404           is the word =schwarzenegger
    index=715            is the word =spielberg

```python
print(f'labels vocab:\n', dict(label_parser.vocab.stoi))
```

    labels vocab:
     {'positive': 0, 'negative': 1, 'neutral': 2}

```python
BATCH_SIZE = 4

# BucketIterator creates batches with samples of similar length
# to minimize the number of <pad> tokens in the batch.
dl_train, dl_valid, dl_test = torchtext.data.BucketIterator.splits(
    (ds_train, ds_valid, ds_test), batch_size=BATCH_SIZE,
    shuffle=True, device=device)

batch = next(iter(dl_train))

X, y = batch.text, batch.label
print('X = \n', X, X.shape, end='\n\n')
print('y = \n', y, y.shape)
```

    X =
     tensor([[    2,     2,     2,     2],
            [ 3744,   131,   420,   192],
            [10772,   243,     4,    18],
            [   55,    16,     3,    39],
            [    9,     7,     1,  1928],
            [    5,   108,     1,  1648],
            [ 4309,     6,     1,    16],
            [    9,     7,     1,     7],
            [  612,    22,     1,  3617],
            [   11,    67,     1,     4],
            [  660,   230,     1,     3],
            [ 5292,   480,     1,     1],
            [    8,    15,     1,     1],
            [14007,   411,     1,     1],
            [ 2106,   209,     1,     1],
            [    7,    62,     1,     1],
            [13348,    15,     1,     1],
            [   11,  1238,     1,     1],
            [ 4227,  5472,     1,     1],
            [14832,     9,     1,     1],
            [ 6616,  1044,     1,     1],
            [   15,    15,     1,     1],
            [ 2832,    48,     1,     1],
            [   20,  1079,     1,     1],
            [  768,  6410,     1,     1],
            [    6, 10378,     1,     1],
            [  877,     5,     1,     1],
            [    8,    65,     1,     1],
            [  487,    11,     1,     1],
            [ 6363,     5,     1,     1],
            [    4,  6552,     1,     1],
            [    3,     3,     1,     1]]) torch.Size([32, 4])

    y =
     tensor([2, 0, 1, 2]) torch.Size([4])

Our sample tensor `X` is of shape `(sentence_length, batch_size)`, in our case batch is 1.

Note that:
1. `sentence_length` is fixed since we picked a max_len, otherwise it will change every batch! You can re-run the previous blocks without max_len.
2. `sentence_length` dimension first,soon you will see why

### Model

We'll now create our sentiment analysis model based on the simple `RNNLayer` we've implemented above.

embedding_layer = nn.Embedding(num_embeddings=5, embedding_dim=8)

token_idx = torch.randint(low=0, high=5, size=(6,))
print(token_idx)
embedding_layer(token_idx)The model will:
- Take an input batch of tokenized sentences.
- Compute a dense word-embedding of each token.
- Process the sentence **sequentially** through the RNN layer.
- Produce a `(B, 3)` tensor, which we'll interpret as class probabilities for each sentence in the batch.

OK, model time:

```python
class SentimentRNN(nn.Module):
    def __init__(self, vocab_dim, embedding_dim, h_dim, out_dim):
        super().__init__()

        # nn.Embedding converts from token index to dense tensor
        self.embedding = nn.Embedding(vocab_dim, embedding_dim)

        # Our own Vanilla RNN layer, without phi_y so it outputs a class score
        self.rnn = RNNLayer(in_dim=embedding_dim, h_dim=h_dim, out_dim=out_dim, phi_y=None)

        # To convert class scores to log-probability we'll apply log-softmax
        self.log_softmax = nn.LogSoftmax(dim=1)

    def forward(self, X):
        # X shape: (S, B) Note batch dim is not first!

        embedded = self.embedding(X) # embedded shape: (S, B, E)

        # Loop over (batch of) tokens in the sentence(s)
        ht = None
        for xt in embedded:           # xt is (B, E)
            yt, ht = self.rnn(xt, ht) # yt is (B, D_out)

        # Class scores to log-probability
        yt_log_proba = self.log_softmax(yt)

        return yt_log_proba
```

Let's instantiate our model.

```python
INPUT_DIM = len(vocab)
EMBEDDING_DIM = 100
HIDDEN_DIM = 128
OUTPUT_DIM = 3

model = SentimentRNN(INPUT_DIM, EMBEDDING_DIM, HIDDEN_DIM, OUTPUT_DIM)
model
```

    SentimentRNN(
      (embedding): Embedding(15482, 100)
      (rnn): RNNLayer(
        (fc_xh): Linear(in_features=100, out_features=128, bias=False)
        (fc_hh): Linear(in_features=128, out_features=128, bias=True)
        (fc_hy): Linear(in_features=128, out_features=3, bias=True)
      )
      (log_softmax): LogSoftmax(dim=1)
    )

Test a manual forward pass:

```python
print(f'model(X) = \n', model(X), model(X).shape)
print(f'labels = ', y)
```

    model(X) =
     tensor([[-1.6985, -1.0844, -0.7362],
            [-1.5440, -1.2102, -0.7168],
            [-1.0753, -0.8943, -1.3868],
            [-1.0753, -0.8943, -1.3868]], grad_fn=<LogSoftmaxBackward>) torch.Size([4, 3])
    labels =  tensor([2, 0, 1, 2])

How big is our model?

```python
def count_parameters(model):
    return sum(p.numel() for p in model.parameters() if p.requires_grad)

print(f'The RNN model has {count_parameters(model):,} trainable weights.')
```

    The RNN model has 1,577,899 trainable weights.

Why so many? We used only one RNN layer.

Where are most of the weights?

### Training

Let's complete the example by showing the regular pytorch-style train loop with this model.

We'll run only a few epochs on a small subset just to test that it works.

```python
def train(model, optimizer, loss_fn, dataloader, max_epochs=100, max_batches=200):
    for epoch_idx in range(max_epochs):
        total_loss, num_correct = 0, 0
        start_time = time.time()

        for batch_idx, batch in enumerate(dataloader):
            X, y = batch.text, batch.label

            # Forward pass
            y_pred_log_proba = model(X)

            # Backward pass
            optimizer.zero_grad()
            loss = loss_fn(y_pred_log_proba, y)
            loss.backward()

            # Weight updates
            optimizer.step()

            # Calculate accuracy
            total_loss += loss.item()
            y_pred = torch.argmax(y_pred_log_proba, dim=1)
            num_correct += torch.sum(y_pred == y).float().item()

            if batch_idx == max_batches-1:
                break

        print(f"Epoch #{epoch_idx}, loss={total_loss /(max_batches):.3f}, accuracy={num_correct /(max_batches*BATCH_SIZE):.3f}, elapsed={time.time()-start_time:.1f} sec")
```

```python
import torch.optim as optim

rnn_model = SentimentRNN(INPUT_DIM, EMBEDDING_DIM, HIDDEN_DIM, OUTPUT_DIM).to(device)

optimizer = optim.Adam(rnn_model.parameters(), lr=1e-3)

# Recall: LogSoftmax + NLL is equiv to CrossEntropy on the class scores
loss_fn = nn.NLLLoss()

train(rnn_model, optimizer, loss_fn, dl_train, max_epochs=3) # Demo, please don't use it for your HW as is!
```

    Epoch #0, loss=1.112, accuracy=0.422, elapsed=4.0 sec
    Epoch #1, loss=1.064, accuracy=0.410, elapsed=3.6 sec
    Epoch #2, loss=1.062, accuracy=0.410, elapsed=3.5 sec

```python
def present_accuracy(model, dataloader,show=True):
    model.eval() # put in evaluation mode
    total_correct = 0
    total = 0
    confusion_matrix = np.zeros([3,3], int)
    with torch.no_grad():
        for data in dataloader:
            X, y = data.text.to(device), data.label.to(device)
            y_pred_log_proba = model(X)
            predicted = torch.argmax(y_pred_log_proba, dim=1)
            total += len(data)
            total_correct += (predicted == y).sum().item()
            for i, l in enumerate(y):
                confusion_matrix[l.item(), predicted[i].item()] += 1

    model_accuracy = total_correct / total * 100
    print("Test accuracy: {:.3f}%".format(model_accuracy))
    if show:
        labels = ('0','1','2')
        fig, ax = plt.subplots(1,1,figsize=(8,6))
        ax.matshow(confusion_matrix, aspect='auto', vmin=0, vmax=1000, cmap=plt.get_cmap('Blues'))
        plt.ylabel('Actual Category')
        plt.yticks(range(3), labels)
        plt.xlabel('Predicted Category')
        plt.xticks(range(3), labels)
        plt.show()

from sklearn import metrics

def print_stats(model, dataloader):
    model.eval() # put in evaluation mode
    trues = []
    preds = []
    with torch.no_grad():
        for data in dataloader:
            X, y = data.text.to(device), data.label.to(device)
            trues+=list(y.cpu())
            y_pred_log_proba = model(X)
            predicted = torch.argmax(y_pred_log_proba, dim=1)
            preds+= list(predicted.cpu())
    print(metrics.classification_report(trues, preds, digits=3))
```

```python
present_accuracy(rnn_model, dl_train)
```

```python
print_stats(rnn_model, dl_train)
```

                  precision    recall  f1-score   support

               0      0.423     0.966     0.588      3610
               1      0.387     0.034     0.062      3310
               2      0.000     0.000     0.000      1624

        accuracy                          0.421      8544
       macro avg      0.270     0.333     0.217      8544
    weighted avg      0.328     0.421     0.272      8544

#### Limitations

As usual this is a very naïve model, just for demonstration.
It lacks many tricks of the NLP trade, such was pre-trained embeddings,
gated RNN units, deep or bi-directional models, dropout, etc.

Don't expect SotA results :)

## Time-Series Forecasting: Predicting Stock Prices Using An LSTM Model
---

Now we learned how to work with words and documents, how about making money by forecasting in NASDAQ?

as a disclaimer, i will try to explain, why it's nearly impossible to beat the market, but it's still a great tool to be familiar with, and who knows, maybe you would think out of the box and beat the market, just don't forget to tip your favorit T.A after :)

we will run this section regardless on the tutorial, so we're going to import all libraries again...

note that `alpha_vantage` just help us to get the stocks data from https://www.alphavantage.co/.

if you don't want it as part of your enviroment, just use `conda update` with the json file again...

### Attention (not the mechanisem :))

**this is not a recommandation to use alpha_vantage API, it's you can use yahoo database from pandas or whatever other API, or even scrape the data yourself**

```python
! pip install alpha_vantage -q

# pip install numpy
import numpy as np

# pip install torch
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data import Dataset
from torch.utils.data import DataLoader

# pip install matplotlib
import matplotlib.pyplot as plt
from matplotlib.pyplot import figure

# pip install alpha_vantage
from alpha_vantage.timeseries import TimeSeries

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
```

now we need to determine some parameters, and we're going to use a dictionary config for that.

when usually we deal only with batch size and learning rate, here we devide the keys to some sections (so we actually have a dict of dicts)

- **alpha_vantage**: what params we want to use (here we use only the closing price and only IBM symbole)
- **data**: we determine window size, as the input seq len for the model and the split size for train and validation
- **plots**: just some parameters for the plots
- **model**: some model params
- **training**: some parameters for the training process

```python
config = {
    "alpha_vantage": {
        "key": "YOUR_API_KEY", # Claim your free API key here: https://www.alphavantage.co/support/#api-key
        "symbol": "IBM",   #can use APPL or anything you would like
        "outputsize": "full",
        "key_adjusted_close": "5. adjusted close",
    },
    "data": {
        "window_size": 20,
        "train_split_size": 0.80,
    },
    "plots": {
        "show_plots": True,
        "xticks_interval": 90,
        "color_actual": "#001f3f",
        "color_train": "#3D9970",
        "color_val": "#0074D9",
        "color_pred_train": "#3D9970",
        "color_pred_val": "#0074D9",
        "color_pred_test": "#FF4136",
    },
    "model": {
        "input_size": 1, # since we are only using 1 feature, close price
        "num_lstm_layers": 2,
        "lstm_size": 32,
        "dropout": 0.2,
    },
    "training": {
        "device": "cpu", # "cuda" or "cpu"
        "batch_size": 64,
        "num_epoch": 100,
        "learning_rate": 0.01,
        "scheduler_step_size": 40,
    }
}
```

```python
def download_data(config, plot=False):
    # get the data from alpha vantage

    ts = TimeSeries(key=config["alpha_vantage"]["key"])
    data, meta_data = ts.get_daily_adjusted(config["alpha_vantage"]["symbol"], outputsize=config["alpha_vantage"]["outputsize"])

    data_date = [date for date in data.keys()]
    data_date.reverse()

    data_close_price = [float(data[date][config["alpha_vantage"]["key_adjusted_close"]]) for date in data.keys()]
    data_close_price.reverse()
    data_close_price = np.array(data_close_price)

    num_data_points = len(data_date)
    display_date_range = "from " + data_date[0] + " to " + data_date[num_data_points-1]
    print("Number data points:", num_data_points, display_date_range)

    if plot:
        fig = figure(figsize=(25, 5), dpi=80)
        fig.patch.set_facecolor((1.0, 1.0, 1.0))
        plt.plot(data_date, data_close_price, color=config["plots"]["color_actual"])
        xticks = [data_date[i] if ((i%config["plots"]["xticks_interval"]==0 and (num_data_points-i) > config["plots"]["xticks_interval"]) or i==num_data_points-1) else None for i in range(num_data_points)] # make x ticks nice
        x = np.arange(0,len(xticks))
        plt.xticks(x, xticks, rotation='vertical')
        plt.title("Daily close price for " + config["alpha_vantage"]["symbol"] + ", " + display_date_range)
        plt.grid(b=None, which='major', axis='y', linestyle='--')
        plt.show()

    return data_date, data_close_price, num_data_points, display_date_range

data_date, data_close_price, num_data_points, display_date_range = download_data(config, plot=config["plots"]["show_plots"])
```

```python
class Normalizer():
    def __init__(self):
        self.mu = None
        self.sd = None

    def fit_transform(self, x):
        self.mu = np.mean(x, axis=(0), keepdims=True)
        self.sd = np.std(x, axis=(0), keepdims=True)
        normalized_x = (x - self.mu)/self.sd
        return normalized_x

    def inverse_transform(self, x):
        return (x*self.sd) + self.mu

# normalize
scaler = Normalizer()
normalized_data_close_price = scaler.fit_transform(data_close_price)
```

fit_transform don't change the distribution, but simply normelize it, so we won't predict too high numbers...

```python
plt.hist(data_close_price);
```

```python
plt.hist(normalized_data_close_price);
```

```python
def prepare_data_x(x, window_size):
    # perform windowing
    n_row = x.shape[0] - window_size + 1
    output = np.lib.stride_tricks.as_strided(x, shape=(n_row,window_size), strides=(x.strides[0],x.strides[0]))
    return output[:-1], output[-1]

def prepare_data_y(x, window_size):
    # # perform simple moving average
    # output = np.convolve(x, np.ones(window_size), 'valid') / window_size

    #even easier, use the next day as label
    output = x[window_size:]
    return output

def prepare_data(normalized_data_close_price, config, plot=False):
    data_x, data_x_unseen = prepare_data_x(normalized_data_close_price, window_size=config["data"]["window_size"])
    data_y = prepare_data_y(normalized_data_close_price, window_size=config["data"]["window_size"])

    # split dataset

    split_index = int(data_y.shape[0]*config["data"]["train_split_size"])
    data_x_train = data_x[:split_index]
    data_x_val = data_x[split_index:]
    data_y_train = data_y[:split_index]
    data_y_val = data_y[split_index:]

    if plot:
        # prepare data for plotting

        to_plot_data_y_train = np.zeros(num_data_points)
        to_plot_data_y_val = np.zeros(num_data_points)

        to_plot_data_y_train[config["data"]["window_size"]:split_index+config["data"]["window_size"]] = scaler.inverse_transform(data_y_train)
        to_plot_data_y_val[split_index+config["data"]["window_size"]:] = scaler.inverse_transform(data_y_val)

        to_plot_data_y_train = np.where(to_plot_data_y_train == 0, None, to_plot_data_y_train)
        to_plot_data_y_val = np.where(to_plot_data_y_val == 0, None, to_plot_data_y_val)

        ## plots

        fig = figure(figsize=(25, 5), dpi=80)
        fig.patch.set_facecolor((1.0, 1.0, 1.0))
        plt.plot(data_date, to_plot_data_y_train, label="Prices (train)", color=config["plots"]["color_train"])
        plt.plot(data_date, to_plot_data_y_val, label="Prices (validation)", color=config["plots"]["color_val"])
        xticks = [data_date[i] if ((i%config["plots"]["xticks_interval"]==0 and (num_data_points-i) > config["plots"]["xticks_interval"]) or i==num_data_points-1) else None for i in range(num_data_points)] # make x ticks nice
        x = np.arange(0,len(xticks))
        plt.xticks(x, xticks, rotation='vertical')
        plt.title("Daily close prices for " + config["alpha_vantage"]["symbol"] + " - showing training and validation data")
        plt.grid(b=None, which='major', axis='y', linestyle='--')
        plt.legend()
        plt.show()

    return split_index, data_x_train, data_y_train, data_x_val, data_y_val, data_x_unseen

split_index, data_x_train, data_y_train, data_x_val, data_y_val, data_x_unseen = prepare_data(normalized_data_close_price, config, plot=config["plots"]["show_plots"])
```

```python
class TimeSeriesDataset(Dataset):
    def __init__(self, x, y):
        x = np.expand_dims(x, 2) # in our case, we have only 1 feature, so we need to convert `x` into [batch, sequence, features] for the model
        self.x = x.astype(np.float32)
        self.y = y.astype(np.float32)

    def __len__(self):
        return len(self.x)

    def __getitem__(self, idx):
        return (self.x[idx], self.y[idx])

dataset_train = TimeSeriesDataset(data_x_train, data_y_train)
dataset_val = TimeSeriesDataset(data_x_val, data_y_val)

print("Train data shape", dataset_train.x.shape, dataset_train.y.shape)
print("Validation data shape", dataset_val.x.shape, dataset_val.y.shape)
```

    Train data shape (4428, 20, 1) (4428,)
    Validation data shape (1107, 20, 1) (1107,)

```python
class GRU_stocks_Model(nn.Module):
    def __init__(self, input_size=1, hidden_layer_size=32, num_layers=2, output_size=1, dropout=0.2):
        super().__init__()
        self.hidden_layer_size = hidden_layer_size

        self.linear_1 = nn.Linear(input_size, hidden_layer_size)
        self.relu = nn.ReLU()
        self.gru = nn.GRU(hidden_layer_size, hidden_size=self.hidden_layer_size, num_layers=num_layers, batch_first=True)
        self.dropout = nn.Dropout(dropout)
        self.linear_2 = nn.Linear(num_layers*hidden_layer_size, output_size)

        self.init_weights()

    def init_weights(self):
        for name, param in self.gru.named_parameters():
            if 'bias' in name:
                 nn.init.constant_(param, 0.0)
            elif 'weight_ih' in name:
                 nn.init.kaiming_normal_(param)
            elif 'weight_hh' in name:
                 nn.init.orthogonal_(param)

    def forward(self, x):
        batchsize = x.shape[0]

        # layer 1
        x = self.linear_1(x)
        x = self.relu(x)

        #gru
        #if you choose to use lstm:
        #lstm_out, (h_n, c_n) = self.lstm(x)
        gru_out, h_n = self.gru(x)

        # reshape output from hidden cell into [batch, features] for `linear_2`
        x = h_n.permute(1, 0, 2).reshape(batchsize, -1)

        # layer 2
        x = self.dropout(x)
        predictions = self.linear_2(x)
        return predictions[:,-1]

stocks_Model = GRU_stocks_Model(input_size=config["model"]["input_size"], hidden_layer_size=config["model"]["lstm_size"], num_layers=config["model"]["num_lstm_layers"], output_size=1, dropout=config["model"]["dropout"])
stocks_Model = stocks_Model.to(config["training"]["device"])
```

```python
def run_epoch(dataloader, is_training=False):
    epoch_loss = 0

    if is_training:
        stocks_Model.train()
    else:
        stocks_Model.eval()

    for idx, (x, y) in enumerate(dataloader):
        if is_training:
            optimizer.zero_grad()

        batchsize = x.shape[0]

        x = x.to(config["training"]["device"])
        y = y.to(config["training"]["device"])

        out = stocks_Model(x)
        loss = criterion(out.contiguous(), y.contiguous())

        if is_training:
            loss.backward()
            optimizer.step()

        epoch_loss += (loss.detach().item() / batchsize)

    lr = scheduler.get_last_lr()[0]

    return epoch_loss, lr

# create `DataLoader`
train_dataloader = DataLoader(dataset_train, batch_size=config["training"]["batch_size"], shuffle=True)
val_dataloader = DataLoader(dataset_val, batch_size=config["training"]["batch_size"], shuffle=True)

# define optimizer, scheduler and loss function
criterion = nn.MSELoss()
optimizer = optim.Adam(stocks_Model.parameters(), lr=config["training"]["learning_rate"], betas=(0.9, 0.98), eps=1e-9)
scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=config["training"]["scheduler_step_size"], gamma=0.1)

# begin training
for epoch in range(config["training"]["num_epoch"]):
    loss_train, lr_train = run_epoch(train_dataloader, is_training=True)
    loss_val, lr_val = run_epoch(val_dataloader)
    scheduler.step()
    if epoch%10 ==0:
        print('Epoch[{}/{}] | loss train:{:.6f}, test:{:.6f} | lr:{:.6f}'
                  .format(epoch+1, config["training"]["num_epoch"], loss_train, loss_val, lr_train))

```

    Epoch[1/100] | loss train:0.053583, test:0.002071 | lr:0.010000
    Epoch[11/100] | loss train:0.009592, test:0.006677 | lr:0.010000
    Epoch[21/100] | loss train:0.008554, test:0.001540 | lr:0.010000
    Epoch[31/100] | loss train:0.007705, test:0.001590 | lr:0.010000
    Epoch[41/100] | loss train:0.007116, test:0.001127 | lr:0.001000
    Epoch[51/100] | loss train:0.006314, test:0.001177 | lr:0.001000
    Epoch[61/100] | loss train:0.007098, test:0.001091 | lr:0.001000
    Epoch[71/100] | loss train:0.006655, test:0.001268 | lr:0.001000
    Epoch[81/100] | loss train:0.006238, test:0.001081 | lr:0.000100
    Epoch[91/100] | loss train:0.006443, test:0.001077 | lr:0.000100

```python
# here we re-initialize dataloader so the data doesn't shuffled, so we can plot the values by date

train_dataloader = DataLoader(dataset_train, batch_size=config["training"]["batch_size"], shuffle=False)
val_dataloader = DataLoader(dataset_val, batch_size=config["training"]["batch_size"], shuffle=False)

stocks_Model.eval()

# predict on the training data, to see how well the model managed to learn and memorize

predicted_train = np.array([])

for idx, (x, y) in enumerate(train_dataloader):
    x = x.to(config["training"]["device"])
    out = stocks_Model(x)
    out = out.cpu().detach().numpy()
    predicted_train = np.concatenate((predicted_train, out))

# predict on the validation data, to see how the model does

predicted_val = np.array([])

for idx, (x, y) in enumerate(val_dataloader):
    x = x.to(config["training"]["device"])
    out = stocks_Model(x)
    out = out.cpu().detach().numpy()
    predicted_val = np.concatenate((predicted_val, out))

if config["plots"]["show_plots"]:

    # prepare data for plotting, show predicted prices

    to_plot_data_y_train_pred = np.zeros(num_data_points)
    to_plot_data_y_val_pred = np.zeros(num_data_points)

    to_plot_data_y_train_pred[config["data"]["window_size"]:split_index+config["data"]["window_size"]] = scaler.inverse_transform(predicted_train)
    to_plot_data_y_val_pred[split_index+config["data"]["window_size"]:] = scaler.inverse_transform(predicted_val)

    to_plot_data_y_train_pred = np.where(to_plot_data_y_train_pred == 0, None, to_plot_data_y_train_pred)
    to_plot_data_y_val_pred = np.where(to_plot_data_y_val_pred == 0, None, to_plot_data_y_val_pred)

    # plots

    fig = figure(figsize=(25, 5), dpi=80)
    fig.patch.set_facecolor((1.0, 1.0, 1.0))
    plt.plot(data_date, data_close_price, label="Actual prices", color=config["plots"]["color_actual"])
    plt.plot(data_date, to_plot_data_y_train_pred, label="Predicted prices (train)", color=config["plots"]["color_pred_train"])
    plt.plot(data_date, to_plot_data_y_val_pred, label="Predicted prices (validation)", color=config["plots"]["color_pred_val"])
    plt.title("Compare predicted prices to actual prices")
    xticks = [data_date[i] if ((i%config["plots"]["xticks_interval"]==0 and (num_data_points-i) > config["plots"]["xticks_interval"]) or i==num_data_points-1) else None for i in range(num_data_points)] # make x ticks nice
    x = np.arange(0,len(xticks))
    plt.xticks(x, xticks, rotation='vertical')
    plt.grid(b=None, which='major', axis='y', linestyle='--')
    plt.legend()
    plt.show()

    # prepare data for plotting, zoom in validation

    to_plot_data_y_val_subset = scaler.inverse_transform(data_y_val)
    to_plot_predicted_val = scaler.inverse_transform(predicted_val)
    to_plot_data_date = data_date[split_index+config["data"]["window_size"]:]

    # plots

    fig = figure(figsize=(25, 5), dpi=80)
    fig.patch.set_facecolor((1.0, 1.0, 1.0))
    plt.plot(to_plot_data_date, to_plot_data_y_val_subset, label="Actual prices", color=config["plots"]["color_actual"])
    plt.plot(to_plot_data_date, to_plot_predicted_val, label="Predicted prices (validation)", color=config["plots"]["color_pred_val"])
    plt.title("Zoom in to examine predicted price on validation data portion")
    xticks = [to_plot_data_date[i] if ((i%int(config["plots"]["xticks_interval"]/5)==0 and (len(to_plot_data_date)-i) > config["plots"]["xticks_interval"]/6) or i==len(to_plot_data_date)-1) else None for i in range(len(to_plot_data_date))] # make x ticks nice
    xs = np.arange(0,len(xticks))
    plt.xticks(xs, xticks, rotation='vertical')
    plt.grid(b=None, which='major', axis='y', linestyle='--')
    plt.legend()
    plt.show()
```

```python
# predict on the unseen data, tomorrow's price

stocks_Model.eval()

x = torch.tensor(data_x_unseen).float().to(config["training"]["device"]).unsqueeze(0).unsqueeze(2) # this is the data type and shape required, [batch, sequence, feature]
prediction = stocks_Model(x)
prediction = prediction.cpu().detach().numpy()
prediction = scaler.inverse_transform(prediction)[0]

if config["plots"]["show_plots"]:

    # prepare plots

    plot_range = 10
    to_plot_data_y_val = np.zeros(plot_range)
    to_plot_data_y_val_pred = np.zeros(plot_range)
    to_plot_data_y_test_pred = np.zeros(plot_range)

    to_plot_data_y_val[:plot_range-1] = scaler.inverse_transform(data_y_val)[-plot_range+1:]
    to_plot_data_y_val_pred[:plot_range-1] = scaler.inverse_transform(predicted_val)[-plot_range+1:]

    to_plot_data_y_test_pred[plot_range-1] = prediction

    to_plot_data_y_val = np.where(to_plot_data_y_val == 0, None, to_plot_data_y_val)
    to_plot_data_y_val_pred = np.where(to_plot_data_y_val_pred == 0, None, to_plot_data_y_val_pred)
    to_plot_data_y_test_pred = np.where(to_plot_data_y_test_pred == 0, None, to_plot_data_y_test_pred)

    # plot

    plot_date_test = data_date[-plot_range+1:]
    plot_date_test.append("next trading day")

    fig = figure(figsize=(25, 5), dpi=80)
    fig.patch.set_facecolor((1.0, 1.0, 1.0))
    plt.plot(plot_date_test, to_plot_data_y_val, label="Actual prices", marker=".", markersize=10, color=config["plots"]["color_actual"])
    plt.plot(plot_date_test, to_plot_data_y_val_pred, label="Past predicted prices", marker=".", markersize=10, color=config["plots"]["color_pred_val"])
    plt.plot(plot_date_test, to_plot_data_y_test_pred, label="Predicted price for next day", marker=".", markersize=20, color=config["plots"]["color_pred_test"])
    plt.title("Predicted close price of the next trading day")
    plt.grid(b=None, which='major', axis='y', linestyle='--')
    plt.legend()
    plt.show()

print("Predicted close price of the next trading day:", round(prediction, 2))
```

## Links
---

[RNN vs LSTM vs GRU](https://medium.com/analytics-vidhya/rnn-vs-gru-vs-lstm-863b0b7b1573)

[LSTM and Bidirectional LSTM for Regression](https://towardsdatascience.com/lstm-and-bidirectional-lstm-for-regression-4fddf910c655)

https://nlp.stanford.edu/pubs/glove.pdf glove

## Credits
---

This tutorial was written by [Moshe Kimhi](https://www.linkedin.com/in/moshekimhi/)

To re-use, please provide attribution and link to the original.

Stock prediction from https://www.alphavantage.co/academy/

some of the RNN images from tutorials given by [Aviv Rosenberg](https://avivr.net).

Some images in this tutorial were taken and/or adapted from the following sources:
- http://d2l.ai
- http://giphy.com
- Fundamentals of Deep Learning, Nikhil Buduma, Oreilly 2017
- Sebastian Ruder, "On word embeddings - Part 1", 2016, https://ruder.io
- http://karpathy.github.io
- Stanford cs231n

```python

```


**Test:** Including numpy functions in package

```python
from pysolve3.model import Model
from pysolve3.utils import SolveSFC
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
```

```python
def teste():
    from sympy import lambdify
    import sympy as sp
    x = sp.Symbol('x')
    mod = Model()
    mod.set_var_default(0)

    mod.var('y')

    mod.add('d(y) = normal(normal(1,1,100)[randint(0,100)],1)')

    return mod
SolveSFC(teste())
```

      _y__1
      y

      0
      NaN
      NaN

      1
      0.000000
      1.136278

      2
      1.136278
      1.136278

      3
      1.136278
      1.136278

      4
      1.136278
      1.136278

      5
      1.136278
      1.136278

      6
      1.136278
      1.136278

      7
      1.136278
      1.136278

      8
      1.136278
      1.136278

      9
      1.136278
      1.136278

      10
      1.136278
      1.136278

      11
      1.136278
      1.136278

      12
      1.136278
      1.136278

      13
      1.136278
      1.136278

      14
      1.136278
      1.136278

      15
      1.136278
      1.136278

      16
      1.136278
      1.136278

      17
      1.136278
      1.136278

      18
      1.136278
      1.136278

      19
      1.136278
      1.136278

      20
      1.136278
      1.136278

      21
      1.136278
      1.136278

      22
      1.136278
      1.136278

      23
      1.136278
      1.136278

      24
      1.136278
      1.136278

      25
      1.136278
      1.136278

      26
      1.136278
      1.136278

      27
      1.136278
      1.136278

      28
      1.136278
      1.136278

      29
      1.136278
      1.136278

      ...
      ...
      ...

      471
      1.136278
      1.136278

      472
      1.136278
      1.136278

      473
      1.136278
      1.136278

      474
      1.136278
      1.136278

      475
      1.136278
      1.136278

      476
      1.136278
      1.136278

      477
      1.136278
      1.136278

      478
      1.136278
      1.136278

      479
      1.136278
      1.136278

      480
      1.136278
      1.136278

      481
      1.136278
      1.136278

      482
      1.136278
      1.136278

      483
      1.136278
      1.136278

      484
      1.136278
      1.136278

      485
      1.136278
      1.136278

      486
      1.136278
      1.136278

      487
      1.136278
      1.136278

      488
      1.136278
      1.136278

      489
      1.136278
      1.136278

      490
      1.136278
      1.136278

      491
      1.136278
      1.136278

      492
      1.136278
      1.136278

      493
      1.136278
      1.136278

      494
      1.136278
      1.136278

      495
      1.136278
      1.136278

      496
      1.136278
      1.136278

      497
      1.136278
      1.136278

      498
      1.136278
      1.136278

      499
      1.136278
      1.136278

      500
      1.136278
      1.136278

501 rows × 2 columns


```
#@title Copyright 2020 The Cirq Developers
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
```

# Quantum approximate optimization algorithm for the Ising model

    View on QuantumAI

    Run in Google Colab

    View source on GitHub

    Download notebook

This notebook provides an introduction to the Quantum Approximate Optimization Algorithm (QAOA) using Cirq. The presentation mostly follows [Farhi et al](https://arxiv.org/abs/1411.4028). We will show how to construct the QAOA circuit and use it to solve optimization problems.

```
try:
    import cirq
except ImportError:
    print("installing cirq...")
    !pip install cirq --quiet
    import cirq
    print("installed cirq.")
```

```
import numpy as np
import sympy
import matplotlib.pyplot as plt
```

## Description of the QAOA

Suppose we have a function $C(z)$ which we want to minimize. Here, $z$ denotes a collection of variables $z = z_1,z_2,\ldots, z_n$, where each $z_j$ can be either $+1$ or $-1$. (The important thing here is that each $z_j$ has *two possible values*. By convention we choose those values to be $\pm 1$). The QAOA is a general-purpose algorithm whose goal is to produce an assignment of the $z_j$ that gives a relatively low value of $C(z)$. It's not guaranteed to give the lowest possible value of $C(z)$ (hence "approximate" in the name) except in a particular limit which we will discuss.

The QAOA acts on $n$ qubits where, as you might guess, each qubit represents one of the variables $z_1, ..., z_n$, and the $2^n$ states of the computational basis correspond to the $2^n$ possible assignments of the $z$ variables. To be more specific, let's agree that the value of $z_j$ corresponds to the measurement outcome of the Pauli-$Z$ operator on the $j$th qubit. (There is a potential confusion here because the state $| 0 \rangle$ corresponds to $z = +1$, while the state $| 1\rangle$ corresponds to $z=-1$. This is unfortunate, but is something that we'll just have to deal with.)

The QAOA is fairly simple to explain, though the reasons behind why it works are not obvious at first glance. As usual, we begin with all of our qubits initialized in the $|0\rangle$ state. The first step is to act with $H^{\otimes n}$, the Hadamard operator on each qubit. This prepares an equal superposition of all bitstrings, i.e., an equal superposition of all possible $z$ assignments:

$$
H^{\otimes n} |0^n\rangle =\frac{1}{2^{n/2}} \sum_{z \in \{0,1\}^n} |z\rangle.
$$

This should be thought of as the "real" initial state of the algorithm (as opposed to thinking of the $|0\rangle$ state as the initial state). The point of the remaining steps is to affect the amplitudes such that those with small $C(z)$ values grow while those with large $C(z)$ values shrink. In this way, when we measure the qubits we'll be more likely to find a bitstring with a small value of $C(z)$.

The meat of the algorithm relies on the following unitary operator:

$$
U(\gamma, C) = e^{i \pi \gamma C(Z) / 2} .
$$

This operator deserves some explanation. First, $\gamma$ is a parameter which we will later treat as a variational parameter, adjusting its value to produce the best possible result. $C$ here is the function we are trying to minimize, and the notation $C(Z)$ is supposed to tell you to plug in the Pauli-$Z$ operator for each qubit in place of the argument $z$. For example, if

$$
C(z) = 3z_1 z_2 - z_2z_3 + z_4 ,
$$

then

$$
C(Z) = 3Z_1 Z_2 - Z_2Z_3 + Z_4.
$$

It doesn't look like much happened, but the point here is that $C(z)$ is a number while $C(Z)$ is a matrix. That matrix is diagonal in the computational basis, and those diagonal entries represent all the possible values of $C(z)$.

After acting with $H^{\otimes n}$, we act with $U(\gamma, C)$. The result is still a sum over all possible bit-strings, but now the coefficients are complex phases which depend on $C$. At this point there is still an equal probability to measure any particular string, though, because Born's rule only depends on the square of the amplitude. So, the algorithm is not done yet.

The next step of the algorithm is to act with the unitary operator

$$
U(\beta,B) = e^{i\pi\beta B/2},~~~ B = \sum_{j=1}^n X_j ,
$$

where $\beta$ is another variational parameter. Since the Pauli-$X$ operators on each qubit commute with each other, we can alternatively write this as

$$
U(\beta, B) = \prod_{j=1}^n e^{i\pi\beta X_j/2} .
$$

So this is just a rotation of each qubit around the $X$-axis on the Bloch sphere by an amount determined by $\beta$. This operation is _not_ diagonal in the computational basis, and the resulting state will not be an equal superposition over all bitstrings. So after this step there will be constructive and destructive interference, which hopefully leads to enhancement of states corresponding to small values of $C$. This $U(\beta, B)$ is sometimes called a "mixing" operation. Note that, up to an inconsequential global phase, we can also write

$$
U(\beta, B) = \prod_{j=1}^n X_j^{\beta} .
$$

The total circuit consists of repeating the previous two steps a total of $p\geq 1$ times, where the choice of $p$ is up to you. The parameters $\gamma$ and $\beta$ can be chosen independently at each step. So at the conclusion of the circuit, the state of the qubits is

$$
|\gamma,\beta\rangle = U(\beta_p,B)U(\gamma_p,C)\cdots U(\beta_1,B)U(\gamma_1,C)H^{\otimes n}|0^n\rangle.
$$

If we choose $\gamma$ and $\beta$ so that the expectation value

$$
F(\gamma,\beta) = \langle \gamma,\beta|C(Z)|\gamma,\beta\rangle
$$

is minimized, then measuring the state $|\gamma,\beta\rangle$ in the computational basis gives us a good candidate bitstring for the minimum of $C(z)$. That's the whole thing!

In summary we have to perform the following tasks in order to implement the QAOA:

1.   Figure out how to perform the $U(\gamma, C)$ operation in Cirq for our choice of $C$.
2.   Create a quantum circuit alternating $U(\gamma, C)$ and $U(\beta, B)$ operations as many times as desired.
3.   Find the optimal value of the variational parameters in our circuit.
4.   Measure the output of our circuit.

## Toy problem: Ground state of the Ising model

The Ising Model defines the energy function

$$
E = -\sum_{\langle i,j \rangle} Z_i Z_j - \sum_i h_i Z_i,
$$

where the notation $\langle i,j\rangle$ means a sum over all nearest-neighbor pairs. The picture here is that the qubits live on the vertices of a graph, and the edges of the graph define which qubits are neighbors. We'll just take our graph to be a rectangular lattice with some number of rows and some number of columns. The numbers $h_i$ have the physical interpretation of an external magnetic field.

We are interested in finding a low-lying state of the Ising Model, by which we mean a state that has a relatively low amount of energy. This is a difficult problem in general. The pairwise interaction terms would tell you that neighboring qubits should be in the same state to lower the energy, while the magnetic field terms tell you that a given qubit wants to point "in the same direction" as its local field, and the strength of that preference depends on the magnitude of the field. These two different kinds of pressure are not always in agreement!

This type of problem is a perfect candidate for the QAOA, where we use the energy $E$ as our cost function $C$.

### Implementing $U(\gamma, C)$

The first thing we need to do is create the operation $U(\gamma, C)$ where $C$ is equal to the Ising model energy function. Note that since all terms in the energy function commute, we can decompose this operation as

$$
U(\gamma, C) = \prod_{\langle i,j\rangle}e^{-i\pi\gamma Z_iZ_j/2} \prod_i e^{-i\pi \gamma h_i Z_i/2}.
$$

This requires that we have the two-qubit gate $\exp(-i\pi\gamma ZZ/2)$ at our disposal. In matrix form, this is

$$
\begin{align}
\exp(-i \pi\gamma Z\otimes Z/2) = \begin{bmatrix}
e^{-i\pi \gamma/2}  & 0  &0&0\\
0 & e^{i\pi \gamma/2}   &0&0\\
0&0& e^{i\pi \gamma/2}  &0 \\
0&0 & 0  & e^{-i\pi \gamma/2}
\end{bmatrix}
\end{align}
$$

Cirq has a built-in gate `cirq.ZZ` which is equivalent to this once you account for a global phase.

```
"""Example of using the ZZ gate."""
# Get two qubits.
a = cirq.NamedQubit("a")
b = cirq.NamedQubit("b")

# Pick a value for gamma.
gamma = 0.3

# Display the circuit.
circuit = cirq.Circuit(cirq.ZZ(a,b)**gamma)
print(circuit)
```

We should also check that the unitary is what we expect.

```
"""Check the unitary of the ZZ gate."""
# Unitary we want to implement.
expected_unitary = np.array([
    [np.exp(-1j * np.pi * gamma / 2),0, 0, 0],
    [0, np.exp(1j * np.pi * gamma / 2), 0, 0],
    [0, 0, np.exp(1j * np.pi * gamma / 2), 0],
    [0, 0, 0, np.exp(-1j * np.pi * gamma / 2)]
])

# Unitary we are implementing.
actual_unitary = cirq.unitary(circuit)

# Check that they are equal up to global phase.
cirq.testing.assert_allclose_up_to_global_phase(
    actual_unitary, expected_unitary, atol=1e-5
)
```

The magnetic field terms can be handled in a similar way. The single-qubit unitary

$$
\exp(-i\pi \gamma hZ/2) = \begin{bmatrix}
e^{-i\pi \gamma h/2} & 0 \\
0 & e^{i\pi \gamma h/2}
\end{bmatrix}
$$

is equivalent to `cirq.Z**(h*gamma)` up to global phase.

```
"""Example of using the Z gate."""
# Value of the external magenetic field.
h = 1.3

# Display the circuit.
circuit = cirq.Circuit(cirq.Z(a)**(gamma*h))
print(circuit)
```

As before, we can check this unitary is what we expect.

```
"""Check the unitary of the Z gate."""
# Unitary we want to implement.
expected_unitary = np.array([
    [np.exp(-1j*np.pi*gamma*h/2), 0],
    [0, np.exp(1j*np.pi*gamma*h/2)]
])

# Unitary we are implementing.
actual_unitary = cirq.unitary(circuit)

# Check that they are equal up to global phase.
cirq.testing.assert_allclose_up_to_global_phase(
    expected_unitary, actual_unitary, atol=1e-5
)
```

Now that we know how to implement both $Z_i Z_j$ and $Z_i$, we can implement the full $U(\gamma, C)$ operator.

### Exercise: More general two-qubit gate

The Ising Model is particularly simple because the nearest-neighbor interaction $Z_i Z_j$ is already given in terms of a product of Pauli matrices. But suppose instead that the cost function was a sum of terms that looked like

$$
C(z_i,z_j) = \begin{cases}
c_{00} \text{ if } z_i =1,~z_j=1,\\
c_{01} \text{ if } z_i =1,~z_j=-1,\\
c_{10} \text{ if } z_i =-1,~z_j=1,\\
c_{11} \text{ if } z_i =-1,~z_j=-1
\end{cases}
$$

for some numbers $c_{ab}$. How would you make the analogous two-qubit gate for this case? You can either make a custom gate from scratch, or build a solution from the standard elementary gates.

```
# Your code here!
```

### Implementing the full circuit

We will focus on the Ising model on a rectangular lattice with an arbitrary number of rows and columns. Here are some things to think about:

1.   `cirq.GridQubit`s are natural because our qubits actually do live on a grid. Cirq does not care what kind of qubit you make, though.
2.   It's a good idea to define separate functions to place the C and B layers for the circuit. Really these should be generators that yield the required gates.
3.   You might consider wrapping everything inside a class. We won't do that here, but if you want to play around with different numbers of rows/columns or different numbers of B/C layers it can be convenient.

First, we'll define the basic parameters of our model and the generators for the different layers.

```
"""Define problem parameters and get a set of GridQubits."""
# Set the dimensions of the grid.
n_cols = 3
n_rows = 3

# Set the value of the external magnetic field at each site.
h = 0.5 * np.ones((n_rows, n_cols))

# Arranging the qubits in a list-of-lists like this makes them easy to refer to later.
qubits = [[cirq.GridQubit(i, j) for j in range(n_cols)] for i in range(n_rows)]
```

```
def gamma_layer(gamma_value, h):
    """Generator for U(gamma, C) layer of QAOA

    Args:
        gamma: Float variational parameter for the circuit
        h: Array of floats of external magnetic field values
    """
    for i in range(n_rows):
        for j in range(n_cols):
            if i < n_rows - 1:
                yield cirq.ZZ(qubits[i][j], qubits[i + 1][j])**gamma_value
            if j < n_cols - 1:
                yield cirq.ZZ(qubits[i][j], qubits[i][j + 1])**gamma_value
            yield cirq.Z(qubits[i][j])**(gamma_value * h[i, j])
```

```
def beta_layer(beta_value):
    """Generator for U(beta, B) layer (mixing layer) of QAOA"""
    for row in qubits:
        for qubit in row:
            yield cirq.X(qubit)**beta_value
```

Let's test these functions by constructing the circuit. Try making a circuit with different numbers of layers. How would you automatically make a circuit with a specified number of layers? Make sure the parameters of these layers are distinct `sympy.Symbol`s for later optimization. Print the circuit to see that it's doing what you want it to do.

```
"""Create the QAOA circuit."""
# Use sympy.Symbols for the 𝛾 and β parameters.
gamma = sympy.Symbol("𝛄")
beta = sympy.Symbol("β")

# Start in the H|0> state.
qaoa = cirq.Circuit(cirq.H.on_each(qubits))

# Your code here!

# Display the QAOA circuit.
qaoa
```

#### Solution

We'll just illustrate the solution for a single $C$ layer and a single $B$ layer.

```
"""Create the QAOA circuit."""
# Use sympy.Symbols for the 𝛾 and β parameters.
gamma = sympy.Symbol("𝛄")
beta = sympy.Symbol("β")

# Start in the H|0> state.
qaoa = cirq.Circuit(cirq.H.on_each(qubits))

# Implement the U(gamma, C) operator.
qaoa.append(gamma_layer(gamma, h))

# Implement the U(beta, B) operator.
qaoa.append(beta_layer(beta), strategy=cirq.InsertStrategy.NEW_THEN_INLINE)

# Display the QAOA circuit.
qaoa
```

### Computing the energy

To train the QAOA circuit (that is, find the optimal values of the parameters) we're going to need to be able to compute the expectation value of the Ising model energy.

If we were using real hardware, the only way to compute the expectation value of the energy would be to estimate it by sampling. Using a simulator we can alternatively compute the wavefunction and then calculate the expectation value from that. Not only does this save us from having to worry about statistical error, it also tends to be faster that simulating the sampling process.

> We divide the total energy by the number of qubits because we expect the energy to scale with the size of the system.

```
def energy_from_wavefunction(wf, h):
    """Computes the energy-per-site of the Ising model directly from the
    a given wavefunction.

    Args:
        wf: Array of size 2**(n_rows * n_cols) specifying the wavefunction.
        h: Array of shape (n_rows, n_cols) giving the magnetic field values.

    Returns:
        energy: Float equal to the expectation value of the energy per site
    """
    n_sites = n_rows * n_cols

    # Z is an array of shape (n_sites, 2**n_sites). Each row consists of the
    # 2**n_sites non-zero entries in the operator that is the Pauli-Z matrix on
    # one of the qubits times the identities on the other qubits. The
    # (i*n_cols + j)th row corresponds to qubit (i,j).
    Z = np.array([(-1)**(np.arange(2**n_sites) >> i) for i in range(n_sites - 1, -1, -1)])

    # Create the operator corresponding to the interaction energy summed over all
    # nearest-neighbor pairs of qubits
    ZZ_filter = np.zeros_like(wf, dtype=float)
    for i in range(n_rows):
        for j in range(n_cols):
            if i < n_rows - 1:
                ZZ_filter += Z[i * n_cols + j] * Z[(i + 1) * n_cols + j]
            if j < n_cols - 1:
                ZZ_filter += Z[i * n_cols + j] * Z[i * n_cols + (j + 1)]

    energy_operator = -ZZ_filter - h.reshape(n_sites).dot(Z)

    # Expectation value of the energy divided by the number of sites
    return np.sum(np.abs(wf)**2 * energy_operator) / n_sites
```

We'll also need a helper function that computes the expected value of the energy given some parameters of the QAOA.

```
def energy_from_params(gamma_value, beta_value, qaoa, h):
    """Returns the energy given values of the parameters."""
    sim = cirq.Simulator()
    params = cirq.ParamResolver({"𝛄": gamma_value, "β": beta_value})
    wf = sim.simulate(qaoa, param_resolver=params).final_state_vector
    return energy_from_wavefunction(wf, h)
```

### Optimizing the parameters

Now we need to figure out the best values of $\gamma$ and $\beta$ by minimizing the expectation value of the energy. We'll start by doing a brute-force search of the parameter space for illustrative purposes.

```
"""Do a grid search over values of 𝛄 and β."""
# Set the grid size and range of parameters.
grid_size = 50
gamma_max = 2
beta_max = 2

# Do the grid search.
energies = np.zeros((grid_size, grid_size))
for i in range(grid_size):
    for j in range(grid_size):
        energies[i, j] = energy_from_params(
            i * gamma_max / grid_size, j * beta_max / grid_size, qaoa, h
        )
```

We can visualize the energy landscape as follows.

```
"""Plot the energy as a function of the parameters 𝛄 and β found in the grid search."""
plt.ylabel(r"$\gamma$")
plt.xlabel(r"$\beta$")
plt.title("Energy as a function of parameters")
plt.imshow(energies, extent=(0, beta_max, gamma_max, 0))
plt.colorbar()
plt.show()
```

We see that the energy function has a number of interesting properties. First, note that the function is periodic in $\beta$ and $\gamma$ with shorter periods than one might naively expect given the definition of the gates. The details of why that's true will take us away from the main content of this tutorial, but it's a good thing to understand so that the parameter space can be efficiently truncated.

The other main thing to notice is that there are many local minima and maxima. This makes it challenging to use gradient-based methods for optimization, which we'll explicitly see next. Part of the challenge for algorithms of this type is finding efficient ways to optimize the parameters.

#### Gradient descent

For practice let's try to minimize the expectation value of the energy using gradient descent. We know that there are local minima that we might get stuck in, depending on initialization, but it's still a worthwhile exercise.

The first step is to define a function which approximates the gradient of the energy. We'll do this by symmetric difference, i.e., $f'(x) \approx (f(x+\epsilon)-f(x-\epsilon))/(2\epsilon)$. You should experiment with different values of $\epsilon$ as well as different formulas for the gradient.

```
def gradient_energy(gamma, beta, qaoa, h):
    """Uses a symmetric difference to calulate the gradient."""
    eps = 10**-3 # Try different values of the discretization parameter

    # Gamma-component of the gradient
    grad_g = energy_from_params(gamma + eps, beta, qaoa, h)
    grad_g -= energy_from_params(gamma - eps, beta, qaoa, h)
    grad_g /= 2*eps

    # Beta-compoonent of the gradient
    grad_b = energy_from_params(gamma, beta + eps, qaoa, h)
    grad_b -= energy_from_params(gamma, beta - eps, qaoa, h)
    grad_b /= 2*eps

    return grad_g, grad_b
```

Now we'll implement a gradient descent algorithm that minimizes the energy. Note that it will get stuck in local minima depending on the initialization.

```
"""Run a simple gradient descent optimizer."""
gamma, beta = 0.2, 0.7 # Try different initializations
eta = 10**-2 # Try adjusting the learning rate.

# Perform gradient descent for a given number of steps.
num_steps = 150
for i in range(num_steps + 1):
    # Compute the gradient.
    grad_g, grad_b = gradient_energy(gamma, beta, qaoa, h)

    # Update the parameters.
    gamma -= eta * grad_g
    beta -= eta * grad_b

    # Status updates.
    if not i % 25:
        print("Step: {} Energy: {}".format(i, energy_from_params(gamma, beta, qaoa, h)))

print("\nLearned gamma: {}\nLearned beta: {}".format(gamma, beta, qaoa, h))
```

### Getting the approximate solutions

We've optimized our parameters. How well did we do?

For a $3\times 3$ grid we have $9$ qubits and $12$ interacting nearest-neighbor pairs. If all of the qubits are in the $|0\rangle$ state or all are in the $|1\rangle$ state, then the energy-per-qubit is $-12/9 = -1.33$ at zero external magnetic field $h$, and will be close to that if the magnetic field is small. Notice that the QAOA algorithm we analyzed above is __not__ getting close to that ground state. Is this a problem?

Well, not really. The QAOA algorithm still succeeds if we can  find the ground state after a small number of measurements. The QAOA prepares a certain state which is a linear combination of the ground state and many other states. When we measure the qubits, we find the ground-state configuration with some probability. If that probability is relatively large, then after a reasonably small number of measurements we'll locate the ground state.

Practically speaking, this means we should measure the state prepared by the QAOA several times and record the lowest-energy state we find. The QAOA can be successful by biasing these measurements toward the ground state, even if they do not produce the ground state with $100\%$ probability.

Let's make a copy of our qaoa circuit for measurement purposes and attach a measurement gate to each qubit.

```
"""Add measurements to the QAOA circuit."""
measurement_circuit = qaoa.copy()
measurement_circuit.append(cirq.measure(*[qubit for row in qubits for qubit in row], key="m"))
measurement_circuit
```

Now we'll measure the output of the circuit repeatedly for a good set of angles $\gamma$ and $\beta$. Note that these are simply found from inspecting the above heatmap of the energy found via grid search.

```
"""Sample from the QAOA circuit."""
num_reps = 1000  # Try different numbers of repetitions.
gamma_value, beta_value = 0.2, 0.25  # Try different values of the parameters.

# Sample from the circuit.
simulator = cirq.Simulator()
params = cirq.ParamResolver({"𝛄": gamma_value, "β": beta_value})
result = simulator.run(measurement_circuit, param_resolver = params, repetitions=num_reps)
```

Finally, we'll compute the energy for each of our measurement outcomes and look at the statistics. We start with a helper function which calculates the energy given a set of measurement outcomes.

```
def compute_energy(meas):
    """Returns the energy computed from measurements.

    Args:
        meas: Measurements/samples.
    """
    Z_vals = 1 - 2 * meas.reshape(n_rows,n_cols)
    energy = 0
    for i in range(n_rows):
        for j in range(n_cols):
            if i < n_rows - 1:
                energy -= Z_vals[i, j] * Z_vals[i + 1, j]
            if j < n_cols - 1:
                energy -= Z_vals[i, j] * Z_vals[i, j + 1]
            energy -= h[i, j] * Z_vals[i, j]
    return energy / (n_rows * n_cols)
```

Now we consider the 10 most common outputs of our measurements and compute the energies of those.

```
"""Compute the energies of the most common measurement results."""
# Get a histogram of the measurement results.
hist = result.histogram(key="m")

# Consider the top 10 of them.
num = 10

# Get the most common measurement results and their probabilities.
configs = [c for c, _ in hist.most_common(num)]
probs = [v / result.repetitions for _, v in hist.most_common(num)]
```

We can now plot the probabilities of the most common measurement results as well as the energies associated with these results.

```
"""Plot the most common measurement results and their energies."""
# Plot probabilities of the most common bitstrings.
plt.title("Probability of {} Most Common Outputs".format(num))
plt.bar([x for x in range(len(probs))],probs)
plt.show()
meas = [[int(s) for s in "".join([str(b) for b in bin(k)[2:]]).zfill(n_rows * n_cols)] for k in configs]
costs = [compute_energy(np.array(m)) for m in meas]

# Plot energies of the most common bitstrings.
plt.title("Energy of {} Most Common Outputs".format(num))
plt.bar([x for x in range(len(costs))], costs)
plt.show()
print("Fraction of outputs displayed: {}".format(np.sum(probs).round(2)))
```

We see that, for a good choice of $\gamma$ and $\beta$, ground state is the most probable outcome.

Try changing the values of $\gamma$ and $\beta$ away from the optimal ones. You'll see that this experiment no longer finds the ground state for us.

### Exercise: Experiment with different numbers of layers

See if you can get a closer to the true ground state (i.e. a larger fraction of measurements yielding the minimal energy) by adding more layers to the circuit.

### Exercise: Try a different graph and/or different interaction strengths

Instead of a square lattice, you can try to formulate the Ising model on any graph you like. This just changes which qubits you link in the $U(\gamma, C)$ layer. Each edge of the graph could also come with a different interaction coefficient, so that instead of $\exp(i\pi \gamma Z_i Z_j/2)$ for that edge you would have $\exp(i\pi \gamma J_{ij} Z_i Z_j / 2)$ for some matrix $J_{ij}$ of coefficients. Note that you have to change both the $U(\gamma, C)$ layer and the definition of the energy function to make this work.

### Exercise: Repeat using sampling

On real hardware we need to use sampling to estimate expectation values. Adjust your code so that sampling is used instead of wavefunction evaluation. How many samples do you need to take to get good results? Try different values.

### Exercise: Transverse field Ising model

The Ising Model with transverse field replaces the $\sum h_i Z_i$ term with a $\sum h_i X_i$ term. Can we use the QAOA here as well? What are the differences? This is no longer a classical problem: in general the ground state will now be a superposition of elements of the computational basis. Can you make a circuit that prepares a state close to the ground state?


**To execute this notebook it's necessary to install the required dependencies:**

1. [Savime](https://github.com/hllustosa/Savime/). run the script `examples/test_savime.sh` to verify if the installing process was successful.
2. Some Python modules are displayed in `tutorial.yml`. The easiest way to install them is to use the Anaconda package manager [Anaconda](https://www.anaconda.com/) and create a new enviroment with the command
    - `conda env create -f tutorial.yml`
    - To activate the enviroment, run `conda activate tutorial`
3. [pysavime](https://github.com/dnasc/PySavime): Use the previously defined enviroment. It won't be necessary to install the suggested packages in PySavime repository.
4. [TFX](https://www.tensorflow.org/tfx/serving/setup): In Ubuntu, run `sudo apt install tensorflow-model-server`. If using another OS, follow this [tutorial](https://www.tensorflow.org/tfx/serving/setup).

# Part 01

In this first notebook, we will generate multiple models, all of them are adapted from [Elastic Net](https://en.wikipedia.org/wiki/Elastic_net_regularization). These will be used on notebooks 2 and 3.
To generate the models, we will take the following steps:

1. Define a learning function (where $d$ is the number of *features*).
$$\begin{align}y &= f(x)\\f: X &\to Y, X \subset \mathbb{R}^d, Y \subset \mathbb{R}\end{align}$$
2. Generate an uniform sample (dataset) $D_X \subset X$.
3. Generate $D_Y$ from $D_X$ and $D = concat(D_X, D_Y)$.
4. Train $m$ (regressor) models $m_i: X \to Y$. Each model $m_i$ must be
trained (train / test/ validation) in one partition from $D^{(i)} = (D_X^{(i)}, D_Y^{(i)}) \subset D$ so that:
$$D = \bigcup_{i=1}^{m} D^{(i)} \;,\; \bigcap_{i=1}^{m} D^{(i)} = \emptyset \; \land \;  i < j \implies \forall x_i \in D_X^{(i)}, x_j \in D_X^{(j)} \left(x_i < x_j \right)$$
5. Use a metric $\phi$ and evaluate each model $m_i$ in each partition $D^{(j)}$.
Here $\phi$ is defined as the mean squared error between the prediction and the true value.
6. Save the models in a appropriat format to posterior call from Tfx. Save $D$ in order to be registered in Savime.

## Definition of $f$:

The definition of $f$ requires that we define some parameters:

- $ X = [\rm {low}, \rm{high}]$;
- $ D_X = (x_1, ..., x_n), x_i \sim U(X)$, where $n$ is the number of observations.
- $D_Y = (f(x),  x \in D_X)$.
- $ D = (D_X, D_Y) $ is partitioned into $m$ partitions $D^{(i)}$ each associated with a $m_i$ model.
- $ A = \rm{linspace} (\alpha_{min} = 1, \alpha_{max} = 3, \rm{num\_partitions} = m) $is an ordered and equally
spaced from numbers between 1 and 3 and size equal to $m$. Each $ d_i $ is associated to an $\alpha_i \in A$.
- $ g (x) $ indicates which $\alpha$ should be associated to $x$. For example, if $ X = [-1, 1] $,
and $ m = 2 $ then $ A = [1, 3] $, $ g (x = -. 5) = 1 $ and $ g (x = .5) = 3 $.
- $ f (x) = g (x_1) x_1 + ... + g (x_d) x_d + \varepsilon $, where
$ \varepsilon \sim \mathcal{N}(\mu, \sigma) $ (e.g., $\mu = 0 $ and $ \sigma = 1 $).
- *Note that $ f $ is a collection of linear functions each associated with an $ X $ partition.

```python
%load_ext autoreload
%autoreload 2

import inspect
import os
import random as rn

if not 'notebooks' in os.listdir('.'):
    current_dir = os.path.abspath(os.getcwd())
    parent_dir = os.path.dirname(current_dir)
    os.chdir(parent_dir)

import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
import tensorflow as tf

from src.functions import f2_x as f
from src.model import ElasticNet, ModelFactory
from src.util import plot_heatmap

# Reproductibility
import numpy as np
import random as rn
import tensorflow as tf

seed = 32
rn.seed(seed)
np.random.seed(seed)
tf.random.set_seed(seed)

sns.set_context('notebook')
sns.set_style('whitegrid')
sns.set_palette(sns.color_palette("Paired"))

tf.get_logger().setLevel('ERROR')
```

    The autoreload extension is already loaded. To reload it, use:
      %reload_ext autoreload

As an example, the behavior of $f$ is shown below. Note that (i) the function domain is $[- 1, 1]$, (ii) the number of partitions is 5, and (iii) the five alphas (coefficients) associated with the partitions are [-1, -0.5, 0, 0.5, 1]. Note also that the function has a stochastic component ($ \varepsilon \in \mathcal {N} \left (0, \frac {1} {10} \right)$):

```python
low_plot, high_plot, num_partitions_plot = -1, 1, 5
x_plot = np.random.uniform(low_plot, high_plot, 10000).reshape(-1, 1)
y_plot = f(x_plot, low_plot, high_plot, num_partitions_plot, 0, .01,
              alpha_min=-1, alpha_max=1)
sns.regplot(x_plot.ravel(), y_plot.ravel(), fit_reg=False)
```

To train the models, we will use the following configurations:
- **Number of features** = 2
- **Number of models and partitions** = 25
- **Observations**:
  - number_of_observations = 100000
  - Range = [-1, 1]
- **Model adjustment**:
   - Percentage of training, validation and testing respectively equal to .5, .3 and .2.
- **Function**:
   - Alphas ranging between 1 and 3.
   - Error $\varepsilon \in \mathcal{N} (0, 0.01)$

```python
# number of features, observations and models
d = 2
n = 100000
m = 25

# Model's Input shape
input_shape = (d, )

# X interval
low = -1
high = 1

# uniform sample between low and high of size n*d
x = np.random.uniform(low, high, n * d)

# Ordering x to get item 4
x = np.reshape(np.sort(x), newshape=(n, d))

# Model training configurations
val_size = .3
test_size = .2

# Name prefix associated to each model
model_name_prefix = 'my_elastic_net'

# Where model and data are stored
output_dir = os.path.join(parent_dir, 'saved_models_elastic_net')

# Function Configuration
f_alpha_min = 1
f_alpha_max = 3
f_mean = 0
f_std = 10e-2
```

**In the cell below, a model factory is instantiated. The factory ('model_factory') will be responsible for:**
1. Create the vector y based on the function f described previously;
2. Divide x into m partitions;
3. Train each model in its proper partition;
4. Train a model across all the domain;
5. Save the models;
6. Save the data.

```python
f_config = dict(low=low, high=high, num_partitions=m, mean=f_mean, std=f_std,
                alpha_min=f_alpha_min, alpha_max=f_alpha_max)

model_config = dict(input_shape=input_shape, loss=tf.keras.losses.mean_squared_error,
                    metrics=[tf.keras.metrics.mean_squared_error],epochs=10)

model_factory = ModelFactory(model_class=ElasticNet, x=x, f_x=f, num_models=m,
                             test_size=test_size, val_size=val_size,
                             model_name_prefix=model_name_prefix,
                             f_kwargs=f_config, model_kwargs=model_config)

model_factory.build_models()
model_factory.fit_models()
```

    Building models: 100%|██████████| 25/25 [00:01<00:00, 13.20it/s]
    Training models by partition: 25it [01:08,  2.72s/it]
    Training a model in the whole domain: 100%|██████████| 1/1 [00:56<00:00, 56.86s/it]

Next we calculate the each models mean squared error on each partition and on the complete domain.

```python
# Saving model and data
info = model_factory.get_metric_info()

model_factory.save_models(output_dir)
model_factory.save_data(output_dir, info)
```

    Building metric matrices: 26it [02:39,  6.14s/it]

```python
fig, ax = plt.subplots()
fig.set_size_inches(15, 9)

matrix = info['mean_squared_error']

plot_heatmap(matrix=matrix, title=f'Heatmap for MSE', x_label='Model ID', y_label='Split ID', ax=ax,
             heatmap_kwargs=dict(linewidths=.1, cmap='Blues', annot=True, fmt='.1f'))
fig.tight_layout()
```

Note: The rightmost column and the lowest row refer respectively to the test set and to the model trained in the complete domain.

Note in the plot above that the farther from the partition the model was trained in, the greater the error that the model obtains in the prediction. For example, model 24 gets MSE of more than 50 for partition 1 while less than 1 for partition 24.