# Generative models - variational auto-encoders

### Author: Philippe Esling (esling@ircam.fr)

In this course we will cover
1. A [quick recap](#recap) on simple probability concepts
2. A formal introduction to [Variational Auto-Encoders](#vae) (VAEs)
3. An explanation of the [implementation](#implem) of VAEs
4. Some [modifications and tips to improve the reconstruction](#improve) of VAEs **(exercise)**

## Quick recap on probability

The field of probability aims to model random or uncertain events. Hence, a random variable $X$ denotes a quantity that is uncertain, such as the result of an experiment (flipping a coin) or the measurement of an uncertain property (measuring the temperature). If we observe several occurrences of the variable $\{\mathbf{x}_{i}\}_{i=1}$, it might take different values on each occasion, but some values may occur more often than others. This information is captured by the _probability distribution_ $p(\mathbf{x})$ of the random variable.

To understand these concepts graphically, we will rely on the `Pytorch Probability` package.

```python
import torch
import torch.nn as nn
import torch.distributions as distrib
import torchvision
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
from helper_plot import hdr_plot_style
hdr_plot_style()
```

### Probability distributions

#### Discrete distributions

Let $\mathbf{x}$ be a discrete random variable with range $R_{X}=\{x_1,\cdots,x_n\}$ (finite or countably infinite). The function
\begin{equation}
    p_{X}(x_{i})=p(X=x_{i}), \forall i\in\{1,\cdots,n\}
\end{equation}
is called the probability mass function (PMF) of $X$.

Hence, the PMF defines the probabilities of all possible values for a random variable. The above notation allows to express that the PMF is defined for the random variable $X$, so that $p_{X}(1)$ gives the probability that $X=1$. For discrete random variables, the PMF is also called the \textit{probability distribution}. The PMF is a probability measure, therefore it satisfies all the corresponding properties
- $0 \leq p_{X}(x_i) < 1, \forall x_i$
- $\sum_{x_i\in R_{X}} p_{X}(x_i) = 1$
- $\forall A \subset R_{X}, p(X \in A)=\sum_{x_a \in A}p_{X}(x_a)$

A very simple example of discrete distribution is the `Bernoulli` distribution. With this distribution, we can model a coin flip. If we throw the coin a very large number of times, we hope to see on average an equal amount of _heads_ and _tails_.

```python
bernoulli = distrib.Bernoulli(0.5)
samples = bernoulli.sample((10000,))
plt.figure(figsize=(10,8))
sns.distplot(samples)
plt.title("Samples from a Bernoulli (coin toss)")
plt.show()
```

However, we can also _sample_ from the distribution to have individual values of a single throw. In that case, we obtain a series of separate events that _follow_ the distribution

```python
vals = ['heads', 'tails']
samples = bernoulli.sample((10,))
for s in samples:
    print('Coin is tossed on ' + vals[int(s)])
```

    Coin is tossed on heads
    Coin is tossed on heads
    Coin is tossed on heads
    Coin is tossed on tails
    Coin is tossed on tails
    Coin is tossed on tails
    Coin is tossed on tails
    Coin is tossed on heads
    Coin is tossed on heads
    Coin is tossed on heads

#### Continuous distributions

The same ideas apply to _continuous_ random variables, which can model for instance the height of human beings. If we try to guess the height of someone that we do not know, there is a higher probability that this person will be around 1m70, instead of 20cm or 3m. For the rest of this course, we will use the shorthand notation $p(\mathbf{x})$ for the distribution $p(\mathbf{x}=x_{i})$,  which expresses for a real-valued random variable $\mathbf{x}$, evaluated at $x_{i}$, the probability that $\mathbf{x}$ takes the value $x_i$.

One notorious example of such distributions is the Gaussian (or Normal) distribution, which is defined as
\begin{equation}
    p(x)=\mathcal{N}(\mu,\sigma)=\frac{1}{\sqrt{2\pi\sigma^{2}}}e^{-\frac{(x-\mu)^{2}}{2\sigma^{2}}}
\end{equation}

Similarly as before, we can observe the behavior of this distribution with the following code

```python
normal = distrib.Normal(loc=0., scale=1.)
samples = normal.sample((10000,))
plt.figure(figsize=(10,8))
sns.distplot(samples)
plt.title("Samples from a standard Normal")
plt.show()
```

### Comparing distributions (KL divergence)
$
\newcommand{\R}{\mathbb{R}}
\newcommand{\bb}[1]{\mathbf{#1}}
\newcommand{\bx}{\bb{x}}
\newcommand{\by}{\bb{y}}
\newcommand{\bz}{\bb{z}}
\newcommand{\KL}[2]{\mathcal{D}_{\text{KL}}\left[#1 \| #2\right]}$
Originally defined in the field of information theory, the _Kullback-Leibler (KL) divergence_ (usually noted $\KL{p(\bx)}{q(\bx)}$) is a dissimilarity measure between two probability distributions $p(\bx)$ and $q(\bx)$. In the view of information theory, it can be understood as the cost in number of bits necessary for coding samples from $p(\bx)$ by using a code optimized for $q(\bx)$ rather than the code optimized for $p(\bx)$. In the view of probability theory, it represents the amount of information lost when we use $q(\bx)$ to approximate the true distribution $p(\bx)$. %that explicit the cost incurred if events were generated by $p(\bx)$ but charged under $q(\bx)$

Given two probability distributions $p(\bx)$ and $q(\bx)$, the Kullback-Leibler divergence of $q(\bx)$ _from_ $p(\bx)$ is defined to be
\begin{equation}
    \KL{p(\bx)}{q(\bx)}=\int_{\R} p(\bx) \log \frac{p(\bx)}{q(\bx)}d\bx
\end{equation}

Note that this dissimilarity measure is \textit{asymmetric}, therefore, we have
\begin{equation}
    \KL{p(\bx)}{q(\bx)}\neq \KL{q(\bx)}{p(\bx)}
\end{equation}
This asymmetry also describes an interesting behavior of the KL divergence, depending on the order to which it is evaluated. The KL divergence can either be a _mode-seeking_ or _mode-coverage} measure.

## Variational auto-encoders

As we have seen in the previous AE course, VAEs are also a form generative models. However, they are defined from a more sound probabilistic perspective. to find the underlying probability distribution of the data $p(\mathbf{x})$ based on a set of examples in $\mathbf{x}\in\mathbb{R}^{d_{x}}$. To do so, we consider *latent variables* defined in a lower-dimensional space $\mathbf{z}\in\mathbb{R}^{d_{z}}$ ($d_{z} \ll d_{x}$) with the joint probability distribution $p(\mathbf{x}, \mathbf{z}) = p(\mathbf{x} \vert \mathbf{z})p(\mathbf{z})$. Unfortunately, for complex distributions this integral is too complex and cannot be found in closed form.

### Variational inference

The idea of *variational inference* (VI) allows to solve this problem through *optimization* by assuming a simpler approximate distribution $q_{\phi}(\mathbf{z}\vert\mathbf{x})\in\mathcal{Q}$ from a family $\mathcal{Q}$ of approximate densities. Hence, the goal is to minimize the difference between this approximation and the real distribution. Therefore, this turns into the optimization problem of minimizing the Kullback-Leibler (KL) divergence between the parametric approximation and the original density

$$
q_{\phi}^{*}(\mathbf{z}\vert \mathbf{x})=\text{argmin}_{q_{\phi}(\mathbf{z} \vert \mathbf{x})\in\mathcal{Q}} \mathcal{D}_{KL} \big[ q_{\phi}\left(\mathbf{z} \vert \mathbf{x}\right) \parallel p\left(\mathbf{z} \vert \mathbf{x}\right) \big]
\tag{2}
$$

By developing this KL divergence and re-arranging terms (the detailed development can be found in [3](#reference1)), we obtain

$$
\log{p(\mathbf{x})} - D_{KL} \big[ q_{\phi}(\mathbf{z} \vert \mathbf{x}) \parallel p(\mathbf{z} \vert \mathbf{x}) \big] =
\mathbb{E}_{\mathbf{z}} \big[ \log{p(\mathbf{x} \vert \mathbf{z})}\big] - D_{KL} \big[ q_{\phi}(\mathbf{z} \vert \mathbf{x}) \parallel p(\mathbf{z}) \big]
\tag{3}
$$

This formulation describes the quantity we want to maximize $\log p(\mathbf{x})$ minus the error we make by using an approximate $q$ instead of $p$. Therefore, we can optimize this alternative objective, called the *evidence lower bound* (ELBO)

$$
\begin{equation}
\mathcal{L}_{\theta, \phi} = \mathbb{E} \big[ \log{ p_\theta (\mathbf{x|z}) } \big] - \beta \cdot D_{KL} \big[ q_\phi(\mathbf{z|x}) \parallel p_\theta(\mathbf{z}) \big]
\end{equation}
\tag{4}
$$

We can see that this equation involves $q_{\phi}(\mathbf{z} \vert \mathbf{x})$ which *encodes* the data $\mathbf{x}$ into the latent representation $\mathbf{z}$ and a *decoder* $p(\mathbf{x} \vert \mathbf{z})$, which allows generating a data vector $\mathbf{x}$ given a latent configuration $\mathbf{z}$. Hence, this structure defines the *Variational Auto-Encoder* (VAE).

The VAE objective can be interpreted intuitively. The first term  increases the likelihood of the data generated given a configuration of the latent, which amounts to minimize the *reconstruction error*. The second term represents the error made by using a simpler posterior distribution $q_{\phi}(\mathbf{z} \vert \mathbf{x})$ compared to the true prior $p_{\theta}(\mathbf{z})$. Therefore, this *regularizes* the choice of approximation $q$ so that it remains close to the true posterior distribution [3].

### Reparametrization trick

Now, while this formulation has some very interesting properties, it involves sampling operations, where we need to draw the latent point $\mathbf{z}$ from the distribution $q_{\phi}(\mathbf{z}\vert\mathbf{x})$.  The simplest choice for this variational approximate posterior is a multivariate Gaussian with a diagonal covariance structure (which leads to independent Gaussians on every dimension, called the *mean-field* family) so that
$$
\text{log}q_\phi(\mathbf{z}\vert\mathbf{x}) = \text{log}\mathcal{N}(\mathbf{z};\mathbf{\mu}^{(i)},\mathbf{\sigma}^{(i)})
\tag{5}
$$
where the mean $\mathbf{\mu}^{(i)}$ and standard deviation $\mathbf{\sigma}^{(i)}$ of the approximate posterior are different for each input point and are produced by our encoder parametrized by its variational parameters $\phi$. Now the KL divergence between this distribution and a simple prior $\mathcal{N}(\mathbf{0}, \mathbf{I})$ can be very simply obtained with
$$
D_{KL} \big[ q_\phi(\mathbf{z|x}) \parallel \mathcal{N}(\mathbf{0}, \mathbf{I}) \big] = \frac{1}{2}\sum_{j=1}^{D}\left(1+\text{log}((\sigma^{(i)}_j)^2)+(\mu^{(i)}_j)^2+(\sigma^{(i)}_j)^2\right)
\tag{6}
$$

While this looks convenient, we will still have to perform gradient descent through a sampling operation, which is non-differentiable. To solve this issue, we can use the *reparametrization trick*, which takes the sampling operation outside of the gradient flow by considering $\mathbf{z}^{(i)}=\mathbf{\mu}^{(i)}+\mathbf{\sigma}^{(i)}\odot\mathbf{\epsilon}^{(l)}$ with $\mathbf{\epsilon}^{(l)}\sim\mathcal{N}(\mathbf{0}, \mathbf{I})$

## VAE implementation

As we have seen, VAEs can be simply implemented by decomposing the above series of operations into an `encoder` which represents the distribution $q_\phi(\mathbf{z}\vert\mathbf{x})$, from which we will sample some values $\tilde{\mathbf{z}}$ (using the reparametrization trick) and compute the Kullback-Leibler (KL) divergence. Then, we use these values as input to a `decoder` which represents the distribution $p_\theta(\mathbf{x}\vert\mathbf{z})$ so that we can produce a reconstruction $\tilde{\mathbf{x}}$ and compute the reconstruction error.

Therefore, we can define the VAE based on our previous implementation of the AE that we recall here

```python
class AE(nn.Module):
    def __init__(self, encoder, decoder, encoding_dim):
        super(AE, self).__init__()
        self.encoding_dims = encoding_dim
        self.encoder = encoder
        self.decoder = decoder

    def forward(self, x):
        encoded = self.encoder(x)
        decoded = self.decoder(encoded)
        return decoded
```

In order to move to a probabilistic version, we need to add the latent space sampling mechanism, and change the behavior of our `call` function. This process is implemented in the following `VAE` class.

Note that we purposedly rely on an implementation of the `encode` function where the `encoder` first produces an intermediate representation of size `encoder_dims`. Then, this representation goes through two separate functions for encoding $\mathbf{\mu}$ and $\mathbf{\sigma}$. This provides a clearer implementation but also the added bonus that we can ensure that $\mathbf{\sigma} > 0$

```python
class VAE(AE):

    def __init__(self, encoder, decoder, encoding_dims, latent_dims):
        super(VAE, self).__init__(encoder, decoder, encoding_dims)
        self.latent_dims = latent_dims
        self.mu = nn.Sequential(nn.Linear(self.encoding_dims, self.latent_dims), nn.ReLU())
        self.sigma = nn.Sequential(nn.Linear(self.encoding_dims, self.latent_dims), nn.Softplus())

    def encode(self, x):

        ######################
        # YOUR CODE GOES HERE
        ######################

        return mu, sigma

    def decode(self, z):
        return self.decoder(z)

    def forward(self, x):
        # Encode the inputs
        z_params = self.encode(x)
        # Obtain latent samples and latent loss
        z_tilde, kl_div = self.latent(x, z_params)
        # Decode the samples
        x_tilde = self.decode(z_tilde)
        return x_tilde.reshape(-1, 1, 28, 28), kl_div

    def latent(self, x, z_params):

        ######################
        # YOUR CODE GOES HERE
        ######################

        return z, kl_div
```

Now the interesting aspect of VAEs is that we can define any parametric function as `encoder` and `decoder`, as long as we can optimize them. Here, we will rely on simple feed-forward neural networks, but these can be largely more complex (with limitations that we will discuss later in the tutorial).

```python
def construct_encoder_decoder(nin, n_latent = 16, n_hidden = 512, n_classes = 1):
    # Encoder network
    encoder = nn.Sequential(
          nn.Flatten(),
          nn.Linear(nin, n_hidden), nn.ReLU(),
          nn.Linear(n_hidden, n_hidden), nn.ReLU(),
          nn.Linear(n_hidden, n_hidden), nn.ReLU(),
    )
    # Decoder network
    decoder = nn.Sequential(
          nn.Linear(n_latent, n_hidden), nn.ReLU(),
          nn.Linear(n_hidden, n_hidden), nn.ReLU(),
          nn.Linear(n_hidden, nin * n_classes), nn.Sigmoid()
    )
    return encoder, decoder
```

### Evaluating the error

In the definition of the `VAE` class, we directly included the computation of the $D_{KL}$ term to regularize our latent space. However, remember that the complete loss of equation (4) also contains a *reconstruction loss* which compares our reconstructed output to the original data.

While there are several options to compare the error between two elements, there are usually two preferred choices among the generative literature depending on how we consider our problem
1. If we consider each dimension (pixel) to be a binary unit (following a Bernoulli distribution), we can rely on the `binary cross entropy` between the two distributions
2. If we turn our problem to a set of classifications, where each dimension can belong to a given set of *intensity classes*, then we can compute the `multinomial loss` between the two distributions

In the following, we define both error functions and regroup them in the `reconstruction_loss` call (depending on the `num_classes` considered). However, as the `multinomial loss` requires a large computational overhead, and for the sake of simplicity, we will train all our first models by relying on the `binary cross entropy`

```python
# Reconstruction criterion
recons_criterion = torch.nn.MSELoss(reduction='sum')

def compute_loss(model, x):

    ######################
    # YOUR CODE GOES HERE
    ######################

    return full_loss

def train_step(model, x, optimizer):
    # Compute the loss.
    loss = compute_loss(model, x)
    # Before the backward pass, zero all of the network gradients
    optimizer.zero_grad()
    # Backward pass: compute gradient of the loss with respect to parameters
    loss.backward()
    # Calling the step function to update the parameters
    optimizer.step()
    return loss
```

### Optimizing a VAE on a real dataset

For this tutorial, we are going to take a quick shot at a real-life problem by trying to train our VAEs on the `FashionMNIST` dataset. This dataset can be natively used in PyTorch by relying on the `torchvision.datasets` classes as follows

```python
dataset_dir = './data'
# Going to use 80%/20% split for train/valid
valid_ratio = 0.2
# Load the dataset for the training/validation sets
train_valid_dataset = torchvision.datasets.FashionMNIST(root=dataset_dir, train=True, transform=torchvision.transforms.ToTensor(), download=True)
# Split it into training and validation sets
nb_train = int((1.0 - valid_ratio) * len(train_valid_dataset))
nb_valid =  int(valid_ratio * len(train_valid_dataset))
train_dataset, valid_dataset = torch.utils.data.dataset.random_split(train_valid_dataset, [nb_train, nb_valid])
# Load the test set
test_dataset = torchvision.datasets.FashionMNIST(root=dataset_dir, transform=torchvision.transforms.ToTensor(),train=False)
# Prepare
num_threads = 4     # Loading the dataset is using 4 CPU threads
batch_size  = 128   # Using minibatches of 128 samples
train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=batch_size, num_workers=num_threads)
valid_loader = torch.utils.data.DataLoader(dataset=valid_dataset, batch_size=batch_size, shuffle=False, num_workers=num_threads)
test_loader = torch.utils.data.DataLoader(dataset=test_dataset,batch_size=batch_size,shuffle=False,num_workers=num_threads)
```

The `FashionMNIST` dataset is composed of simple 28x28 black and white images of different items of clothings (such as shoes, bags, pants and shirts). We put a simple function here to display one batch of the test set (note that we keep a fixed batch from the test set in order to evaluate the different variations that we will try in this tutorial).

```python
print("The train set contains {} images, in {} batches".format(len(train_loader.dataset), len(train_loader)))
print("The validation set contains {} images, in {} batches".format(len(valid_loader.dataset), len(valid_loader)))
print("The test set contains {} images, in {} batches".format(len(test_loader.dataset), len(test_loader)))
nsamples = 10
classes_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat', 'Sandal','Shirt', 'Sneaker', 'Bag', 'Ankle boot']
imgs_test, labels = next(iter(test_loader))
fig = plt.figure(figsize=(20,5))
for i in range(nsamples):
    ax = plt.subplot(1,nsamples, i+1)
    plt.imshow(imgs_test[i, 0, :, :], vmin=0, vmax=1.0, cmap=matplotlib.cm.gray)
    ax.set_title("{}".format(classes_names[labels[i]]), fontsize=15)
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
```

Now based on our proposed implementation, the optimization aspects are defined in a very usual way

```python
# Using Bernoulli or Multinomial loss
num_classes = 1
# Number of hidden and latent
n_hidden = 512
n_latent = 2
# Compute input dimensionality
nin = imgs_test.shape[2] * imgs_test.shape[3]
# Construct encoder and decoder
encoder, decoder = construct_encoder_decoder(nin, n_hidden = n_hidden, n_latent = n_latent, n_classes = num_classes)
# Build the VAE model
model = VAE(encoder, decoder, n_hidden, n_latent)
# Construct the optimizer
optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)
```

Now all that is left to do is train the model. We define here a `train_vae` function that we will reuse along the future implementations and variations of VAEs and flows. Note that this function is set to run for only a very few number of `epochs` and also most importantly, *only considers a subsample of the full dataset at each epoch*. This option is just here so that you can test the different models very quickly on any CPU or laptop.

```python
def generate_and_save_images(model, epoch, test_sample):
    predictions, _ = model(test_sample)
    fig = plt.figure(figsize=(4, 4))
    for i in range(predictions.shape[0]):
        plt.subplot(4, 4, i + 1)
        plt.imshow(predictions[i, 0, :, :].detach(), cmap='gray')
        plt.axis('off')
    # Tight_layout minimizes the overlap between 2 sub-plots
    #plt.savefig('image_at_epoch_{:04d}.png'.format(epoch))
    plt.show()

epochs=50
test_sample = imgs_test[0:16, :, :, :]
for epoch in range(1, epochs + 1):
    full_loss = torch.Tensor([0])
    # Forward pass: compute predicted y by passing x to the model.
    for i, (x, _) in enumerate(train_loader):
        full_loss += train_step(model, x, optimizer)
    #for i, (x, _) in enumerate(valid_loader):
    #    train_step(model, x, optimizer)
    print('Epoch: {}, Test set ELBO: {}'.format(epoch, full_loss))
    generate_and_save_images(model, epoch, test_sample)
```

### Evaluating generative models

In order to evaluate our upcoming generative models, we will rely on the computation of the Negative Log-Likelihood. This code for the following `evaluate_nll_bpd` is inspired by the [Sylvester flow repository](https://github.com/riannevdberg/sylvester-flows)

```python
from scipy.special import logsumexp

def evaluate_nll_bpd(data_loader, model, batch = 500, R = 5):
    # Set of likelihood tests
    likelihood_test = []
    # Go through dataset
    for batch_idx, (x, _) in enumerate(data_loader):
        for j in range(x.shape[0]):
            a = []
            for r in range(0, R):
                cur_x = x[j].unsqueeze(0)
                # Repeat it as batch
                x = cur_x.expand(batch, *cur_x.size()[1:]).contiguous()
                x = x.view(batch, -1)
                x_tilde, kl_div = model(x)
                rec = reconstruction_loss(x_tilde, x, average=False)
                a_tmp = (rec + kl_div)
                a.append(- a_tmp.cpu().data.numpy())
            # calculate max
            a = np.asarray(a)
            a = np.reshape(a, (a.shape[0] * a.shape[1], 1))
            likelihood_x = logsumexp(a)
            likelihood_test.append(likelihood_x - np.log(len(a)))
    likelihood_test = np.array(likelihood_test)
    nll = - np.mean(likelihood_test)
    # Compute the bits per dim (but irrelevant for binary data)
    bpd = nll / (np.prod(nin) * np.log(2.))
    return nll, bpd
```

Now we can evaluate our VAE model more formally as follows.

```python
# Plot final loss
plt.figure()
plt.plot(losses_kld[:, 0].numpy());
# Evaluate log-likelihood and bits per dim
nll, _ = evaluate_nll_bpd(test_loader, model)
print('Negative Log-Likelihood : ' + str(nll))
```

We can also evaluate the latent space of our model, which should be organized (being the overall point of using a VAE instead of a common AE).

```python
x = np.linspace(-3, 3, 8)
y = np.linspace(-3, 3, 8)
fig = plt.figure(figsize=(10, 8))
for i in range(8):
    for j in range(8):
        plt.subplot(8, 8, (i * 8) + j + 1)
        final_tensor = torch.zeros(2)
        final_tensor[0] = x[i]
        final_tensor[1] = y[j]
        plt.imshow(model.decode(final_tensor).detach().reshape(28, 28), cmap='gray')
        plt.axis('off')
```

### Limitations of VAEs - (**exercise**)

Although VAEs are extremely powerful tools, they still have some limitations. Here we list the three most important and known limitations (all of them are still debated and topics of active research).
1. **Blurry reconstructions.** As can be witnessed directly in the results of the previous vanilla VAE implementation, the reconstructions appear to be blurry. The precise origin of this phenomenon is still debated, but the proposed explanation are
    1. The use of the KL regularization
    2. High variance regions of the latent space
    3. The reconstruction criterion (expectation)
    4. The use of simplistic latent distributions
2. **Posterior collapse.** The previous *blurry reconstructions* issue can be mitigated by using a more powerful decoder. However, relying on a decoder with a large capacity causes the phenomenon of *posterior collapse* where the latent space becomes useless. A nice intuitive explanation can be found [here](https://ermongroup.github.io/blog/a-tutorial-on-mmd-variational-autoencoders/)
3. **Simplistic Gaussian approximation**. In the derivation of the VAE objective, recall that the KL divergence term needs to be computed analytically. Therefore, this forces us to rely on quite simplistic families. However, the Gaussian family might be too simplistic to model real world data

In the present tutorial, we show how normalizing flows can be used to mostly solve the third limitation, while also adressing the two first problems. Indeed, we will see that normalizing flows also lead to sharper reconstructions and also act on preventing posterior collapse

## Improving the quality of VAEs

As we discussed in the previous section, several known issues have been reported when using the vanilla VAE implementation. We listed some of the major issues as being
1. **Blurry reconstructions.**
2. **Posterior collapse.**
3. **Simplistic Gaussian approximation**.

Here, we discuss some recent developments that were proposed in the VAE literature and simple adjustments that can be made to (at least partly) alleviate these issues. However, note that some more advanced proposals such as PixelVAE [5](#reference1) and VQ-VAE [6](#reference1) can lead to wider increases in quality

### Reducing the bluriness of reconstructions

In this tutorial, we relied on extremely simple decoder functions, to show how we could easily define VAEs and normalizing flows together. However, the capacity of the decoder obviously directly influences the quality of the final reconstruction. Therefore, we could address this issue naively by using deep networks and of course convolutional layers as we are currently dealing with images.

First you need to construct a more complex encoder and decoder

```python
def construct_encoder_decoder_complex(nin, n_latent = 16, n_hidden = 512, n_params = 0, n_classes = 1):

    ######################
    # YOUR CODE GOES HERE
    ######################

    # Encoder network
    encoder = ...
    # Decoder network
    decoder = ...
    return encoder, decoder
```

### Preventing posterior collapse with Wasserstein-VAE-MMD (InfoVAE)

As we discussed earlier, the reason behind posterior collapse mostly relates to the KL divergence criterion (a nice intuitive explanation can be found [here](https://ermongroup.github.io/blog/a-tutorial-on-mmd-variational-autoencoders/). This can be mitigated by relying on a different criterion, such as regularizing the latent distribution by using the *Maximum Mean Discrepancy* (MMD) instead of the KL divergence. This model was independently proposed as the *InfoVAE* and later also as the *Wasserstein-VAE*.

Here we provide a simple implementation of the `InfoVAEMMD` class based on our previous implementations.

```python
######################
# YOUR CODE GOES HERE
######################

def compute_kernel(x, y):
    return ...

def compute_mmd(x, y):
    return ...

class InfoVAEMMD(VAE):

    def __init__(self, encoder, decoder):
        super(InfoVAEMMD, self).__init__(encoder, decoder)

    def latent(self, x, z_params):
        return ...
```

### Putting it all together

Here we combine all these ideas (except for the MMD, which is not adequate as the flow definition already regularizes the latent space without the KL divergence) to perform a more advanced optimization of the dataset. Hence, we will rely on the complex encoder and decoder with gated convolutions, the multinomial loss and the normalizing flows in order to improve the overall quality of our reconstructions.

```python
# Size of latent space
n_latent = 16
# Number of hidden units
n_hidden = 256
# Rely on Bernoulli or multinomial
num_classes = 128

######################
# YOUR CODE GOES HERE
######################

# Construct encoder and decoder
encoder, decoder = ...
# Create VAE or (InfoVAEMMD - WAE) model
model_flow_p = ...
# Create optimizer algorithm
optimizer = ...
# Add learning rate scheduler
scheduler = ...
# Launch our optimization
losses_flow_param = ...
```

*NB*: It seems that the multinomial version have a hard time converging. Although I only let this run for 200 epochs and only for a subsampling of 5000 examples, it might need more time, but this might also come from a mistake somewhere in my code ... If you spot something odd please let me know :)

### References

[1] Rezende, Danilo Jimenez, and Shakir Mohamed. "Variational inference with normalizing flows." _arXiv preprint arXiv:1505.05770_ (2015). [link](http://arxiv.org/pdf/1505.05770)

[2] Kingma, Diederik P., Tim Salimans, and Max Welling. "Improving Variational Inference with Inverse Autoregressive Flow." _arXiv preprint arXiv:1606.04934_ (2016). [link](https://arxiv.org/abs/1606.04934)

[3] Kingma, D. P., & Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114. (2013). [link](https://arxiv.org/pdf/1312.6114)

[4] Rezende, D. J., Mohamed, S., & Wierstra, D. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401.4082. (2014). [link](https://arxiv.org/pdf/1401.4082)

[5] Gulrajani, I., Kumar, K., Ahmed, F., Taiga, A. A., Visin, F., Vazquez, D., & Courville, A. (2016). Pixelvae: A latent variable model for natural images. arXiv preprint arXiv:1611.05013. [link](https://arxiv.org/pdf/1611.05013)

[6] Van den Oord, A., & Vinyals, O. (2017). Neural discrete representation learning. In NIPS 2017 (pp. 6306-6315). [link](http://papers.nips.cc/paper/7210-neural-discrete-representation-learning.pdf)

### Inspirations and resources

https://blog.evjang.com/2018/01/nf1.html
https://github.com/ex4sperans/variational-inference-with-normalizing-flows
https://akosiorek.github.io/ml/2018/04/03/norm_flows.html
https://github.com/abdulfatir/normalizing-flows
https://github.com/riannevdberg/sylvester-flows


# Sympy - Symbolic algebra in Python

Travis N. Vaught (travis@vaught.net) - minor edits to perform plotting using the Bokeh package. 2015-05-17

J.R. Johansson (robert@riken.jp) http://dml.riken.jp/~rob/

The latest version of the original [IPython notebook](http://ipython.org/notebook.html) lecture is available at [http://github.com/jrjohansson/scientific-python-lectures](http://github.com/jrjohansson/scientific-python-lectures).

The other notebooks in this lecture series are indexed at [http://jrjohansson.github.com](http://jrjohansson.github.com).

```python
from bokeh.plotting import output_notebook
output_notebook()
```

    Loading BokehJS ...

```python
from bokeh.plotting import figure, show
```

## Introduction

There are two notable Computer Algebra Systems (CAS) for Python:

* [SymPy](http://sympy.org/en/index.html) - A python module that can be used in any Python program, or in an IPython session, that provides powerful CAS features.
* [Sage](http://www.sagemath.org/) - Sage is a full-featured and very powerful CAS enviroment that aims to provide an open source system that competes with Mathematica and Maple. Sage is not a regular Python module, but rather a CAS environment that uses Python as its programming language.

Sage is in some aspects more powerful than SymPy, but both offer very comprehensive CAS functionality. The advantage of SymPy is that it is a regular Python module and integrates well with the IPython notebook.

In this lecture we will therefore look at how to use SymPy with IPython notebooks. If you are interested in an open source CAS environment I also recommend to read more about Sage.

To get started using SymPy in a Python program or notebook, import the module `sympy`:

```python
from sympy import *
```

To get nice-looking $\LaTeX$ formatted output run:

```python
init_printing()

# or with older versions of sympy/ipython, load the IPython extension
#%load_ext sympy.interactive.ipythonprinting
# or
#%load_ext sympyprinting
```

## Symbolic variables

In SymPy we need to create symbols for the variables we want to work with. We can create a new symbol using the `Symbol` class:

```python
x = Symbol('x')
```

```python
(pi + x)**2
```

```python
# alternative way of defining symbols
a, b, c = symbols("a, b, c")
```

```python
type(a)
```

    sympy.core.symbol.Symbol

We can add assumptions to symbols when we create them:

```python
x = Symbol('x', real=True)
```

```python
x.is_imaginary
```

    False

```python
x = Symbol('x', positive=True)
```

```python
x > 0
```

### Complex numbers

The imaginary unit is denoted `I` in Sympy.

```python
1+1*I
```

```python
I**2
```

```python
(x * I + 1)**2
```

### Rational numbers

There are three different numerical types in SymPy: `Real`, `Rational`, `Integer`:

```python
r1 = Rational(4,5)
r2 = Rational(5,4)
```

```python
r1
```

```python
r1+r2
```

```python
r1/r2
```

## Numerical evaluation

SymPy uses a library for arbitrary precision as numerical backend, and has predefined SymPy expressions for a number of mathematical constants, such as: `pi`, `e`, `oo` for infinity.

To evaluate an expression numerically we can use the `evalf` function (or `N`). It takes an argument `n` which specifies the number of significant digits.

```python
pi.evalf(n=50)
```

```python
y = (x + pi)**2
```

```python
N(y, 5) # same as evalf
```

When we numerically evaluate algebraic expressions we often want to substitute a symbol with a numerical value. In SymPy we do that using the `subs` function:

```python
y.subs(x, 1.5)
```

```python
N(y.subs(x, 1.5))
```

The `subs` function can of course also be used to substitute Symbols and expressions:

```python
y.subs(x, a+pi)
```

We can also combine numerical evolution of expressions with NumPy arrays:

```python
import numpy
```

```python
x_vec = numpy.arange(0, 10, 0.1)
```

```python
y_vec = numpy.array([N(((x + pi)**2).subs(x, xx)) for xx in x_vec], dtype=float)
#TODO: note, weird issue with "Float" object (vs numpy float type) serialization to JSON
```

```python
N(y, 5)
```

```python
title = N(y, 5).__str__()
p = figure(title=title, x_axis_label='x', y_axis_label='y')
p.line(x_vec, y_vec)
```

    <bokeh.models.renderers.GlyphRenderer at 0x1006759d0>

```python
show(p)
```

However, this kind of numerical evolution can be very slow, and there is a much more efficient way to do it: Use the function `lambdify` to "compile" a Sympy expression into a function that is much more efficient to evaluate numerically:

```python
f = lambdify([x], (x + pi)**2, 'numpy')  # the first argument is a list of variables that
                                         # f will be a function of: in this case only x -> f(x)
```

```python
y_vec = f(x_vec)  # now we can directly pass a numpy array and f(x) is efficiently evaluated
# Note: this does not have the same "Float" object problem as above.
```

The speedup when using "lambdified" functions instead of direct numerical evaluation can be significant, often several orders of magnitude. Even in this simple example we get a significant speed up:

```python
%%timeit

y_vec = numpy.array([N(((x + pi)**2).subs(x, xx)) for xx in x_vec])
```

    10 loops, best of 3: 32.3 ms per loop

```python
%%timeit

y_vec = f(x_vec)
```

    The slowest run took 13.65 times longer than the fastest. This could mean that an intermediate result is being cached.
    100000 loops, best of 3: 2.48 µs per loop

## Algebraic manipulations

One of the main uses of an CAS is to perform algebraic manipulations of expressions. For example, we might want to expand a product, factor an expression, or simply an expression. The functions for doing these basic operations in SymPy are demonstrated in this section.

### Expand and factor

The first steps in an algebraic manipulation

```python
(x+1)*(x+2)*(x+3)
```

```python
expand((x+1)*(x+2)*(x+3))
```

This is "reversible" with the `factor` command.

The `expand` function takes a number of keywords arguments which we can tell the functions what kind of expansions we want to have performed. For example, to expand trigonometric expressions, use the `trig=True` keyword argument:

```python
sin(a+b)
```

```python
expand(sin(a+b), trig=True)
```

See `help(expand)` for a detailed explanation of the various types of expansions the `expand` functions can perform.

The opposite of product expansion is of course factoring. The factor an expression in SymPy use the `factor` function:

```python
factor(x**3 + 6 * x**2 + 11*x + 6)
```

### Simplify

The `simplify` tries to simplify an expression into a nice looking expression, using various techniques. More specific alternatives to the `simplify` functions also exists: `trigsimp`, `powsimp`, `logcombine`, etc.

The basic usages of these functions are as follows:

```python
# simplify expands a product
simplify((x+1)*(x+2)*(x+3))
```

```python
# simplify uses trigonometric identities
simplify(sin(a)**2 + cos(a)**2)
```

```python
simplify(cos(x)/sin(x))
```

### apart and together

To manipulate symbolic expressions of fractions, we can use the `apart` and `together` functions:

```python
f1 = 1/((a+1)*(a+2))
```

```python
f1
```

```python
apart(f1)
```

```python
f2 = 1/(a+2) + 1/(a+3)
```

```python
f2
```

```python
together(f2)
```

Simplify usually combines fractions but does not factor:

```python
simplify(f2)
```

## Calculus

In addition to algebraic manipulations, the other main use of CAS is to do calculus, like derivatives and integrals of algebraic expressions.

### Differentiation

Differentiation is usually simple. Use the `diff` function. The first argument is the expression to take the derivative of, and the second argument is the symbol by which to take the derivative:

```python
y
```

```python
y**2
```

```python
diff(y**2, x)
```

For higher order derivatives we can do:

```python
diff(y**2, x, x)
```

```python
diff(y**2, x, 2) # same as above
```

To calculate the derivative of a multivariate expression, we can do:

```python
x, y, z = symbols("x,y,z")
```

```python
f = sin(x*y) + cos(y*z)
```

$\frac{d^3f}{dxdy^2}$

```python
diff(f, x, 1, y, 2)
```

## Integration

Integration is done in a similar fashion:

```python
f
```

```python
integrate(f, x)
```

$$x \cos{\left (y z \right )} + \begin{cases} 0 & \text{for}\: y = 0 \\- \frac{1}{y} \cos{\left (x y \right )} & \text{otherwise} \end{cases}$$

By providing limits for the integration variable we can evaluate definite integrals:

```python
integrate(f, (x, -1, 1))
```

and also improper integrals

```python
integrate(exp(-x**2), (x, -oo, oo))
```

Remember, `oo` is the SymPy notation for inifinity.

### Sums and products

We can evaluate sums and products using the functions: 'Sum'

```python
n = Symbol("n")
```

```python
Sum(1/n**2, (n, 1, 10))
```

```python
Sum(1/n**2, (n,1, 10)).evalf()
```

```python
Sum(1/n**2, (n, 1, oo)).evalf()
```

Products work much the same way:

```python
Product(n, (n, 1, 10)) # 10!
```

## Limits

Limits can be evaluated using the `limit` function. For example,

```python
limit(sin(x)/x, x, 0)
```

We can use 'limit' to check the result of derivation using the `diff` function:

```python
f
```

```python
diff(f, x)
```

$\displaystyle \frac{\mathrm{d}f(x,y)}{\mathrm{d}x} = \frac{f(x+h,y)-f(x,y)}{h}$

```python
h = Symbol("h")
```

```python
limit((f.subs(x, x+h) - f)/h, h, 0)
```

OK!

We can change the direction from which we approach the limiting point using the `dir` keywork argument:

```python
limit(1/x, x, 0, dir="+")
```

```python
limit(1/x, x, 0, dir="-")
```

## Series

Series expansion is also one of the most useful features of a CAS. In SymPy we can perform a series expansion of an expression using the `series` function:

```python
series(exp(x), x)
```

By default it expands the expression around $x=0$, but we can expand around any value of $x$ by explicitly include a value in the function call:

```python
series(exp(x), x, 1)
```

And we can explicitly define to which order the series expansion should be carried out:

```python
series(exp(x), x, 1, 10)
```

The series expansion includes the order of the approximation, which is very useful for keeping track of the order of validity when we do calculations with series expansions of different order:

```python
s1 = cos(x).series(x, 0, 5)
s1
```

```python
s2 = sin(x).series(x, 0, 2)
s2
```

```python
expand(s1 * s2)
```

If we want to get rid of the order information we can use the `removeO` method:

```python
expand(s1.removeO() * s2.removeO())
```

But note that this is not the correct expansion of $\cos(x)\sin(x)$ to $5$th order:

```python
(cos(x)*sin(x)).series(x, 0, 6)
```

## Linear algebra

### Matrices

Matrices are defined using the `Matrix` class:

```python
m11, m12, m21, m22 = symbols("m11, m12, m21, m22")
b1, b2 = symbols("b1, b2")
```

```python
A = Matrix([[m11, m12],[m21, m22]])
A
```

$$\left[\begin{matrix}m_{11} & m_{12}\\m_{21} & m_{22}\end{matrix}\right]$$

```python
b = Matrix([[b1], [b2]])
b
```

$$\left[\begin{matrix}b_{1}\\b_{2}\end{matrix}\right]$$

With `Matrix` class instances we can do the usual matrix algebra operations:

```python
A**2
```

$$\left[\begin{matrix}m_{11}^{2} + m_{12} m_{21} & m_{11} m_{12} + m_{12} m_{22}\\m_{11} m_{21} + m_{21} m_{22} & m_{12} m_{21} + m_{22}^{2}\end{matrix}\right]$$

```python
A * b
```

$$\left[\begin{matrix}b_{1} m_{11} + b_{2} m_{12}\\b_{1} m_{21} + b_{2} m_{22}\end{matrix}\right]$$

And calculate determinants and inverses, and the like:

```python
A.det()
```

```python
A.inv()
```

$$\left[\begin{matrix}\frac{1}{m_{11}} + \frac{m_{12} m_{21}}{m_{11}^{2} \left(m_{22} - \frac{m_{12} m_{21}}{m_{11}}\right)} & - \frac{m_{12}}{m_{11} \left(m_{22} - \frac{m_{12} m_{21}}{m_{11}}\right)}\\- \frac{m_{21}}{m_{11} \left(m_{22} - \frac{m_{12} m_{21}}{m_{11}}\right)} & \frac{1}{m_{22} - \frac{m_{12} m_{21}}{m_{11}}}\end{matrix}\right]$$

## Solving equations

For solving equations and systems of equations we can use the `solve` function:

```python
solve(x**2 - 1, x)
```

```python
solve(x**4 - x**2 - 1, x)
```

System of equations:

```python
solve([x + y - 1, x - y - 1], [x,y])
```

In terms of other symbolic expressions:

```python
solve([x + y - a, x - y - c], [x,y])
```

## Quantum mechanics: noncommuting variables

How about non-commuting symbols? In quantum mechanics we need to work with noncommuting operators, and SymPy has a nice support for noncommuting symbols and even a subpackage for quantum mechanics related calculations!

```python
from sympy.physics.quantum import *
```

## States

We can define symbol states, kets and bras:

```python
Ket('psi')
```

```python
Bra('psi')
```

```python
u = Ket('0')
d = Ket('1')

a, b = symbols('alpha beta', complex=True)
```

```python
phi = a * u + sqrt(1-abs(a)**2) * d; phi
```

```python
Dagger(phi)
```

```python
Dagger(phi) * d
```

Use `qapply` to distribute a mutiplication:

```python
qapply(Dagger(phi) * d)
```

$$\overline{\alpha} \left\langle 0 \right. {\left|1\right\rangle } + \overline{\sqrt{- \left\lvert{\alpha}\right\rvert^{2} + 1}} \left\langle 1 \right. {\left|1\right\rangle }$$

```python
qapply(Dagger(phi) * u)
```

$$\overline{\alpha} \left\langle 0 \right. {\left|0\right\rangle } + \overline{\sqrt{- \left\lvert{\alpha}\right\rvert^{2} + 1}} \left\langle 1 \right. {\left|0\right\rangle }$$

### Operators

```python
A = Operator('A')
B = Operator('B')
```

Check if they are commuting!

```python
A * B == B * A
```

    False

```python
expand((A+B)**3)
```

$$A B A + A \left(B\right)^{2} + \left(A\right)^{2} B + \left(A\right)^{3} + B A B + B \left(A\right)^{2} + \left(B\right)^{2} A + \left(B\right)^{3}$$

```python
c = Commutator(A,B)
c
```

$$\left[A,B\right]$$

We can use the `doit` method to evaluate the commutator:

```python
c.doit()
```

$$A B - B A$$

We can mix quantum operators with C-numbers:

```python
c = Commutator(a * A, b * B)
c
```

$$\alpha \beta \left[A,B\right]$$

To expand the commutator, use the `expand` method with the `commutator=True` keyword argument:

```python
c = Commutator(A+B, A*B)
c.expand(commutator=True)
```

$$- \left[A,B\right] B + A \left[A,B\right]$$

```python
Dagger(Commutator(A, B))
```

$$- \left[A^{\dagger},B^{\dagger}\right]$$

```python
ac = AntiCommutator(A,B)
```

```python
ac.doit()
```

$$A B + B A$$

#### Example: Quadrature commutator

Let's look at the commutator of the electromagnetic field quadatures $x$ and $p$. We can write the quadrature operators in terms of the creation and annihilation operators as:

$\displaystyle x = (a + a^\dagger)/\sqrt{2}$

$\displaystyle p = -i(a - a^\dagger)/\sqrt{2}$

```python
X = (A + Dagger(A))/sqrt(2)
X
```

$$\frac{\sqrt{2}}{2} \left(A^{\dagger} + A\right)$$

```python
P = -I * (A - Dagger(A))/sqrt(2)
P
```

$$- \frac{\sqrt{2} i}{2} \left(- A^{\dagger} + A\right)$$

Let's expand the commutator $[x,p]$

```python
Commutator(X, P).expand(commutator=True).expand(commutator=True)
```

$$- i \left[A^{\dagger},A\right]$$

Here we see directly that the well known commutation relation for the quadratures

$[x,p]=i$

is a directly related to

$[A, A^\dagger]=1$

(which SymPy does not know about, and does not simplify).

For more details on the quantum module in SymPy, see:

* http://docs.sympy.org/0.7.2/modules/physics/quantum/index.html
* http://nbviewer.ipython.org/urls/raw.github.com/ipython/ipython/master/docs/examples/notebooks/sympy_quantum_computing.ipynb

## Further reading

* http://sympy.org/en/index.html - The SymPy projects web page.
* https://github.com/sympy/sympy - The source code of SymPy.
* http://live.sympy.org - Online version of SymPy for testing and demonstrations.

## Versions

```python
# must have version_information.py file in local dir or appropriate PATH
%reload_ext version_information

%version_information numpy, sympy, bokeh
```

Software	Version

Python	2.7.9 64bit [GCC 4.2.1 (Apple Inc. build 5577)]

IPython	3.1.0

OS	Darwin 14.3.0 x86_64 i386 64bit

numpy	1.9.2

sympy	0.7.6

bokeh	0.8.2

Sun May 17 18:42:45 2015 CDT

```python

```


# Section 4.9 $\quad$ Rank of a Matrix (continued)

### Example 1

Let $A$ be a $6\times 4$ matrix.
- What is the largest possible value of rank (A)?
- Are rows of $A$ linearly independent?
- Are columns of $A$ linearly independent?

### Example 2

Let $A$ be a $4\times 5$ matrix.
- List all possible values of the rank of $A$.
- If the rank of $A$ is 4, what is the dimension of its column space?
- If the rank of $A$ is 3, what is the dimension of the solution space of $A\mathbf{x}=\mathbf{0}$?

### Example 3

Let
\begin{equation*}
  A =
  \left[
    \begin{array}{cccc}
      1 & 1 & 4 & 1  \\
      0 & 1 & 2 & 1  \\
      0 & 0 & 0 & 1  \\
      1& -1 & 0 & 0  \\
      2 & 1 & 6 & 0  \\
    \end{array}
  \right]
\end{equation*}
Find the rank of $A$ and the dimension of the solution space of $A\mathbf{x}=\mathbf{0}$.

```python
from sympy import *

A = Matrix([[1, 1, 4, 1], [0, 1, 2, 1], [0, 0, 0, 1], [1, -1, 0, 0], [2, 1, 6, 0]]);

A.rank()
```

    3

```python
from sympy import *

A = Matrix([[1, 1, 4, 1], [0, 1, 2, 1], [0, 0, 0, 1], [1, -1, 0, 0], [2, 1, 6, 0]]);
x1, x2, x3, x4 = symbols('x1 x2 x3 x4');
x = Matrix([x1, x2, x3, x4]);

solve(A.dot(x))
```

    [{x2: -2*x3, x1: -2*x3, x4: 0}]

>**Theorem** If $A$ is an $m\times n$ matrix, then

## Properties of Rank

Let $A$ be an $n\times n$ matrix.
- rank(A) = n if and only if $A$ is row equivalent to $\underline{\hspace{.3in}}$.

- $A$ is nonsingular if and only if rank($A$) =  $\underline{\hspace{.3in}}$.

- rank(A) = n if and only if $\det(A)$ $\underline{\hspace{.3in}}$.

- The homogeneous system $A\mathbf{x} = \mathbf{0}$ has a nontrivial solution if and only if rank($A$) $\underline{\hspace{.3in}}$.

- The nonhomogeneous system $A\mathbf{x} = \mathbf{b}$ has a unique solution if and only if rank($A$) $\underline{\hspace{.3in}}$.

### Example 4

Determine whether the homogeneous system has a nontrivial solution.
\begin{equation*}
    \left[
    \begin{array}{cccc}
      1 & 1 & 2 & -1  \\
      1 & 3 & -1& 2  \\
      1 & 1 & 1 & 3  \\
      1 & 2 & 3 & 1  \\
    \end{array}
  \right]
  \left[
    \begin{array}{c}
      x_1 \\ x_2 \\ x_3 \\ x_4  \\
    \end{array}
  \right] =
  \left[
    \begin{array}{c}
      0 \\ 0 \\ 0 \\ 0  \\
    \end{array}
  \right]
\end{equation*}

```python
from sympy import *

A = Matrix([[1, 1, 2, -1], [1, 3, -1, 2], [1, 1, 1, 3], [1, 2, 3, 1]]);

A.rank()
```

    4

## Application to Nonhomogeneous System $A\mathbf{x}=\mathbf{b}$

>**Theorem** The linear system $A\mathbf{x} = \mathbf{b}$ has a solution if and only if

### Example 5

Determine whether the nonhomogeneous system has a solution.
\begin{equation*}
    \left[
    \begin{array}{ccc}
      1 & 2 & 3   \\
      1 & -3 &4  \\
      2 & -1 &7  \\
    \end{array}
  \right]
  \left[
    \begin{array}{c}
      x_1 \\ x_2 \\ x_3  \\
    \end{array}
  \right] =
  \left[
    \begin{array}{c}
      4 \\ 5 \\ 6 \\
    \end{array}
  \right]
\end{equation*}

```python
from sympy import *

A = Matrix([[1, 2, 3], [1, -3, 4], [2, -1, 7]]);
x1, x2, x3 = symbols('x1 x2 x3');
x = Matrix([x1, x2, x3]);

solve(Matrix(A.dot(x)) - Matrix([4, 5, 6]))
```

    []

**Remark** For the nonhomogeneous system $A\mathbf{x}= \mathbf{b}$, where $A$ is an $n\times n$ matrix,
- If $rank(A) = n$,

- If $rank(A) < n$, and $rank(A) = rank([A~|~\mathbf{b}])$,

- If $rank(A) < n$, and $rank(A) \neq rank([A~|~\mathbf{b}])$,


## Breakeven Analysis: 3D Printing vs. Injection Molding ##

Detemine the breakeven point when comparing the production of the plastic enclosure for the SomniCloud
 - __Given__: Enclosure volume is $2.57 in^3$

```python
part_vol = 2.57 # in^3
```

### 3D Printing Specs ###

- \$4.25 / cubic inch of ABS
- Tooling Cost: \$0.00
- Machine Time Cost: \$12.00 / hour
- Print Time: 4 hours

```python
# Set 3D ABS variable constants
usd_in3_3d = 4.25 # $/in^3
setup_3d = 0.00 # $ Setup Cost
mc_3d = 12.00 # $/hr
time_3d = 4.0 # hrs
```

### Injection Mold Data ###

- \$4.50 / lb ABS
- Mold Cost: \$6500.00
- Machine Time Cost: \$80.00 / hour
- Cycle Time: 30 seconds

```python
# Set Injection Mold ABS variable constants
usd_lb_im = 6.50 # $/lb
setup_im = 50000.00 # $ Setup Cost
mc_im = 120.00 # $/hr
time_im = 60 / 3600 # hrs
```

### Step 1: Convert \$ / lb to \$ / cubic inch ###
- Density of ABS: $\rho_{ABS} = 1.07 g/cm^{3}$

\begin{equation}
\left(\frac{\$4.50}{lb}\right) \
\left(\frac{1.07g}{cm^{3}}\right) \
\left(\frac{2.2lb}{1000g}\right) \
\left(\frac{2.54cm}{1in}\right)^{3} \
= X \frac{\$}{in_{3}}
\end{equation}

```python
# Create variables for given constants
rho_abs = 1.07 # g/cm^3
```

```python
# Calculate $ per in^3 of injection molded ABS
usd_in3_im = (usd_lb_im) * (rho_abs) * (2.2 / 1000) * ((2.54 / 1)**3) # $/in^3
print(f"Injection Molded ABS [$/in^3]: {usd_in3_im}")
```

    Injection Molded ABS [$/in^3]: 0.25073846626400004

### Step 2: Calculate Cost Functions ###

- $Total\ Cost = (Cost\ per\ Unit) * (Number\ of\ Units) + (Setup\ Cost)$

- $Cost\ per\ Unit = (ABS\ Cost\ per\ Unit) + (Machine\ Time\ Cost\ per\ Unit)$

- $ABS\ Cost\ per\ Unit = (ABS\ Cost\ per\ in^{3})  * (Part Volume)$

- $Cost\ per\ Unit = (Machine\ Time\ Cost\ per\ Hour) + (Machine\ Time)$

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import math
N = np.linspace(0,1000,10000)
```

### Cost Function for 3D Printing ###

```python
# Cost per unit for 3D printing
abs_cpu_3d = usd_in3_3d * part_vol # ABS Cost per Unit
mc_cpu_3d = mc_3d * time_3d # Machine Cost per Unit
t_cpu_3d = abs_cpu_3d + mc_cpu_3d # Total per unit cost

# Cost function for 3D printing
C_3d = (t_cpu_3d)*N + setup_3d

print(f"3D Printing Cost per Unit: ${t_cpu_3d:.2f}")
print(f"3D Printing Setup Cost: ${setup_3d:.2f}")
```

    3D Printing Cost per Unit: $58.92
    3D Printing Setup Cost: $0.00

### Cost Function for Injection Molding Printing ###

```python
# Cost per unit for 3D printing
abs_cpu_im = usd_in3_im * part_vol # ABS Cost per Unit
mc_cpu_im = mc_im * time_im # Machine Cost per Unit
t_cpu_im = abs_cpu_im + mc_cpu_im # Total per unit cost

# Cost function for 3D printing
C_im = (t_cpu_im)*N + setup_im

print(f"3D Printing Cost per Unit: ${t_cpu_im:.2f}")
print(f"3D Printing Setup Cost: ${setup_im:.2f}")
```

    3D Printing Cost per Unit: $2.64
    3D Printing Setup Cost: $50000.00

### Step 3: Plot Results ###

```python
fig, ax = plt.subplots(figsize=(12,8))

ax.plot(N,C_3d, label=r"3D Printing")
ax.plot(N,C_im, label=r"Injection Molding")

idx = np.argwhere(np.diff(np.sign(C_3d - C_im)) != 0).reshape(-1) + 0
ax.plot(N[idx], C_3d[idx], 'o', markersize=8, color='black')

ax.legend(loc=2) # upper left corner
ax.set_xlabel(r'# of Units', fontsize=18)
ax.set_ylabel(r'Total Cost', fontsize=18)
ax.set_title('Breakeven Analysis')

ax.set_xlim([min(N), max(N)])
ax.set_ylim([0, max(C_3d)])
plt.show()

print(f"Breakeven point = {math.ceil(N[idx])} units")
```

```python

```


A $\textbf {Binomial Distribution}$ can be thought of as simply the probability of a SUCCESS or FAILURE outcome in an experiment or survey that is repeated multiple times. The binomial is a type of distribution that has two possible outcomes (the prefix “bi” means two, or twice).

$\textbf {For example,}$ a coin toss has only two possible outcomes: heads or tails and taking a test could have two possible outcomes: pass or fail.

The Probability Density Function (PDF) for $\textbf {Binomial Distribution}$ is defined as::

\begin{equation}
P(X=K) = {N\choose K}{p}^K{q}^N-K
\end{equation}

$\textbf {Coin Tossing Example:}$

If we toss a coin 5 times and find the probability of getting a 1 head.
The following events can happen are:

 H,T,T,T,T

 T,H,T,T,T

 T,T,H,T,T

 T,T,T,H,T

 T,T,T,T,H

$Probability(P) = P(H{\cap T \cap T \cap T \cap T})$

We also know that coin tossing is a independent probability and it will not affect each other's probability.So, each probabilities will be multiplied.

$Probability(P) = P(H).P(T).P(T).P(T).P(T)$

Let's assume a coin is tossed for 25 crores times and Head (H) comes for 65% times.

Probability of getting Head = P(H) = p (Success of getting Head) = 0.65

Probability of getting Tail = P(T) = q (Failure of getting Head or Success of getting Tail)
                                   = 1-p = 0.35

Conditions for Binomial Random Variable:

 The outcomes of the random variable should be 2.

 They should an independent event

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.stats as S
```

```python
p = 0.5
N = 10
q = 1-p
```

```python
RandomValues = np.arange(0,N+1)
```

```python
RandomValues
```

    array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10])

```python
BinomialValuesPDF = S.binom.pmf(RandomValues,N,p)
```

```python
BinomialValuesPDF
```

    array([0.00097656, 0.00976563, 0.04394531, 0.1171875 , 0.20507813,
           0.24609375, 0.20507813, 0.1171875 , 0.04394531, 0.00976563,
           0.00097656])

```python
plt.stem(RandomValues,BinomialValuesPDF)
```

The skewness at the x-axis of the graph will be dependent on the values of the p (Success) and q(Failure).

 If p is greater than q the skewness towards Right side

 If p is less than q the skewness towards Left side

Let's see this by this example.

```python
p = 0.7
N = 10
q = 1-p
```

```python
RandomValues = np.arange(0,N+1)
```

```python
BinomialValuesPDF = S.binom.pmf(RandomValues,N,p)
```

```python
plt.stem(RandomValues,BinomialValuesPDF)
```

```python
p = 0.3
N = 10
q = 1-p
```

```python
RandomValues = np.arange(0,N+1)
```

```python
BinomialValuesPDF = S.binom.pmf(RandomValues,N,p)
```

```python
plt.stem(RandomValues,BinomialValuesPDF)
```

The mean of the binomial distribution is:
$ E[X]=Np $

The Variance of the binomial distribution is : $ E[(X-E[X]^2)] = Npq $

The Standard deviation of the binomial distribution is : $\sqrt {Npq} $

```python

```


# Black-Scholes European Option Pricing Script

```python
# File Contains: Python code containing closed-form solutions for the valuation of European Options,
# for backward compatability with Python 2.7
from __future__ import division

# import necessary libaries
import math
import numpy as np
from scipy.stats import norm
from scipy.stats import mvn

# Plotting
import matplotlib.pylab as pl
import numpy as np
```

# Option Pricing Theory: Black-Scholes model
Black Scholes genre option models widely used to value European options. The original “Black Scholes” model was published in 1973 for non-dividend paying stocks. Since that time, a wide variety of extensions to the original Black Scholes model have been created. Modifications of the formula are used to price other financial instruments like dividend paying stocks, commodity futures, and FX forwards. Mathematically, these formulas are nearly identical. The primary difference between these models is whether the asset has a carrying cost (if the asset has a cost or benefit associated with holding it) and how the asset gets present valued. To illustrate this relationship, a “generalized” form of the Black Scholes equation is shown below.

The Black Scholes model is based on number of assumptions about how financial markets operate. Black Scholes style models assume:

1.	**Arbitrage Free Markets**. Black Scholes formulas assume that traders try to maximize their personal profits and don’t allow arbitrage opportunities (riskless opportunities to make a profit) to persist.
2.	**Frictionless, Continuous Markets**. This assumption of frictionless markets assumes that it is possible to buy and sell any amount of the underlying at any time without transaction costs.
3.	**Risk Free Rates**. It is possible to borrow and lend money at a risk-free interest rate
4.	**Log-normally Distributed Price Movements**. Prices are log-normally distributed and described by Geometric Brownian Motion
5.	**Constant Volatility**. The Black Scholes genre options formulas assume that volatility is constant across the life of the option contract.

In practice, these assumptions are not particularly limiting. The primary limitation imposed by these models is that it is possible to (reasonably) describe the dispersion of prices at some point in the future in a mathematical equation.

An important concept of Black Scholes models is that the actual way that the underlying asset drifts over time isn't important to the valuation. Since European options can only be exercised when the contract expires, it is only the distribution of possible prices on that date that matters - the path that the underlying took to that point doesn't affect the value of the option. This is why the primary limitation of the model is being able to describe the dispersion of prices at some point in the future, not that the dispersion process is simplistic.

The generalized Black-Scholes formula can found below (see *Figure 1 – Generalized Black Scholes Formula*). While these formulas may look complicated at first glance, most of the terms can be found as part of an options contract or are prices readily available in the market.  The only term that is difficult to calculate is the implied volatility (σ). Implied volatility is typically calculated using prices of other options that have recently been traded.

>*Call Price*
>\begin{equation}
C = Fe^{(b-r)T} N(D_1) - Xe^{-rT} N(D_2)
\end{equation}

>*Put Price*
>\begin{equation}
P = Xe^{-rT} N(-D_2) - Fe^{(b-r)T} N(-D_1)
\end{equation}

>*with the following intermediate calculations*

>\begin{equation}
D_1 = \frac{ln\frac{F}{X} + (b+\frac{V^2}{2})T}{V*\sqrt{T}}
\end{equation}

>\begin{equation}
D_2 = D_1 - V\sqrt{T}
\end{equation}

>*and the following inputs*

>|    Symbol    |    Meaning    |
>|--------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
>|    F or S   |    **Underlying Price**. The price of the underlying asset on the valuation date. S is used commonly used to represent a spot price, F a forward price     |
>|    X    |    **Strike Price**.   The strike, or exercise, price of the option.    |
>|    T    |    **Time to expiration**. The time to expiration in years. This can be calculated   by comparing the time between the expiration date and the valuation date.   T   = (t_1 - t_0)/365    |
>|    t_0    |    **Valuation Date**.   The date on which the option is being valued. For example, it might be   today’s date if the option we being valued today.    |
>|    t_1    |    **Expiration Date**.   The date on which the option must be exercised.    |
>|    V    |    **Volatility**.   The volatility of the underlying security. This factor usually cannot be directly observed in the market. It is most often calculated by looking at the prices for recent option transactions and back-solving a Black Scholes style equation to find the volatility that would result in the observed price. This is commonly abbreviated with the greek letter sigma,σ, although V is used here for consistency with the code below.    |
>|    q    |    **Continuous Yield**.   Used in the Merton model, this is the continuous yield of the underlying   security. Option holders are typically not paid dividends or other payments   until they exercise the option. As a result, this factor decreases the value   of an option.    |
>|    r    |    **Risk Free Rate**. This is expected return on a risk-free investment. This   is commonly a approximated by the yield on a low-risk government bond or the   rate that large banks borrow between themselves (LIBOR). The rate depends on   tenor of the cash flow. For example, a 10-year risk-free bond is likely to   have a different rate than a 20-year risk-free bond.[DE1]     |
>|    rf    |    **Foreign   Risk Free Rate**. Used in the Garman Kohlhagen model, this is the risk   free rate of the foreign currency. Each currency will have a risk free rate.    |
>*Figure 1 - Generalized Black Scholes Formula*

## Model Implementation
These functions encapsulate a generic version of the pricing formulas. They are primarily intended to be called by the other functions within this libary. The following functions will have a fixed interface so that they can be called directly for academic applicaitons that use the cost-of-carry (b) notation:

    _GBS()                  A generalized European option model
    _GBS_ImpliedVol()       A generalized European option implied vol calculator

The other functions in this libary are called by the four main functions and are not expected to be interface safe (the implementation and interface may change over time).

### Implementation for European Options

```python
# The primary class for calculating Generalized Black Scholes option prices and deltas
# It is not intended to be part of this module's public interface

# Inputs: option_type = "p" or "c", fs = price of underlying, x = strike, t = time to expiration, r = risk free rate
#         b = cost of carry, v = implied volatility
# Outputs: value, delta, gamma, theta, vega, rho
def _gbs(option_type, fs, x, t, r, b, v):
    _debug("Debugging Information: _gbs()")

    # -----------
    # Create preliminary calculations
    t__sqrt = math.sqrt(t)
    d1 = (math.log(fs / x) + (b + (v * v) / 2) * t) / (v * t__sqrt)
    d2 = d1 - v * t__sqrt

    if option_type == "c":
        # it's a call
        _debug("     Call Option")
        value = fs * math.exp((b - r) * t) * norm.cdf(d1) - x * math.exp(-r * t) * norm.cdf(d2)
        delta = math.exp((b - r) * t) * norm.cdf(d1)
        gamma = math.exp((b - r) * t) * norm.pdf(d1) / (fs * v * t__sqrt)
        theta = -(fs * v * math.exp((b - r) * t) * norm.pdf(d1)) / (2 * t__sqrt) - (b - r) * fs * math.exp(
            (b - r) * t) * norm.cdf(d1) - r * x * math.exp(-r * t) * norm.cdf(d2)
        vega = math.exp((b - r) * t) * fs * t__sqrt * norm.pdf(d1)
        rho = x * t * math.exp(-r * t) * norm.cdf(d2)
    else:
        # it's a put
        _debug("     Put Option")
        value = x * math.exp(-r * t) * norm.cdf(-d2) - (fs * math.exp((b - r) * t) * norm.cdf(-d1))
        delta = -math.exp((b - r) * t) * norm.cdf(-d1)
        gamma = math.exp((b - r) * t) * norm.pdf(d1) / (fs * v * t__sqrt)
        theta = -(fs * v * math.exp((b - r) * t) * norm.pdf(d1)) / (2 * t__sqrt) + (b - r) * fs * math.exp(
            (b - r) * t) * norm.cdf(-d1) + r * x * math.exp(-r * t) * norm.cdf(-d2)
        vega = math.exp((b - r) * t) * fs * t__sqrt * norm.pdf(d1)
        rho = -x * t * math.exp(-r * t) * norm.cdf(-d2)

    _debug("     d1= {0}\n     d2 = {1}".format(d1, d2))
    _debug("     delta = {0}\n     gamma = {1}\n     theta = {2}\n     vega = {3}\n     rho={4}".format(delta, gamma,
                                                                                theta, vega,
                                                                                rho))

    return value, delta, gamma, theta, vega, rho
```

### Implementation: Implied Volatility
This section implements implied volatility calculations. It contains implementation of a **Newton-Raphson Search.** This is a fast implied volatility search that can be used when there is a reliable estimate of Vega (i.e., European options)

```python
# ----------
# Find the Implied Volatility of an European (GBS) Option given a price
# using Newton-Raphson method for greater speed since Vega is available

#def _gbs_implied_vol(option_type, fs, x, t, r, b, cp, precision=.00001, max_steps=100):
#    return _newton_implied_vol(_gbs, option_type, x, fs, t, b, r, cp, precision, max_steps)
```

### Public Interface for valuation functions
This section encapsulates the functions that user will call to value certain options. These function primarily figure out the cost-of-carry term (b) and then call the generic version of the function (like _GBS() or _American). All of these functions return an array containg the premium and the greeks.

```python
# ---------------------------
# Black Scholes: stock Options (no dividend yield)
# Inputs:
#    option_type = "p" or "c"
#    fs          = price of underlying
#    x           = strike
#    t           = time to expiration
#    v           = implied volatility
#    r           = risk free rate
#    q           = dividend payment
#    b           = cost of carry
# Outputs:
#    value       = price of the option
#    delta       = first derivative of value with respect to price of underlying
#    gamma       = second derivative of value w.r.t price of underlying
#    theta       = first derivative of value w.r.t. time to expiration
#    vega        = first derivative of value w.r.t. implied volatility
#    rho         = first derivative of value w.r.t. risk free rates
def BlackScholes(option_type, fs, x, t, r, v):
    b = r
    return _gbs(option_type, fs, x, t, r, b, v)
```

### Public Interface for implied Volatility Functions

```python
# Inputs:
#    option_type = "p" or "c"
#    fs          = price of underlying
#    x           = strike
#    t           = time to expiration
#    v           = implied volatility
#    r           = risk free rate
#    q           = dividend payment
#    b           = cost of carry
# Outputs:
#    value       = price of the option
#    delta       = first derivative of value with respect to price of underlying
#    gamma       = second derivative of value w.r.t price of underlying
#    theta       = first derivative of value w.r.t. time to expiration
#    vega        = first derivative of value w.r.t. implied volatility
#    rho         = first derivative of value w.r.t. risk free rates
```

```python
#def euro_implied_vol(option_type, fs, x, t, r, q, cp):
#    b = r - q
#    return _gbs_implied_vol(option_type, fs, x, t, r, b, cp)
```

### Implementation: Helper Functions
These functions aren't part of the main code but serve as utility function mostly used for debugging

```python
# ---------------------------
# Helper Function for Debugging

# Prints a message if running code from this module and _DEBUG is set to true
# otherwise, do nothing

# Developer can toggle _DEBUG to True for more messages
# normally this is set to False
_DEBUG = False

def _debug(debug_input):
    if (__name__ is "__main__") and (_DEBUG is True):
        print(debug_input)
```

## Real Calculations of Options Prices

```python
bs = BlackScholes('c', fs=60, x=65, t=0.25, r=0.08, v=0.30)
optionPrice = bs[0]
optionPrice
```

    2.1333684449162007

## Option prices charts

```python
stockPrices = np.arange(50, 100, 1)
prices = stockPrices * 0

stockPrice = 60
strike = 65
timeToExpiration = 0.25
impliedVolatility = 0.30
riskFreeRate = 0.05
pl.title('Stock Option Price')

for i in range(len(stockPrices)):
    prices[i] = BlackScholes('c', stockPrices[i], strike, t = timeToExpiration, r = riskFreeRate, v = impliedVolatility)[0]

pl.plot(stockPrices, prices, label = 'Option Price')
pl.xlabel("Stock Price")
pl.ylabel("Option Price")
pl.grid(True)
pl.show()
```

```python
timeToExpiration = np.arange(0.1, 1, 0.05)
prices = timeToExpiration * 0

stockPrice = 60
strike = 65
#timeToExpiration = 0.25
impliedVolatility = 0.30
riskFreeRate = 0.05
pl.title('Stock Option Price')

for i in range(len(prices)):
    prices[i] = BlackScholes('c', stockPrice, strike, t = timeToExpiration[i], r = riskFreeRate, v = impliedVolatility)[0]

pl.plot(timeToExpiration, prices, label = 'Option Price')
pl.xlabel("Time to Expiry")
pl.ylabel("Option Price")
pl.grid(True)
pl.show()
```

```python
strikes = np.arange(50, 80, 1)
prices = strikes * 0

stockPrice = 60
strike = 65
timeToExpiration = 0.25
impliedVolatility = 0.30
riskFreeRate = 0.05
pl.title('Stock Option Price')

for i in range(len(prices)):
    prices[i] = BlackScholes('c', stockPrice, strikes[i], t = timeToExpiration, r = riskFreeRate, v = impliedVolatility)[0]

pl.plot(strikes, prices, label = 'Option Price')
pl.xlabel("Striking Price")
pl.ylabel("Option Price")
pl.grid(True)
pl.show()
```

```python
strikes = np.arange(50, 80, 1)
prices = strikes * 0

stockPrice = 60
strike = 65
timeToExpiration = 0.25
impliedVolatility = 0.30
riskFreeRate = 0.05
pl.title('Stock Put Option Price')

for i in range(len(prices)):
    prices[i] = BlackScholes('p', stockPrice, strikes[i], t = timeToExpiration, r = riskFreeRate, v = impliedVolatility)[0]

pl.plot(strikes, prices, label = 'Option Price')
pl.xlabel("Striking Price")
pl.ylabel("Option Price")
pl.grid(True)
pl.show()
```

```python

```


## 2. Nonstationary Bandit
The k-armed bandit problem in 'simple_bandit.ipynb) was stationary in that the expected reward for taking an action did not change with time (i.e. the mean and variance was fixed). In some problems, it is possible that the expected reward changes with time. Such problems are said to be nonstationary

From the incremental update rule,
\begin{align}
Q_{n+1} & = Q_n + \alpha[R_n - Q_n] \\
& = \alpha R_n + (1 - \alpha)Q_n \\
& = \alpha R_n + (1 - \alpha)\left[\alpha R_{n-1} + (1-\alpha)Q_{n-1} \right] \\
& = \alpha R_n + (1 - \alpha)\alpha R_{n-1} + (1-\alpha)^2Q_{n-1} \\
& = \alpha R_n + (1 - \alpha)\alpha R_{n-1} + (1-\alpha)^2\alpha R_{n-2} + ... + (1-\alpha)^{n-1}\alpha R_1 + (1-\alpha)^n Q_1 \\
& = (1-\alpha)^n Q_1 + \sum_{i=1}^n \alpha(1-\alpha)^{n-i} R_i
\end{align}

This is called a **weighted average**. Note that the weight given to a previous reward decays exponentially as the number of intervening rewards increases. $\alpha$ is called the step-size parameter. How do we set $\alpha$? In some cases, it is conveneint to vary it with time. Let $\alpha_n(a)$ denote the step-size parameter used to process the reward received after the nth selection of action a. In the sample average method, $\alpha_n(a) = \frac{1}{n}$. Another often used variant is the constant step-size parameter, $\alpha_n(a) = \alpha$.

Below we investigate the effect of $\alpha$ on the performance of an agent in a nonstationary k-armed bandit problem.

#### Exercise 2.3  $^1$
Design and conduct an experiment to demonstrate the difficulties that sample-average methods have for nonstationary problems. Use a modified version of the 10-armed testbed in which all the $q_*(a)$ start out equal and then take independent random walks. Prepare plots like Figure 2.2 for an action-value method using sample averages, incrementally computed by $\alpha = \frac{1}{n}$, and another action-value method using a constant step-size parameter, $\alpha = 0.1$. Use $\epsilon = 0.1$ and if necessary, runs longer than 1000 steps

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
def sample_average_step_size(N, i):
    """
    :param N: N[i] is the no. of times action i has been executed
    :param i: The index of the selected action
    """
    return 1.0 / N[i]

def constant_step_size(N, i):
    return 0.1
```

```python
def get_reward(true_values, a):
    """
    Returns the reward for selecting action a.
    Reward is selected around true_values[a] with unit variance (as in problem description)
    :param true_values: true_values[i] is the expected reward for action i
    :param a: index of action to return reward for
    """
    reward = np.random.normal(true_values[a], size=1)[0]
    return reward
```

```python
def random_walk(true_values):
    """
    Updates each expected reward by random amounts to imitate a nonstationary reward
    :param true_values: true_values[i] is the expected reward for action i
    """
    true_values += np.random.uniform(-1.0, 1.0, true_values.shape)
```

```python
# We get rid of the sample_average() method and modify k_armed_bandit as follows:
def k_armed_bandit(k, epsilon, iterations, step_fn):
    """
    Performs a single run of the k-armed bandit experiment
    :param k: the number of arms
    :param epsilon: Value of epsilon for epoch-greedy action selection
    :param iterations: number of steps in a single run
    :param step_fn: step-size function
    """
    # Equal action values at start
    true_values = np.ones(k) * np.random.uniform(-1.0, 1.0)

    # Estimates of action values
    Q = np.zeros(k)

    # N[i] is the no. of times action i has been taken
    N = np.zeros(k)

    # Store the rewards received for this experiment
    rewards = []

    # Track how often the optimal action was selected
    optimal = []

    for _ in range(iterations):
        prob = np.random.rand(1)

        if prob > epsilon:
            # Greedy (exploit current knowledge)
            a = np.random.choice(np.flatnonzero(Q == Q.max()))
        else:
            # Explore (take random action)
            a = np.random.randint(0, k)

        reward = get_reward(true_values, a)

        # Update statistics for executed action
        N[a] += 1
        Q[a] += step_fn(N, a) * (reward - Q[a])

        rewards.append(reward)
        optimal.append(1 if a == true_values.argmax() else 0)

        # Update the rewards after each step
        random_walk(true_values)

    return rewards, optimal
```

```python
def experiment(k, epsilon, iters, epochs, step_fn):
    """
    Runs the k-armed bandit experiment
    :param k: the number of arms
    :param epsilon: the value of epsilon for epoch-greedy action selection
    :param iters: the number of steps in a single run
    :param epochs: the number of runs to execute
    :param step_fn: the step-size function
    """
    rewards = []
    optimal = []

    for i in range(epochs):
        r, o = k_armed_bandit(k, epsilon, iters, step_fn)
        rewards.append(r)
        optimal.append(o)

    print('Experiment with \u03b5 = {} completed.'.format(epsilon))

    # Compute the mean reward for each iteration
    r_means = np.mean(rewards, axis=0)
    o_means = np.mean(optimal, axis=0)

    return r_means, o_means
```

```python
k = 10
epsilon = 0.1
iters = 5000
runs = 2000

# Experiment with sample-average step-size and constant step-size
r_exp1, o_exp1 = experiment(k, epsilon, iters, runs, sample_average_step_size)
r_exp2, o_exp2 = experiment(k, epsilon, iters, runs, constant_step_size)
```

    Experiment with ε = 0.1 completed.
    Experiment with ε = 0.1 completed.

```python
x = range(iters)
plt.plot(x, r_exp1, label='sample average step-size')
plt.plot(x, r_exp2, label='constant step-size')
plt.xlabel('Steps')
plt.ylabel('Average reward')
plt.legend()
plt.show()
```

```python
plt.plot(x, o_exp1, label='sample average step-size')
plt.plot(x, o_exp2, label='constant step-size')
plt.xlabel('Steps')
plt.ylabel('% Optimal action')
plt.legend()
plt.show()
```

The results show that indeed, the sample average method suffers when used in a nonstationary environment. As n increases, the update to Q[a] decreases and so our estimates suffer leading to poor results.

## References
1. Richard S. Sutton, Andrew G. Barto (1998). Reinforcement Learning: An Introduction. MIT Press.


```python
surf_choice = 'torus'
```

```python
from sympy import init_printing; init_printing();
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"
```

```python
from silkpy.symbolic.surface.surface import ParametricSurface
from sympy import symbols, sin, cos, pi, cot, Array, refine, Q
from silkpy.sympy_utility import dot
u, v = symbols('u, v', real=True)
```

```python
if surf_choice=='cylindrical':
    R = symbols('R', positive=True)
    s = ParametricSurface([u, v], [R*cos(u), R*sin(u), v])
elif surf_choice=='cone':
    w = symbols('omega', real=True)
    s = ParametricSurface([u, v], [v*cos(u), v*sin(u), v*cot(w)])
elif surf_choice=='Mobius':
    theta = symbols('theta', real=True)
    s = ParametricSurface([theta, v],
        Array([cos(theta),                sin(theta),                0                ]) +
        Array([sin(theta/2) * cos(theta), sin(theta/2) * sin(theta), cos(theta/2)]) * v)
elif surf_choice=='torus':
    from sympy import Q, ask
    from sympy.assumptions import global_assumptions
    a, r = symbols('a, r', real=True, positive=True)
    global_assumptions.add(Q.positive(a + r*cos(u)))
    s = ParametricSurface([u, v], [ (a+r*cos(u)) * cos(v), (a+r*cos(u)) * sin(v), r*sin(u)])
```

```python
s.christoffel_symbol.tensor()
```

    /home/wenyin/pyvenv/vsym/lib/python3.8/site-packages/IPython/lib/latextools.py:126: MatplotlibDeprecationWarning:
    The to_png function was deprecated in Matplotlib 3.4 and will be removed two minor releases later. Use mathtext.math_to_image instead.
      mt.to_png(f, s, fontsize=12, dpi=dpi, color=color)
    /home/wenyin/pyvenv/vsym/lib/python3.8/site-packages/IPython/lib/latextools.py:126: MatplotlibDeprecationWarning:
    The to_rgba function was deprecated in Matplotlib 3.4 and will be removed two minor releases later. Use mathtext.math_to_image instead.
      mt.to_png(f, s, fontsize=12, dpi=dpi, color=color)
    /home/wenyin/pyvenv/vsym/lib/python3.8/site-packages/IPython/lib/latextools.py:126: MatplotlibDeprecationWarning:
    The to_mask function was deprecated in Matplotlib 3.4 and will be removed two minor releases later. Use mathtext.math_to_image instead.
      mt.to_png(f, s, fontsize=12, dpi=dpi, color=color)
    /home/wenyin/pyvenv/vsym/lib/python3.8/site-packages/IPython/lib/latextools.py:126: MatplotlibDeprecationWarning:
    The MathtextBackendBitmap class was deprecated in Matplotlib 3.4 and will be removed two minor releases later. Use mathtext.math_to_image instead.
      mt.to_png(f, s, fontsize=12, dpi=dpi, color=color)

$\displaystyle \left[\begin{matrix}\left[\begin{matrix}0 & 0\\0 & \frac{\left(a + r \cos{\left(u \right)}\right) \sin{\left(u \right)}}{r}\end{matrix}\right] & \left[\begin{matrix}0 & - \frac{r \left(a + r \cos{\left(u \right)}\right) \sin{\left(u \right)}}{a^{2} + 2 a r \cos{\left(u \right)} + r^{2} \cos^{2}{\left(u \right)}}\\- \frac{r \left(a + r \cos{\left(u \right)}\right) \sin{\left(u \right)}}{a^{2} + 2 a r \cos{\left(u \right)} + r^{2} \cos^{2}{\left(u \right)}} & 0\end{matrix}\right]\end{matrix}\right]$

```python

```

```python
s.metric_tensor.tensor()
s.metric_tensor.change_config('uu').tensor()
s.christoffel_symbol.tensor()
r_u, r_v = s.exprs.diff(u), s.exprs.diff(v); r_u, r_v
a_, b_ = r_u, r_v
s.weingarten_matrix
```

$\displaystyle \left[\begin{matrix}r^{2} & 0\\0 & \left(a + r \cos{\left(u \right)}\right)^{2}\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}\frac{1}{r^{2}} & 0\\0 & \frac{1}{\left(a + r \cos{\left(u \right)}\right)^{2}}\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}\left[\begin{matrix}0 & 0\\0 & \frac{\left(a + r \cos{\left(u \right)}\right) \sin{\left(u \right)}}{r}\end{matrix}\right] & \left[\begin{matrix}0 & - \frac{r \left(a + r \cos{\left(u \right)}\right) \sin{\left(u \right)}}{a^{2} + 2 a r \cos{\left(u \right)} + r^{2} \cos^{2}{\left(u \right)}}\\- \frac{r \left(a + r \cos{\left(u \right)}\right) \sin{\left(u \right)}}{a^{2} + 2 a r \cos{\left(u \right)} + r^{2} \cos^{2}{\left(u \right)}} & 0\end{matrix}\right]\end{matrix}\right]$

$\displaystyle \left( \left[\begin{matrix}- r \sin{\left(u \right)} \cos{\left(v \right)} & - r \sin{\left(u \right)} \sin{\left(v \right)} & r \cos{\left(u \right)}\end{matrix}\right], \  \left[\begin{matrix}- \left(a + r \cos{\left(u \right)}\right) \sin{\left(v \right)} & \left(a + r \cos{\left(u \right)}\right) \cos{\left(v \right)} & 0\end{matrix}\right]\right)$

$\displaystyle \left[\begin{matrix}\frac{1}{r} & 0\\0 & \frac{\cos{\left(u \right)}}{a + r \cos{\left(u \right)}}\end{matrix}\right]$

```python
Wa = s.weingarten_transform(a_)
Wb = s.weingarten_transform(b_)
dot(Wa, b_), dot(a_, Wb)
s.K_H
s.prin_curvature_and_vector
from silkpy.sympy_utility import dot
(_, vec1), (_, vec2) = s.prin_curvature_and_vector
dot(vec1, vec2) # The two principal curvature vectors are perpendicular to each other.
```

```python
InteractiveShell.ast_node_interactivity = "last"
```

```python
from sympy import sin, cos, pi
from silkpy.numeric.surface.geodesic import geodesic_ncurve
theta = pi / 24 # symbols('theta', real=True)
t_arr, (u_arr, v_arr) = geodesic_ncurve(
    s.subs({a:5, r:2}), [pi/4, pi/4], [cos(theta), sin(theta)])
```

```python
from sympy import sin, cos, pi
from silkpy.numeric.surface.geodesic import geodesic_polar_ncoordinate
rho_arr, theta_arr, u_grid, v_grid = geodesic_polar_ncoordinate(
    s.subs({a:5, r:2}), [pi/4, pi/4], rho1=2.4, nrho=12, ntheta=48)

x_grid, y_grid, z_grid = s.subs({a:5, r:2}).lambdified()(u_grid, v_grid)
```

```python
from silkpy.symbolic.surface.draw import draw_surface_plotly
import plotly.graph_objects as go

if surf_choice=='torus':
    fig = draw_surface_plotly(s.subs({a: 5, r:2}), domain=[(-float(pi), float(pi)), (-float(pi), float(pi))])
else:
    raise NotImplementedError()

fig.add_trace(go.Scatter3d(
    x=x_arr, y=y_arr, z=z_arr,
    mode='lines',
    line=dict(color=t_arr, width=2)
))

# import numpy as np
# for i in range(len(theta_arr)):
#     fig.add_trace(go.Scatter3d(
#         x=x_grid[:, i],
#         y=y_grid[:, i],
#         z=z_grid[:, i],
#         mode='lines',
#         line=dict(#color=rho_arr,
#                   width=2)
#     ))

# for i in range(len(rho_arr)):
#     fig.add_trace(go.Scatter3d(
#         x=np.r_[x_grid[i,:], x_grid[i,:]],
#         y=np.r_[y_grid[i,:], y_grid[i,:]],
#         z=np.r_[z_grid[i,:], z_grid[i,:]],
#         mode='lines',
#         line=dict(#color=rho_arr[i],
#             width=2)
#     ))
# fig.show()
```

## Not yet done

```python
from sympy import series, Eq
t0 = symbols('t_0', real=True)
```

```python
t0 = 0
exprs[0].subs(t, t0) + (t-t0) * exprs[0].diff(t, 1).subs(t, t0)
exprs[1].subs(t, t0) + (t-t0) * exprs[1].diff(t, 1).subs(t, t0)
```

```python
exprs[0].evalf(subs={t:0}) + exprs[0].diff(t, 1).evalf(subs={t:0})
```

```python
from sympy import Eq
import sympy.solvers.ode as ode
ode.systems.dsolve_system([
    Eq(linearized_exprs[0], 0),
    Eq(linearized_exprs[1], 0)], funcs=[u1, u2])
```

```python

```

```python
def curvature_curve(surface):
    from sympy import Matrix, Array, Eq
    from sympy import Function, symbols
    import sympy.solvers.ode as ode
    t = symbols('t', real=True)
    # u1, u2 = symbols('u1, u2', real=True, cls=Function)
    u1 = Function(surface.sym(0), real=True)(t)
    u2 = Function(surface.sym(1), real=True)(t)
    curvature_curve_mat = Matrix([
        [u1.diff(t)**2, -u1.diff(t) * u2.diff(t), u2.diff(t)**2],
        Array(surface.E_F_G).subs(surface.sym(0), u1),
        Array(surface.L_M_N).subs(surface.sym(1), u2)])
    # typically there would be two solutions
    sol_with_u1_equal_t = ode.systems.dsolve_system(
        [Eq(curvature_curve_mat.det(), 0 ), Eq(u1.diff(t), 1)])[0]
    sol_with_u2_equal_t = ode.systems.dsolve_system(
        [Eq(curvature_curve_mat.det(), 0 ), Eq(u2.diff(t), 1)])[0]
    return [sol_with_u1_equal_t, sol_with_u2_equal_t]
```

```python
curvature_curve(s)
```

```python

```


# Function Representation and Manipulation

```
%matplotlib inline
```

```
import numpy as np
import matplotlib
matplotlib.rcParams.update({'font.size': 14})
import matplotlib.pyplot as plt
```

From a mathematical point of view, a central point in numerical methods is how we represent a general function $f(x)$. As $x$ is a real number the function encodes an (uncountably) infinite amount of information: its value $f(x)$ for every point $x$. A computer can only usefully store and manipulate a finite amount of information, so it would appear impossible to represent even the simplest function of one variable.

Of course, most functions can be represented straightforwardly using little information. We often see functions that can be represented symbolically - $f(x) = x^2 + x$ is a simple example - but numerical methods are precisely for cases where symbolic methods are not applicable. Instead we must consider how a function is represented, or approximated, in terms of a finite amount of information.

Two standard representations can be used to make the central point. The first representation is in terms of polynomials, using Taylor series:

\begin{equation}
  f(x ; x_0) = \sum_{n=0}^{\infty} \frac{c_n}{n!} (x - x_0)^n.
\end{equation}

Here $x_0$ is a parameter about which the Taylor series is centred.

The second representation is in terms of trigonometric functions, using Fourier series. For simplicity we consider only the Fourier cosine series

\begin{equation}
  f(x ; L) = \frac{1}{2} a_0 + \sum_{n=1}^{\infty} a_n \cos \left( \frac{n \pi x}{L} \right).
\end{equation}

The Fourier series representation is $2L$-periodic.

Given a set of coefficients ($\{c_n\}$ for the Taylor series or $\{a_n\}$ for the Fourier series) we define a function.

```
an = np.zeros((10, 1))
cn = np.zeros_like(an)
for n in range(len(an)):
    an[n] = 1.0 / (n + 1)**2
    cn[n] = an[n]
x = np.linspace(-1.0, 1.0, 100)
taylor = an[0]*np.ones_like(x)
fourier = cn[0]*np.ones_like(x)
for n in range(1, len(an)):
    taylor += an[n] * x**n
    fourier += cn[n] * np.cos(n * np.pi * x)
fig = plt.figure(figsize = (8,6))
ax = fig.add_subplot(111)
ax.plot(x,taylor,label = 'Taylor')
ax.plot(x,fourier, label = 'Fourier')
plt.legend()
ax.set_xlabel('$x$')
plt.show()
```

Given a function, the standard analytical methods will give you the coefficients. This can rarely be directly applied in numerical methods - as noted above, it is usual that you're dealing with a function that isn't known in symbolic form.

Above, when we defined the function $f(x)$ from the coefficients, we were noting that we could evaluate the representation for any point $x$ given

1. the form of the representation (e.g., as a Taylor series about $x_0=0$), and
2. the value of the coefficients.

We can (nearly always) go in the opposite direction: given the value of the function at enough points $x_n$ we can compute the coefficients of the representation.

For example, let us assume that

1. the function will be represented as a polynomial of order 2, i.e. a Taylor series about $x_0=0$ such that all coefficients $c_n$ vanish for $n > 2$, and
2. the value of the function is known at three points $x_1, x_2, x_3$.

This is precisely the information needed to compute the coefficients $c_0, c_1, c_2$ and hence write the function as

\begin{equation}
  f(x) \approx c_0 + c_1 x + \frac{c_2}{2} x^2.
\end{equation}

Explicitly, we note that the function representation matches the known value of the function at the three points only if

\begin{equation}
  \begin{pmatrix} 1 & x_1 & \tfrac{1}{2} x_1^2 \\ 1 & x_2 & \tfrac{1}{2} x_2^2 \\ 1 & x_3 & \tfrac{1}{2} x_3^2 \end{pmatrix} \begin{pmatrix} c_0 \\ c_1 \\ c_2 \end{pmatrix} = \begin{pmatrix} f(x_1) \\ f(x_2) \\ f(x_3) \end{pmatrix}
\end{equation}

which, given the information we know, is a linear system for the coefficients.

Note in particular that the number of points at which the function is known must match the number of coefficients in the function representation.

The previous example hints at one of the standard methods of numerically representing functions. First, we take a representation that we know, given an infinite amount of information, converges to the function we want (in some sense). Second, assume that taking a *finite* number of terms is sufficient: e.g., truncate the sum in the Taylor series expansion after a finite number of terms. Then, using a finite amount of information about the function (such at its value at an appropriate number of points) we compute the coefficients of the representation. We then manipulate the representation *as if* it were the true function.

## From data to derivative

A standard example of manipulating "functions" would be the following. The *value* of the function $f$ is known at three points $x_{1,2,3}$ with $x_1 < x_2 < x_3$. Find the derivative of the function at some location in $[x_1, x_3]$.

One method is to use finite differences. However, it is instructive to go through the steps in terms of function representations.

Here we will use the Taylor series representation above. That is, we assume that $f(x) \simeq g(x)$, where $g(x)$ is the *unique* polynomial of degree 2 that matches the known values of $f$, i.e. $g(x_i) = f(x_i)$ for $i = 1, 2, 3$, and

$$
  g(x) = c_0 + c_1 x + \frac{c_2}{2} x^2
$$

as above. For simplicity we'll assume an evenly spaced grid of points, i.e. $x_1 = x_2 - h$ and $x_3 = x_2 + h$.

```
import sympy
from sympy.matrices import *
x1,x2,x3=sympy.symbols(("x1","x2","x3"))
h = sympy.symbols("h")
M = Matrix([[1, x1, x1**2/2],[1, x2, x2**2/2],[1, x3, x3**2/2]])
M = M.subs(x1, x2-h)
M = M.subs(x3, x2+h)
f1,f2,f3=sympy.symbols(("f1","f2","f3"))
b = Matrix([f1, f2, f3])
c = M.solve(b)
sympy.pretty_print(c.expand())
```

This gives us the $c_i$ coefficients. Now, as $f(x) \simeq g(x)$ we *assume* that $f'(x) \simeq g'(x)$. As we have the explicit form for $g$, we immediately know

$$
  \begin{align}
    && g(x) & = c_0 + c_1 x + \frac{c_2}{2} x^2 \\
    \implies && g'(x) & = c_1 + c_2 x \\
    && & = \frac{1}{h} \left( \frac{f_3 - f_1}{2} + \frac{x - x_2}{h} \left( f_1 - 2 f_2 + f_3 \right) \right)
  \end{align}
$$

We immediately see that

$$
  f'(x_2) \simeq \frac{f_3 - f_1}{2 h}
$$

which is the standard central differencing formula.

Let us try it for a simple example - $f(x) = \exp(x)$ on $[0, 1]$. We see that the derivative is reasonable, but varies linearly with $x$. This is to be expected - $g(x)$ is a quadratic, so its derivative must be a linear polynomial.

```
x = np.array([0.0, 0.5, 1.0])
xp = np.linspace(0.0, 1.0) # Used for plotting
h = 0.5
f = np.exp(x) # Samples of f
df = np.exp(xp) # The derivative of f is again the exponential; this is for plotting
dg = 1/h * ((f[2] - f[0])/2.0 + (xp - x[1]) / h * (f[0] - 2.0*f[1] + f[2]))
fig = plt.figure(figsize = (8,6))
ax = fig.add_subplot(111)
ax.plot(xp, df, 'b-', label=r"$\exp(x)$")
ax.plot(xp, dg, 'r--', label=r"$g'(x)$")
ax.set_xlabel(r"$x$")
plt.legend(loc=2);
```

We should note that there is a distinction between what we have done above and the standard approach to finite differencing. In finite differencing we evaluate the result at a *single* point. Here, as we have the functional form of the approximation $g(x)$ at all points in the interval, we can evaluate the result *anywhere*.

## From data to integral

If we wanted to integrate the function $f(x)$ over the interval, a simple approximation would be to integrate the approximate function $g(x)$. This is the basis for Simpson's rule, in the case of a second order polynomial interpolating function. In the general case, where the interpolating function is a (piecewise) polynomial of some order, it is the basis of the Newton-Cotes methods.

By using the form of $g(x)$ above we have

$$
  \begin{align}
    && g(x) & = c_0 + c_1 x + \frac{c_2}{2} x^2 \\
    \implies && \int_0^{2h} g(x) \, dx & = 2 h c_0 + 2 h^2 c_1 + \frac{4}{3} h^3 c_2
  \end{align}
$$

which we can evaluate (as $x_2 = h$, using the results for $c_i$ above)

$$
\begin{equation}
  \int_0^{2 h} g(x) \, dx =  \frac{h}{3} \left( f_1 + 4 f_2 + f_3 \right).
\end{equation}
$$

This is precisely Simpson's rule.

## From equations to algorithms

Suppose we want to solve a differential equation $y' = f(x, y)$.

We can *assume* that the solution that we want can be approximated by a simple form; again, let us assume that in the interval $[x_n, x_{n+1}]$ it can be approximated by a linear polynomial. Therefore $y(x) \simeq Y(x)$ where

$$\begin{equation}
  Y(x) = c_0 + c_1 (x - x_{n}).
\end{equation}$$

This immediately tells us that $y' \simeq Y' = c_1$, which is constant across the entire interval $[x_n, x_{n+1}]$.

Now, if we assume that we *know* the value of the solution at the start of the interval, $y_n = y(x_n)$, then we can insist that

$$\begin{equation}
  Y(x_n) = c_0 = y_n.
\end{equation}$$

As we also *know*, given the value of $y_n$, that

$$\begin{equation}
  y'(x_n) = f(x_n, y_n) \simeq Y'(x_n) = c_1,
\end{equation}$$

we have fixed all the parameters determining $Y(x)$.

Therefore we can *evaluate*

$$\begin{equation}
  y_{n+1} = y(x_{n+1}) \simeq Y(x_{n+1}) = c_0 + c_1 (x_{n+1} - x_n) = y_n + (x_{n+1} - x_n) f(x_n, y_n).
\end{equation}$$

If we denote $h = (x_{n+1} - x_n)$ then we see the standard formula for *Euler's method* for solving initial value problems

$$\begin{equation}
  y_{n+1} =  y_n + h f(x_n, y_n).
\end{equation}$$

Lectures/Function Representation and Manipulation.ipynb


```python
from IPython.display import display, Markdown, Latex
import linear_systems as ls
import numpy as np
```

```python

```

Using KVL in 3 meshes we have the relations:-
$$
\begin{align}
-10+10I_1+20+10(I_1-I_2)-10I_1  = 0 \\
-20 +10(I_2-I_1) + 30I_2+10(I_2-I_3) = 0 \\
-10 -20I_3-10(I_3-I_2)= 0
\end{align}
$$
In matrix form we have the following relation
$$
\begin{pmatrix}
4 & -1 & 0 \\
-1 & 5 & -1 \\
0 & 1 & -3 \\
\end{pmatrix}
\begin{pmatrix}
I_1 \\
I_2\\
I_3 \\
\end{pmatrix}
=
\begin{pmatrix}
-1 \\
2\\
1 \\
\end{pmatrix}
$$
$$

\left[\begin{array}{ccc|c}
4 & -1 & 0 &-1 \\
-1 & 5 & -1 &2\\
0 & 1 & -3& 1 \\
\end{array}\right]
$$

Using Gaussian elimination we have,

```python
A = np.array(
    [[4,-1,0],
    [-1,5,-1],
    [0,1,-3]] , dtype=float
)
b = np.array([-1,2,1])

print(ls.gauss_seidel(A,b,tol=1e-13)[1])
np.linalg.solve(A,b)
```

    2.4452662117369073e-14

    array([-0.16981132,  0.32075472, -0.22641509])

```python

```