# Generative models - variational auto-encoders

### Author: Philippe Esling (esling@ircam.fr)

In this course we will cover
1. A [quick recap](#recap) on simple probability concepts (and in TensorFlow)
2. A formal introduction to [Variational Auto-Encoders](#vae) (VAEs)
3. An explanation of the [implementation](#implem) of VAEs
4. Some [modifications and tips to improve the reconstruction](#improve) of VAEs **(exercise)**

## Quick recap on probability

The field of probability aims to model random or uncertain events. Hence, a random variable $X$ denotes a quantity that is uncertain, such as the result of an experiment (flipping a coin) or the measurement of an uncertain property (measuring the temperature). If we observe several occurrences of the variable $\{\mathbf{x}_{i}\}_{i=1}$, it might take different values on each occasion, but some values may occur more often than others. This information is captured by the _probability distribution_ $p(\mathbf{x})$ of the random variable.

To understand these concepts graphically, we will rely on the `Tensorflow Probability` package.

```python
import tensorflow_probability as tfp
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
```

### Probability distributions

#### Discrete distributions

Let $\mathbf{x}$ be a discrete random variable with range $R_{X}=\{x_1,\cdots,x_n\}$ (finite or countably infinite). The function
\begin{equation}
    p_{X}(x_{i})=p(X=x_{i}), \forall i\in\{1,\cdots,n\}
\end{equation}
is called the probability mass function (PMF) of $X$.

Hence, the PMF defines the probabilities of all possible values for a random variable. The above notation allows to express that the PMF is defined for the random variable $X$, so that $p_{X}(1)$ gives the probability that $X=1$. For discrete random variables, the PMF is also called the \textit{probability distribution}. The PMF is a probability measure, therefore it satisfies all the corresponding properties
- $0 \leq p_{X}(x_i) < 1, \forall x_i$
- $\sum_{x_i\in R_{X}} p_{X}(x_i) = 1$
- $\forall A \subset R_{X}, p(X \in A)=\sum_{x_a \in A}p_{X}(x_a)$

A very simple example of discrete distribution is the `Bernoulli` distribution. With this distribution, we can model a coin flip. If we throw the coin a very large number of times, we hope to see on average an equal amount of _heads_ and _tails_.

```python
bernoulli = tfp.distributions.Bernoulli(probs=0.5)
samples = bernoulli.sample(10000)
sns.distplot(samples)
plt.title("Samples from a Bernoulli (coin toss)")
plt.show()
```

However, we can also _sample_ from the distribution to have individual values of a single throw. In that case, we obtain a series of separate events that _follow_ the distribution

```python
vals = ['heads', 'tails']
samples = bernoulli.sample(10)
for s in samples:
    print('Coin is tossed on ' + vals[s])
```

    Coin is tossed on heads
    Coin is tossed on heads
    Coin is tossed on tails
    Coin is tossed on heads
    Coin is tossed on heads
    Coin is tossed on tails
    Coin is tossed on tails
    Coin is tossed on heads
    Coin is tossed on tails
    Coin is tossed on tails

#### Continuous distributions

The same ideas apply to _continuous_ random variables, which can model for instance the height of human beings. If we try to guess the height of someone that we do not know, there is a higher probability that this person will be around 1m70, instead of 20cm or 3m. For the rest of this course, we will use the shorthand notation $p(\mathbf{x})$ for the distribution $p(\mathbf{x}=x_{i})$,  which expresses for a real-valued random variable $\mathbf{x}$, evaluated at $x_{i}$, the probability that $\mathbf{x}$ takes the value $x_i$.

One notorious example of such distributions is the Gaussian (or Normal) distribution, which is defined as
\begin{equation}
    p(x)=\mathcal{N}(\mu,\sigma)=\frac{1}{\sqrt{2\pi\sigma^{2}}}e^{-\frac{(x-\mu)^{2}}{2\sigma^{2}}}
\end{equation}

Similarly as before, we can observe the behavior of this distribution with the following code

```python
normal = tfp.distributions.Normal(loc=0., scale=1.)
samples = normal.sample(10000)
sns.distplot(samples)
plt.title("Samples from a standard Normal")
plt.show()
```

### Comparing distributions (KL divergence)
$
\newcommand{\R}{\mathbb{R}}
\newcommand{\bb}[1]{\mathbf{#1}}
\newcommand{\bx}{\bb{x}}
\newcommand{\by}{\bb{y}}
\newcommand{\bz}{\bb{z}}
\newcommand{\KL}[2]{\mathcal{D}_{\text{KL}}\left[#1 \| #2\right]}$
Originally defined in the field of information theory, the _Kullback-Leibler (KL) divergence_ (usually noted $\KL{p(\bx)}{q(\bx)}$) is a dissimilarity measure between two probability distributions $p(\bx)$ and $q(\bx)$. In the view of information theory, it can be understood as the cost in number of bits necessary for coding samples from $p(\bx)$ by using a code optimized for $q(\bx)$ rather than the code optimized for $p(\bx)$. In the view of probability theory, it represents the amount of information lost when we use $q(\bx)$ to approximate the true distribution $p(\bx)$. %that explicit the cost incurred if events were generated by $p(\bx)$ but charged under $q(\bx)$

Given two probability distributions $p(\bx)$ and $q(\bx)$, the Kullback-Leibler divergence of $q(\bx)$ _from_ $p(\bx)$ is defined to be
\begin{equation}
    \KL{p(\bx)}{q(\bx)}=\int_{\R} p(\bx) \log \frac{p(\bx)}{q(\bx)}d\bx
\end{equation}

Note that this dissimilarity measure is \textit{asymmetric}, therefore, we have
\begin{equation}
    \KL{p(\bx)}{q(\bx)}\neq \KL{q(\bx)}{p(\bx)}
\end{equation}
This asymmetry also describes an interesting behavior of the KL divergence, depending on the order to which it is evaluated. The KL divergence can either be a _mode-seeking_ or _mode-coverage} measure.

## Variational auto-encoders

As we have seen in the previous AE course, VAEs are also a form generative models. However, they are defined from a more sound probabilistic perspective. to find the underlying probability distribution of the data $p(\mathbf{x})$ based on a set of examples in $\mathbf{x}\in\mathbb{R}^{d_{x}}$. To do so, we consider *latent variables* defined in a lower-dimensional space $\mathbf{z}\in\mathbb{R}^{d_{z}}$ ($d_{z} \ll d_{x}$) with the joint probability distribution $p(\mathbf{x}, \mathbf{z}) = p(\mathbf{x} \vert \mathbf{z})p(\mathbf{z})$. Unfortunately, for complex distributions this integral is too complex and cannot be found in closed form.

### Variational inference

The idea of *variational inference* (VI) allows to solve this problem through *optimization* by assuming a simpler approximate distribution $q_{\phi}(\mathbf{z}\vert\mathbf{x})\in\mathcal{Q}$ from a family $\mathcal{Q}$ of approximate densities. Hence, the goal is to minimize the difference between this approximation and the real distribution. Therefore, this turns into the optimization problem of minimizing the Kullback-Leibler (KL) divergence between the parametric approximation and the original density

$$
q_{\phi}^{*}(\mathbf{z}\vert \mathbf{x})=\text{argmin}_{q_{\phi}(\mathbf{z} \vert \mathbf{x})\in\mathcal{Q}} \mathcal{D}_{KL} \big[ q_{\phi}\left(\mathbf{z} \vert \mathbf{x}\right) \parallel p\left(\mathbf{z} \vert \mathbf{x}\right) \big]
\tag{2}
$$

By developing this KL divergence and re-arranging terms (the detailed development can be found in [3](#reference1)), we obtain

$$
\log{p(\mathbf{x})} - D_{KL} \big[ q_{\phi}(\mathbf{z} \vert \mathbf{x}) \parallel p(\mathbf{z} \vert \mathbf{x}) \big] =
\mathbb{E}_{\mathbf{z}} \big[ \log{p(\mathbf{x} \vert \mathbf{z})}\big] - D_{KL} \big[ q_{\phi}(\mathbf{z} \vert \mathbf{x}) \parallel p(\mathbf{z}) \big]
\tag{3}
$$

This formulation describes the quantity we want to maximize $\log p(\mathbf{x})$ minus the error we make by using an approximate $q$ instead of $p$. Therefore, we can optimize this alternative objective, called the *evidence lower bound* (ELBO)

$$
\begin{equation}
\mathcal{L}_{\theta, \phi} = \mathbb{E} \big[ \log{ p_\theta (\mathbf{x|z}) } \big] - \beta \cdot D_{KL} \big[ q_\phi(\mathbf{z|x}) \parallel p_\theta(\mathbf{z}) \big]
\end{equation}
\tag{4}
$$

We can see that this equation involves $q_{\phi}(\mathbf{z} \vert \mathbf{x})$ which *encodes* the data $\mathbf{x}$ into the latent representation $\mathbf{z}$ and a *decoder* $p(\mathbf{x} \vert \mathbf{z})$, which allows generating a data vector $\mathbf{x}$ given a latent configuration $\mathbf{z}$. Hence, this structure defines the *Variational Auto-Encoder* (VAE).

The VAE objective can be interpreted intuitively. The first term  increases the likelihood of the data generated given a configuration of the latent, which amounts to minimize the *reconstruction error*. The second term represents the error made by using a simpler posterior distribution $q_{\phi}(\mathbf{z} \vert \mathbf{x})$ compared to the true prior $p_{\theta}(\mathbf{z})$. Therefore, this *regularizes* the choice of approximation $q$ so that it remains close to the true posterior distribution [3].

### Reparametrization trick

Now, while this formulation has some very interesting properties, it involves sampling operations, where we need to draw the latent point $\mathbf{z}$ from the distribution $q_{\phi}(\mathbf{z}\vert\mathbf{x})$.  The simplest choice for this variational approximate posterior is a multivariate Gaussian with a diagonal covariance structure (which leads to independent Gaussians on every dimension, called the *mean-field* family) so that
$$
\text{log}q_\phi(\mathbf{z}\vert\mathbf{x}) = \text{log}\mathcal{N}(\mathbf{z};\mathbf{\mu}^{(i)},\mathbf{\sigma}^{(i)})
\tag{5}
$$
where the mean $\mathbf{\mu}^{(i)}$ and standard deviation $\mathbf{\sigma}^{(i)}$ of the approximate posterior are different for each input point and are produced by our encoder parametrized by its variational parameters $\phi$. Now the KL divergence between this distribution and a simple prior $\mathcal{N}(\mathbf{0}, \mathbf{I})$ can be very simply obtained with
$$
D_{KL} \big[ q_\phi(\mathbf{z|x}) \parallel \mathcal{N}(\mathbf{0}, \mathbf{I}) \big] = \frac{1}{2}\sum_{j=1}^{D}\left(1+\text{log}((\sigma^{(i)}_j)^2)+(\mu^{(i)}_j)^2+(\sigma^{(i)}_j)^2\right)
\tag{6}
$$

While this looks convenient, we will still have to perform gradient descent through a sampling operation, which is non-differentiable. To solve this issue, we can use the *reparametrization trick*, which takes the sampling operation outside of the gradient flow by considering $\mathbf{z}^{(i)}=\mathbf{\mu}^{(i)}+\mathbf{\sigma}^{(i)}\odot\mathbf{\epsilon}^{(l)}$ with $\mathbf{\epsilon}^{(l)}\sim\mathcal{N}(\mathbf{0}, \mathbf{I})$

## VAE implementation

As we have seen, VAEs can be simply implemented by decomposing the above series of operations into an `encoder` which represents the distribution $q_\phi(\mathbf{z}\vert\mathbf{x})$, from which we will sample some values $\tilde{\mathbf{z}}$ (using the reparametrization trick) and compute the Kullback-Leibler (KL) divergence. Then, we use these values as input to a `decoder` which represents the distribution $p_\theta(\mathbf{x}\vert\mathbf{z})$ so that we can produce a reconstruction $\tilde{\mathbf{x}}$ and compute the reconstruction error.

Therefore, we can define the VAE based on our previous implementation of the AE that we recall here

```python
import tensorflow as tf
from sklearn.metrics import accuracy_score, precision_score, recall_score
from sklearn.model_selection import train_test_split
from tensorflow.keras import layers, losses
from tensorflow.keras.datasets import fashion_mnist
from tensorflow.keras.models import Model

class AE(Model):
    def __init__(self, encoder, decoder, encoding_dim):
        super(AE, self).__init__()
        self.encoding_dim = encoding_dim
        self.encoder = encoder
        self.decoder = decoder

    def call(self, x):
        encoded = self.encoder(x)
        decoded = self.decoder(encoded)
        return decoded
```

In order to move to a probabilistic version, we need to add the latent space sampling mechanism, and change the behavior of our `call` function. This process is implemented in the following `VAE` class.

Note that we purposedly rely on an implementation of the `encode` function where the `encoder` first produces an intermediate representation of size `encoder_dims`. Then, this representation goes through two separate functions for encoding $\mathbf{\mu}$ and $\mathbf{\sigma}$. This provides a clearer implementation but also the added bonus that we can ensure that $\mathbf{\sigma} > 0$

```python
class VAE(AE):

    def __init__(self, encoder, decoder, encoding_dims, latent_dims):
        super(VAE, self).__init__(encoder, decoder, encoding_dims)
        self.latent_dims = latent_dims
        self.mu = layers.Dense(self.latent_dims, activation='relu')
        self.sigma = layers.Dense(self.latent_dims, activation='softplus')

    def encode(self, x):
        x = self.encoder(x)
        mu = self.mu(x)
        sigma = self.sigma(x)
        return mu, sigma

    def decode(self, z):
        return self.decoder(z)

    def call(self, x):
        # Encode the inputs
        z_params = self.encode(x)
        # Obtain latent samples and latent loss
        z_tilde, kl_div = self.latent(x, z_params)
        # Decode the samples
        x_tilde = self.decode(z_tilde)
        return x_tilde, kl_div

    def latent(self, x, z_params):
        n_batch = x.shape[0]
        # Retrieve mean and var
        mu, sigma = z_params
        # Re-parametrize
        q = tfp.distributions.Normal(np.zeros(mu.shape[1]), np.ones(sigma.shape[1]))
        z = (sigma * tf.cast(q.sample(n_batch), 'float32')) + mu
        # Compute KL divergence
        kl_div = -0.5 * tf.reduce_sum(1 + sigma - tf.pow(mu, 2) - tf.exp(sigma))
        kl_div = kl_div / n_batch
        return z, kl_div
```

Now the interesting aspect of VAEs is that we can define any parametric function as `encoder` and `decoder`, as long as we can optimize them. Here, we will rely on simple feed-forward neural networks, but these can be largely more complex (with limitations that we will discuss later in the tutorial).

```python
def construct_encoder_decoder(nin, n_latent = 16, n_hidden = 512, n_classes = 1):
    # Encoder network
    encoder = tf.keras.Sequential([
          layers.Flatten(),
          layers.Dense(n_hidden, activation='relu'),
          layers.Dense(n_hidden, activation='relu'),
          layers.Dense(n_hidden, activation='relu'),
    ])
    # Decoder network
    decoder = tf.keras.Sequential([
          layers.Dense(n_hidden, activation='relu'),
          layers.Dense(n_hidden, activation='relu'),
          layers.Dense(nin * n_classes, activation='sigmoid'),
          layers.Reshape((28, 28))
    ])
    return encoder, decoder
```

### Evaluating the error

In the definition of the `VAE` class, we directly included the computation of the $D_{KL}$ term to regularize our latent space. However, remember that the complete loss of equation (4) also contains a *reconstruction loss* which compares our reconstructed output to the original data.

While there are several options to compare the error between two elements, there are usually two preferred choices among the generative literature depending on how we consider our problem
1. If we consider each dimension (pixel) to be a binary unit (following a Bernoulli distribution), we can rely on the `binary cross entropy` between the two distributions
2. If we turn our problem to a set of classifications, where each dimension can belong to a given set of *intensity classes*, then we can compute the `multinomial loss` between the two distributions

In the following, we define both error functions and regroup them in the `reconstruction_loss` call (depending on the `num_classes` considered). However, as the `multinomial loss` requires a large computational overhead, and for the sake of simplicity, we will train all our first models by relying on the `binary cross entropy`

```python
optimizer = tf.keras.optimizers.Adam(1e-4)

def compute_loss(model, x):
    x_tilde, kl_div = model(x)
    cross_ent = tf.nn.sigmoid_cross_entropy_with_logits(logits=x_tilde, labels=x)
    logpx_z = -tf.reduce_sum(cross_ent, axis=[1, 2])
    return -tf.reduce_mean(logpx_z + kl_div)

@tf.function
def train_step(model, x, optimizer):
    """Executes one training step and returns the loss."""
    with tf.GradientTape() as tape:
        loss = compute_loss(model, x)
    gradients = tape.gradient(loss, model.trainable_variables)
    optimizer.apply_gradients(zip(gradients, model.trainable_variables))
```

### Optimizing a VAE on a real dataset

For this tutorial, we are going to take a quick shot at a real-life problem by trying to train our VAEs on the `FashionMNIST` dataset. This dataset can be natively used in PyTorch by relying on the `torchvision.datasets` classes as follows

```python
# Load (and eventually download) the dataset
(x_train, _), (x_test, _) = fashion_mnist.load_data()
# Normalize the dataset in the [0, 1] range]
x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.
```

The `FashionMNIST` dataset is composed of simple 28x28 black and white images of different items of clothings (such as shoes, bags, pants and shirts). We put a simple function here to display one batch of the test set (note that we keep a fixed batch from the test set in order to evaluate the different variations that we will try in this tutorial).

```python
def plot_batch(batch, nslices=8):
    # Create one big image for plot
    img = np.zeros(((batch.shape[1] + 1) * nslices, (batch.shape[2] + 1) * nslices))
    for b in range(batch.shape[0]):
        row = int(b / nslices); col = int(b % nslices)
        r_p = row * batch.shape[1] + row; c_p = col * batch.shape[2] + col
        img[r_p:(r_p+batch.shape[1]),c_p:(c_p+batch.shape[2])] = batch[b]
    im = plt.imshow(img, cmap='Greys', interpolation='nearest'),
    return im
# Select a random set of fixed data
fixed_batch = x_test[:64]
print(x_test.shape)
plt.figure(figsize=(10, 10))
plot_batch(fixed_batch);
```

Now based on our proposed implementation, the optimization aspects are defined in a very usual way

```python
# Using Bernoulli or Multinomial loss
num_classes = 1
# Number of hidden and latent
n_hidden = 512
n_latent = 2
# Compute input dimensionality
nin = fixed_batch.shape[1] * fixed_batch.shape[2]
# Construct encoder and decoder
encoder, decoder = construct_encoder_decoder(nin, n_hidden = n_hidden, n_latent = n_latent, n_classes = num_classes)
# Build the VAE model
model = VAE(encoder, decoder, n_hidden, n_latent)
```

Now all that is left to do is train the model. We define here a `train_vae` function that we will reuse along the future implementations and variations of VAEs and flows. Note that this function is set to run for only a very few number of `epochs` and also most importantly, *only considers a subsample of the full dataset at each epoch*. This option is just here so that you can test the different models very quickly on any CPU or laptop.

```python
def generate_and_save_images(model, epoch, test_sample):
    predictions, _ = model(test_sample)
    fig = plt.figure(figsize=(4, 4))
    for i in range(predictions.shape[0]):
        plt.subplot(4, 4, i + 1)
        plt.imshow(predictions[i, :, :], cmap='gray')
        plt.axis('off')
    # tight_layout minimizes the overlap between 2 sub-plots
    plt.savefig('image_at_epoch_{:04d}.png'.format(epoch))
    plt.show()

epochs=50
test_sample = x_test[0:16, :, :]
for epoch in range(1, epochs + 1):
    for train_x in x_train:
        train_step(model, tf.expand_dims(train_x, axis=0), optimizer)
    loss = tf.keras.metrics.Mean()
    for test_x in x_test:
        loss(compute_loss(model, tf.expand_dims(test_x, axis=0)))
    elbo = -loss.result()
    print('Epoch: {}, Test set ELBO: {}'.format(epoch, elbo))
    generate_and_save_images(model, epoch, test_sample)
```

### Evaluating generative models

In order to evaluate our upcoming generative models, we will rely on the computation of the Negative Log-Likelihood. This code for the following `evaluate_nll_bpd` is inspired by the [Sylvester flow repository](https://github.com/riannevdberg/sylvester-flows)

```python
from scipy.special import logsumexp

def evaluate_nll_bpd(data_loader, model, batch = 500, R = 5):
    # Set of likelihood tests
    likelihood_test = []
    # Go through dataset
    for batch_idx, (x, _) in enumerate(data_loader):
        for j in range(x.shape[0]):
            a = []
            for r in range(0, R):
                cur_x = x[j].unsqueeze(0)
                # Repeat it as batch
                x = cur_x.expand(batch, *cur_x.size()[1:]).contiguous()
                x = x.view(batch, -1)
                x_tilde, kl_div = model(x)
                rec = reconstruction_loss(x_tilde, x, average=False)
                a_tmp = (rec + kl_div)
                a.append(- a_tmp.cpu().data.numpy())
            # calculate max
            a = np.asarray(a)
            a = np.reshape(a, (a.shape[0] * a.shape[1], 1))
            likelihood_x = logsumexp(a)
            likelihood_test.append(likelihood_x - np.log(len(a)))
    likelihood_test = np.array(likelihood_test)
    nll = - np.mean(likelihood_test)
    # Compute the bits per dim (but irrelevant for binary data)
    bpd = nll / (np.prod(nin) * np.log(2.))
    return nll, bpd
```

Now we can evaluate our VAE model more formally as follows.

```python
# Plot final loss
plt.figure()
plt.plot(losses_kld[:, 0].numpy());
# Evaluate log-likelihood and bits per dim
nll, _ = evaluate_nll_bpd(test_loader, model)
print('Negative Log-Likelihood : ' + str(nll))
```

### Limitations of VAEs - (**exercise**)

Although VAEs are extremely powerful tools, they still have some limitations. Here we list the three most important and known limitations (all of them are still debated and topics of active research).
1. **Blurry reconstructions.** As can be witnessed directly in the results of the previous vanilla VAE implementation, the reconstructions appear to be blurry. The precise origin of this phenomenon is still debated, but the proposed explanation are
    1. The use of the KL regularization
    2. High variance regions of the latent space
    3. The reconstruction criterion (expectation)
    4. The use of simplistic latent distributions
2. **Posterior collapse.** The previous *blurry reconstructions* issue can be mitigated by using a more powerful decoder. However, relying on a decoder with a large capacity causes the phenomenon of *posterior collapse* where the latent space becomes useless. A nice intuitive explanation can be found [here](https://ermongroup.github.io/blog/a-tutorial-on-mmd-variational-autoencoders/)
3. **Simplistic Gaussian approximation**. In the derivation of the VAE objective, recall that the KL divergence term needs to be computed analytically. Therefore, this forces us to rely on quite simplistic families. However, the Gaussian family might be too simplistic to model real world data

In the present tutorial, we show how normalizing flows can be used to mostly solve the third limitation, while also adressing the two first problems. Indeed, we will see that normalizing flows also lead to sharper reconstructions and also act on preventing posterior collapse

## Improving the quality of VAEs

As we discussed in the previous section, several known issues have been reported when using the vanilla VAE implementation. We listed some of the major issues as being
1. **Blurry reconstructions.**
2. **Posterior collapse.**
3. **Simplistic Gaussian approximation**.

Here, we discuss some recent developments that were proposed in the VAE literature and simple adjustments that can be made to (at least partly) alleviate these issues. However, note that some more advanced proposals such as PixelVAE [5](#reference1) and VQ-VAE [6](#reference1) can lead to wider increases in quality

### Reducing the bluriness of reconstructions

In this tutorial, we relied on extremely simple decoder functions, to show how we could easily define VAEs and normalizing flows together. However, the capacity of the decoder obviously directly influences the quality of the final reconstruction. Therefore, we could address this issue naively by using deep networks and of course convolutional layers as we are currently dealing with images.

First you need to construct a more complex encoder and decoder

```python
def construct_encoder_decoder_complex(nin, n_latent = 16, n_hidden = 512, n_params = 0, n_classes = 1):
    # Encoder network
    encoder = ...
    # Decoder network
    decoder = ...
    return encoder, decoder
```

### Preventing posterior collapse with Wasserstein-VAE-MMD (InfoVAE)

As we discussed earlier, the reason behind posterior collapse mostly relates to the KL divergence criterion (a nice intuitive explanation can be found [here](https://ermongroup.github.io/blog/a-tutorial-on-mmd-variational-autoencoders/). This can be mitigated by relying on a different criterion, such as regularizing the latent distribution by using the *Maximum Mean Discrepancy* (MMD) instead of the KL divergence. This model was independently proposed as the *InfoVAE* and later also as the *Wasserstein-VAE*.

Here we provide a simple implementation of the `InfoVAEMMD` class based on our previous implementations.

```python
def compute_kernel(x, y):
    return ...

def compute_mmd(x, y):
    return ...

class InfoVAEMMD(VAE):

    def __init__(self, encoder, decoder):
        super(InfoVAEMMD, self).__init__(encoder, decoder)

    def latent(self, x, z_params):
        return ...
```

### Putting it all together

Here we combine all these ideas (except for the MMD, which is not adequate as the flow definition already regularizes the latent space without the KL divergence) to perform a more advanced optimization of the dataset. Hence, we will rely on the complex encoder and decoder with gated convolutions, the multinomial loss and the normalizing flows in order to improve the overall quality of our reconstructions.

```python
# Size of latent space
n_latent = 16
# Number of hidden units
n_hidden = 256
# Rely on Bernoulli or multinomial
num_classes = 128
# Construct encoder and decoder
encoder, decoder = ...
# Create VAE or (InfoVAEMMD - WAE) model
model_flow_p = ...
# Create optimizer algorithm
optimizer = ...
# Add learning rate scheduler
scheduler = ...
# Launch our optimization
losses_flow_param = ...
```

*NB*: It seems that the multinomial version have a hard time converging. Although I only let this run for 200 epochs and only for a subsampling of 5000 examples, it might need more time, but this might also come from a mistake somewhere in my code ... If you spot something odd please let me know :)

### References

[1] Rezende, Danilo Jimenez, and Shakir Mohamed. "Variational inference with normalizing flows." _arXiv preprint arXiv:1505.05770_ (2015). [link](http://arxiv.org/pdf/1505.05770)

[2] Kingma, Diederik P., Tim Salimans, and Max Welling. "Improving Variational Inference with Inverse Autoregressive Flow." _arXiv preprint arXiv:1606.04934_ (2016). [link](https://arxiv.org/abs/1606.04934)

[3] Kingma, D. P., & Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114. (2013). [link](https://arxiv.org/pdf/1312.6114)

[4] Rezende, D. J., Mohamed, S., & Wierstra, D. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401.4082. (2014). [link](https://arxiv.org/pdf/1401.4082)

[5] Gulrajani, I., Kumar, K., Ahmed, F., Taiga, A. A., Visin, F., Vazquez, D., & Courville, A. (2016). Pixelvae: A latent variable model for natural images. arXiv preprint arXiv:1611.05013. [link](https://arxiv.org/pdf/1611.05013)

[6] Van den Oord, A., & Vinyals, O. (2017). Neural discrete representation learning. In NIPS 2017 (pp. 6306-6315). [link](http://papers.nips.cc/paper/7210-neural-discrete-representation-learning.pdf)

### Inspirations and resources

https://blog.evjang.com/2018/01/nf1.html
https://github.com/ex4sperans/variational-inference-with-normalizing-flows
https://akosiorek.github.io/ml/2018/04/03/norm_flows.html
https://github.com/abdulfatir/normalizing-flows
https://github.com/riannevdberg/sylvester-flows


# 15 Partial Differential Equations — 2

## Solving Laplace's or Poisson's equation

Still solving **Poisson's equation** for the electric potential $\Phi(\mathbf{r})$ and the charge density $\rho(\mathbf{r})$:

$$
\nabla^2 \Phi(x, y, z) = -4\pi\rho(x, y, z)\\
$$

For a region of space without charges ($\rho = 0$) this reduces to **Laplace's equation**

$$
\nabla^2 \Phi(x, y, z) = 0
$$

General solution by iteration:
$$
\Phi_{i,j} = \frac{1}{4}\Big(\Phi_{i+1,j} + \Phi_{i-1,j} + \Phi_{i,j+1} + \Phi_{i,j-1}\Big)
     + \pi\rho_{i,j} \Delta^2
$$

#### Jacobi method
Do not change $\Phi_{i,j}$ until a complete sweep has been completed.

```python
def Laplace_Jacobi_slow(Phi):
    # Don't use, very slow AND inefficient
    Phi_new = Phi.copy()
    Nx, Ny = Phi.shape
    for xi in range(1, Nx-1):
        for yj in range(1, Ny-1):
            Phi_new[xi, yj] = 0.25*(Phi[xi+1, yj] + Phi[xi-1, yj]
                                  + Phi[xi, yj+1] + Phi[xi, yj-1])
    Phi[:, :] = Phi_new
    return Phi
```

Fast implementation using numpy array operations (vectorized, run at C speed, not Python speed) (see the [board notes (PDF)](https://github.com/ASU-CompMethodsPhysics-PHY494/PHY494-resources/blob/master/16_PDEs/16_PDEs-2-LectureNotes.pdf) or [local PDF](16_PDEs-2-LectureNotes.pdf) for an explanation for how to set up the numpy array operations that give you a 100-fold speed up over `Laplace_Jacobi_slow()`):

```python
def Laplace_Jacobi(Phi):
    """One update in the Jacobi algorithm"""
    Phi[1:-1, 1:-1] = 0.25*(Phi[2:, 1:-1] + Phi[0:-2, 1:-1] + Phi[1:-1, 2:] + Phi[1:-1, 0:-2])
    return Phi
```

#### Gauss-Seidel method
Immediately use updated new values for $\Phi_{i-1, j}$ and $\Phi_{i, j-1}$ (if starting from $\Phi_{1, 1}$).

Leads to *accelerated convergence* and therefore *less round-off error* (but distorts symmetry of boundary conditions... hopefully irrelevant when converged but check!)

```python
def Laplace_Gauss_Seidel(Phi):
    """One update in the Gauss-Seidel algorithm"""
    Nx, Ny = Phi.shape
    for xi in range(1, Nx-1):
        for yj in range(1, Ny-1):
            Phi[xi, yj] = 0.25*(Phi[xi+1, yj] + Phi[xi-1, yj]
                                + Phi[xi, yj+1] + Phi[xi, yj-1])
    return Phi
```

#### Fast Gauss-Seidel-like

Divide the lattice into a checkerboard of black and white cells. Update the odd cells first (like Jacobi) but then use the odd cells to update the even ones (Gauss-Seidel-like; see the [board notes (PDF)](https://github.com/ASU-CompMethodsPhysics-PHY494/PHY494-resources/blob/master/16_PDEs/16_PDEs-2-LectureNotes.pdf) or [local PDF](16_PDEs-2-LectureNotes.pdf) ). This leads to faster convergence *and* can be done with numpy array operations.

```python
def Laplace_Gauss_Seidel_odd_even(Phi):
    """One update in the Gauss-Seidel algorithm on odd or even fields"""
    # odd update (uses old even)
    Phi[1:-2:2, 1:-2:2] = 0.25*(Phi[2::2, 1:-2:2] + Phi[0:-2:2, 1:-2:2] + Phi[1:-2:2, 2::2] + Phi[1:-2:2, 0:-2:2])
    Phi[2:-1:2, 2:-1:2] = 0.25*(Phi[3::2, 2:-1:2] + Phi[1:-2:2, 2:-1:2] + Phi[2:-1:2, 3::2] + Phi[2:-1:2, 1:-2:2])

    # even update (uses new odd)
    Phi[1:-2:2, 2:-1:2] = 0.25*(Phi[2::2, 2:-1:2] + Phi[0:-2:2, 2:-1:2] + Phi[1:-2:2, 3::2] + Phi[1:-2:2, 1:-1:2])
    Phi[2:-1:2, 1:-2:2] = 0.25*(Phi[3::2, 1:-2:2] + Phi[1:-2:2, 1:-2:2] + Phi[2:-1:2, 2::2] + Phi[2:-1:2, 0:-2:2])
    return Phi
```

### *Converged* solution of the wire-in-a-box problem

Solve the box-wire problem and **make sure that the solution is converged to  `tol = 1e-3`.**

Check convergence with the [_Frobenius Norm_](http://mathworld.wolfram.com/FrobeniusNorm.html):

$$
||\mathsf{\Phi}|| := \sqrt{\sum_{i,j}|\Phi_{ij}|^2}
$$

which is implemented as [numpy.linalg.norm()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.norm.html) when the argument is a matrix.

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
```

Interactive plotting (with ipympl):

```python
# for interactive work
%matplotlib widget
```

Only execute the next line if you *don't* want interactive plotting (e.g., when exporting to LaTeX/PDF or html):

```python
%matplotlib inline
```

#### Convience plotting functions.
If you provide a filename then output is only written to a file and figures are close to conserve memory. This allows you to plot files in loops and later assemble them into movies using other programs such as ffmpeg, ImageMagick, mencoder, QuickTime 7, ...

```python
def plot_contour(Phi, filename=None, zlabel=r"potential $\Phi$ (V)",
                 cmap=plt.cm.coolwarm):
    """Plot Phi as a contour plot.

    Arguments
    ---------
    Phi : 2D array
          potential on lattice
    filename : string or None, optional (default: None)
          If `None` then show the figure and return the axes object.
          If a string is given (like "contour.png") it will only plot
          to the filename and close the figure but return the filename.
    cmap : colormap
          pick one from matplotlib.cm
    """
    fig = plt.figure(figsize=(5,4))
    ax = fig.add_subplot(111)

    x = np.arange(Phi.shape[0])
    y = np.arange(Phi.shape[1])
    X, Y = np.meshgrid(x, y)
    Z = Phi[X, Y]
    cset = ax.contourf(X, Y, Z, 20, cmap=cmap)
    ax.set_xlabel('X')
    ax.set_ylabel('Y')
    ax.set_aspect(1)

    cb = fig.colorbar(cset, shrink=0.5, aspect=5)
    cb.set_label(zlabel)

    if filename:
        fig.savefig(filename)
        plt.close(fig)
        return filename
    else:
        return ax

def plot_surf(Phi, filename=None, offset=-20, zlabel=r'potential $\Phi$ (V)',
             elevation=40, azimuth=-65, cmap=plt.cm.coolwarm):
    """Plot Phi as a 3D plot with contour plot underneath.

    Arguments
    ---------
    Phi : 2D array
          potential on lattice
    filename : string or None, optional (default: None)
          If `None` then show the figure and return the axes object.
          If a string is given (like "contour.png") it will only plot
          to the filename and close the figure but return the filename.
    offset : float, optional (default: 20)
          position the 2D contour plot by offset along the Z direction
          under the minimum Z value
    zlabel : string, optional
          label for the Z axis and color scale bar
    elevation : float, optional
          choose elevation for initial viewpoint
    azimuth : float, optional
          chooze azimuth angle for initial viewpoint
    cmap : colormap
          pick one from matplotlib.cm
    """

    x = np.arange(Phi.shape[0])
    y = np.arange(Phi.shape[1])
    X, Y = np.meshgrid(x, y)
    Z = Phi[X, Y]

    fig = plt.figure()
    ax = fig.add_subplot(111, projection='3d')
    ax.plot_wireframe(X, Y, Z, rstride=2, cstride=2, linewidth=0.5, color="gray")
    surf = ax.plot_surface(X, Y, Z, cmap=cmap, alpha=0.6)
    cset = ax.contourf(X, Y, Z, 20, zdir='z', offset=offset+Z.min(), cmap=cmap)

    ax.set_xlabel('X')
    ax.set_ylabel('Y')
    ax.set_zlabel(zlabel)
    ax.set_zlim(offset + Z.min(), Z.max())

    ax.view_init(elev=elevation, azim=azimuth)

    cb = fig.colorbar(surf, shrink=0.5, aspect=5)
    cb.set_label(zlabel)

    if filename:
        fig.savefig(filename)
        plt.close(fig)
        return filename
    else:
        return ax
```

#### Wire problem

Use the fastest Poisson solver that we have at the moment: `Laplace_Gauss_Seidel_odd_even()`

```python
Max_iter=30000
tol = 1e-3
Nmax = 100
Phi = np.zeros((Nmax, Nmax), dtype=np.float64)
Phi_old = np.zeros_like(Phi)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[0, :] = 100     # wire at x=0 at 100 V

for n_iter in range(Max_iter):
    Phi_old[:, :] = Phi
    Phi = Laplace_Gauss_Seidel_odd_even(Phi)
    DeltaPhi = np.linalg.norm(Phi - Phi_old)
    if DeltaPhi < tol:
        print("Laplace_Gauss_Seidel_odd_even converged in {0} iterations to {1}".format(n_iter+1, DeltaPhi))
        break
else:
    print("Laplace_Gauss_Seidel_odd_even did NOT converge in {0} iterations, DeltaPhi={1}".format(n_iter+1, DeltaPhi))

```

    Laplace_Gauss_Seidel_odd_even converged in 7558 iterations to 0.0009995267090286367

Plot the result and visualy compare to what we had before:

```python
plot_contour(Phi)
```

```python
plot_surf(Phi)
```

## Successive Over Relaxation (SOR)
Accelerate convergence with the scheme

\begin{align}
r_{i, j} &= \Phi_{i,j}^\text{new} - \Phi_{i, j}^\text{old}\\
\Phi_{i,j}^\text{new} &= \Phi_{i,j}^\text{old} + \omega r_{i,j}
\end{align}

where the new solution is computed with the Gauss-Seidel scheme.

Values of $1 \leq \omega \leq 2$ may work well, $\omega > 2$ can lead to numerical instabilities. Experiment!

### Solve Wire-in-a-box with SOR

Run to convergence (tol = 1e-03). Record the number of steps required (and visually check the solution).
* Start with $\omega = 1$. What should you get?
* Change $\omega$ and try to get faster convergence? Who can get it to converge in the fewest number of steps?

```python
Max_iter=10000
tol = 1e-3
Nmax = 100
omega = 1.99

Phi = np.zeros((Nmax, Nmax), dtype=np.float64)
Phi_old = np.zeros_like(Phi)
residual = np.zeros_like(Phi)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[0, :] = 100     # wire at x=0 at 100 V

for n_iter in range(Max_iter):
    Phi_old[:, :] = Phi
    Phi = Laplace_Gauss_Seidel_odd_even(Phi)
    residual[:, :] = Phi - Phi_old
    DeltaPhi = np.linalg.norm(residual)
    if DeltaPhi < tol:
        print("SOR converged in {0} iterations to {1}".format(n_iter+1, DeltaPhi))
        break
    # SOR
    Phi[:, :] = Phi_old + omega*residual  # = omega*Phi + (1-omega)*Phi_old
else:
    print("SOR did NOT converge in {0} iterations, DeltaPhi={1}".format(n_iter+1, DeltaPhi))

```

    SOR converged in 3797 iterations to 0.0009987117944452034

```python
plot_surf(Phi)
```

## Simple example of Poisson's equation: Wire and charge
Add a positive charge and a negative charge in the box.

Now we need to solve **Poisson's equation**.

$$
\nabla^2 \Phi(x, y, z) = -4\pi\rho(x, y, z)\\
$$

Finite difference solution:
$$
\Phi_{i,j} = \frac{1}{4}\Big(\Phi_{i+1,j} + \Phi_{i-1,j} + \Phi_{i,j+1} + \Phi_{i,j-1}\Big)
     + \pi\rho_{i,j} \Delta^2
$$

### Poisson Solver
Modify the Laplace solvers to now solve Poisson's equation:

```python
import numpy as np

def Poisson_Jacobi(Phi, rho, Delta=1.):
    """One update in the Jacobi algorithm for Poisson's equation"""
    Phi[1:-1, 1:-1] = 0.25*(Phi[2:, 1:-1] + Phi[0:-2, 1:-1] + Phi[1:-1, 2:] + Phi[1:-1, 0:-2]) \
                      + np.pi * Delta**2 * rho[1:-1, 1:-1]
    return Phi

def Poisson_Gauss_Seidel(Phi, rho, Delta=1.):
    """One update in the Gauss-Seidel algorithm for Poisson's equation"""
    Nx, Ny = Phi.shape
    for xi in range(1, Nx-1):
        for yj in range(1, Ny-1):
            Phi[xi, yj] = 0.25*(Phi[xi+1, yj] + Phi[xi-1, yj]
                                + Phi[xi, yj+1] + Phi[xi, yj-1]) \
                            + np.pi * Delta**2 * rho[xi, yj]
    return Phi

def Poisson_Gauss_Seidel_odd_even(Phi, rho, Delta=1.):
    """One update in the Gauss-Seidel algorithm on odd or even fields"""
    a = np.pi * Delta**2
    # odd update (uses old even)
    Phi[1:-2:2, 1:-2:2] = 0.25*(Phi[2::2, 1:-2:2] + Phi[0:-2:2, 1:-2:2]
                                + Phi[1:-2:2, 2::2] + Phi[1:-2:2, 0:-2:2]) + a * rho[1:-2:2, 1:-2:2]
    Phi[2:-1:2, 2:-1:2] = 0.25*(Phi[3::2, 2:-1:2] + Phi[1:-2:2, 2:-1:2]
                                + Phi[2:-1:2, 3::2] + Phi[2:-1:2, 1:-2:2]) + a * rho[2:-1:2, 2:-1:2]

    # even update (uses new odd)
    Phi[1:-2:2, 2:-1:2] = 0.25*(Phi[2::2, 2:-1:2] + Phi[0:-2:2, 2:-1:2]
                                + Phi[1:-2:2, 3::2] + Phi[1:-2:2, 1:-1:2]) + a * rho[1:-2:2, 2:-1:2]
    Phi[2:-1:2, 1:-2:2] = 0.25*(Phi[3::2, 1:-2:2] + Phi[1:-2:2, 1:-2:2]
                                + Phi[2:-1:2, 2::2] + Phi[2:-1:2, 0:-2:2]) + a * rho[2:-1:2, 1:-2:2]
    return Phi
```

### Solve wire with dipole

Solve the Poisson problem with SOR (as above), but now employing the Poisson solvers (we are using the fastest one that we have):

```python
Nmax = 100
Max_iter = 10000
omega = 1.99

Phi = np.zeros((Nmax, Nmax), dtype=np.float64)
Phi_old = np.zeros_like(Phi)
rho = np.zeros_like(Phi)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[:, 0] = 100     # wire at y=0 at 100 V
rho[25:27, 39:41] = 5.0
rho[75:77, 39:41] = -5.0

Delta = 1.0

for n_iter in range(Max_iter):
    Phi_old[:, :] = Phi
    Phi = Poisson_Gauss_Seidel_odd_even(Phi, rho, Delta=Delta)
    residual[:, :] = Phi - Phi_old
    DeltaPhi = np.linalg.norm(residual)
    if DeltaPhi < tol:
        print("SOR converged in {0} iterations to {1}".format(n_iter+1, DeltaPhi))
        break
    # SOR
    Phi[:, :] = Phi_old + omega*residual # = omega*Phi + (1-omega)*Phi_old
else:
    print("SOR did NOT converge in {0} iterations, DeltaPhi={1}".format(n_iter+1, DeltaPhi))

```

    SOR converged in 3823 iterations to 0.0009991605868815044

```python
plot_surf(Phi, elevation=40, azimuth=20)
```

## Charge density
Given a potential $\Phi$, Poisson's equation gives the charge density:
$$
\rho(\mathbf{x}) = -\frac{1}{4\pi} \nabla^2\Phi(\mathbf{x})
$$

Discretized Laplacian
$$
\nabla^2 \Phi \approx \frac{\Phi(x+\Delta x,y) + \Phi(x-\Delta x,y) +\Phi(x,y+\Delta y) +\, \Phi(x,y-\Delta y) - 4\Phi(x, y)}{\Delta^2}
$$

### Laplacian in Python

```python
def laplacian2d(f, Delta=1):
    """Finite difference approximation of Del^2 f.

    Arguments
    ---------
    f : M x N matrix
    Delta : float

    Returns
    -------
    M x N matrix, boundaries set to 0
    """

    L = np.zeros_like(f, dtype=np.float64)
    L[1:-1, 1:-1] = f[2:, 1:-1] + f[:-2, 1:-1] + f[1:-1, 2:] + f[1:-1, :-2] - 4*f[1:-1, 1:-1]
    return L/Delta**2

def laplacian2dsimple(f, Delta=1):
    L = np.zeros_like(f, dtype=np.float64)
    for i in range(1, L.shape[0]-1):
        for j in range(1, L.shape[1]-1):
            L[i, j] = f[i+1, j] + f[i-1, j] + f[i, j+1] + f[i, j-1] - 4*f[i, j]
    return L/Delta**2
```

#### Testing the implementation of Laplacian

Making sure that our fancy numpy-version of the Laplacian is producing the same results as the naive implementation with Python loops:

```python
import numpy as np

def test_laplacian2d():
    ftest = np.random.random((200, 200))
    assert np.allclose(laplacian2d(ftest), laplacian2dsimple(ftest))

test_laplacian2d()
```

### Charge density from potential

Compute the charge density from the converged potential
$$
\rho(x, y) = -\frac{1}{4\pi} \nabla^2 \Phi(x, y)
$$

```python
rhox = - laplacian2d(Phi)/(4*np.pi)
```

Does `rhox` show the charges of $+5$ and $-5$ that we introduced with the charge density $\rho$?

```python
print(rhox.min())
print(rhox.max())
```

    -5.0000041867249845
    5.000000000000001

Indeed, the point charges are recovered from the potential.

Plot the charge density:

```python
plot_contour(rhox, zlabel=r"charge density $\rho$")
```

The position of the point charges are found correctly but the charge on the wire appear to be absent. This is expected because the Laplacian is not defined on the boundary. If you wanted to see the charge on the wire you would need to include it inside the integration domain and also assign a thickness (e.g., one or two grid cells).

The 3D plot shows the "point" charges as spikes of the size of one grid cell:

```python
plot_surf(rhox, zlabel=r"charge density $\rho$", elevation=20, azimuth=20, offset=-3)
```

The position of the point charges are found correctly but the charge on the wire appear to be absent. The problem is that the Laplacian is not defined on the boundary.

Also, if the charges are only defined on a single grid cell then the 3D visualization is sometimes not able to show it correctly.

```python

```


# SymPy
La bibliothèque **SymPy** est spécialisée dans le **calcul symbolique**.  Elle ne dépend d'aucune bibliothèque supplémentaire. Elle permet de faire

- arithmétique
- algèbre
- calcul matriciel
- résolution d'équations

[Source](https://docs.sympy.org/latest/tutorial)

Tout d'abord nous importons toutes les méthodes (`*`) depuis le module `sympy`.

```python
from sympy import *
```

**Exercice**
Si vous rencontrez une erreur **ModuleNotFoundError** vous devez d'abord installer le module **sympy**.

## Calcul symbolique ?
C'est quoi le calcul symbolique?

La fonction `sqrt` du module `math` permet de calculer une racine carrée. Le nombre affiché n'est pas la valeur exacte. La racine de 2 ne peux pas être représenté par un nombre à virgule flottante.

```python
import math
math.sqrt(2)
```

    1.4142135623730951

Si nous faisons l'opération inverses, le carré, il reste une petite erreur.

```python
2 - math.sqrt(2) ** 2
```

    -4.440892098500626e-16

Par contre si nous utilisons la fonction `sqrt` que nous venons d'importer du module `sympy` nous gardons une expression symbolique pour la racine.

```python
sqrt(2)
```

$\displaystyle \sqrt{2}$

Si nous faisons l'opération inverse, il n'y a pas d'erreur d'arrondi.

```python
2 - sqrt(2) ** 2
```

$\displaystyle 0$

En plus, le calcul symbolique peut être simplifié.

```python
sqrt(8)
```

$\displaystyle 2 \sqrt{2}$

## Un exemple plus intéressant.

Nous définissons deux symboles

```python
x, y = symbols('x y')
expr = x + 2*y
expr
```

$\displaystyle x + 2 y$

Voici une fonction composée.

```python
f = sin(x)*exp(x)
f
```

$\displaystyle e^{x} \sin{\left(x \right)}$

Nous pouvons la différencier par rapport à x.

```python
diff(f, x)
```

$\displaystyle e^{x} \sin{\left(x \right)} + e^{x} \cos{\left(x \right)}$

## Développement

Définissons les symboles a et b et formons une expression.

```python
a, b = symbols('a b')
e = (a + b)**5
e
```

$\displaystyle \left(a + b\right)^{5}$

La méthode `expand` permet de développer l'expression.

```python
e.expand()
```

$\displaystyle a^{5} + 5 a^{4} b + 10 a^{3} b^{2} + 10 a^{2} b^{3} + 5 a b^{4} + b^{5}$

La méthode `factor` permet de la factoriser.

```python
_.factor()
```

$\displaystyle \left(a + b\right)^{5}$

## Dérivée

Définissons deux variables x et y et formons une fonction f.

```python
x, y = symbols('x, y')
f = x**2 / y + 2 * x - ln(y)
```

```python
f
```

$\displaystyle \frac{x^{2}}{y} + 2 x - \log{\left(y \right)}$

Nous pouvons différencier la fonction par rapport à x.

```python
diff(f, x)
```

$\displaystyle \frac{2 x}{y} + 2$

Et une deuxième fois par rapport à y.

```python
diff((diff(f, x)), y)
```

$\displaystyle - \frac{2 x}{y^{2}}$

## Intégrale

Nous pouvons calculer une intégrale.

```python
a = Integral(cos(x)*exp(x), x)
Eq(a, a.doit())
```

$\displaystyle \int e^{x} \cos{\left(x \right)}\, dx = \frac{e^{x} \sin{\left(x \right)}}{2} + \frac{e^{x} \cos{\left(x \right)}}{2}$

## Expansion, factorisation

Nous définissons une expression simple.

```python
x, y = symbols('x, y')
expr = x + 2*y
expr
```

$\displaystyle x + 2 y$

Nous poumvons soustraire un terme.

expr - x

Nous pouvons multplier par un terme.

```python
x*expr
```

$\displaystyle x \left(x + 2 y\right)$

Nous pouvons développer le résultat.

```python
expand(x*expr)
```

$\displaystyle x^{2} + 2 x y$

Nous pouvons factoriser une expression.

```python
factor(x**2-y**2)
```

$\displaystyle \left(x - y\right) \left(x + y\right)$

Nous pouvons différencier une expression.

```python
x, t, z, nu = symbols('x t z nu')
diff(sin(x)*exp(x), x)
```

$\displaystyle e^{x} \sin{\left(x \right)} + e^{x} \cos{\left(x \right)}$

Et nous pouvons l'intégrer de nouveau.

```python
integrate(_, x)
```

$\displaystyle e^{x} \sin{\left(x \right)}$

Nous pouvons intégrer sur une intervalle infinie.

```python
integrate(sin(x**2), (x, -oo, oo))
```

$\displaystyle \frac{\sqrt{2} \sqrt{\pi}}{2}$

Calculer une limite.

```python
limit(sin(x)/x, x, 0)
```

$\displaystyle 1$

Résoudre une écuation quadratique

```python
solve(x**2 - 3*x - 2, x)
```

    [3/2 - sqrt(17)/2, 3/2 + sqrt(17)/2]

```python
y = Function('y')
dsolve(Eq(y(t).diff(t, t) - y(t), exp(t)), y(t))
```

$\displaystyle y{\left(t \right)} = C_{2} e^{- t} + \left(C_{1} + \frac{t}{2}\right) e^{t}$

Définir une matrice.

```python
M = Matrix([[1, 2], [2, 2]])
M
```

$\displaystyle \left[\begin{matrix}1 & 2\\2 & 2\end{matrix}\right]$

Calculer sa valeur propre.

```python
M.eigenvals()
```

    {3/2 - sqrt(17)/2: 1, 3/2 + sqrt(17)/2: 1}

## Simplification

Les méthodes `simplify`, `factor` et `expand` permettent de transformer des expressions.

```python
e = sin(x)**2 + cos(x)**2
e
```

$\displaystyle \sin^{2}{\left(x \right)} + \cos^{2}{\left(x \right)}$

```python
e.simplify()
```

$\displaystyle 1$

```python
e2 = x**2 + 2*x + 1
e2
```

$\displaystyle x^{2} + 2 x + 1$

```python
e2.factor()
```

$\displaystyle \left(x + 1\right)^{2}$

```python
e5 = (x-1)**5
e5
```

$\displaystyle \left(x - 1\right)^{5}$

```python
e5.expand()
```

$\displaystyle x^{5} - 5 x^{4} + 10 x^{3} - 10 x^{2} + 5 x - 1$

```python
factor(x**5 + x**4 + x**3 + x**2 + x + 1)
```

$\displaystyle \left(x + 1\right) \left(x^{2} - x + 1\right) \left(x^{2} + x + 1\right)$

## Matrices
Pour créer une **matrice** en SymPy utilisez le constructeur `Matrix()` et en donnant une liste des vecteurs-ligne comme argument.

```python
M = Matrix([[1, -1], [3, 4], [0, 2]])
M
```

$\displaystyle \left[\begin{matrix}1 & -1\\3 & 4\\0 & 2\end{matrix}\right]$

Pour créer une **matrice-colonne**, une liste simple d'éléments est suffisant.

```python
Matrix([1, 2, 3])
```

$\displaystyle \left[\begin{matrix}1\\2\\3\end{matrix}\right]$

Pour créer une **matrice-ligne**, une double-liste d'éléments est nécessaire.

```python
Matrix([[1, 2, 3]])
```

$\displaystyle \left[\begin{matrix}1 & 2 & 3\end{matrix}\right]$

Matrices peuvet être multipliées avec l'opérateur `*`.

```python
M = Matrix([[1, 2, 3], [3, 2, 1]])
M
```

$\displaystyle \left[\begin{matrix}1 & 2 & 3\\3 & 2 & 1\end{matrix}\right]$

```python
N = Matrix([0, 1, 1])
N
```

$\displaystyle \left[\begin{matrix}0\\1\\1\end{matrix}\right]$

Voici le **produit matriciel** de deux matrices.

```python
M*N
```

$\displaystyle \left[\begin{matrix}5\\3\end{matrix}\right]$

Voici le **produit avec un scalaire**.

```python
k = Symbol('k')
M*k
```

$\displaystyle \left[\begin{matrix}k & 2 k & 3 k\\3 k & 2 k & k\end{matrix}\right]$

Contrairement aux autres objets dans SymPy, les matrice sont mutables. Donc elles peuvent être modifiés en place.

### Opérations de base

Considérons la matrice

```python
M = Matrix([[1, 2, 3], [-2, 0, 4]])
M
```

$\displaystyle \left[\begin{matrix}1 & 2 & 3\\-2 & 0 & 4\end{matrix}\right]$

La méthode `shape` retourne le nombre de lignes et colonnes $(n, m)$

```python
M.shape
```

    (2, 3)

La méthode `len` retourne le nombre total de coéfficients ($n \times m$)

```python
len(M)
```

    6

On peut accéder à un élément de la matrice par son indices.

```python
M[5]
```

$\displaystyle 4$

Itérer avec l'instruction `for` sur une matrice accède à tout ses éléments, lignes par ligne.

```python
for m in M:
    print(m, end='  ')
```

    1  2  3  -2  0  4

### Accéder aux lignes et colonnes

La méthode `row` permet d'extraire une ligne.

```python
M.row(0)
```

$\displaystyle \left[\begin{matrix}1 & 2 & 3\end{matrix}\right]$

La méthode `col` permet d'extraire une colonne.

```python
M.col(0)
```

$\displaystyle \left[\begin{matrix}1\\-2\end{matrix}\right]$

La méthode `col_del`permet de supprimer une colonne.

```python
M.col_del(0)
M
```

$\displaystyle \left[\begin{matrix}2 & 3\\0 & 4\end{matrix}\right]$

La méthode `row_del` permet de supprimer une ligne. Cette opération modifie la matrice **en place** et retourne `None`.

```python
M.row_del(1)
M
```

$\displaystyle \left[\begin{matrix}2 & 3\end{matrix}\right]$

Les méthodes `row_insert` et `col_insert` insèrent une ligne ou une colonne, mais pas en pace. Elle retournent une nouvelle matrice.

```python
M.row_insert(1, Matrix([[0, 4]]))
```

$\displaystyle \left[\begin{matrix}2 & 3\\0 & 4\end{matrix}\right]$

Vérfions que la matrice $M$ n'a pas changé.

```python
M
```

$\displaystyle \left[\begin{matrix}2 & 3\end{matrix}\right]$

Pour changer la matrice, nous devons affecter l'opération à son symbole.

```python
M = M.row_insert(1, Matrix([[0, 4]]))
M
```

$\displaystyle \left[\begin{matrix}2 & 3\\0 & 4\end{matrix}\right]$

### Opérations de base
Soient deux matrices $M$ et $N$ de taille $2 \times 2$.

```python
M = Matrix([[1, 3], [-2, 3]])
M
```

$\displaystyle \left[\begin{matrix}1 & 3\\-2 & 3\end{matrix}\right]$

```python
N = Matrix([[0, 3], [0, 7]])
N
```

$\displaystyle \left[\begin{matrix}0 & 3\\0 & 7\end{matrix}\right]$

Voici l'**addition**

```python
M+N
```

$\displaystyle \left[\begin{matrix}1 & 6\\-2 & 10\end{matrix}\right]$

Voici la **soustraction**

```python
M-N
```

$\displaystyle \left[\begin{matrix}1 & 0\\-2 & -4\end{matrix}\right]$

Voici la **multiplication**

```python
M*N
```

$\displaystyle \left[\begin{matrix}0 & 24\\0 & 15\end{matrix}\right]$

Voici la **puissance 2**

```python
M**2
```

$\displaystyle \left[\begin{matrix}-5 & 12\\-8 & 3\end{matrix}\right]$

La puissance -1 donne l'**inverse** d'une matrice.

```python
M**(-1)
```

$\displaystyle \left[\begin{matrix}\frac{1}{3} & - \frac{1}{3}\\\frac{2}{9} & \frac{1}{9}\end{matrix}\right]$

### La transposé
La méthode `T` retourne la transposé d'une matrice.

```python
M = Matrix([[1, 2, 3], [4, 5, 6]])
M
```

$\displaystyle \left[\begin{matrix}1 & 2 & 3\\4 & 5 & 6\end{matrix}\right]$

```python
M.T
```

$\displaystyle \left[\begin{matrix}1 & 4\\2 & 5\\3 & 6\end{matrix}\right]$

### Matrice particuliers
La fonction `eye(n)` retourne une **matrice d'identité**.

```python
eye(3)
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{matrix}\right]$

La fonction `zeros(n, m)` retourne une **matrice zéro**.

```python
zeros(3, 5)
```

$\displaystyle \left[\begin{matrix}0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0\end{matrix}\right]$

La fonction `ones(n, m)` retourne une matrice dont tous les éléments valent 1.

```python
ones(2, 4)
```

$\displaystyle \left[\begin{matrix}1 & 1 & 1 & 1\\1 & 1 & 1 & 1\end{matrix}\right]$

La fonction `diag` retourne une matrice diagonale.

```python
diag(1, 2, 3)
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0\\0 & 2 & 0\\0 & 0 & 3\end{matrix}\right]$

Tous ces méthodes fonction aussi avec des symboles.

```python
x, y, z = symbols('x y z')
diag(x, y, z)
```

$\displaystyle \left[\begin{matrix}x & 0 & 0\\0 & y & 0\\0 & 0 & z\end{matrix}\right]$

### Déterminant
La méthode `det()` calcule le déterminant d'une matrice.

```python
M = Matrix([[1, 0, 1], [2, -1, 3], [4, 3, 2]])
M
```

$\displaystyle \left[\begin{matrix}1 & 0 & 1\\2 & -1 & 3\\4 & 3 & 2\end{matrix}\right]$

```python
M.det()
```

$\displaystyle -1$

Dans le cas d'une matrice $2 \times 2$ le déterminant est:

```python
a, b, c, d = symbols('a b c d')
M = Matrix([[a, b], [c, d]])
M.det()
```

$\displaystyle a d - b c$

Le déterminant d'une matrice identié est

```python
eye(3).det()
```

$\displaystyle 1$

```python
M.nullspace()
```

    []

```python
A = Matrix([[1, 2], [-1, 3]])
A
```

$\displaystyle \left[\begin{matrix}1 & 2\\-1 & 3\end{matrix}\right]$

```python
A.eigenvals()
```

    {2 - I: 1, 2 + I: 1}

```python
A.eigenvects()
```

    [(2 - I,
      1,
      [Matrix([
       [1 + I],
       [    1]])]),
     (2 + I,
      1,
      [Matrix([
       [1 - I],
       [    1]])])]

## Equations

```python
Eq(x, y)
```

$\displaystyle x = y$

```python
eq = Eq(x**2 + 2*x + 1, 0)
eq
```

$\displaystyle x^{2} + 2 x + 1 = 0$

```python
solveset(eq, x)
```

$\displaystyle \left\{-1\right\}$

```python
linsolve([x + y + z -1, x + y + 2*z - 3], (x, y, z))
```

$\displaystyle \left\{\left( - y - 1, \  y, \  2\right)\right\}$

```python
Eq(x + y + z -1, x + y + 2*z - 3)
```

$\displaystyle x + y + z - 1 = x + y + 2 z - 3$

```python
diag(1, 2) * M
```

$\displaystyle \left[\begin{matrix}a & b\\2 c & 2 d\end{matrix}\right]$

```python
N
```

$\displaystyle \left[\begin{matrix}0 & 3\\0 & 7\end{matrix}\right]$

## Permutation de lignes et colonnes

```python
M = Matrix([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
M
```

$\displaystyle \left[\begin{matrix}1 & 2 & 3\\4 & 5 & 6\\7 & 8 & 9\end{matrix}\right]$

Une matrice *anti-diagonale* permet de **basculer** une matrice.

```python
A = Matrix([[0, 0, 1], [0, 1, 0], [1, 0, 0]])
A
```

$\displaystyle \left[\begin{matrix}0 & 0 & 1\\0 & 1 & 0\\1 & 0 & 0\end{matrix}\right]$

Permutation des colonnes de $(2, 1, 0)$

```python
M * A
```

$\displaystyle \left[\begin{matrix}3 & 2 & 1\\6 & 5 & 4\\9 & 8 & 7\end{matrix}\right]$

Permutation des lignes de (2, 1, 0)

```python
A * M
```

$\displaystyle \left[\begin{matrix}7 & 8 & 9\\4 & 5 & 6\\1 & 2 & 3\end{matrix}\right]$

Horizontalement et verticalement

```python
A * M * A
```

$\displaystyle \left[\begin{matrix}9 & 8 & 7\\6 & 5 & 4\\3 & 2 & 1\end{matrix}\right]$

```python
B = Matrix([[0, 1, 0], [0, 0, 1], [1, 0, 0]])
B
```

$\displaystyle \left[\begin{matrix}0 & 1 & 0\\0 & 0 & 1\\1 & 0 & 0\end{matrix}\right]$

Les colonnes sont permuté de $(2, 0, 1)$

```python
M * B
```

$\displaystyle \left[\begin{matrix}3 & 1 & 2\\6 & 4 & 5\\9 & 7 & 8\end{matrix}\right]$

Les lignes sont permuté de $(1, 2, 0)$

```python
B * M
```

$\displaystyle \left[\begin{matrix}4 & 5 & 6\\7 & 8 & 9\\1 & 2 & 3\end{matrix}\right]$

```python
C = Matrix([[1, 0, 0], [0, 0, 1], [0, 1, 0]])
C
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0\\0 & 0 & 1\\0 & 1 & 0\end{matrix}\right]$

Permutation des colonnes $(0, 2, 1)$

```python
M * C
```

$\displaystyle \left[\begin{matrix}1 & 3 & 2\\4 & 6 & 5\\7 & 9 & 8\end{matrix}\right]$

```python

```


# Deep Learning - Assignment 1

## Outline (15 points)
#### In this assignment, you will learn:
* How to generate random data using python.
* Building linear models for simple regression problem on the generated data.
* Training the linear models with gradient descent algorithm.
* How to alleviate over-fitting for your model.
* **Concepts** you will learn: ***Regularization***, ***Model selection***, ***Gradient descent***, ***Over-fitting***, ***Weight decay***, ***Training/Validation/Testing***.

#### Tasks
In this assignment, we are going to solve a basic linear regression problem by fitting a polynomial function, which we shall use as a running example to motivate a number of key concepts mentioned above.
* **Part 1.** Generate training and testing data using python.
* **Part 2.** Linear regression with polynomials without regularization.
* **Part 3.** Linear regression with polynomials with regularization.

#### Environment
Python 3.

Other libraries should be installed correctly such as numpy, matplotlib, *etc*., according to the dependencies of the assignment.

#### Database
* Randomly synthesized data.

## Part 1. Generate training and testing data using python
Suppose we observe a real-valued input variable $x$ and we wish to use this observation to predict the value of a real-valued target variable $t$. For the present purposes, it is instructive to consider an artificial example using synthetically generated data because we then know the precise process that generated the data for comparison against any learned model.

We therefore generate our training data which consists of 10 pairs of $\{x_i, y_i\}$ where $\{x_i\}$ are generated uniformly in range (0, 1), and the corresponding target values $\{y_i\}$ are obtained by first computing the corresponding values of the function $sin(2\pi x)$ and then adding random noise with a Gaussian distribution having standard deviation 0.3.
\begin{align*}
y_i = sin(2\pi x_i) + \varepsilon, \varepsilon\sim \mathcal{N}(0,\,0.3^{2}).
\end{align*}

The testing data is created the same way as training data but the number of pairs is 100, in order to give a more complete evaluation of the model to be trained.

Please create the required training and testing data below and plot them. You could use numpy to create the data and matplotlib.pyplot to show them.

```python
# import libraries
import numpy as np
import matplotlib.pyplot as plt

# your code here
x_train = np.random.rand(10, 1)
y_train = np.sin(2 * np.pi * x_train) + np.random.normal(0, 0.3, (10, 1))

x_test = np.random.rand(100, 1)
y_test = np.sin(2 * np.pi * x_test) + np.random.normal(0, 0.3, (100, 1))

plt.plot(x_train, y_train, marker='o', linestyle='none')
plt.plot(x_test, y_test, fillstyle='none', marker='o', color='g', linestyle='none')
plt.show()
```

## Part 2. Linear regression with polynomials without regularization
Congratulations if you generate the data correctly! However, to make sure all of you use the same data, we have generated it in advance.

```python
import pickle
with open('data/rawdata.pickle', 'rb') as f:
    data = pickle.load(f)

x_train = data['x_train']
y_train = data['y_train']

x_test = data['x_test']
y_test = data['y_test']
```

The task in this part is to train a linear model to fit the curve on the training data. Now that we have $\{x, y\}$, we consider using a polynomial function of the form
\begin{align}
f(x, \pmb{w}) = \omega_0 + \omega_1x + \omega_2x^2 + ... + \omega_Mx^M = \sum_{j=0}^{M}\omega_jx^j,
\tag{1}
\end{align}
where $M$ is the *order* of the polynomial, and $x^j$ denotes $x$ raised to the power of j. The values of the coefficients will be determined by fitting the polynomial to the training data. To do so, we introduce the error function to be minimized:
\begin{equation}
E(\pmb{w}) = \frac{1}{2}\sum_{i=1}^{N}\{f(x_i, \pmb{w}) - y_i\}^2,
\tag{2}
\end{equation}
where, $N$ is 10 in our case.

First of all, to use matrix operation in this case, we re-write the polynomial function (1) in the form of matrix multiplication:
$$
\begin{bmatrix}
f(x_1, \pmb{w}) \\ f(x_2, \pmb{w}) \\ . \\ . \\ . \\ f(x_N, \pmb{w})
\end{bmatrix}
=
\begin{bmatrix}
1 & x_1 & x_1^2 & . & . & . & x_1^M \\ 1 & x_2 & x_2^2 & . & . & . & x_2^M \\ . & . & . & . & . & . & . \\ . & . & . & . & . & . & . \\ . & . & . & . & . & . & . \\ 1 & x_N & x_N^2 & . & . & . & x_N^M
\end{bmatrix}
\begin{bmatrix}
\omega_0 \\ \omega_1 \\ . \\ . \\ . \\ \omega_M
\end{bmatrix}
$$
We call the first matrix on the right hand side feature matrix. We next build the feature matrix and try to solve above minimization problem using [`Gradient Descent Algorithm`](https://towardsdatascience.com/an-introduction-to-gradient-descent-c9cca5739307) and the closed-form analytical solution. Because the result from the closed-form solution is theoretically optimal for the problem, by doing this, we could know how well our gradient descent algorithm performs by comparing the results using gradient descent and closed-form methods respectively.

Anyway, I'll give an guidance using $M=3$ as following.

### Part 2.1. Method based on closed-form analysis
First, let's quickly solve this using the closed-form analytical solution. In this case, we luckily have this solution to solve the polynomial regression problem. However, in practical problems using deep learning models, which are non-linear and often very complex, there doesn't exist a closed-form solution or extremely difficult to find the solution analytically, the most common way is using gradient descent algorithm.

In our case, we will first solve the regression problem using closed-form analysis to get the optimal solution, which can also be seen as the upper bound the gradient descent algorithm goes (because gradient descent algorithm will alwayse find an approximate solution which close to the optimal).

First, we calculate the feature matrix based on the matrix formulation and apply [`sklearn.linear_model.LinearRegression`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html) to solve the regression problem.

```python
feature_matrix = np.ones_like(x_train)
M = 3
for i in range(1, M+1):
    feature_matrix = np.concatenate((feature_matrix, x_train ** i), axis=1)

from sklearn.linear_model import LinearRegression

model = LinearRegression(fit_intercept=False).fit(feature_matrix, y_train)
```

Then, show the original training data points and the draw the curve of the function obtained on a single figure.

```python
x_curve = np.linspace(0, 1, 10000).reshape(-1, 1)
features_curve = np.ones_like(x_curve)
for i in range(1, M+1):
    features_curve = np.concatenate((features_curve, x_curve ** i), axis=1)

y_curve = model.predict(features_curve)
plt.plot(x_train, y_train, marker='o', linestyle='None')
plt.plot(x_curve, y_curve, 'r')
plt.show()
```

Now, we have got all the coefficients (noted as $\pmb{w}^*$) of the polynomial function by minimizing the error function (2). Therefore, we can evaluate the residual value of $E(\pmb{w}^*)$ given by (2) for the training and testing data. However, it is sometimes more convenient to use the *root-mean-square* (RMS) error defined by
$$
E_{RMS} = \sqrt{2E(\pmb{w}^*)/N}
\tag{3}
$$
in which the division by $N$ allows us to compare different sizes of data sets on an equal footing, also with the same scale (and in the same units) as the target variable $y$.

In order to evaluate the polynomial function we obtained, we do the following:
* Calculate and print the RMS error on training data based on equation (3),
* Calculate and print the RMS error on testing data based on equation (3),
* Get and print the coefficients of the trained function $\omega_0$ ~ $\omega_M$ using [`LinearRegression.coef_`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html#sklearn.linear_model.LinearRegression).

```python
prediction_train = model.predict(feature_matrix)
rms_train = np.sqrt(np.sum((prediction_train - y_train) ** 2) / len(prediction_train))

feature_matrix_test = np.ones_like(x_test)
for i in range(1, M+1):
    feature_matrix_test = np.concatenate((feature_matrix_test, x_test ** i), axis=1)

prediction_test = model.predict(feature_matrix_test)
rms_test = np.sqrt(np.sum((prediction_test - y_test) ** 2) / len(prediction_test))

print('rms error on training data: ', rms_train)
print('rms error on testing data: ', rms_test)
print('coefficients of the trained function: ', model.coef_)
```

    rms error on training data:  0.16066897173745506
    rms error on testing data:  0.30943910311337297
    coefficients of the trained function:  [[ -0.16311694  10.9380811  -30.08577056  19.11555865]]

### Part 2.2. Method based on Gradient Descent Algorithm
Once we get the closed-form solution, which is also the optimal solution, the next thing we are supposed to do is to re-solve the problem by gradient descent algorithm, because gradient descent matters a lot in practical applications since we cannot have a closed-form solution for most of the practical problems. Generally, the ingredients make up a gradient descent algorithm are:
* defining the learning rate to update our parameters,
* defining a loss function or objective function which you should minimize (here we use the error function based on equation (3)),
* calculating the gradients of parameters w.r.t. the loss,
* update parameters according to the learning rate and gradients to a direction that makes the loss smaller.

So, let's do this in that way.

```python
# defining a learning rate, you could change it
learning_rate = 0.1

# initialize the parameters, here we simply assign each parameter to 1
# becuase M=3, so we totally have 4 parameters (w0, w1, w2, w3)
W = np.ones((M+1, 1))

# calculate the feature matrix as before, in order to compute the loss
feature_matrix = np.ones_like(x_train)
for i in range(1, 4):
    feature_matrix = np.concatenate((feature_matrix, x_train ** i), axis=1)

# iteratively do gradient descent, here we just iterate it 50 times
for i in range(50):
    # calculate the values of the polynomial function
    F = np.matmul(feature_matrix, W)

    # calculate the loss, print it every 10 iterations
    if i % 10 ==0:
        loss = 0.5 * np.sum((F - y_train) ** 2)
        print('loss: ', loss)

    # calcualte the gradients of the parameters
    # here we have 10 data samples, so we calculate the mean.
    # please understand this formulation by inferencing the gradients
    # by yourself with what you learnt in undergraduate school.
    G = np.mean((F - y_train) * feature_matrix, axis=0).reshape(-1, 1)

    # update parameters to a direction that makes the loss smaller
    # so we use '-' here, it is also why we call gradient "descent"
    W = W - learning_rate * G
```

    loss:  34.380066478803926
    loss:  4.012740605942455
    loss:  2.436307391471592
    loss:  1.958278267953077
    loss:  1.6626874106866916

Again, we show the original training data points and the draw the curve of the function obtained on a single figure.

```python
x_curve = np.linspace(0, 1, 10000).reshape(-1, 1)
features_curve = np.ones_like(x_curve)
for i in range(1, M+1):
    features_curve = np.concatenate((features_curve, x_curve ** i), axis=1)

#y_curve = model.predict(features_curve)
y_curve = features_curve.dot(W)
plt.plot(x_train, y_train, marker='o', linestyle='None')
plt.plot(x_curve, y_curve, 'r')
plt.show()
```

Also, in order to evaluate the polynomial function we obtained, we do the following:
* Calculate and print the RMS error on training data based on equation (3),
* Calculate and print the RMS error on testing data based on equation (3),
* Print the coefficients (parameters).

```python
prediction_train = np.matmul(feature_matrix, W)
rms_train = np.sqrt(np.sum((prediction_train - y_train) ** 2) / len(prediction_train))

prediction_test = np.matmul(feature_matrix_test, W)
rms_test = np.sqrt(np.sum((prediction_test - y_test) ** 2) / len(prediction_test))

print('rms error on training data: ', rms_train)
print('rms error on testing data: ', rms_test)
print('coefficients of the trained function: ', W)
```

    rms error on training data:  0.5422276467732126
    rms error on testing data:  0.5918422558573816
    coefficients of the trained function:  [[ 0.23861297]
     [-0.26124322]
     [-0.24765614]
     [-0.13137266]]

#### Question 1. What's the difference between closed-form method and gradient descent method?

#### Your answer:

### Part 2.3. Model selection with closed-form solution
There remains the problem of choosing the order $M$ of the polynomial, this is actually an example of the concept *model selection*. To see how changing $M$ affects the regression problem being solved, please change $M=$ from 1 to 9, and get the following objectives based on **closed-form analysis**:
* Draw all the 9 figures as done above (make sure the function curve and training data points are in the same figure for each case, using plt.subplots),
* Calculate all the error values using equation (3) with respect to trainig and testing data for all the 9 cases, and plot this values on a single figure (that means 18 points would be drawn in the figure, with x axis representing $M$ and y axis representing the error value, please use different colors to distinguish training and testing),
* Print the coefficients for the 9 polynomial functions.

```python
# your code here
rmss_train = []
rmss_test = []
fig, ax = plt.subplots(3, 3, figsize=(15, 15))
for M in range(1, 10):
    feature_matrix = np.ones_like(x_train)
    for i in range(1, M + 1):
        feature_matrix = np.concatenate((feature_matrix, x_train ** i), axis=1)

    model = LinearRegression(fit_intercept=False).fit(feature_matrix, y_train)
    x_curve = np.linspace(0, 1, 10000).reshape(-1, 1)
    features_curve = np.ones_like(x_curve)
    for i in range(1, M + 1):
        features_curve = np.concatenate((features_curve, x_curve ** i), axis=1)

    # your code here
    y_curve = model.predict(features_curve)
    row = (M-1) // 3
    col = (M-1) % 3
    ax[row,col].plot(x_train, y_train, marker='o', linestyle='None')
    ax[row,col].plot(x_curve, y_curve, 'r')

    prediction_train = model.predict(feature_matrix)
    rms_train = np.sqrt(np.sum((prediction_train - y_train) ** 2) / len(prediction_train))

    feature_matrix_test = np.ones_like(x_test)
    for i in range(1, M + 1):
        feature_matrix_test = np.concatenate((feature_matrix_test, x_test ** i), axis=1)

    prediction_test = model.predict(feature_matrix_test)
    rms_test = np.sqrt(np.sum((prediction_test - y_test) ** 2) / len(prediction_test))

    rmss_train.append(rms_train)
    rmss_test.append(rms_test)

    print('\n Results when M = %d' % M)
    print('rms error on training / testing data: %f / %f' % (rms_train, rms_test))
    print('coefficients of the trained function: ', model.coef_)

fig.tight_layout()
plt.show()
M_x = np.linspace(1, 9, 9)
```

```python
plt.plot(M_x, rmss_train, 'b', marker='o', linestyle='None')
plt.plot(M_x, rmss_test, 'r', marker='o', linestyle='None')
plt.show()
```

In fact, when $M = 9$, the 10 training points would be exactly fitted, and the training loss becomes zero, since the degree of freedom for this regression problem is 10. However, the fitted curve oscillates wildly and gives a very poor representation of the function $sin(2\pi x)$. This behavior is known as **over-fitting**.
#### Question 2. Which one is the best one and why? Why the over-fitting happens when $M$ is big?

#### Your answer:

### Part 2.4. Gradient descent practice
Set $M=9$, based on **gradient descent algorithm**, do the following (During training, print the loss every $n$ iterations to show you are running the gradient descent algorithm to train the model, you could select a $n$ as you like):
* Change learning rate and see how fast the training converges,
* Change number of iteration to see whether the training loss would be smaller,
* Draw the figure as done above when you feel the training is good enough (make sure the function curve and training data points are in the same figure),
* Calculate the error values using equation (3) with respect to trainig and testing data,
* Print the coefficients (parameters) for the polynomial function.

```python
# your code here
# defining a learning rate
learning_rate = 0.1

# initialize the parameters, here we simply assign each parameter to 1
# becuase M=3, so we totally have 4 parameters (w0, w1, w2, w3)
W = np.ones((M+1, 1))

# calculate the feature matrix as before, in order to compute the loss
feature_matrix = np.ones_like(x_train)
for i in range(1, M+1):
    feature_matrix = np.concatenate((feature_matrix, x_train ** i), axis=1)

# iteratively do gradient descent
for i in range(50):
    # calculate the values of the polynomial function
    F = np.matmul(feature_matrix, W)

    # calculate the loss
    if i % 10 ==0:
        loss = 0.5 * np.sum((F - y_train) ** 2)
        print('loss: ', loss)

    # calcualte the gradients of the parameters
    # here we have 10 data samples, so we calculate the mean.
    # please understand this formulation by inferencing the gradients
    # by yourself with what you learnt in undergraduate school.
    G = np.mean((F - y_train) * feature_matrix, axis=0).reshape(-1, 1)

    # update parameters to a direction that makes the loss smaller
    # so we use '-' here, it is also why we call gradient "descent"
    W = W - learning_rate * G
```

    loss:  108.9576867114645
    loss:  4.306569760761631
    loss:  2.1385030210300267
    loss:  1.5563558849819616
    loss:  1.312244278798215

```python
x_curve = np.linspace(0, 1, 10000).reshape(-1, 1)
features_curve = np.ones_like(x_curve)
for i in range(1, M+1):
    features_curve = np.concatenate((features_curve, x_curve ** i), axis=1)

#y_curve = model.predict(features_curve)
y_curve = features_curve.dot(W)
plt.plot(x_train, y_train, marker='o', linestyle='None')
plt.plot(x_curve, y_curve, 'r')
plt.show()
```

```python
prediction_train = np.matmul(feature_matrix, W)
rms_train = np.sqrt(np.sum((prediction_train - y_train) ** 2) / len(prediction_train))

prediction_test = np.matmul(feature_matrix_test, W)
rms_test = np.sqrt(np.sum((prediction_test - y_test) ** 2) / len(prediction_test))

print('rms error on training data: ', rms_train)
print('rms error on testing data: ', rms_test)
print('coefficients of the trained function: ', W)
```

    rms error on training data:  0.48981780339698056
    rms error on testing data:  0.5338034958265429
    coefficients of the trained function:  [[ 0.35652297]
     [-0.29036022]
     [-0.34839046]
     [-0.27086086]
     [-0.17482634]
     [-0.08679106]
     [-0.01139232]
     [ 0.05213066]
     [ 0.10563873]
     [ 0.15094612]]

#### Question 3. Actually, when using gradient descent algorithm, even in the case that $M=9$, it is hard to observe the over-fitting phenomenon. It is because it's difficult to reach the theoretically optimal solution that makes the training loss zero. Regardless of over-fitting, please give your idea to make the training loss smaller when using gradient descent algorithm in this case.

#### Your answer:

## Part 3. Linear regression with polynomials with regularization
Before introducing regularization, let's think about a way to solve the over-fitting problem.

Definitely, through the experiments done above, you somewhat have found that the most direct way is reducing $M$. However, what if we use more training data?

### Part 3.1. Re-visit polynomial regression with extra training data and $M=9$
Finish this step as in Part 2.1 based on closed-form analysis. (*note*: please calculate the RMS error on the original training and testing data)

```python
with open('data/extradata.pickle', 'rb') as f:
    extra_data = pickle.load(f)

x_extra = extra_data['x_extra']
y_extra = extra_data['y_extra']

# your code here
x_extra_train = np.concatenate((x_train, x_extra), axis=0)
y_extra_train = np.concatenate((y_train, y_extra), axis=0)
feature_extra_matrix = np.ones_like(x_extra_train)
for i in range(1, 10):
    feature_extra_matrix = np.concatenate((feature_extra_matrix, x_extra_train ** i), axis=1)

model = LinearRegression(fit_intercept=False).fit(feature_extra_matrix, y_extra_train)

x_curve = np.linspace(0, 1, 10000).reshape(-1, 1)
features_curve = np.ones_like(x_curve)
for i in range(1, 10):
    features_curve = np.concatenate((features_curve, x_curve ** i), axis=1)
y_curve = model.predict(features_curve)
plt.plot(x_train, y_train, marker='o', linestyle='None')
plt.plot(x_curve, y_curve, 'r')
plt.show()

prediction_train = model.predict(feature_matrix)
rms_train = np.sqrt(np.sum((prediction_train - y_train) ** 2) / len(prediction_train))

feature_matrix_test = np.ones_like(x_test)
for i in range(1, 10):
    feature_matrix_test = np.concatenate((feature_matrix_test, x_test ** i), axis=1)

prediction_test = model.predict(feature_matrix_test)
rms_test = np.sqrt(np.sum((prediction_test - y_test) ** 2) / len(prediction_test))

print('rms error on training data: ', rms_train)
print('rms error on testing data: ', rms_test)
print('coefficients of the trained function: ', model.coef_)
```

#### Question 4. Does introducing more training data in this case alleviate over-fitting? Why?

#### Your answer:

### Part 3.2. Re-visit polynomial regression with regularization and $M=9$
We may alleviate the over-fitting problem through introducing more training data, but what if there is no extra training data? In fact, we donnot want to limit the number of parameters in a model according to the size of the available training set, because it also limits the capacity and flexibility of the model. One technique that is often used to control over the over-fitting phenomenon in such cases is that of **regularization**, which involves adding a penalty term to the error function (2) in order to discourage the coefficients from reaching large values. The simplest such penalty term takes the form of a sum of squares of all of the coefficients, leading to a modified error function:
$$
\widetilde{E}(\pmb{w})=\frac{1}{2}\sum_{i=1}^{N}\{f(x_i, \pmb{w}) - y_i\}^2 + \frac{\lambda}{2}||\pmb{w}||^2
\tag{4}
$$
where $||\pmb{w}||^2\equiv \pmb{w}^T\pmb{w} = \omega_0^2 + \omega_1^2 + ... + \omega_M^2$, and the coefficient $\lambda$ governs the relative importance of the regularization term compared with the sum-of-squares error term.

Again, this error function can be minimized exactly in closed form. The regression with such a quadratic regularizer is called *ridge regression*. In the context of neural networks, this approcah is known as **weight decay**.

Apart from training and testing data, we also need another data set to test how suitable the $\lambda$ is before we use the model on the testing data. We call this data set **validation** set. Usually, when we have large amount of training data, we create a validation set splitted from the original training data and use the rest of the training data as the new training data. In our case, we only have the training data consisting of 10 data points, so here we use the extra data on Part 3.1 as the validation data.

Please solve the ridge regression problem with the polynomial function using **training** data in the following 7 conditions:
* $M=9$, $ln\lambda=-35$ ($\lambda=e^{-35}$)
* $M=9$, $ln\lambda=-25$
* $M=9$, $ln\lambda=-20$
* $M=9$, $ln\lambda=-15$
* $M=9$, $ln\lambda=-10$
* $M=9$, $ln\lambda=-5$
* $M=9$, $ln\lambda=0$

with the following tasks:
1. Calculate the root-mean-square error based on (3) (not the modified one) on the **training** and **validation** set. Plot the curves of RMS error versus $\ln\lambda$ for training and validation set in a single figure (i.e., in this figure, two curves will be plotted, one for training data and one for validation data. x axis represents $\ln\lambda$ and y axis represents RMS error).
2. Print the coefficients for the obtained 7 ploynomial functions.
3. Find the best value for $\lambda$ among these conditions (`best` means its corresponding RMS on the **validation** set is the smallest), calculate the corresponding RMS error on **testing** data and print it.
4. Draw the curve of the polynomial function in a figure w.r.t the best $\lambda$, in which the 10 original training data points and the 100 testing data points should also be plotted.

To sovle the ridge regression problem, we could use [`sklearn.linear_model.Ridge`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html).

```python
# your code here
feature_matrix = np.ones_like(x_train)
for i in range(1, 10):
    feature_matrix = np.concatenate((feature_matrix, x_train ** i), axis=1)

feature_matrix_valid = np.ones_like(x_extra)
for i in range(1, 10):
    feature_matrix_valid = np.concatenate((feature_matrix_valid, x_extra ** i), axis=1)

from sklearn.linear_model import Ridge

rmss_train = []
rmss_valid = []
best_model = None
best_valid = np.finfo(float).max

ln_lambdas = [-35, -25, -20, -15, -10, -5, 0]
for ln_lambda in ln_lambdas:
    model = Ridge(alpha=np.exp(ln_lambda), fit_intercept=False).fit(feature_matrix, y_train)

    prediction_train = model.predict(feature_matrix)
    rms_train = np.sqrt(np.sum((prediction_train - y_train) ** 2) / len(prediction_train))

    prediction_valid = model.predict(feature_matrix_valid)
    rms_valid = np.sqrt(np.sum((prediction_valid - y_extra) ** 2) / len(prediction_valid))

    rmss_train.append(rms_train)
    rmss_valid.append(rms_valid)

    print('\n Results when M = %d' % M)
    print('coefficients of the trained function: ', model.coef_)

    if rms_valid < best_valid:
        best_valid = rms_valid
        best_model = model

plt.plot(ln_lambdas, rmss_train, 'b', marker='o')
plt.plot(ln_lambdas, rmss_valid, 'r', marker='o')
plt.xlabel('ln(lambda)')
plt.ylabel('RMS error')
plt.show()
print(rmss_valid)
```

```python
feature_matrix_test = np.ones_like(x_test)
for i in range(1, 10):
    feature_matrix_test = np.concatenate((feature_matrix_test, x_test ** i), axis=1)

prediction_test = best_model.predict(feature_matrix_test)
rms_test = np.sqrt(np.sum((prediction_test - y_test) ** 2) / len(prediction_test))
print('rms error on testing data: ', rms_test)

x_curve = np.linspace(0, 1, 10000).reshape(-1, 1)
features_curve = np.ones_like(x_curve)
for i in range(1, 10):
    features_curve = np.concatenate((features_curve, x_curve ** i), axis=1)

y_curve = best_model.predict(features_curve)
plt.plot(x_train, y_train, marker='o', linestyle='None')
plt.plot(x_test, y_test, fillstyle='none', marker='o', color='g', linestyle='None')
plt.plot(x_curve, y_curve, 'r')
plt.show()
```

#### Question 5. Does regularization alleviate over-fitting? Why?

#### Your answer:

### Part 3.3. Gradient descent practice
Choose a $\lambda$ as you like and set $M=9$, calculate and print its rms error on training, validation and testing data respectively, draw the function curve with all the training and testing data on it, based on **gradient descent algorithm**.

***Note***: the loss function is equation (4), when calculating the gradients, remember to also consider the regularizatin component.

```python
# your code here
# your code here
# defining a learning rate
M = 9
learning_rate = 0.1
lmda = np.exp(-10)

# initialize the parameters, here we simply assign each parameter to 1
# becuase M=3, so we totally have 4 parameters (w0, w1, w2, w3)
W = np.ones((M+1, 1))

# calculate the feature matrix as before, in order to compute the loss
feature_matrix = np.ones_like(x_train)
for i in range(1, M+1):
    feature_matrix = np.concatenate((feature_matrix, x_train ** i), axis=1)

# iteratively do gradient descent
for i in range(50):
    # calculate the values of the polynomial function
    F = np.matmul(feature_matrix, W)

    # calculate the loss
    if i % 10 ==0:
        loss = 0.5 * np.sum((F - y_train) ** 2) + 0.5 * lmda * np.sum(W ** 2)
        print('loss: ', loss)

    # calcualte the gradients of the parameters
    # here we have 10 data samples, so we calculate the mean.
    # please understand this formulation by inferencing the gradients
    # by yourself with what you learnt in undergraduate school.
    G = np.mean((F - y_train) * feature_matrix, axis=0).reshape(-1, 1) + lmda * W

    # update parameters to a direction that makes the loss smaller
    # so we use '-' here, it is also why we call gradient "descent"
    W = W - learning_rate * G

x_curve = np.linspace(0, 1, 10000).reshape(-1, 1)
features_curve = np.ones_like(x_curve)
for i in range(1, M+1):
    features_curve = np.concatenate((features_curve, x_curve ** i), axis=1)

#y_curve = model.predict(features_curve)
y_curve = features_curve.dot(W)
plt.plot(x_train, y_train, marker='o', linestyle='None')
plt.plot(x_curve, y_curve, 'r')
plt.show()

prediction_train = np.matmul(feature_matrix, W)
rms_train = np.sqrt(np.sum((prediction_train - y_train) ** 2) / len(prediction_train))

prediction_test = np.matmul(feature_matrix_test, W)
rms_test = np.sqrt(np.sum((prediction_test - y_test) ** 2) / len(prediction_test))

print('rms error on training data: ', rms_train)
print('rms error on testing data: ', rms_test)
print('coefficients of the trained function: ', W)
```

```python

```


# Algorithms Exercise 2

## Imports

```python
%matplotlib inline
from matplotlib import pyplot as plt
import seaborn as sns
import numpy as np
```

## Peak finding

Write a function `find_peaks` that finds and returns the indices of the local maxima in a sequence. Your function should:

* Properly handle local maxima at the endpoints of the input array.
* Return a Numpy array of integer indices.
* Handle any Python iterable as input.

```python
def find_peaks(a):
    """Find the indices of the local maxima in a sequence."""
    # YOUR CODE HERE
    #raise NotImplementedError()
    ind=[]
    #next two if checks end points
    if a[0]> a[1]:
            ind.append(0)
    if a[len(a)-1]>a[len(a)-2]:
        ind.append(len(a)-1)
    #finds local maxima in string by comparing adjacent
    for i in range(1,len(a)-1):
        if a[i]>a[i-1] and a[i]>a[i+1]:
            ind.append(i)
    #sorts by increasing order
    return sorted(ind)
find_peaks([2,0,1,0,2,0,1])
```

    [0, 2, 4, 6]

```python
p1 = find_peaks([2,0,1,0,2,0,1])
assert np.allclose(p1, np.array([0,2,4,6]))
p2 = find_peaks(np.array([0,1,2,3]))
assert np.allclose(p2, np.array([3]))
p3 = find_peaks([3,2,1,0])
assert np.allclose(p3, np.array([0]))
```

Here is a string with the first 10000 digits of $\pi$ (after the decimal). Write code to perform the following:

* Convert that string to a Numpy array of integers.
* Find the indices of the local maxima in the digits of $\pi$.
* Use `np.diff` to find the distances between consequtive local maxima.
* Visualize that distribution using an appropriately customized histogram.

```python
from sympy import pi, N
pi_digits_str = str(N(pi, 10001))[2:]
```

```python
# YOUR CODE HERE
#raise NotImplementedError()
def pimax(x):
    '''uses find_peaks to find the local maxima then finds the space between the maxima and
    plots the distribution of space between local maxima'''
    pi=np.ones(10000)
    for i in range(len(x)):
        pi[i]=int(x[i])
    m = find_peaks(pi)
    dist = np.diff(m)
    p = plt.hist(dist,bins=17)
    plt.title('Distances Between Local Maxima in First 10000 digtis of $\pi$')
    plt.xlabel('Distance Between Local Maxima')
    plt.ylabel('Number of Times Occured')
    plt.grid(False)
    plt.xlim([1,19])
    a=range(2,19)
    plt.xticks(a[::2])
    plt.ylim(0,1100)
    plt.show()
pimax(pi_digits_str)
```

```python
assert True # use this for grading the pi digits histogram
```


# Estadísticos principales

- Esperanzas, varianza y ley débil de los grandes números
- Variables aleatorias especiales

## Esperanza
La esperanza o  valor esperado de una v.a. $X$ se denota $E[X]$ y se calcula como:

$\begin{array}{ll}
E[X] =
\left\{\begin{array}{ll} \sum_i x_i P(X=x_i) & si\,X\, discreta\\
                          \int x f_X(x)dx & si\,X\, continua\\
\end{array} \right .\\
\end{array}$

Consideremos $g$ una función a valores reales, entonces:

$\begin{array}{lll}
E[g(X)] & = &
\left\{\begin{array}{ll} \sum_i g(x_i) P(X=x_i) & si\,X\, discreta\\
                          \int g(x) f_X(x)dx & si\,X\, continua\\
\end{array}\right .\\
\end{array}$

Para el caso especial de $g(x) = x^n$ se define el n-ésimo momento de X como:

$\begin{array}{lll}
E[X^n] & = &
\left\{\begin{array}{ll} \sum_i x_i^n P(X=x_i) & si\,X\, discreta\\
                          \int x^n f_X(x)dx & si\,X\, continua\\
\end{array}\right .\\
\end{array}$

La esperanza es el primer momento y se denota $\mu$.

**Propiedades**

Sean $a,b \in \cal{R}$ entonces:

$\begin{array}{lll}
E[aX+b] & = & aE[X] + b \\
E[X + Y] & = & E[X] + E[Y]\\
\end{array}$

## Varianza y covarianza
La varianza mide la variación de la v.a. entorno a la esperanza o media $\mu$, y se define como

$\begin{equation}
\begin{array}{ll}
Var(X) = E[(X-\mu)^2] = E[X^2] - \mu^2
\end{array}
\end{equation}$

Se cumple que:

$\begin{equation}
\begin{array}{ll}
Var(aX+b) = a^2 Var(X)
\end{array}
\end{equation}$

Se define además la desviación estándar $\sigma = \sqrt{Var(X)}$

La covarianza mide la relación (lineal) que hay entre dos v.a. $X$ e $Y$. Si denotamos $\mu_X = E[X]$ y $ \mu_Y= E[Y]$ entonces:

$\begin{equation}
\begin{array}{lll}
Cov(X,Y) & = & E[(X-\mu_X)(Y-\mu_y)]
\end{array}
\end{equation}$

La correlación es una medida normalizada:

$\begin{equation}
\begin{array}{lll}
Corr(X,Y) & = & \frac{Cov(X,Y)}{Var(X) Var(Y)}
\end{array}
\end{equation}$

**Propiedades**

$\begin{array}{lll}
Cov(X,Y) & = & Cov(Y,X) \\
Cov(X,X) & = & Var(X)\\
Cov(X+Z,Y) & = & Cov(X,Y) + Cov(Z,Y)\\
Cov(\sum_i \limits X_i,Y) & = & \sum_i \limits Cov(X_i,Y)\\
Var(X+Y) & = & Var(X) + Var(Y) + 2Cov(X,Y)\\
Var(\sum_i \limits X_i) & = &
\sum_i \limits Var(X_i) + \sum_i \limits \sum_{j\neq i} \limits Cov(X_i,X_j)
\end{array}$

## Otros estadísticos
$\begin{array}{lll}
\text{ Asimetría (skewness) } & = &  \frac{E[(X-\mu)^3]}{\sigma^3} = \frac{E[X^3]-3\mu\sigma^2 - \mu^3}{\sigma^3}\\
&&\\
\text{ Curtosis }& = &\frac{E[(X-\mu)^4]}{\sigma^4} = \frac{E[X^4] - 4\mu E[X^3] + 6\mu^2\sigma^2 + 3\mu^4}{\sigma^4}\\
\end{array}$

```R

```


```python
import sympy
import numpy
from math import cos, sin

import sympybotics
from sympybotics._compatibility_ import exec_

pi = sympy.pi
q = sympybotics.robotdef.q

a1, a2, d3, d4 = sympy.symbols('a1, a2, d3, d4')
```

```python
scara = sympybotics.robotdef.RobotDef(
    'SCARA - Spong',
    [( 0, a1,  0, q),
     (pi, a2,  0, q),
     ( 0,  0,  q, 0),
     ( 0,  0, d4, q)],
    dh_convention='standard')
```

```python
scara_geo = sympybotics.geometry.Geometry(scara)
scara_kin = sympybotics.kinematics.Kinematics(scara, scara_geo)

cos, sin = sympy.cos, sympy.sin
q1, q2, q3, q4 = sympy.flatten(scara.q)
```

```python
T_spong = sympy.Matrix([
    [(-sin(q1)*sin(q2) + cos(q1)*cos(q2))*cos(q4) + (sin(q1)*cos(q2) + sin(q2)*cos(q1))*sin(q4),
     -(-sin(q1)*sin(q2) + cos(q1)*cos(q2))*sin(q4) + (sin(q1)*cos(q2) + sin(q2)*cos(q1))*cos(q4),
     0, a1*cos(q1) - a2*sin(q1)*sin(q2) + a2*cos(q1)*cos(q2)],
    [(sin(q1)*sin(q2) - cos(q1)*cos(q2))*sin(q4) + (sin(q1)*cos(q2) + sin(q2)*cos(q1))*cos(q4),
     (sin(q1)*sin(q2) - cos(q1)*cos(q2))*cos(q4) - (sin(q1)*cos(q2) + sin(q2)*cos(q1))*sin(q4),
     0, a1*sin(q1) + a2*sin(q1)*cos(q2) + a2*sin(q2)*cos(q1)],
    [0, 0, -1, -d4 - q3],
    [0, 0, 0, 1]])

print(scara_geo.T[-1])

assert (scara_geo.T[-1] - T_spong).expand() == sympy.zeros(4)
```

    Matrix([[(-sin(q1)*sin(q2) + cos(q1)*cos(q2))*cos(q4) + (sin(q1)*cos(q2) + sin(q2)*cos(q1))*sin(q4), -(-sin(q1)*sin(q2) + cos(q1)*cos(q2))*sin(q4) + (sin(q1)*cos(q2) + sin(q2)*cos(q1))*cos(q4), 0, a1*cos(q1) - a2*sin(q1)*sin(q2) + a2*cos(q1)*cos(q2)], [(sin(q1)*sin(q2) - cos(q1)*cos(q2))*sin(q4) + (sin(q1)*cos(q2) + sin(q2)*cos(q1))*cos(q4), (sin(q1)*sin(q2) - cos(q1)*cos(q2))*cos(q4) - (sin(q1)*cos(q2) + sin(q2)*cos(q1))*sin(q4), 0, a1*sin(q1) + a2*sin(q1)*cos(q2) + a2*sin(q2)*cos(q1)], [0, 0, -1, -d4 - q3], [0, 0, 0, 1]])

```python
J_spong = sympy.Matrix(
    [[-a1*sin(q1) - a2*sin(q1)*cos(q2) - a2*sin(q2)*cos(q1), -a2*sin(q1)*cos(q2) - a2*sin(q2)*cos(q1), 0, 0],
     [a1*cos(q1) - a2*sin(q1)*sin(q2) + a2*cos(q1)*cos(q2), -a2*sin(q1)*sin(q2) + a2*cos(q1)*cos(q2), 0, 0],
     [0, 0, -1, 0],
     [0, 0, 0, 0],
     [0, 0, 0, 0],
     [1, 1, 0, -1]])

print(scara_kin.J[-1])

assert (scara_kin.J[-1] - J_spong).expand() == sympy.zeros(6, 4)
```

    Matrix([[-a1*sin(q1) - a2*sin(q1)*cos(q2) - a2*sin(q2)*cos(q1), -a2*sin(q1)*cos(q2) - a2*sin(q2)*cos(q1), 0, 0], [a1*cos(q1) - a2*sin(q1)*sin(q2) + a2*cos(q1)*cos(q2), -a2*sin(q1)*sin(q2) + a2*cos(q1)*cos(q2), 0, 0], [0, 0, -1, 0], [0, 0, 0, 0], [0, 0, 0, 0], [1, 1, 0, -1]])


# Lognormal distribution EF (Error Factor)
$$EF=\sqrt{\frac{p95}{p05}}=\frac{p95}{p50}=\frac{p50}{p5}$$

```python
# Загрузка нужных библиотек
import numpy as np
from scipy.stats import lognorm
from sympy import symbols, solve, exp, log
import matplotlib.pyplot as plt
```

```python
ef, mean, sigma, median, ci90 = symbols('ef mean sigma median ci90', Positive=True)
sigma = log(ef)/ci90
```

```python
solve(exp(log(median)+(sigma**2)/2)-mean,median)
```

```python
solve(solve(log(ef)/ci90-sigma,,sigma)
```

```python
# Настройка стиля рисунков
# Включение модного, черного стиля для рисунков
#plt.style.use('dark_background')
# Изменение стандартного шрифта на Linux-своместимый
#plt.rcParams["font.family"] = 'Liberation Mono'
# Изменяем стандартный размер шрифта
plt.rcParams.update({'font.size': 12})
# Изменяем стандартный размер картинки
plt.rcParams['figure.figsize'] = [16, 8]
```

```python
m = 3e-5
x = np.linspace(0,1)
qq = np.linspace(0.01,0.99)
x2 = np.linspace(0, 3*m)
fig, axs = plt.subplots(nrows=1, ncols=2)
for ef in range(2,6):
    ci90 = 1.64485
    sigma = np.log(ef)/ci90
    med = m*np.exp(-(sigma**2)/2)
    axs[0].plot(x, lognorm.ppf(qq, s=sigma, scale= med), '--')
    axs[0].ticklabel_format(axis="both", style="sci", scilimits=(0,0))
    axs[1].plot(x2, lognorm.pdf(x2, s=sigma, scale= med), '--')
    mean = lognorm.mean(s=sigma, scale= med)
    median = lognorm.median(s=sigma, scale= med)
    axs[1].plot(mean, lognorm.pdf(mean,s=sigma, scale= med), 'o', c='grey')
    axs[1].plot(median, lognorm.pdf(median,s=sigma, scale= med), '^', c='grey')
    print(mean)
    axs[1].ticklabel_format(axis="both", style="sci", scilimits=(0,0))
axs[0].axhline(m, ls=':', color='grey')
axs[0].text(0, m*1.01, f'mean={m}')
axs[1].axvline(m, ls= ':', color='grey')
axs[1].text(m*1.01, 0, f'mean={m}')
axs[0].legend(range(2,6))
axs[0].grid(linestyle=':')
axs[1].grid(linestyle=':')
plt.show()
```