```R
install.packages("factoextra", dependencies = TRUE)

```

    also installing the dependencies ‘pixmap’, ‘ade4’, ‘ca’, ‘igraph’, ‘mclust’

    Updating HTML index of packages in '.Library'
    Making 'packages.html' ... done

```R
library(tidyverse)
library(quantmod)
library(factoextra)
```

    Welcome! Want to learn more? See two factoextra-related books at https://goo.gl/ve3WBa

```R
library(repr)
options(repr.plot.width=5, repr.plot.height=3)
```

K-means is based on unsupervised learning – you give the data and k-means decides how to classify it. The idea is to split data into clusters based on cluster center and assign each point to nearest center.  There is drawback with such approach – the algorithm tries to establish the centers of  clusters with initial data set. If the data is very noisy and the centers are not stable, then every try will give you different results.

\begin{equation}
W(C_k) = \sum\limits_{x_i \in C_k} (x_i - \mu_k)^2
\end{equation}

- xi design a data point belonging to the cluster Ck
- μk is the mean value of the points assigned to the cluster Ck

Each observation (xi) is assigned to a given cluster such that the sum of squares (SS) distance of the observation to their assigned cluster centers μk is a minimum.

We define the total within-cluster variation as follow:

\begin{equation}
tot.withinss = \sum\limits_{k=1}^k W(C_k) = \sum\limits_{k=1}^k \sum\limits_{x_i \in C_k} (x_i - \mu_k)^2
\end{equation}

Kmeans
 - Specify the number of clusters (K) to be created (by the analyst)
 - Select randomly k objects from the dataset as the initial cluster centers or means
 - Assigns each observation to their closest centroid, based on the Euclidean distance between the object and the centroid
 - For each of the k clusters update the cluster centroid by calculating the new mean values of all the data points in the cluster. The centoid of a Kth cluster is a vector of length p containing the means of all variables for the observations in the kth cluster; p is the number of variables.
 - Iteratively minimize the total within sum of square. That is, iterate steps 3 and 4 until the cluster assignments stop changing or the maximum number of iterations is reached. By default, the R software uses 10 as the default value for the maximum number of iterations.

```R
data("iris")
df <- iris
head(df, 3)
```

Sepal.Length	Sepal.Width	Petal.Length	Petal.Width	Species

5.1   	3.5   	1.4   	0.2   	setosa

4.9   	3.0   	1.4   	0.2   	setosa

4.7   	3.2   	1.3   	0.2   	setosa

```R
# Compute k-means with k = 3
set.seed(123)
res.km <- kmeans(scale(df[, -5]), 3, nstart = 25)
# K-means clusters showing the group of each individuals
res.km$cluster
```

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

3

3

3

2

2

2

3

2

2

2

2

2

2

2

2

3

2

2

2

2

3

2

2

2

2

3

3

3

2

2

2

2

2

2

2

3

3

2

2

2

2

2

2

2

2

2

2

2

2

2

3

2

3

3

3

3

2

3

3

3

3

3

3

2

2

3

3

3

3

2

3

2

3

2

3

3

2

3

3

3

3

3

3

2

2

3

3

3

2

3

3

3

2

3

3

3

2

3

3

2

```R
install.packages("remotes")
```

    Updating HTML index of packages in '.Library'
    Making 'packages.html' ... done

```R
packageurl <- "https://cran.r-project.org/src/contrib/broom_0.7.3.tar.gz"
install.packages(packageurl, repos=NULL, type="source")
```

fviz_cluster(res.km, data = df[, -5],
             palette = c("#2E9FDF", "#00AFBB", "#E7B800"),
             geom = "point",
             ellipse.type = "convex",
             ggtheme = theme_bw()
             )

```R
data("USArrests")      # Loading the data set
df <- scale(USArrests) # Scaling the data

# View the firt 3 rows of the data
head(df, n = 3)
```

	Murder	Assault	UrbanPop	Rape

Alabama	1.24256408  	0.7828393   	-0.5209066  	-0.003416473

Alaska	0.50786248  	1.1068225   	-1.2117642  	 2.484202941

Arizona	0.07163341  	1.4788032   	 0.9989801  	 1.042878388

```R
km.res <- kmeans(df, 4, nstart = 25)
```

```R
print(km.res)
```

    K-means clustering with 4 clusters of sizes 8, 16, 13, 13

    Cluster means:
          Murder    Assault   UrbanPop        Rape
    1  1.4118898  0.8743346 -0.8145211  0.01927104
    2 -0.4894375 -0.3826001  0.5758298 -0.26165379
    3  0.6950701  1.0394414  0.7226370  1.27693964
    4 -0.9615407 -1.1066010 -0.9301069 -0.96676331

    Clustering vector:
           Alabama         Alaska        Arizona       Arkansas     California
                 1              3              3              1              3
          Colorado    Connecticut       Delaware        Florida        Georgia
                 3              2              2              3              1
            Hawaii          Idaho       Illinois        Indiana           Iowa
                 2              4              3              2              4
            Kansas       Kentucky      Louisiana          Maine       Maryland
                 2              4              1              4              3
     Massachusetts       Michigan      Minnesota    Mississippi       Missouri
                 2              3              4              1              3
           Montana       Nebraska         Nevada  New Hampshire     New Jersey
                 4              4              3              4              2
        New Mexico       New York North Carolina   North Dakota           Ohio
                 3              3              1              4              2
          Oklahoma         Oregon   Pennsylvania   Rhode Island South Carolina
                 2              2              2              2              1
      South Dakota      Tennessee          Texas           Utah        Vermont
                 4              1              3              2              4
          Virginia     Washington  West Virginia      Wisconsin        Wyoming
                 2              2              4              4              2

    Within cluster sum of squares by cluster:
    [1]  8.316061 16.212213 19.922437 11.952463
     (between_SS / total_SS =  71.2 %)

    Available components:

    [1] "cluster"      "centers"      "totss"        "withinss"     "tot.withinss"
    [6] "betweenss"    "size"         "iter"         "ifault"

```R
dd <- cbind(USArrests, cluster = km.res$cluster)
head(dd)
```

	Murder	Assault	UrbanPop	Rape	cluster

Alabama	13.2	236 	58  	21.2	1

Alaska	10.0	263 	48  	44.5	3

Arizona	 8.1	294 	80  	31.0	3

Arkansas	 8.8	190 	50  	19.5	1

California	 9.0	276 	91  	40.6	3

Colorado	 7.9	204 	78  	38.7	3

```R
km.res$centers
```

Murder	Assault	UrbanPop	Rape

 1.4118898 	 0.8743346 	-0.8145211 	 0.01927104

-0.4894375 	-0.3826001 	 0.5758298 	-0.26165379

 0.6950701 	 1.0394414 	 0.7226370 	 1.27693964

-0.9615407 	-1.1066010 	-0.9301069 	-0.96676331

```R
install.packages("broom")
```

    Updating HTML index of packages in '.Library'
    Making 'packages.html' ... done

# Show points only
fviz_cluster(km.res, iris[, -5], geom = "point")
# Show text only
fviz_cluster(km.res, iris[, -5], geom = "text")

```R

```


# Lab: Batchnormalization Layer

## What is a batchnormalization layer?
It is a layer that normalize the output before the activation layer. [The original paper](https://arxiv.org/abs/1502.03167) was proposed by Sergey Ioffe in 2015.

Batch Normalization Layer looks like this:

## Why batchnormalization?
The distribution of each layer's input changes because the weights of the previous layer change as we update weights by the gradient descent. This is called a covariance shift, which makes the network training difficult.

For example, if the activation layer is a relu layer and the input of the activation layer is shifted to less than zeros, no weights will be activated!

One thing also worth mentioning is that $\gamma$ and $\beta$ parameters in $$ y = \gamma \hat{x} + \beta $$ are also trainable.

**What it means is that if we don't need the batchnormalization, its parameters will be updated such that it offsets the normalization step.**

For example, assume that

\begin{align}
\gamma &= \sqrt{\sigma^2_B + \epsilon}\\
\beta &= \mu_B
\end{align}

then

$$ y_i = \gamma \hat{x_i} + \beta = x_i $$

Also note that $\mu$ and $\sigma$ are computed using moving averages during the training step. However, during the test time, the computed $\mu$ and $\sigma$ will be used as fixed

## Conclusion
- Always use the batch normalization!

## Enough Talk: how to implement in Tensorflow

### 1. Load Library
- We use the famous MNIST data

```python
import tensorflow as tf
import matplotlib.pyplot as plt
from tensorflow.examples.tutorials.mnist import input_data
%matplotlib inline

mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)
```

    Extracting MNIST_data/train-images-idx3-ubyte.gz
    Extracting MNIST_data/train-labels-idx1-ubyte.gz
    Extracting MNIST_data/t10k-images-idx3-ubyte.gz
    Extracting MNIST_data/t10k-labels-idx1-ubyte.gz

```python
mnist.train.images.shape
```

    (55000, 784)

### 2. Define Model & Solver Class
- Object-Oriented-Programming allows to define multiple model easily
- Why do we separate model and solver classes?
  - We can just swap out the model class in the Solver class when we need a different network architecture
  - Usually we need one solver class

```python
class Model:
    """Network Model Class

    Note that this class has only the constructor.
    The actual model is defined inside the constructor.

    Attributes
    ----------
    X : tf.float32
        This is a tensorflow placeholder for MNIST images
        Expected shape is [None, 784]

    y : tf.float32
        This is a tensorflow placeholder for MNIST labels (one hot encoded)
        Expected shape is [None, 10]

    mode : tf.bool
        This is used for the batch normalization
        It's `True` at training time and `False` at test time

    loss : tf.float32
        The loss function is a softmax cross entropy

    train_op
        This is simply the training op that minimizes the loss

    accuracy : tf.float32
        The accuracy operation

    Examples
    ----------
    >>> model = Model("Batch Norm", 32, 10)

    """
    def __init__(self, name, input_dim, output_dim, hidden_dims=[32, 32], use_batchnorm=True, activation_fn=tf.nn.relu, optimizer=tf.train.AdamOptimizer, lr=0.01):
        """ Constructor

        Parameters
        --------
        name : str
            The name of this network
            The entire network will be created under `tf.variable_scope(name)`

        input_dim : int
            The input dimension
            In this example, 784

        output_dim : int
            The number of output labels
            There are 10 labels

        hidden_dims : list (default: [32, 32])
            len(hidden_dims) = number of layers
            each element is the number of hidden units

        use_batchnorm : bool (default: True)
            If true, it will create the batchnormalization layer

        activation_fn : TF functions (default: tf.nn.relu)
            Activation Function

        optimizer : TF optimizer (default: tf.train.AdamOptimizer)
            Optimizer Function

        lr : float (default: 0.01)
            Learning rate

        """
        with tf.variable_scope(name):
            # Placeholders are defined
            self.X = tf.placeholder(tf.float32, [None, input_dim], name='X')
            self.y = tf.placeholder(tf.float32, [None, output_dim], name='y')
            self.mode = tf.placeholder(tf.bool, name='train_mode')

            # Loop over hidden layers
            net = self.X
            for i, h_dim in enumerate(hidden_dims):
                with tf.variable_scope('layer{}'.format(i)):
                    net = tf.layers.dense(net, h_dim)

                    if use_batchnorm:
                        net = tf.layers.batch_normalization(net, training=self.mode)

                    net = activation_fn(net)

            # Attach fully connected layers
            net = tf.contrib.layers.flatten(net)
            net = tf.layers.dense(net, output_dim)

            self.loss = tf.nn.softmax_cross_entropy_with_logits(logits=net, labels=self.y)
            self.loss = tf.reduce_mean(self.loss, name='loss')

            # When using the batchnormalization layers,
            # it is necessary to manually add the update operations
            # because the moving averages are not included in the graph
            update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS, scope=name)
            with tf.control_dependencies(update_ops):
                self.train_op = optimizer(lr).minimize(self.loss)

            # Accuracy etc
            softmax = tf.nn.softmax(net, name='softmax')
            self.accuracy = tf.equal(tf.argmax(softmax, 1), tf.argmax(self.y, 1))
            self.accuracy = tf.reduce_mean(tf.cast(self.accuracy, tf.float32))
```

```python
class Solver:
    """Solver class

    This class will contain the model class and session

    Attributes
    ----------
    model : Model class
    sess : TF session

    Methods
    ----------
    train(X, y)
        Run the train_op and Returns the loss

    evalulate(X, y, batch_size=None)
        Returns "Loss" and "Accuracy"
        If batch_size is given, it's computed using batch_size
        because most GPU memories cannot handle the entire training data at once

    Example
    ----------
    >>> sess = tf.InteractiveSession()
    >>> model = Model("BatchNorm", 32, 10)
    >>> solver = Solver(sess, model)

    # Train
    >>> solver.train(X, y)

    # Evaluate
    >>> solver.evaluate(X, y)
    """
    def __init__(self, sess, model):
        self.model = model
        self.sess = sess

    def train(self, X, y):
        feed = {
            self.model.X: X,
            self.model.y: y,
            self.model.mode: True
        }
        train_op = self.model.train_op
        loss = self.model.loss

        return self.sess.run([train_op, loss], feed_dict=feed)

    def evaluate(self, X, y, batch_size=None):
        if batch_size:
            N = X.shape[0]

            total_loss = 0
            total_acc = 0

            for i in range(0, N, batch_size):
                X_batch = X[i:i + batch_size]
                y_batch = y[i:i + batch_size]

                feed = {
                    self.model.X: X_batch,
                    self.model.y: y_batch,
                    self.model.mode: False
                }

                loss = self.model.loss
                accuracy = self.model.accuracy

                step_loss, step_acc = self.sess.run([loss, accuracy], feed_dict=feed)

                total_loss += step_loss * X_batch.shape[0]
                total_acc += step_acc * X_batch.shape[0]

            total_loss /= N
            total_acc /= N

            return total_loss, total_acc

        else:
            feed = {
                self.model.X: X,
                self.model.y: y,
                self.model.mode: False
            }

            loss = self.model.loss
            accuracy = self.model.accuracy

            return self.sess.run([loss, accuracy], feed_dict=feed)
```

### 3. Instantiate Model/Solver classes

```python
input_dim = 784
output_dim = 10
N = 55000

tf.reset_default_graph()
sess = tf.InteractiveSession()

# We create two models: one with the batch norm and other without
bn = Model('batchnorm', input_dim, output_dim, use_batchnorm=True)
nn = Model('no_norm', input_dim, output_dim, use_batchnorm=False)

# We create two solvers: to train both models at the same time for comparison
# Usually we only need one solver class
bn_solver = Solver(sess, bn)
nn_solver = Solver(sess, nn)
```

```python
epoch_n = 10
batch_size = 32

# Save Losses and Accuracies every epoch
# We are going to plot them later
train_losses = []
train_accs = []

valid_losses = []
valid_accs = []
```

### 4. Run the train step

```python
init = tf.global_variables_initializer()
sess.run(init)

for epoch in range(epoch_n):
    for _ in range(N//batch_size):
        X_batch, y_batch = mnist.train.next_batch(batch_size)

        _, bn_loss = bn_solver.train(X_batch, y_batch)
        _, nn_loss = nn_solver.train(X_batch, y_batch)

    b_loss, b_acc = bn_solver.evaluate(mnist.train.images, mnist.train.labels, batch_size)
    n_loss, n_acc = nn_solver.evaluate(mnist.train.images, mnist.train.labels, batch_size)

    # Save train losses/acc
    train_losses.append([b_loss, n_loss])
    train_accs.append([b_acc, n_acc])
    print(f'[Epoch {epoch}-TRAIN] Batchnorm Loss(Acc): {b_loss:.5f}({b_acc:.2%}) vs No Batchnorm Loss(Acc): {n_loss:.5f}({n_acc:.2%})')

    b_loss, b_acc = bn_solver.evaluate(mnist.validation.images, mnist.validation.labels)
    n_loss, n_acc = nn_solver.evaluate(mnist.validation.images, mnist.validation.labels)

    # Save valid losses/acc
    valid_losses.append([b_loss, n_loss])
    valid_accs.append([b_acc, n_acc])
    print(f'[Epoch {epoch}-VALID] Batchnorm Loss(Acc): {b_loss:.5f}({b_acc:.2%}) vs No Batchnorm Loss(Acc): {n_loss:.5f}({n_acc:.2%})')
    print()
```

    [Epoch 0-TRAIN] Batchnorm Loss(Acc): 0.18456(94.19%) vs No Batchnorm Loss(Acc): 0.31917(91.01%)
    [Epoch 0-VALID] Batchnorm Loss(Acc): 0.19054(94.10%) vs No Batchnorm Loss(Acc): 0.31920(91.00%)

    [Epoch 1-TRAIN] Batchnorm Loss(Acc): 0.10349(96.78%) vs No Batchnorm Loss(Acc): 0.16142(95.34%)
    [Epoch 1-VALID] Batchnorm Loss(Acc): 0.11720(96.48%) vs No Batchnorm Loss(Acc): 0.18348(94.96%)

    [Epoch 2-TRAIN] Batchnorm Loss(Acc): 0.11239(96.43%) vs No Batchnorm Loss(Acc): 0.17737(94.79%)
    [Epoch 2-VALID] Batchnorm Loss(Acc): 0.12829(96.30%) vs No Batchnorm Loss(Acc): 0.20401(94.34%)

    [Epoch 3-TRAIN] Batchnorm Loss(Acc): 0.07526(97.69%) vs No Batchnorm Loss(Acc): 0.15240(95.65%)
    [Epoch 3-VALID] Batchnorm Loss(Acc): 0.09549(97.12%) vs No Batchnorm Loss(Acc): 0.20025(95.16%)

    [Epoch 4-TRAIN] Batchnorm Loss(Acc): 0.07339(97.68%) vs No Batchnorm Loss(Acc): 0.15641(95.53%)
    [Epoch 4-VALID] Batchnorm Loss(Acc): 0.10588(96.96%) vs No Batchnorm Loss(Acc): 0.19816(94.86%)

    [Epoch 5-TRAIN] Batchnorm Loss(Acc): 0.08164(97.38%) vs No Batchnorm Loss(Acc): 0.15969(95.67%)
    [Epoch 5-VALID] Batchnorm Loss(Acc): 0.11476(96.52%) vs No Batchnorm Loss(Acc): 0.22123(95.10%)

    [Epoch 6-TRAIN] Batchnorm Loss(Acc): 0.05879(98.10%) vs No Batchnorm Loss(Acc): 0.18191(94.92%)
    [Epoch 6-VALID] Batchnorm Loss(Acc): 0.09402(97.30%) vs No Batchnorm Loss(Acc): 0.25907(94.50%)

    [Epoch 7-TRAIN] Batchnorm Loss(Acc): 0.05014(98.38%) vs No Batchnorm Loss(Acc): 0.23831(93.59%)
    [Epoch 7-VALID] Batchnorm Loss(Acc): 0.08446(97.58%) vs No Batchnorm Loss(Acc): 0.28310(93.46%)

    [Epoch 8-TRAIN] Batchnorm Loss(Acc): 0.04956(98.41%) vs No Batchnorm Loss(Acc): 0.12616(96.48%)
    [Epoch 8-VALID] Batchnorm Loss(Acc): 0.08479(97.48%) vs No Batchnorm Loss(Acc): 0.18636(95.44%)

    [Epoch 9-TRAIN] Batchnorm Loss(Acc): 0.04351(98.61%) vs No Batchnorm Loss(Acc): 0.12277(96.54%)
    [Epoch 9-VALID] Batchnorm Loss(Acc): 0.08275(97.66%) vs No Batchnorm Loss(Acc): 0.19641(95.74%)

### 5. Performance Comparison
* With the batchnormalization, the loss is lower and it's more accurate too!

```python
bn_solver.evaluate(mnist.test.images, mnist.test.labels)
```

    [0.089340471, 0.97370011]

```python
nn_solver.evaluate(mnist.test.images, mnist.test.labels)
```

    [0.20733583, 0.95130014]

```python
def plot_compare(loss_list: list, ylim=None, title=None) -> None:

    bn = [i[0] for i in loss_list]
    nn = [i[1] for i in loss_list]

    plt.figure(figsize=(15, 10))
    plt.plot(bn, label='With BN')
    plt.plot(nn, label='Without BN')
    if ylim:
        plt.ylim(ylim)

    if title:
        plt.title(title)
    plt.legend()
    plt.grid('on')
    plt.show()
```

```python
plot_compare(train_losses, title='Training Loss at Epoch')
```

```python
plot_compare(train_accs, [0, 1.0], title="Training Acc at Epoch")
```

```python
plot_compare(valid_losses, title='Validation Loss at Epoch')
```

```python
plot_compare(valid_accs, [0, 1.], title='Validation Acc at Epoch')
```


 # Steady State Cornering Example
 \begin{equation}\label{eq:DynamicBicycleModel}
        \begin{aligned}
            \dot v_x &=  \frac{1}{m}\big(F_{x}^r \text{cos}(\delta^r)-F_{y}^r \text{sin}(\delta^r)-F_{y}^f \text{sin}(\delta^f)\big) + \omega_z v_y \\
            \dot v_y &= \frac{1}{m}\big (F_{y}^f\text{cos}(\delta^f) + F_{y}^r \text{cos}(\delta^r)+F_{x}^r \text{sin}(\delta^r) \big) - \omega_z v_x \\
            \dot \omega_z &= \frac{1}{I_z} \big(l_f F_{y}^f \text{cos}(\delta^f) - l_r F_{y}^r \text{cos}(\delta^r) - l_r  F_{x}^r \text{sin}(\delta^r) \big) \\
            \dot X &= v_x \text{cos}(\psi) - v_y \text{sin}(\psi)  \\
        \dot Y &= v_x \text{sin}(\psi) + v_y \text{cos}(\psi)  \\
        \dot \psi &= \omega_z  \\
        \end{aligned}
    \end{equation}

```python
import numpy as np
#from scipy.integrate import odeint
import matplotlib.pyplot as plt
import matplotlib.patches as patches
from matplotlib import animation
from IPython.display import display
from ipywidgets import interactive, fixed

# This function simulates the one-step  vehicle dynamics
def _DynModel(x, u, deltaT,constantvx):

    x_next= np.zeros(6)
    # Vehicle Parameters BARC
    m  = 1.98
    lf = 0.125
    lr = 0.125
    Iz = 0.024
    Df = 0.8 * m * 9.81 / 2.0
    Cf = 1.25
    Bf = 1.0
    Dr = 0.8 * m * 9.81 / 2.0
    Cr = 1.25
    Br = 1.0

    # # Vehicle Parameters {Passsenger car}
    m=2300 # in Kg
    Iz=4400
    lf=1.5 #length from CoG to front axle in meters
    lr=1.4 #length from CoG to back axle in meters
    lf=1.5 #length from CoG to front axle in meters
    lr=1.4 #length from CoG to back axle in meters
    #TIRE MODEL
    Fz = (1/2)*(m*9.81)/1000; #force in K-newtons
    a1 = -22.1
    a2 = 1011
    a3 = 1078
    a4 = 1.82
    a5 = 0.208
    a6 = 0.000
    a7 = -0.354
    a8 = 0.707;
    Df = a1*(Fz*Fz) + a2*Fz
    BCD = a3*np.sin(a4*np.arctan(a5*Fz))
    Cf = 1.30;
    Bf = BCD/(Cf*Df)
    #Ef = a6*(Fz^2) + a7*Fz + a8;
    Dr=Df
    Cr=Cf
    Br=Bf

    # Extract the value of the inputs and states
    delta = u[0]
    a     = u[1]
    if constantvx > 0:
        vx = constantvx
    else:
        vx    = x[0]
    vy    = x[1]
    wz    = x[2]
    psi   = x[3]
    X     = x[4]
    Y     = x[5]

    # Compute tire slip angle
    alpha_f = delta - np.arctan2( vy + lf * wz, vx )
    alpha_r = - np.arctan2( vy - lf * wz , vx)

    # Compute lateral force at front and rear tire
    Fyf = 2 * Df * np.sin( Cf * np.arctan(Bf * alpha_f ) )
    Fyr = 2 * Dr * np.sin( Cr * np.arctan(Br * alpha_r ) )

    # Propagate the dynamics
    if constantvx>0:
        x_next[0] = constantvx
    else:
        x_next[0] = vx  + deltaT * (a - 1 / m * Fyf * np.sin(delta) + wz*vy)
    x_next[1] = vy  + deltaT * (1 / m * (Fyf * np.cos(delta) + Fyr) - wz * vx)
    x_next[2] = wz  + deltaT * (1 / Iz *(lf * Fyf * np.cos(delta) - lr * Fyr) )
    x_next[3] = psi + deltaT * (wz)
    x_next[4] =   X + deltaT * ((vx * np.cos(psi) - vy * np.sin(psi)))
    x_next[5] =   Y + deltaT * (vx * np.sin(psi)  + vy * np.cos(psi))

    # Addd Noises
    #noise_vx = np.maximum(-0.05, np.minimum(np.random.randn() * 0.01, 0.05))
    #noise_vy = np.maximum(-0.1, np.minimum(np.random.randn() * 0.01, 0.1))
    #noise_wz = np.maximum(-0.05, np.minimum(np.random.randn() * 0.005, 0.05))
    #x_next[0] = x_next[0] + 0.1*noise_vx
    #x_next[1] = x_next[1] + 0.1*noise_vy
    #x_next[2] = x_next[2] + 0.1*noise_wz

    return x_next,alpha_f,alpha_r
```

```python
def sim(constantsteering,constantvx):
    # Sampling time
    deltaT = 0.01
    # Simulation steps
    Tsim=100
    simSteps=int(Tsim/deltaT)

    # initialization
    x= np.zeros((simSteps,6))
    u= np.zeros((simSteps,2))
    alpha_f= np.zeros(simSteps)
    alpha_r= np.zeros(simSteps)
    x[0, :]=[0,0,0,0,0,0]

    # simulate system
    for t in range(0,simSteps-1):
        u[t, :] = [constantsteering * np.pi / 180, 0]
        x[t + 1, :], alpha_f[t], alpha_r[t] = _DynModel(x[t, :], u[t, :],deltaT,constantvx)

    # plot results
    plt.figure(1)
    plt.plot(x[:,4],x[:,5],'-',linewidth=2,label='CarTrajectory'+'_try:'+str(constantvx))
    plt.xlabel('X')
    plt.ylabel('Y')
    plt.legend()
    plt.figure(2)
    plt.plot(alpha_f,'-',linewidth=2,label='alpha_f'+'_try:'+str(constantvx))
    plt.plot(alpha_r,'-',linewidth=2,label='alpha_r'+'_try:'+str(constantvx))
    plt.xlabel('time')
    plt.ylabel('alpha')
    plt.legend()
    #plt.show()
    return x,u,simSteps

w=interactive(sim,constantsteering=(-20.0,20.0),constantvx=(0.0,3.0))
display(w)
```

    interactive(children=(FloatSlider(value=0.0, description='constantsteering', max=20.0, min=-20.0), FloatSlider…

```python
x,u,simSteps=sim(7,1)
from IPython.display import HTML
def animation_xy(x,u,simSteps):
    # for infor read: https: // jakevdp.github.io / blog / 2013 / 05 / 28 / a - simple - animation - the - magic - triangle /
    P1 = np.array([[1, 1], [1,-1], [-1, -1], [-1, 1]])
    fig=plt.figure()
    ax = plt.axes(xlim=(-30, 30), ylim=(-10, 50))
    plt.plot(x[:,4],x[:,5],'--',linewidth=0.5)
    kwds = dict(ec='k', alpha=0.7, closed=True,zorder=10)
    rec = [ax.add_patch(plt.Polygon(0 * P1, fc='g', **kwds)),
                          ax.add_patch(plt.Polygon(0 * P1, fc='b', **kwds))]

    l = 3; w = 1.5

    def animate(t):
        t=t*20 #speedup animation
        if t<simSteps:
            lw=0.4
            ww=0.1
            psi   = x[t,3]
            X     = x[t,4]
            Y     = x[t,5]
            delta = u[t,0]
            car_x = [ X + l * np.cos(psi) - w * np.sin(psi), X + l*np.cos(psi) + w * np.sin(psi),
                      X - l * np.cos(psi) + w * np.sin(psi), X - l * np.cos(psi) - w * np.sin(psi)]
            car_y = [ Y + l * np.sin(psi) + w * np.cos(psi), Y + l * np.sin(psi) - w * np.cos(psi),
                      Y - l * np.sin(psi) - w * np.cos(psi), Y - l * np.sin(psi) + w * np.cos(psi)]
            lfwheel_x = [ car_x[0] + lw * np.cos(psi+delta) - ww * np.sin(psi+delta), car_x[0]+ lw*np.cos(psi+delta) + ww * np.sin(psi+delta),
                      car_x[0] - lw * np.cos(psi+delta) + ww * np.sin(psi+delta), car_x[0] - lw * np.cos(psi+delta) - ww * np.sin(psi+delta)]
            lfwheel_y = [car_y[0] + lw * np.sin(psi+delta) + ww * np.cos(psi+delta), car_y[0] + lw * np.sin(psi+delta) - ww * np.cos(psi+delta),
                     car_y[0] - lw * np.sin(psi+delta) - ww * np.cos(psi+delta), car_y[0] - lw * np.sin(psi+delta) + ww * np.cos(psi+delta)]

            rec[0].set_xy(np.array([car_x, car_y]).T)
            rec[1].set_xy(np.array([lfwheel_x, lfwheel_y ]).T)
        return rec
    anim = animation.FuncAnimation(fig, animate, frames=simSteps, interval=1)
    # save the animation as an mp4.  This requires ffmpeg or mencoder to be
    # installed.  The extra_args ensure that the x264 codec is used, so that
    # the video can be embedded in html5.  You may need to adjust this for
    # your system: for more information, see
    # http://matplotlib.sourceforge.net/api/animation_api.html
    #anim.save('basic_animation.mp4', fps=30, extra_args=['-vcodec', 'libx264'])
    HTML(anim.to_html5_video())
    plt.show()

animation_xy(x,u,simSteps)
```

```python

```

```python

```


```python
from sympy import init_session
init_session()
```

    IPython console for SymPy 1.5.1 (Python 3.6.9-64-bit) (ground types: gmpy)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at https://docs.sympy.org/1.5.1/

```python
#init_printing?
```

```python
# this is for rac[4,2]
a, b, o, d, g, k, l0 = symbols('a b o d g k l0', real=True, positive=True)
```

```python
# Roman's new factorization:
f1 = k**2 + 2*a**2*k + a**4+b**2
f2 = 1 + g**2*k + d**2*k**2
A = a**4+b**2
B = 2*a**2 + g**2*A

f = l0 * f1 * f2 / (A + B*k + o**2*k**2)
f
```

```python
diff(f,l0)
```

```python
da=diff(f,a)
da.factor()
```

```python
db=diff(f,b)
db.factor()
```

```python
dg=diff(f,g)
dg.factor()
```

```python
dd=diff(f,d)
dd.factor()
```

```python
do=diff(f,o)
do.factor()
```

```python

```

```python
solveset(f1, k)
```

```python
expand((-a**2 + 1j*b)**2)
```

```python
#
#  in f2:  d2 = 1/(a**4+b**2)  and g2=2*a**2/(a**4+b**2)
#
f3 = k**2/(a**4+b**2) + k*2*a**2/(a**4+b**2) + 1
solveset(f3, k)
```

```python

```


Sistema: dS/dt=-bS, dI/dt=bI, b es positivo (b=beta)

```python
from sympy import *
from sympy.abc import S,I,t,b
```

```python
#puntos criticos
P=-b*S
Q=b*I
#establecer P(S,I)=0 y Q(S,I)=0
Peqn=Eq(P,0)
Qeqn=Eq(Q,0)
print(solve((Peqn,Qeqn),S,I))
#Eigenvalores y eigenvectores
M=Matrix([[-b,0],[0,b]])
print(M.eigenvals())
pprint(M.eigenvects())
```

    {I: 0, S: 0}
    {-b: 1, b: 1}
    ⎡⎛       ⎡⎡1⎤⎤⎞  ⎛      ⎡⎡0⎤⎤⎞⎤
    ⎢⎜-b, 1, ⎢⎢ ⎥⎥⎟, ⎜b, 1, ⎢⎢ ⎥⎥⎟⎥
    ⎣⎝       ⎣⎣0⎦⎦⎠  ⎝      ⎣⎣1⎦⎦⎠⎦

El sistema tiene un punto critico en el origen el cual tiene eigenvalores -b y b, ya que b es postivo los eigenvalores son reales distintos uno positivo y otro negativo por lo que el punto critico es un punto silla, el cual es inestable.

Ya que el sistema tiene un punto critico que es un punto silla, por la definicion 9 el indice del punto critico es -1 y por el teorema 1 entonces no hay ciclos limites en el sistema, es decir no existe una solucion periodica aislada del sistema.

```python
import matplotlib.pyplot as plt
import numpy as np
from scipy.integrate import odeint
import pylab as pl
import matplotlib
```

```python
def dx_dt(x,t):
    return [ -0.5*x[0] , 0.5*x[1] ]
#trayectorias en tiempo hacia adelante
ts=np.linspace(0,10,500)
ic=np.linspace(-1,1,6)
for r in ic:
    for s in ic:
        x0=[r,s]
        xs=odeint(dx_dt,x0,ts)
        plt.plot(xs[:,0],xs[:,1],"r-")
#trayectorias en tiempo hacia atras
ts=np.linspace(0,-10,500)
ic=np.linspace(-1,1,6)
for r in ic:
    for s in ic:
        x0=[r,s]
        xs=odeint(dx_dt,x0,ts)
        plt.plot(xs[:,0],xs[:,1],"r-")
#etiquetas de ejes y estilo de letra
plt.xlabel('S',fontsize=10)
plt.ylabel('I',fontsize=10)
plt.tick_params(labelsize=10)
plt.xlim(-1,1)
plt.ylim(-1,1)
#campo vectorial
X,Y=np.mgrid[-1:1:20j,-1:1:20j]
u=-0.5*X
v=0.5*Y
pl.quiver(X,Y,u,v,color='b')
plt.show()
```

Analisis de existencia de Bifurcaciones

El punto critico del sistema no varia con el cambio en el valor de b (beta) ya que es idependiente de este parametro.

Comportamiento del sistema cuando beta<0

```python
#sistema dS/dt=-b*S, dI/dt=b*I con beta<0
def dx_dt(x,t):
    return [ 1*x[0] , -1*x[1] ]
#trayectorias en tiempo hacia adelante
ts=np.linspace(0,10,500)
ic=np.linspace(20000,100000,3)
for r in ic:
    for s in ic:
        x0=[r,s]
        xs=odeint(dx_dt,x0,ts)
        plt.plot(xs[:,0],xs[:,1],"-", color="orangered", lw=1.5)
#trayectorias en tiempo hacia atras
ts=np.linspace(0,-10,500)
ic=np.linspace(20000,100000,3)
for r in ic:
    for s in ic:
        x0=[r,s]
        xs=odeint(dx_dt,x0,ts)
        plt.plot(xs[:,0],xs[:,1],"-", color="orangered", lw=1.5)
#etiquetas de ejes y estilo de letra
plt.xlabel('S',fontsize=20)
plt.ylabel('I',fontsize=20)
plt.tick_params(labelsize=12)
plt.ticklabel_format(style="sci", scilimits=(0,0))
plt.xlim(0,100000)
plt.ylim(0,100000)
#campo vectorial
X,Y=np.mgrid[0:100000:15j,0:100000:15j]
u=1*X
v=-1*Y
pl.quiver(X,Y,u,v,color='dimgray')
plt.show()
```

Comportamiento del sistema cuando beta>0

```python
#sistema dS/dt=-b*S, dI/dt=b*I con beta>0
def dx_dt(x,t):
    return [ -1*x[0] , 1*x[1] ]
#trayectorias en tiempo hacia adelante
ts=np.linspace(0,10,500)
ic=np.linspace(0,100000,4)
for r in ic:
    for s in ic:
        x0=[r,s]
        xs=odeint(dx_dt,x0,ts)
        plt.plot(xs[:,0],xs[:,1],"r-")
#trayectorias en tiempo hacia atras
ts=np.linspace(0,-10,500)
ic=np.linspace(0,100000,4)
for r in ic:
    for s in ic:
        x0=[r,s]
        xs=odeint(dx_dt,x0,ts)
        plt.plot(xs[:,0],xs[:,1],"r-")
#etiquetas de ejes y estilo de letra
plt.xlabel('S',fontsize=10)
plt.ylabel('I',fontsize=10)
plt.tick_params(labelsize=10)
plt.xlim(0,100000)
plt.ylim(0,100000)
#campo vectorial
X,Y=np.mgrid[0:100000:20j,0:100000:20j]
u=-1*X
v=1*Y
pl.quiver(X,Y,u,v,color='b')
plt.show()
```

hay un cambio en la direccion del flujo en el sistema.

```python

```


# [Linear Maps](https://www.youtube.com/watch?v=dtvM-CzNe50&list=PLJHszsWbB6hrkmmq57lX8BV-o-YIOFsiG&index=10)
### transform input vectors , don't transform the basis.
- keep gridlines parallel
- keep gridlines evenly spaced
- keep the origin stationary
- translations are not linear maps
- Maps vectors to vectors $ L:V \to W$
- Linearity

> ### - Coordinate Definition(arrays)
>> ### Matirx
>>> ### are the coordinate version of linear maps
>>> ### transform column vectors
>> ### Column Vectors
>>> ### Vectors
>> ### Row Vectors
>>> ### Covectors
> ### - Geometric Definition(pictures)
> ### - Abstact Definition(algebra)

```python

```

> ### $ \quad L: V \to V \quad \{\vec{e_1}, \vec{e_2}\}$
>> ### $\vec{W} = L(\vec{v}) = L(v^1 \vec{e_1} + v^2 \vec{e_2}) = v^1 L(\vec{e_1}) + v^2 L(\vec{e_2})$
>>> ### $ L(\vec{e_1}) = L_{1}^{1} \vec{e_1} + L_{1}^{2} \vec{e_2}$
>>> ### $ L(\vec{e_2}) = L_{2}^{1} + L_{2}^{2} \vec{e_2}$

> ### $\vec{W} = L(\vec{v}) \\
= v^1(L_{1}^{1}\vec{e_1} + L_{1}^{2} \vec{e_2}) +
 v^2(L_{2}^{1}\vec{e_1} + L_{2}^{2} \vec{e_2}) \\
= (L_{1}^{1}v^1 + L_{2}^{1} {v^2})\vec{e_1} +
 (L_{1}^{2}{v^1} + L_{2}^{2} {v^2}) \vec{e_2} \\
= w^1 \vec{e_1} + w^2 \vec{e_2}$

> ### $\therefore w^1 = L_{1}^{1} v^1 + L_{2}^{1} v^2 $
> ### $\therefore w^2 = L_{1}^{2} v^1 + L_{2}^{2} v^2 $
> ### $\therefore w^i = \sum_{j=1}^{n} L_{j}^{i} v^j $
> ### $\therefore \vec{w} = L(\vec{v}) = \sum_{i=1}^{n}w^i \vec{e_i}$
> ### $\therefore L(\vec{e_i}) = \sum_{j=1}^{n}L_{i}^{j}\vec{e_j}$

```python
import sympy as sm
x,y,z,t = sm.symbols('x y z t')
A = sm.Array([[x,y],[z,t]])
B = sm.Array([[2,1],[0,-1]])
sm.tensorproduct(A,B)
```

$\displaystyle \left[\begin{matrix}\left[\begin{matrix}2 x & x\\0 & - x\end{matrix}\right] & \left[\begin{matrix}2 y & y\\0 & - y\end{matrix}\right]\\\left[\begin{matrix}2 z & z\\0 & - z\end{matrix}\right] & \left[\begin{matrix}2 t & t\\0 & - t\end{matrix}\right]\end{matrix}\right]$

```python
sm.tensorproduct(A,A)
```

$\displaystyle \left[\begin{matrix}\left[\begin{matrix}x^{2} & x y\\x z & t x\end{matrix}\right] & \left[\begin{matrix}x y & y^{2}\\y z & t y\end{matrix}\right]\\\left[\begin{matrix}x z & y z\\z^{2} & t z\end{matrix}\right] & \left[\begin{matrix}t x & t y\\t z & t^{2}\end{matrix}\right]\end{matrix}\right]$

```python

```


# Exercise Sheet 7
## Ex. 1 Clustering

### The aim of this exercise is to familiarise yourself with the analysis of clustering hyperparameters. For this purpose, let us consider the AgglomerativeClustering algorithm in sklearn. Using make circles, make moons, make blobs visualise the different performance of different linkages (ward, complete, average, single). Perform a little bit of hyperparameter tuning and show for the different linkage models the best model you have found. In the second part of this exercise, compare these results with the performance of t-SNE and k-means.

```python
# Imports useful libraries

import numpy as np
import matplotlib.pyplot as plt
import tensorflow

from sklearn.cluster import AgglomerativeClustering
from sklearn.datasets import make_moons, make_circles, make_blobs
from sklearn.manifold import TSNE
from sklearn.cluster import KMeans
from pandas import DataFrame

from tensorflow.keras.models import Model, Sequential # imports network type
from tensorflow.keras.layers import Dense, Input # imports our layers want to use

from tensorflow.python.keras.losses import binary_crossentropy, categorical_crossentropy # loss function
from tensorflow.python.keras.optimizers import Adam, SGD # optimizers

```

In the following we generate labelled data using the suggested libraries and we then implement the requested clustering algorithms. We start with blobs and then switch to circles and moons.

```python
# Generate sample data (BLOBS)
X, y = make_blobs(n_samples=100, centers=3, n_features=2, random_state=4)

# Plot sample data with true classification
df = DataFrame(dict(x=X[:, 0], y=X[:, 1], label=y))
colors = {0: 'red', 1: 'blue', 2: 'green'}
fig, ax = plt.subplots()
grouped = df.groupby('label')
for key, group in grouped:
    group.plot(ax=ax, kind='scatter', x='x', y='y', color=colors[key])
plt.title('True Classification')
plt.show()
```

```python
for linkage in ('ward', 'complete', 'average', 'single'):
    model = AgglomerativeClustering(affinity='euclidean', compute_full_tree='auto', connectivity=None, linkage=linkage, n_clusters=3, pooling_func='deprecated')
    model.fit(X)
    y_predicted = model.fit_predict(X)

    # Plot data with predicted classification
    df_predicted = DataFrame(dict(x=X[:, 0], y=X[:, 1], label=y_predicted))
    grouped_predicted = df_predicted.groupby('label')
    colors_predicted = {0: 'red', 1: 'green', 2: 'blue'}
    fig_predicted, ax_predicted = plt.subplots()
    for key, group in grouped_predicted:
        group.plot(ax=ax_predicted, kind='scatter', x='x', y='y', color=colors_predicted[key])
    performance = str(np.floor(100 - np.linalg.norm(y-y_predicted)))
    plt.title('Classification ' + '"' + linkage + '"')
    plt.show()
    print('Accuracy ' + performance + '%')

# Comparison with k-Means
kmeans = KMeans(n_clusters=3, random_state=0).fit(X)
y_kmeans = kmeans.labels_
performance_kmeans = str(np.floor(100 - np.linalg.norm(y-y_kmeans)))
print('KMeans Accuracy ' + performance_kmeans + '%')
```

```python
# Generate sample data (CIRCLES)
X, y = make_circles(n_samples=100, noise=0.05, random_state=4)

# Plot sample data with true classification
df = DataFrame(dict(x=X[:, 0], y=X[:, 1], label=y))
fig, ax = plt.subplots()
grouped = df.groupby('label')
for key, group in grouped:
    group.plot(ax=ax, kind='scatter', x='x', y='y', color=colors[key])
plt.title('True Classification')
plt.show()

```

```python
for linkage in ('ward', 'complete', 'average', 'single'):
    model = AgglomerativeClustering(affinity='euclidean', compute_full_tree='auto',
                                    connectivity=None, linkage=linkage, n_clusters=2,
                                    pooling_func='deprecated')

    model.fit(X)
    y_predicted = model.fit_predict(X)

    # Plot data with predicted classification
    df_predicted = DataFrame(dict(x=X[:, 0], y=X[:, 1], label=y_predicted))
    grouped_predicted = df_predicted.groupby('label')
    colors_predicted = {0: 'red', 1: 'blue'}
    fig_predicted, ax_predicted = plt.subplots()
    for key, group in grouped_predicted:
        group.plot(ax=ax_predicted, kind='scatter', x='x', y='y', color=colors_predicted[key])
    performance = str(np.floor(100 - np.linalg.norm(y-y_predicted)))
    plt.title('Classification ' + '"' + linkage + '"')
    plt.show()
    print('Accuracy ' + performance + '%')

# Comparison with k-Means
kmeans = KMeans(n_clusters=3, random_state=0).fit(X)
y_kmeans = kmeans.labels_
performance_kmeans = str(np.floor(100 - np.linalg.norm(y-y_kmeans)))
print('KMeans Accuracy ' + performance_kmeans + '%')

```

```python
# Generate sample data (MOONS)
X, y = make_moons(n_samples=100, noise=0.05, random_state=4)

# Plot sample data with true classification
df = DataFrame(dict(x=X[:, 0], y=X[:, 1], label=y))
fig, ax = plt.subplots()
grouped = df.groupby('label')
for key, group in grouped:
    group.plot(ax=ax, kind='scatter', x='x', y='y', color=colors[key])
plt.title('True Moon Classification')
plt.show()

```

```python
for linkage in ('ward', 'complete', 'average', 'single'):
    model = AgglomerativeClustering(affinity='euclidean', compute_full_tree='auto', connectivity=None, linkage=linkage, n_clusters=2, pooling_func='deprecated')

    model.fit(X)
    y_predicted = model.fit_predict(X)

    # Plot data with predicted classification
    df_predicted = DataFrame(dict(x=X[:, 0], y=X[:, 1], label=y_predicted))
    grouped_predicted = df_predicted.groupby('label')
    colors_predicted = {0: 'red', 1: 'blue'}
    fig_predicted, ax_predicted = plt.subplots()
    for key, group in grouped_predicted:
        group.plot(ax=ax_predicted, kind='scatter', x='x', y='y', color=colors_predicted[key])
    performance = str(np.floor(100 - np.linalg.norm(y-y_predicted)))
    plt.title('Classification ' + '"' + linkage + '"')
    plt.show()
    print('Accuracy ' + performance + '%')

# Comparison with k-Means
kmeans = KMeans(n_clusters=3, random_state=0).fit(X)
y_kmeans = kmeans.labels_
performance_kmeans = str(np.floor(100 - np.linalg.norm(y-y_kmeans)))
print('KMeans Accuracy ' + performance_kmeans + '%')

```

## Ex. 2 AutoEncoder

### 2.1 Generate 2-d images (1-channel) showing polynomials up to a maximum degree (e.g. 40x40) in two variables.

Here we create 40x40 images of two variable, given maximum degree polynomials. For semplicity we exclude cross terms in the two variables $x$ and $y$ as for example $xy, x^2y$, etc.

```python
# CREATES 40X40 IMAGES OF POLYNOMIALS OF MAXIMUM DEGREE
x=np.linspace(-1,1,40)
xx,yy=np.meshgrid(x,x)

def random_poly(deg, xx=xx, yy=yy): #function that creates polinomial of 2 variables of given degree with no cross terms
    x_params=np.random.uniform(-5.,5., deg+1)
    y_params=np.random.uniform(-5.,5., deg+1)

    p=np.zeros((40,40))

    for i in range(deg+1):
        p+=x_params[i]*xx**i + y_params[i]*yy**i

    return p

# CREATES DATASET
size = 6000
data=[]

for i in range(size):
    degree=int(np.random.uniform(1,9))
    data.append(random_poly(degree))
data=np.array(data)

plt.imshow(data[1298]) # shows element 1298 of the dataset
plt.show()
```

### 2.2 Build two autoencoder architectures (e.g. similar to the ones presented in the lecture) which involve a single hidden dense layer and several hidden dense layers.

```python
# Autoencoder with a single dense hidden layer

model1=Sequential()
model1.add(Dense(800, input_dim=1600, activation='relu')) # encoder
model1.add(Dense(1600, activation='relu')) # decoder

model1.compile(loss='binary_crossentropy', optimizer='adadelta', metrics=['accuracy'])

# Arranges dataset for training

data=data.reshape((len(data), np.prod(data.shape[1:])))
x_train=data[:5900]
x_test=data[5900:]

```

```python
history1 = model1.fit(x_train, x_train, epochs=100, batch_size=256, shuffle=True, verbose=1, validation_data=(x_test, x_test))
```

    Train on 5900 samples, validate on 100 samples
    Epoch 1/100
    5900/5900 [==============================] - 1s 239us/sample - loss: -41.3699 - acc: 0.0000e+00 - val_loss: -51.5172 - val_acc: 0.0000e+00
    Epoch 2/100
    5900/5900 [==============================] - 1s 180us/sample - loss: -52.1401 - acc: 0.0000e+00 - val_loss: -54.0396 - val_acc: 0.0000e+00
    Epoch 3/100
    5900/5900 [==============================] - 1s 203us/sample - loss: -53.8143 - acc: 0.0000e+00 - val_loss: -55.6495 - val_acc: 0.0000e+00
    Epoch 4/100
    5900/5900 [==============================] - 1s 182us/sample - loss: -54.9864 - acc: 0.0000e+00 - val_loss: -56.4199 - val_acc: 0.0000e+00
    Epoch 5/100
    5900/5900 [==============================] - 1s 174us/sample - loss: -55.6094 - acc: 0.0000e+00 - val_loss: -56.5758 - val_acc: 0.0000e+00
    Epoch 6/100
    5900/5900 [==============================] - 1s 177us/sample - loss: -55.9509 - acc: 0.0000e+00 - val_loss: -57.1898 - val_acc: 0.0000e+00
    Epoch 7/100
    5900/5900 [==============================] - 1s 178us/sample - loss: -56.3112 - acc: 0.0000e+00 - val_loss: -57.6111 - val_acc: 0.0000e+00
    Epoch 8/100
    5900/5900 [==============================] - 1s 178us/sample - loss: -56.6761 - acc: 0.0000e+00 - val_loss: -57.8418 - val_acc: 0.0000e+00
    Epoch 9/100
    5900/5900 [==============================] - 1s 192us/sample - loss: -56.9034 - acc: 0.0000e+00 - val_loss: -58.0123 - val_acc: 0.0000e+00
    Epoch 10/100
    5900/5900 [==============================] - 1s 178us/sample - loss: -57.1116 - acc: 0.0000e+00 - val_loss: -58.0966 - val_acc: 0.0000e+00
    Epoch 11/100
    5900/5900 [==============================] - 1s 175us/sample - loss: -57.1533 - acc: 0.0000e+00 - val_loss: -58.0138 - val_acc: 0.0000e+00
    Epoch 12/100
    5900/5900 [==============================] - 1s 170us/sample - loss: -57.1266 - acc: 0.0000e+00 - val_loss: -58.1201 - val_acc: 0.0000e+00
    Epoch 13/100
    5900/5900 [==============================] - 1s 174us/sample - loss: -57.3582 - acc: 0.0000e+00 - val_loss: -58.3518 - val_acc: 0.0000e+00
    Epoch 14/100
    5900/5900 [==============================] - 1s 171us/sample - loss: -57.4367 - acc: 0.0000e+00 - val_loss: -58.3681 - val_acc: 0.0000e+00
    Epoch 15/100
    5900/5900 [==============================] - 1s 176us/sample - loss: -57.3753 - acc: 0.0000e+00 - val_loss: -58.3041 - val_acc: 0.0000e+00
    Epoch 16/100
    5900/5900 [==============================] - 1s 174us/sample - loss: -57.4576 - acc: 0.0000e+00 - val_loss: -58.4106 - val_acc: 0.0000e+00
    Epoch 17/100
    5900/5900 [==============================] - 1s 171us/sample - loss: -57.6183 - acc: 0.0000e+00 - val_loss: -58.5919 - val_acc: 0.0000e+00
    Epoch 18/100
    5900/5900 [==============================] - 1s 174us/sample - loss: -57.7162 - acc: 0.0000e+00 - val_loss: -58.7092 - val_acc: 0.0000e+00
    Epoch 19/100
    5900/5900 [==============================] - 1s 172us/sample - loss: -57.8023 - acc: 0.0000e+00 - val_loss: -58.7681 - val_acc: 0.0000e+00
    Epoch 20/100
    5900/5900 [==============================] - 1s 171us/sample - loss: -57.8966 - acc: 0.0000e+00 - val_loss: -58.7744 - val_acc: 0.0000e+00
    Epoch 21/100
    5900/5900 [==============================] - 1s 172us/sample - loss: -57.9659 - acc: 0.0000e+00 - val_loss: -59.0148 - val_acc: 0.0000e+00
    Epoch 22/100
    5900/5900 [==============================] - 1s 173us/sample - loss: -58.0786 - acc: 0.0000e+00 - val_loss: -58.9938 - val_acc: 0.0000e+00
    Epoch 23/100
    5900/5900 [==============================] - 1s 172us/sample - loss: -58.1149 - acc: 0.0000e+00 - val_loss: -59.1661 - val_acc: 0.0000e+00
    Epoch 24/100
    5900/5900 [==============================] - 1s 174us/sample - loss: -58.2197 - acc: 0.0000e+00 - val_loss: -59.1331 - val_acc: 0.0000e+00
    Epoch 25/100
    5900/5900 [==============================] - 1s 172us/sample - loss: -58.2960 - acc: 0.0000e+00 - val_loss: -59.2596 - val_acc: 0.0000e+00
    Epoch 26/100
    5900/5900 [==============================] - 1s 170us/sample - loss: -58.3483 - acc: 0.0000e+00 - val_loss: -59.2829 - val_acc: 0.0000e+00
    Epoch 27/100
    5900/5900 [==============================] - 1s 173us/sample - loss: -58.3844 - acc: 0.0000e+00 - val_loss: -59.3474 - val_acc: 0.0000e+00
    Epoch 28/100
    5900/5900 [==============================] - 1s 175us/sample - loss: -58.4101 - acc: 0.0000e+00 - val_loss: -59.3449 - val_acc: 0.0000e+00
    Epoch 29/100
    5900/5900 [==============================] - 1s 188us/sample - loss: -58.4813 - acc: 0.0000e+00 - val_loss: -59.4544 - val_acc: 0.0000e+00
    Epoch 30/100
    5900/5900 [==============================] - 1s 205us/sample - loss: -58.4728 - acc: 0.0000e+00 - val_loss: -59.4029 - val_acc: 0.0000e+00
    Epoch 31/100
    5900/5900 [==============================] - 1s 182us/sample - loss: -58.5240 - acc: 0.0000e+00 - val_loss: -59.3736 - val_acc: 0.0000e+00
    Epoch 32/100
    5900/5900 [==============================] - 1s 175us/sample - loss: -58.5176 - acc: 0.0000e+00 - val_loss: -59.4502 - val_acc: 0.0000e+00
    Epoch 33/100
    5900/5900 [==============================] - 1s 178us/sample - loss: -58.5929 - acc: 0.0000e+00 - val_loss: -59.5327 - val_acc: 0.0000e+00
    Epoch 34/100
    5900/5900 [==============================] - 1s 177us/sample - loss: -58.6112 - acc: 0.0000e+00 - val_loss: -59.5525 - val_acc: 0.0000e+00
    Epoch 35/100
    5900/5900 [==============================] - 1s 174us/sample - loss: -58.6778 - acc: 0.0000e+00 - val_loss: -59.5545 - val_acc: 0.0000e+00
    Epoch 36/100
    5900/5900 [==============================] - 1s 168us/sample - loss: -58.7180 - acc: 0.0000e+00 - val_loss: -59.6006 - val_acc: 0.0000e+00
    Epoch 37/100
    5900/5900 [==============================] - 1s 177us/sample - loss: -58.6124 - acc: 0.0000e+00 - val_loss: -59.5575 - val_acc: 0.0000e+00
    Epoch 38/100
    5900/5900 [==============================] - 1s 167us/sample - loss: -58.6633 - acc: 0.0000e+00 - val_loss: -59.5139 - val_acc: 0.0000e+00
    Epoch 39/100
    5900/5900 [==============================] - 1s 169us/sample - loss: -58.7129 - acc: 0.0000e+00 - val_loss: -59.5655 - val_acc: 0.0000e+00
    Epoch 40/100
    5900/5900 [==============================] - 1s 176us/sample - loss: -58.7079 - acc: 0.0000e+00 - val_loss: -59.6893 - val_acc: 0.0000e+00
    Epoch 41/100
    5900/5900 [==============================] - 1s 171us/sample - loss: -58.7878 - acc: 0.0000e+00 - val_loss: -59.7030 - val_acc: 0.0000e+00
    Epoch 42/100
    5900/5900 [==============================] - 1s 177us/sample - loss: -58.8165 - acc: 0.0000e+00 - val_loss: -59.7000 - val_acc: 0.0000e+00
    Epoch 43/100
    5900/5900 [==============================] - 1s 169us/sample - loss: -58.8109 - acc: 0.0000e+00 - val_loss: -59.7437 - val_acc: 0.0000e+00
    Epoch 44/100
    5900/5900 [==============================] - 1s 168us/sample - loss: -58.8226 - acc: 0.0000e+00 - val_loss: -59.5998 - val_acc: 0.0000e+00
    Epoch 45/100
    5900/5900 [==============================] - 1s 169us/sample - loss: -58.8167 - acc: 0.0000e+00 - val_loss: -59.8018 - val_acc: 0.0000e+00
    Epoch 46/100
    5900/5900 [==============================] - 1s 166us/sample - loss: -58.9101 - acc: 0.0000e+00 - val_loss: -59.7912 - val_acc: 0.0000e+00
    Epoch 47/100
    5900/5900 [==============================] - 1s 166us/sample - loss: -58.8981 - acc: 0.0000e+00 - val_loss: -59.8116 - val_acc: 0.0000e+00
    Epoch 48/100
    5900/5900 [==============================] - 1s 165us/sample - loss: -58.9296 - acc: 0.0000e+00 - val_loss: -59.8731 - val_acc: 0.0000e+00
    Epoch 49/100
    5900/5900 [==============================] - 1s 166us/sample - loss: -58.9903 - acc: 0.0000e+00 - val_loss: -59.9149 - val_acc: 0.0000e+00
    Epoch 50/100
    5900/5900 [==============================] - 1s 169us/sample - loss: -59.0208 - acc: 0.0000e+00 - val_loss: -59.9017 - val_acc: 0.0000e+00
    Epoch 51/100
    5900/5900 [==============================] - 1s 173us/sample - loss: -59.0546 - acc: 0.0000e+00 - val_loss: -59.9686 - val_acc: 0.0000e+00
    Epoch 52/100
    5900/5900 [==============================] - 1s 165us/sample - loss: -59.0244 - acc: 0.0000e+00 - val_loss: -59.9307 - val_acc: 0.0000e+00
    Epoch 53/100
    5900/5900 [==============================] - 1s 172us/sample - loss: -58.9963 - acc: 0.0000e+00 - val_loss: -59.9115 - val_acc: 0.0000e+00
    Epoch 54/100
    5900/5900 [==============================] - 1s 166us/sample - loss: -59.0692 - acc: 0.0000e+00 - val_loss: -60.0280 - val_acc: 0.0000e+00
    Epoch 55/100
    5900/5900 [==============================] - 1s 167us/sample - loss: -59.1623 - acc: 0.0000e+00 - val_loss: -60.0836 - val_acc: 0.0000e+00
    Epoch 56/100
    5900/5900 [==============================] - 1s 167us/sample - loss: -59.1792 - acc: 0.0000e+00 - val_loss: -60.0554 - val_acc: 0.0000e+00
    Epoch 57/100
    5900/5900 [==============================] - 1s 172us/sample - loss: -59.1451 - acc: 0.0000e+00 - val_loss: -60.0009 - val_acc: 0.0000e+00
    Epoch 58/100
    5900/5900 [==============================] - 1s 172us/sample - loss: -59.1453 - acc: 0.0000e+00 - val_loss: -60.0095 - val_acc: 0.0000e+00
    Epoch 59/100
    5900/5900 [==============================] - 1s 171us/sample - loss: -59.1104 - acc: 0.0000e+00 - val_loss: -60.0043 - val_acc: 0.0000e+00
    Epoch 60/100
    5900/5900 [==============================] - 1s 170us/sample - loss: -59.1564 - acc: 0.0000e+00 - val_loss: -60.0715 - val_acc: 0.0000e+00
    Epoch 61/100
    5900/5900 [==============================] - 1s 172us/sample - loss: -59.2144 - acc: 0.0000e+00 - val_loss: -60.1546 - val_acc: 0.0000e+00
    Epoch 62/100
    5900/5900 [==============================] - 1s 172us/sample - loss: -59.2966 - acc: 0.0000e+00 - val_loss: -60.1995 - val_acc: 0.0000e+00
    Epoch 63/100
    5900/5900 [==============================] - 1s 166us/sample - loss: -59.3232 - acc: 0.0000e+00 - val_loss: -60.1512 - val_acc: 0.0000e+00
    Epoch 64/100
    5900/5900 [==============================] - 1s 172us/sample - loss: -59.2863 - acc: 0.0000e+00 - val_loss: -60.1556 - val_acc: 0.0000e+00
    Epoch 65/100
    5900/5900 [==============================] - 1s 170us/sample - loss: -59.3041 - acc: 0.0000e+00 - val_loss: -60.2407 - val_acc: 0.0000e+00
    Epoch 66/100
    5900/5900 [==============================] - 1s 173us/sample - loss: -59.3199 - acc: 0.0000e+00 - val_loss: -60.2447 - val_acc: 0.0000e+00
    Epoch 67/100
    5900/5900 [==============================] - 1s 166us/sample - loss: -59.3358 - acc: 0.0000e+00 - val_loss: -60.2930 - val_acc: 0.0000e+00
    Epoch 68/100
    5900/5900 [==============================] - 1s 168us/sample - loss: -59.4303 - acc: 0.0000e+00 - val_loss: -60.3145 - val_acc: 0.0000e+00
    Epoch 69/100
    5900/5900 [==============================] - 1s 169us/sample - loss: -59.4517 - acc: 0.0000e+00 - val_loss: -60.3450 - val_acc: 0.0000e+00
    Epoch 70/100
    5900/5900 [==============================] - 1s 172us/sample - loss: -59.4799 - acc: 0.0000e+00 - val_loss: -60.4106 - val_acc: 0.0000e+00
    Epoch 71/100
    5900/5900 [==============================] - 1s 166us/sample - loss: -59.4728 - acc: 0.0000e+00 - val_loss: -60.3915 - val_acc: 0.0000e+00
    Epoch 72/100
    5900/5900 [==============================] - 1s 170us/sample - loss: -59.5214 - acc: 0.0000e+00 - val_loss: -60.3803 - val_acc: 0.0000e+00
    Epoch 73/100
    5900/5900 [==============================] - 1s 173us/sample - loss: -59.5011 - acc: 0.0000e+00 - val_loss: -60.3880 - val_acc: 0.0000e+00
    Epoch 74/100
    5900/5900 [==============================] - 1s 171us/sample - loss: -59.5172 - acc: 0.0000e+00 - val_loss: -60.3908 - val_acc: 0.0000e+00
    Epoch 75/100
    5900/5900 [==============================] - 1s 176us/sample - loss: -59.5130 - acc: 0.0000e+00 - val_loss: -60.4026 - val_acc: 0.0000e+00
    Epoch 76/100
    5900/5900 [==============================] - 1s 170us/sample - loss: -59.5484 - acc: 0.0000e+00 - val_loss: -60.4494 - val_acc: 0.0000e+00
    Epoch 77/100
    5900/5900 [==============================] - 1s 171us/sample - loss: -59.5454 - acc: 0.0000e+00 - val_loss: -60.3865 - val_acc: 0.0000e+00
    Epoch 78/100
    5900/5900 [==============================] - 1s 169us/sample - loss: -59.5400 - acc: 0.0000e+00 - val_loss: -60.4541 - val_acc: 0.0000e+00
    Epoch 79/100
    5900/5900 [==============================] - 1s 172us/sample - loss: -59.5703 - acc: 0.0000e+00 - val_loss: -60.3837 - val_acc: 0.0000e+00
    Epoch 80/100
    5900/5900 [==============================] - 1s 170us/sample - loss: -59.5437 - acc: 0.0000e+00 - val_loss: -60.3861 - val_acc: 0.0000e+00
    Epoch 81/100
    5900/5900 [==============================] - 1s 171us/sample - loss: -59.5338 - acc: 0.0000e+00 - val_loss: -60.4060 - val_acc: 0.0000e+00
    Epoch 82/100
    5900/5900 [==============================] - 1s 173us/sample - loss: -59.5698 - acc: 0.0000e+00 - val_loss: -60.4858 - val_acc: 0.0000e+00
    Epoch 83/100
    5900/5900 [==============================] - 1s 171us/sample - loss: -59.5952 - acc: 0.0000e+00 - val_loss: -60.4463 - val_acc: 0.0000e+00
    Epoch 84/100
    5900/5900 [==============================] - 1s 172us/sample - loss: -59.6276 - acc: 0.0000e+00 - val_loss: -60.4686 - val_acc: 0.0000e+00
    Epoch 85/100
    5900/5900 [==============================] - 1s 174us/sample - loss: -59.6167 - acc: 0.0000e+00 - val_loss: -60.4989 - val_acc: 0.0000e+00
    Epoch 86/100
    5900/5900 [==============================] - 1s 168us/sample - loss: -59.6196 - acc: 0.0000e+00 - val_loss: -60.5454 - val_acc: 0.0000e+00
    Epoch 87/100
    5900/5900 [==============================] - 1s 174us/sample - loss: -59.6533 - acc: 0.0000e+00 - val_loss: -60.5507 - val_acc: 0.0000e+00
    Epoch 88/100
    5900/5900 [==============================] - 1s 172us/sample - loss: -59.6760 - acc: 0.0000e+00 - val_loss: -60.5587 - val_acc: 0.0000e+00
    Epoch 89/100
    5900/5900 [==============================] - 1s 169us/sample - loss: -59.6701 - acc: 0.0000e+00 - val_loss: -60.5137 - val_acc: 0.0000e+00
    Epoch 90/100
    5900/5900 [==============================] - 1s 171us/sample - loss: -59.6459 - acc: 0.0000e+00 - val_loss: -60.5117 - val_acc: 0.0000e+00
    Epoch 91/100
    5900/5900 [==============================] - 1s 172us/sample - loss: -59.6563 - acc: 0.0000e+00 - val_loss: -60.5115 - val_acc: 0.0000e+00
    Epoch 92/100
    5900/5900 [==============================] - 1s 172us/sample - loss: -59.5761 - acc: 0.0000e+00 - val_loss: -60.4883 - val_acc: 0.0000e+00
    Epoch 93/100
    5900/5900 [==============================] - 1s 175us/sample - loss: -59.6278 - acc: 0.0000e+00 - val_loss: -60.5226 - val_acc: 0.0000e+00
    Epoch 94/100
    5900/5900 [==============================] - 1s 172us/sample - loss: -59.6530 - acc: 0.0000e+00 - val_loss: -60.5311 - val_acc: 0.0000e+00
    Epoch 95/100
    5900/5900 [==============================] - 1s 171us/sample - loss: -59.6702 - acc: 0.0000e+00 - val_loss: -60.5404 - val_acc: 0.0000e+00
    Epoch 96/100
    5900/5900 [==============================] - 1s 169us/sample - loss: -59.7096 - acc: 0.0000e+00 - val_loss: -60.6490 - val_acc: 0.0000e+00
    Epoch 97/100
    5900/5900 [==============================] - 1s 174us/sample - loss: -59.7746 - acc: 0.0000e+00 - val_loss: -60.6222 - val_acc: 0.0000e+00
    Epoch 98/100
    5900/5900 [==============================] - 1s 173us/sample - loss: -59.7720 - acc: 0.0000e+00 - val_loss: -60.6670 - val_acc: 0.0000e+00
    Epoch 99/100
    5900/5900 [==============================] - 1s 171us/sample - loss: -59.7832 - acc: 0.0000e+00 - val_loss: -60.6786 - val_acc: 0.0000e+00
    Epoch 100/100
    5900/5900 [==============================] - 1s 174us/sample - loss: -59.8122 - acc: 0.0000e+00 - val_loss: -60.6775 - val_acc: 0.0000e+00

```python
# Autoencoder with multiple hidden layers

model2=Sequential()
model2.add(Dense(800, input_dim=1600, activation='relu')) # encoder
model2.add(Dense(400, input_dim=1600, activation='relu')) # encoder
model2.add(Dense(200, input_dim=1600, activation='relu')) # encoder

model2.add(Dense(400, activation='relu')) # decoder
model2.add(Dense(800, activation='relu')) # decoder
model2.add(Dense(1600, activation='relu')) # decoder

model2.compile(loss='binary_crossentropy', optimizer='adadelta', metrics=['accuracy'])
```

```python
history2 = model2.fit(x_train, x_train, epochs=100, batch_size=256, shuffle=True, verbose=1, validation_data=(x_test, x_test))
```

    Train on 5900 samples, validate on 100 samples
    Epoch 1/100
    5900/5900 [==============================] - 2s 329us/sample - loss: -15.8014 - acc: 0.0000e+00 - val_loss: -24.3783 - val_acc: 0.0000e+00
    Epoch 2/100
    5900/5900 [==============================] - 1s 243us/sample - loss: -22.5769 - acc: 0.0000e+00 - val_loss: -27.2968 - val_acc: 0.0000e+00
    Epoch 3/100
    5900/5900 [==============================] - 1s 243us/sample - loss: -25.9306 - acc: 0.0000e+00 - val_loss: -29.6808 - val_acc: 0.0000e+00
    Epoch 4/100
    5900/5900 [==============================] - 1s 245us/sample - loss: -30.5874 - acc: 0.0000e+00 - val_loss: -36.0546 - val_acc: 0.0000e+00
    Epoch 5/100
    5900/5900 [==============================] - 2s 266us/sample - loss: -36.1050 - acc: 0.0000e+00 - val_loss: -38.3320 - val_acc: 0.0000e+00
    Epoch 6/100
    5900/5900 [==============================] - 1s 245us/sample - loss: -39.0942 - acc: 0.0000e+00 - val_loss: -42.6779 - val_acc: 0.0000e+00
    Epoch 7/100
    5900/5900 [==============================] - 1s 251us/sample - loss: -41.1157 - acc: 0.0000e+00 - val_loss: -43.9092 - val_acc: 0.0000e+00
    Epoch 8/100
    5900/5900 [==============================] - 2s 292us/sample - loss: -41.8582 - acc: 0.0000e+00 - val_loss: -44.6236 - val_acc: 0.0000e+00
    Epoch 9/100
    5900/5900 [==============================] - 2s 256us/sample - loss: -42.9916 - acc: 0.0000e+00 - val_loss: -45.7453 - val_acc: 0.0000e+00
    Epoch 10/100
    5900/5900 [==============================] - 2s 261us/sample - loss: -43.9281 - acc: 0.0000e+00 - val_loss: -45.7952 - val_acc: 0.0000e+00
    Epoch 11/100
    5900/5900 [==============================] - 2s 260us/sample - loss: -44.2750 - acc: 0.0000e+00 - val_loss: -45.7761 - val_acc: 0.0000e+00
    Epoch 12/100
    5900/5900 [==============================] - 1s 247us/sample - loss: -44.6806 - acc: 0.0000e+00 - val_loss: -45.7115 - val_acc: 0.0000e+00
    Epoch 13/100
    5900/5900 [==============================] - 1s 241us/sample - loss: -45.2055 - acc: 0.0000e+00 - val_loss: -47.5321 - val_acc: 0.0000e+00
    Epoch 14/100
    5900/5900 [==============================] - 1s 245us/sample - loss: -45.2277 - acc: 0.0000e+00 - val_loss: -47.5805 - val_acc: 0.0000e+00
    Epoch 15/100
    5900/5900 [==============================] - 1s 248us/sample - loss: -45.6985 - acc: 0.0000e+00 - val_loss: -48.0495 - val_acc: 0.0000e+00
    Epoch 16/100
    5900/5900 [==============================] - 1s 242us/sample - loss: -45.9409 - acc: 0.0000e+00 - val_loss: -48.0182 - val_acc: 0.0000e+00
    Epoch 17/100
    5900/5900 [==============================] - 1s 239us/sample - loss: -46.2608 - acc: 0.0000e+00 - val_loss: -48.2679 - val_acc: 0.0000e+00
    Epoch 18/100
    5900/5900 [==============================] - 1s 247us/sample - loss: -46.5463 - acc: 0.0000e+00 - val_loss: -48.7393 - val_acc: 0.0000e+00
    Epoch 19/100
    5900/5900 [==============================] - 1s 246us/sample - loss: -46.4950 - acc: 0.0000e+00 - val_loss: -48.5602 - val_acc: 0.0000e+00
    Epoch 20/100
    5900/5900 [==============================] - 1s 240us/sample - loss: -46.4548 - acc: 0.0000e+00 - val_loss: -48.0272 - val_acc: 0.0000e+00
    Epoch 21/100
    5900/5900 [==============================] - 1s 235us/sample - loss: -46.1785 - acc: 0.0000e+00 - val_loss: -48.5086 - val_acc: 0.0000e+00
    Epoch 22/100
    5900/5900 [==============================] - 1s 241us/sample - loss: -46.6614 - acc: 0.0000e+00 - val_loss: -49.1523 - val_acc: 0.0000e+00
    Epoch 23/100
    5900/5900 [==============================] - 1s 247us/sample - loss: -47.2044 - acc: 0.0000e+00 - val_loss: -48.6845 - val_acc: 0.0000e+00
    Epoch 24/100
    5900/5900 [==============================] - 1s 244us/sample - loss: -47.2726 - acc: 0.0000e+00 - val_loss: -48.6632 - val_acc: 0.0000e+00
    Epoch 25/100
    5900/5900 [==============================] - 1s 246us/sample - loss: -47.0119 - acc: 0.0000e+00 - val_loss: -48.8105 - val_acc: 0.0000e+00
    Epoch 26/100
    5900/5900 [==============================] - 1s 243us/sample - loss: -47.7243 - acc: 0.0000e+00 - val_loss: -49.8883 - val_acc: 0.0000e+00
    Epoch 27/100
    5900/5900 [==============================] - 1s 253us/sample - loss: -48.0804 - acc: 0.0000e+00 - val_loss: -49.7896 - val_acc: 0.0000e+00
    Epoch 28/100
    5900/5900 [==============================] - 1s 237us/sample - loss: -48.1825 - acc: 0.0000e+00 - val_loss: -50.1131 - val_acc: 0.0000e+00
    Epoch 29/100
    5900/5900 [==============================] - 1s 240us/sample - loss: -48.4826 - acc: 0.0000e+00 - val_loss: -47.8882 - val_acc: 0.0000e+00
    Epoch 30/100
    5900/5900 [==============================] - 1s 236us/sample - loss: -48.2353 - acc: 0.0000e+00 - val_loss: -49.8448 - val_acc: 0.0000e+00
    Epoch 31/100
    5900/5900 [==============================] - 1s 238us/sample - loss: -48.4069 - acc: 0.0000e+00 - val_loss: -50.9988 - val_acc: 0.0000e+00
    Epoch 32/100
    5900/5900 [==============================] - 1s 236us/sample - loss: -49.1431 - acc: 0.0000e+00 - val_loss: -51.1243 - val_acc: 0.0000e+00
    Epoch 33/100
    5900/5900 [==============================] - 1s 237us/sample - loss: -49.3025 - acc: 0.0000e+00 - val_loss: -50.8822 - val_acc: 0.0000e+00
    Epoch 34/100
    5900/5900 [==============================] - 1s 237us/sample - loss: -49.6351 - acc: 0.0000e+00 - val_loss: -51.5722 - val_acc: 0.0000e+00
    Epoch 35/100
    5900/5900 [==============================] - 1s 239us/sample - loss: -50.3220 - acc: 0.0000e+00 - val_loss: -52.0955 - val_acc: 0.0000e+00
    Epoch 36/100
    5900/5900 [==============================] - 1s 238us/sample - loss: -49.9216 - acc: 0.0000e+00 - val_loss: -51.5950 - val_acc: 0.0000e+00
    Epoch 37/100
    5900/5900 [==============================] - 1s 240us/sample - loss: -49.8444 - acc: 0.0000e+00 - val_loss: -51.8612 - val_acc: 0.0000e+00
    Epoch 38/100
    5900/5900 [==============================] - 1s 250us/sample - loss: -49.9779 - acc: 0.0000e+00 - val_loss: -52.0206 - val_acc: 0.0000e+00
    Epoch 39/100
    5900/5900 [==============================] - 1s 250us/sample - loss: -50.4257 - acc: 0.0000e+00 - val_loss: -52.2551 - val_acc: 0.0000e+00
    Epoch 40/100
    5900/5900 [==============================] - 1s 239us/sample - loss: -50.4816 - acc: 0.0000e+00 - val_loss: -51.0603 - val_acc: 0.0000e+00
    Epoch 41/100
    5900/5900 [==============================] - 2s 257us/sample - loss: -50.0889 - acc: 0.0000e+00 - val_loss: -52.2745 - val_acc: 0.0000e+00
    Epoch 42/100
    5900/5900 [==============================] - 1s 242us/sample - loss: -50.6514 - acc: 0.0000e+00 - val_loss: -52.6449 - val_acc: 0.0000e+00
    Epoch 43/100
    5900/5900 [==============================] - 1s 240us/sample - loss: -51.0457 - acc: 0.0000e+00 - val_loss: -52.6834 - val_acc: 0.0000e+00
    Epoch 44/100
    5900/5900 [==============================] - 2s 256us/sample - loss: -51.0312 - acc: 0.0000e+00 - val_loss: -52.3356 - val_acc: 0.0000e+00
    Epoch 45/100
    5900/5900 [==============================] - 2s 278us/sample - loss: -51.2279 - acc: 0.0000e+00 - val_loss: -53.0152 - val_acc: 0.0000e+00
    Epoch 46/100
    5900/5900 [==============================] - 2s 257us/sample - loss: -51.4714 - acc: 0.0000e+00 - val_loss: -53.1950 - val_acc: 0.0000e+00
    Epoch 47/100
    5900/5900 [==============================] - 1s 252us/sample - loss: -51.6633 - acc: 0.0000e+00 - val_loss: -53.3408 - val_acc: 0.0000e+00
    Epoch 48/100
    5900/5900 [==============================] - 1s 253us/sample - loss: -51.7740 - acc: 0.0000e+00 - val_loss: -53.3132 - val_acc: 0.0000e+00
    Epoch 49/100
    5900/5900 [==============================] - 2s 275us/sample - loss: -50.9159 - acc: 0.0000e+00 - val_loss: -52.6635 - val_acc: 0.0000e+00
    Epoch 50/100
    5900/5900 [==============================] - 2s 274us/sample - loss: -51.7035 - acc: 0.0000e+00 - val_loss: -53.2599 - val_acc: 0.0000e+00
    Epoch 51/100
    5900/5900 [==============================] - 2s 254us/sample - loss: -51.8638 - acc: 0.0000e+00 - val_loss: -53.0353 - val_acc: 0.0000e+00
    Epoch 52/100
    5900/5900 [==============================] - 1s 249us/sample - loss: -51.8713 - acc: 0.0000e+00 - val_loss: -53.3739 - val_acc: 0.0000e+00
    Epoch 53/100
    5900/5900 [==============================] - 1s 249us/sample - loss: -51.7521 - acc: 0.0000e+00 - val_loss: -53.0908 - val_acc: 0.0000e+00
    Epoch 54/100
    5900/5900 [==============================] - 1s 252us/sample - loss: -51.8559 - acc: 0.0000e+00 - val_loss: -53.2747 - val_acc: 0.0000e+00
    Epoch 55/100
    5900/5900 [==============================] - 1s 242us/sample - loss: -51.6408 - acc: 0.0000e+00 - val_loss: -53.3073 - val_acc: 0.0000e+00
    Epoch 56/100
    5900/5900 [==============================] - 1s 241us/sample - loss: -51.4734 - acc: 0.0000e+00 - val_loss: -52.8544 - val_acc: 0.0000e+00
    Epoch 57/100
    5900/5900 [==============================] - 1s 240us/sample - loss: -52.1706 - acc: 0.0000e+00 - val_loss: -53.4280 - val_acc: 0.0000e+00
    Epoch 58/100
    5900/5900 [==============================] - 1s 246us/sample - loss: -51.9305 - acc: 0.0000e+00 - val_loss: -53.2462 - val_acc: 0.0000e+00
    Epoch 59/100
    5900/5900 [==============================] - 1s 248us/sample - loss: -52.3968 - acc: 0.0000e+00 - val_loss: -53.7054 - val_acc: 0.0000e+00
    Epoch 60/100
    5900/5900 [==============================] - 1s 239us/sample - loss: -52.2760 - acc: 0.0000e+00 - val_loss: -54.0302 - val_acc: 0.0000e+00
    Epoch 61/100
    5900/5900 [==============================] - 1s 244us/sample - loss: -52.1973 - acc: 0.0000e+00 - val_loss: -53.8358 - val_acc: 0.0000e+00
    Epoch 62/100
    5900/5900 [==============================] - 1s 241us/sample - loss: -52.3488 - acc: 0.0000e+00 - val_loss: -53.6788 - val_acc: 0.0000e+00
    Epoch 63/100
    5900/5900 [==============================] - 1s 247us/sample - loss: -52.5307 - acc: 0.0000e+00 - val_loss: -53.9658 - val_acc: 0.0000e+00
    Epoch 64/100
    5900/5900 [==============================] - 1s 245us/sample - loss: -52.4804 - acc: 0.0000e+00 - val_loss: -53.3072 - val_acc: 0.0000e+00
    Epoch 65/100
    5900/5900 [==============================] - 1s 247us/sample - loss: -52.4243 - acc: 0.0000e+00 - val_loss: -53.0098 - val_acc: 0.0000e+00
    Epoch 66/100
    5900/5900 [==============================] - 1s 241us/sample - loss: -51.9397 - acc: 0.0000e+00 - val_loss: -53.9208 - val_acc: 0.0000e+00
    Epoch 67/100
    5900/5900 [==============================] - 1s 243us/sample - loss: -52.5599 - acc: 0.0000e+00 - val_loss: -50.3642 - val_acc: 0.0000e+00
    Epoch 68/100
    5900/5900 [==============================] - 1s 245us/sample - loss: -51.5617 - acc: 0.0000e+00 - val_loss: -53.9310 - val_acc: 0.0000e+00
    Epoch 69/100
    5900/5900 [==============================] - 2s 259us/sample - loss: -52.3065 - acc: 0.0000e+00 - val_loss: -53.8182 - val_acc: 0.0000e+00
    Epoch 70/100
    5900/5900 [==============================] - 2s 262us/sample - loss: -52.6629 - acc: 0.0000e+00 - val_loss: -54.1253 - val_acc: 0.0000e+00
    Epoch 71/100
    5900/5900 [==============================] - 1s 254us/sample - loss: -52.7865 - acc: 0.0000e+00 - val_loss: -54.2756 - val_acc: 0.0000e+00
    Epoch 72/100
    5900/5900 [==============================] - 2s 260us/sample - loss: -52.5579 - acc: 0.0000e+00 - val_loss: -53.9994 - val_acc: 0.0000e+00
    Epoch 73/100
    5900/5900 [==============================] - 1s 245us/sample - loss: -52.9169 - acc: 0.0000e+00 - val_loss: -54.3589 - val_acc: 0.0000e+00
    Epoch 74/100
    5900/5900 [==============================] - 1s 242us/sample - loss: -52.8026 - acc: 0.0000e+00 - val_loss: -53.9635 - val_acc: 0.0000e+00
    Epoch 75/100
    5900/5900 [==============================] - 1s 245us/sample - loss: -52.8044 - acc: 0.0000e+00 - val_loss: -54.1079 - val_acc: 0.0000e+00
    Epoch 76/100
    5900/5900 [==============================] - 1s 249us/sample - loss: -52.7159 - acc: 0.0000e+00 - val_loss: -53.7626 - val_acc: 0.0000e+00
    Epoch 77/100
    5900/5900 [==============================] - 2s 274us/sample - loss: -52.7891 - acc: 0.0000e+00 - val_loss: -54.0904 - val_acc: 0.0000e+00
    Epoch 78/100
    5900/5900 [==============================] - 2s 256us/sample - loss: -52.7985 - acc: 0.0000e+00 - val_loss: -53.5519 - val_acc: 0.0000e+00
    Epoch 79/100
    5900/5900 [==============================] - 2s 266us/sample - loss: -52.1951 - acc: 0.0000e+00 - val_loss: -53.9199 - val_acc: 0.0000e+00
    Epoch 80/100
    5900/5900 [==============================] - 1s 247us/sample - loss: -52.1961 - acc: 0.0000e+00 - val_loss: -52.0944 - val_acc: 0.0000e+00
    Epoch 81/100
    5900/5900 [==============================] - 1s 243us/sample - loss: -52.0280 - acc: 0.0000e+00 - val_loss: -53.7747 - val_acc: 0.0000e+00
    Epoch 82/100
    5900/5900 [==============================] - 1s 245us/sample - loss: -52.0588 - acc: 0.0000e+00 - val_loss: -54.2256 - val_acc: 0.0000e+00
    Epoch 83/100
    5900/5900 [==============================] - 1s 241us/sample - loss: -52.7455 - acc: 0.0000e+00 - val_loss: -54.4492 - val_acc: 0.0000e+00
    Epoch 84/100
    5900/5900 [==============================] - 1s 240us/sample - loss: -53.1167 - acc: 0.0000e+00 - val_loss: -54.3670 - val_acc: 0.0000e+00
    Epoch 85/100
    5900/5900 [==============================] - 1s 241us/sample - loss: -53.2219 - acc: 0.0000e+00 - val_loss: -54.7337 - val_acc: 0.0000e+00
    Epoch 86/100
    5900/5900 [==============================] - 1s 241us/sample - loss: -53.0747 - acc: 0.0000e+00 - val_loss: -54.1139 - val_acc: 0.0000e+00
    Epoch 87/100
    5900/5900 [==============================] - 1s 236us/sample - loss: -52.7974 - acc: 0.0000e+00 - val_loss: -53.7877 - val_acc: 0.0000e+00
    Epoch 88/100
    5900/5900 [==============================] - 1s 241us/sample - loss: -52.7835 - acc: 0.0000e+00 - val_loss: -54.3729 - val_acc: 0.0000e+00
    Epoch 89/100
    5900/5900 [==============================] - 1s 240us/sample - loss: -53.1602 - acc: 0.0000e+00 - val_loss: -54.4426 - val_acc: 0.0000e+00
    Epoch 90/100
    5900/5900 [==============================] - 1s 241us/sample - loss: -52.8173 - acc: 0.0000e+00 - val_loss: -54.2285 - val_acc: 0.0000e+00
    Epoch 91/100
    5900/5900 [==============================] - 1s 242us/sample - loss: -53.0386 - acc: 0.0000e+00 - val_loss: -54.5728 - val_acc: 0.0000e+00
    Epoch 92/100
    5900/5900 [==============================] - 1s 240us/sample - loss: -53.0664 - acc: 0.0000e+00 - val_loss: -54.1093 - val_acc: 0.0000e+00
    Epoch 93/100
    5900/5900 [==============================] - 1s 241us/sample - loss: -53.0088 - acc: 0.0000e+00 - val_loss: -54.5935 - val_acc: 0.0000e+00
    Epoch 94/100
    5900/5900 [==============================] - 1s 241us/sample - loss: -53.2231 - acc: 0.0000e+00 - val_loss: -53.9696 - val_acc: 0.0000e+00
    Epoch 95/100
    5900/5900 [==============================] - 1s 242us/sample - loss: -52.9255 - acc: 0.0000e+00 - val_loss: -54.6012 - val_acc: 0.0000e+00
    Epoch 96/100
    5900/5900 [==============================] - 1s 238us/sample - loss: -53.0851 - acc: 0.0000e+00 - val_loss: -54.5833 - val_acc: 0.0000e+00
    Epoch 97/100
    5900/5900 [==============================] - 2s 261us/sample - loss: -52.7857 - acc: 0.0000e+00 - val_loss: -52.5972 - val_acc: 0.0000e+00
    Epoch 98/100
    5900/5900 [==============================] - 2s 277us/sample - loss: -52.3286 - acc: 0.0000e+00 - val_loss: -54.1621 - val_acc: 0.0000e+00
    Epoch 99/100
    5900/5900 [==============================] - 2s 275us/sample - loss: -52.9432 - acc: 0.0000e+00 - val_loss: -54.4515 - val_acc: 0.0000e+00
    Epoch 100/100
    5900/5900 [==============================] - 2s 273us/sample - loss: -53.2856 - acc: 0.0000e+00 - val_loss: -54.7798 - val_acc: 0.0000e+00

### 2.3 For quadratic polynomials and two latent dimensions, visualise the results of your latent dimensions. Is an interpretation of your latent parameters easily visible?

```python
# CREATES 40X40 IMAGES OF POLYNOMIALS OF MAXIMUM DEGREE
x=np.linspace(-1,1,40)
xx,yy=np.meshgrid(x,x)

def random_poly(deg, xx=xx, yy=yy): #function that creates polinomial of 2 variables of given degree with no cross terms
    x_params=np.random.uniform(-5.,5., deg+1)
    y_params=np.random.uniform(-5.,5., deg+1)

    p=np.zeros((40,40))

    for i in range(deg+1):
        p+=x_params[i]*xx**i + y_params[i]*yy**i

    return p

# CREATES DATASET
size = 6000
data=[]

for i in range(size):
    degree=2
    data.append(random_poly(degree))
data=np.array(data)

plt.imshow(data[1298]) # shows element 1298 of the dataset
plt.show()

# Arranges dataset for training

data=data.reshape((len(data), np.prod(data.shape[1:])))
x_train=data[:5900]
x_test=data[5900:]
```

### * Add a custom loss function which de-correlates the latent parameters. Is it possible to find an interpretation for the latent parameters in this case?

## Ex.  3 KL – divergence

### Show by example that $D_{KL}(Q||P) \neq D_{KL}(P||Q)$ in general.

We can for example take $Q= \mathcal{G}(s,S)$ and $P= \mathcal{G}(s,S')$ where $\mathcal{G}(s,S)$ denotes the one-dimensional Gaussian with $0$ mean and variance $S$. Then

\begin{align}
D_{KL}(Q||P)=\int_\mathbb{R} G(\tilde{s},S)\ \ln\left[\frac{G(\tilde{s},S)}{G(\tilde{s},S')} \right] d\tilde{s} = \frac{1}{2} \int_\mathbb{R} \mathcal{G}(\tilde{s},S)\ \left(\ln\left[\frac{S}{S'}\right]-\frac{\tilde{s}^2}{S}+\frac{\tilde{s}^2}{S'} \right) d\tilde{s} = \frac{1}{2}\left(\ln\left[\frac{S}{S'}\right] + \frac{S}{S'} - 1\right)
\end{align}

Exchanging $\ Q \leftrightarrow P$ it becomes clear that

\begin{equation}
    D_{KL}(P||Q) = \frac{1}{2}\left(\ln\left[\frac{S'}{S}\right] + \frac{S'}{S} - 1\right) \neq \frac{1}{2}\left(\ln\left[\frac{S}{S'}\right] + \frac{S}{S'} - 1\right) = D_{KL}(P||Q)
\end{equation}

and since in principle $S \neq S'$, for generic $P$s and $Q$s $\boxed{D_{𝐾𝐿}(P||Q)\neq 𝐷_{𝐾𝐿}(Q||P)}.$


# Initial-Value Problems

This section focuses on analytical solutions for initial-value problems, meaning problems where we know the values of $y(t)$ and $\frac{dy}{dt}$ at $t=0$ (or $y(x)$ and $\frac{dy}{dx}$ at $x=0$): $y(0)$ and $y^{\prime}(0)$.

## Equations with constant coefficents

A common category of 2nd-order homogeneous ODEs are equations with constant coefficients, of the form:
\begin{equation}
y^{\prime\prime} + a y^{\prime} + by = 0
\end{equation}
Note that these are unforced, and the right-hand side is zero.

Solutions to these equations take the form $y(x) = e^{\lambda x}$, and inserting this into the ODE gives us the characteristic equation
\begin{equation}
\lambda^2 + a \lambda + b = 0
\end{equation}
which we can solve to find the solution for given coefficients $a$ and $b$ and initial conditions. Depending on those coefficients and the solution to the characteristic equation, our solution can fall into one of three cases:

* Real roots: $\lambda_1$ and $\lambda_2$. This is an **overdamped** system and the full solution takes the form
\begin{equation}
y(x) = c_1 e^{\lambda_1 x} + c_2 e^{\lambda_2 x}
\end{equation}

* Repeated roots: $\lambda_1 = \lambda_2 = \lambda$. This is a **critically damped** system and the full solution is
\begin{equation}
y(x) = c_1 e^{\lambda x} + c_2 x e^{\lambda x}
\end{equation}
(Where does that second part come from, you might ask? Well, we know that $y_1$ is $e^{\lambda x}$, but the second part cannot also be $e^{\lambda x}$ because those are linearly dependent. So, we use *reduction of order* to find $y_2$, which is $x e^{\lambda x}$.

* Imaginary roots: $\lambda = \frac{-a}{2} \pm \beta i$, where $\beta = \frac{1}{2} \sqrt{4b - a^2}$. This is an **underdamped** system and the solution takes the form
\begin{equation}
y(x) = e^{-ax/2} \left( c_1 \sin \beta x + c_2 \cos \beta x \right)
\end{equation}

Some examples:

1. $y^{\prime\prime} + 3 y^{\prime} + 2y = 0$
\begin{align}
\rightarrow \lambda^2 + 3\lambda + 2 &= 0 \\
(\lambda + 2)(\lambda + 1) &= 0 \\
\lambda &= -2, -1 \\
y(x) &= c_1 e^{-x} + c_2 e^{-2x}
\end{align}
Then, we would use the initial conditions given for $y(0)$ and $y^{\prime}(0)$ to find $c_1$ and $c_2$.

2. $y^{\prime\prime} + 6 y^{\prime} + 9y = 0$
\begin{align}
\rightarrow \lambda^2 + 6\lambda + 9 &= 0 \\
(\lambda + 3)(\lambda + 3) &= 0 \\
\lambda &= -3 \\
y(x) &= c_1 e^{-3x} + c_2 x e^{-3x}
\end{align}

3. $y^{\prime\prime} + 6 y^{\prime} + 25 y = 0$
\begin{align}
\rightarrow \lambda^2 + 6\lambda + 25 &= 0 \\
\lambda &= -3 \pm 4i \\
y(x) &= e^{-3x} \left( c_1 \sin 4x + c_2 \cos 4x \right)
\end{align}

## Euler-Cauchy equations

Euler-Cauchy equations are of the form
\begin{equation}
x^2 y^{\prime\prime} + axy^{\prime} + by = 0
\end{equation}

Solutions take the form $y = x^m$, which when plugged into the ODE leads to a different characterisic equation to find $m$:
\begin{align}
y &= x^m \\
y^{\prime} &= m x^{m-1} \\
y^{\prime\prime} &= m (m-1) x^{m-2} \\
\rightarrow x^2 m (m-1) x^{m-2} + axmx^{m-1} + bx^m &= 0 \\
m^2 + (a-1)m + b &= 0
\end{align}
This is our new characteristic formula for these problems, and solving for the roots of this equation gives us $m$ and thus our general solution.

Like equations with constant coefficients, we have three solution forms depending on the roots of the characteristic equation:

* Real roots: $y(x) = c_1 x^{m_1} + c_2 x^{m_2}$
* Repeated roots: $y(x) = c_1 x^m + c_2 x^m \ln x$
* Imaginary roots: $m = \alpha \pm \beta i$, and $y(x) = x^{\alpha} \left[c_1 \cos (\beta \ln x) + c_2 \sin (\beta \ln x)\right]$

## Inhomogeneous 2nd-order ODEs

Inhomogeneous, or forced, 2nd-order ODEs with constant coefficients take the form
\begin{equation}
y^{\prime\prime} + a y^{\prime} + by = F(t)
\end{equation}
with initial conditions $y(0) = y_0$ and $y^{\prime}(0) = y_0^{\prime}$. Depending on the form of the forcing function $F(t)$, we can solve with techniques such as

 - the method of undetermined coefficients
 - variation of parameters
 - LaPlace transforms

The solution in general to inhomogeneous ODEs includes two parts:
\begin{equation}
y(t) = y_{\text{H}} + y_{\text{IH}} = c_1 y_1 + c_2 y_2 + y_{\text{IH}} \;,
\end{equation}
where $y_{\text{H}}$ is the solution from the equivalent homogeneous ODE $y^{\prime\prime} + a y^{\prime} + b y = 0$.

The forcing function $F(t)$ may be

 - continuous
 - periodic
 - aperiodic/discontinuous

### Continuous $F(t)$: method of undetermined coefficients

For continuous forcing functions, we have two solution methods: the method of undetermined coefficients, and variation of parameters.

Generally you'll want to use the method of undetermined coefficients when possible, which depends on if $F(t)$ matches one of a set of functions. In that case, the form of the inhomogeneous solution $y_{\text{IH}}(t)$ follows that of the forcing function $F(t)$, with one or more unknown constants:

| $F(t)$        | $y_{\text{IH}}(t)$ |
| ------------- | :-------:|
| constant      | $K$ |
| $\cos \omega t$ | $K_1 \cos \omega t + K_2 \sin \omega t$ |
| $\sin \omega t$ | $K_1 \cos \omega t + K_2 \sin \omega t$ |
| $e^{-at}$ | $K e^{-at}$   |
| $(A) t$   | $K_0 + K_1 t$ |
| $t^n$     | $K_0 + K_1 t + K_2 t^2 + \ldots + K_n t^n$|

For combinations of these functions, we can combine functions; for example, given
\begin{align}
F(t) &= e^{-at} \cos \omega t \quad \text{or} e^{-at} \sin \omega t \\
y_{\text{IH}} &= K_1 e^{-at} \cos \omega t + K_2 e^{-at} \sin \omega t
\end{align}
(Note how in all the above cases how the inhomogeneous solution follows the functional form of the forcing function; for example, the exponential decay rate $a$ or the sinusoidal frequency $\omega$ match.

The method of undetermined coefficients works by plugging the candidate inhomogeneous solutionn $y_{\text{IH}}$ into the full ODE, and solving for the constants (e.g., $K$)—but **not** from the initial conditions.

For example, let's solve
\begin{equation}
y^{\prime\prime} + 2y^{\prime} + y = e^{-x}
\end{equation}
with initial conditions $y(0) = y^{\prime}(0) = 0$. First, we should find the solution to the homogeneous equation
\begin{equation}
y^{\prime\prime} + 2y^{\prime} + y = 0 \;.
\end{equation}
We can do this by using the associated characteristic formula
\begin{align}
\lambda^2 + 2 \lambda + 1 &= 0 \\
(\lambda + 1)(\lambda + 1) &= 0 \\
\rightarrow y_{\text{H}} &= c_1 e^{-x} + c_2 x e^{-x}
\end{align}

To find the inhomogeneous solution, we would look at the table above to find what matches the forcing function $e^{-x}$. Normally, we'd grab $K e^{-x}$, but that would not be linearly independent from the first part of the homogeneous solution $y_{\text{H}}$. The same is true for $K x e^{-x}$, which is linearly dependent with the second part of $y_{\text{H}}$, but $K x^2 e^{-x}$ works! Then, we just need to find $K$ by plugging this into the ODE:
\begin{align}
y_{\text{IH}} &= K x^2 e^{-x} \\
y^{\prime} &= K e^{-x} (2x - x^2) \\
y^{\prime\prime} &= K e^{-x} (x^2 - 4x + 2) \\
2 K &= 1 \\
\rightarrow K &= \frac{1}{2} \\
y_{\text{IH}} &= \frac{1}{2} x^2 e^{-x}
\end{align}
Thus, the overall general solution is
\begin{equation}
y(x) = c_1 e^{-x} + c_2 x e^{-x} + \frac{1}{2} x^2 e^{-x}
\end{equation}
and we would solve for the integration constants $c_1$ and $c_2$ using the initial conditions.

Important points to remember:

- The constants of the inhomogeneous solution $y_{\text{IH}}$ come from the ODE, **not** the initial conditions.
- Only solve for the integration constants $c_1$ and $c_2$ (part of the homogeneous solution) once you have the full general solution $y = c_1 y_1 + c_2 y_2 + y_{\text{IH}}$.

### Continuous $F(t)$: variation of parameters

We have the variation of parameters approach to solve for inhomogeneous 2nd-order ODEs that are more general:
\begin{equation}
y^{\prime\prime} + p(x) y^{\prime} + q(x) y = r(x)
\end{equation}
In this case, we can assume a solution $y(x) = y_1 u_1 + y_2 u_2$.

The solution procedure is:

1. Obtain $y_1$ and $y_2$ by solving the homogeneous equation: $y^{\prime\prime} + p(x) y^{\prime} + q(x) y = 0$

2. Solve for $u_1$ and $u_2$:
\begin{align}
u_1 &= - \int \frac{y_2 r(x)}{W} dx + c_1 \\
u_2 &= \int \frac{y_1 r(x)}{W} dx + c_2 \\
W &= \begin{vmatrix}
y_1 & y_2\\ y_1^{\prime} & y_2^{\prime}\\
\end{vmatrix} = y_1 y_2^{\prime} - y_2 y_1^{\prime} \;,
\end{align}
where $W$ is the Wronksian.

3. Then, we have the general solution:
\begin{align}
y &= u_1 y_1 + u_2 y_2 \\
&= \left( -\int \frac{y_2 r(x)}{W} dx + c_1 \right) y_1 + \left( \int \frac{y_1 r(x)}{W} dx + c_2 \right) y_2 \;,
\end{align}
where we solve for $c_1$ and $c_2$ using the two initial conditions.

#### Example 1: variation of parameters

First, let's try the same example we used for the method of undetermined coefficients above:
\begin{equation}
y^{\prime\prime} + 2 y^{\prime} + y = e^{-x}
\end{equation}
We already found the homogeneous solution, so we know that $y_1 = e^{-x}$ and $y_2 = x e^{-x}$.
Next, let's get the Wronksian, and then $u_1$ and $u_2$.
\begin{align}
W &= \begin{vmatrix} y_1 & y_2 \\ y_1^{\prime} & y_2^{\prime} \end{vmatrix} = e^{-x} e^{-x}(1-x) - x e^{-x} (-e^{-x}) = e^{-2x} \\
%
u_1 &= -\int \frac{x e^{-x} e^{-x}}{e^{-2x}} dx + c_1 = -\int x dx + c_1 = -\frac{1}{2} x^2 + c_1 \\
u_2 &= \int \frac{e^{-x} e^{-x}}{e^{-2x}} dx + c_2 = \int dx + c_2 = x + c_2 \\
y(x) &= \left(-\frac{1}{2} x^2 + c_1\right) e^{-x} + (x + c_2) x e^{-x} \\
\end{align}
After simplifying, we obtain the same solution as via the method of undetermined coefficients (but with a bit more work):
\begin{equation}
y(x) = x_1 e^{-x} + c_2 x e^{-x} + \frac{1}{2} x^2 e^{-x}
\end{equation}

#### Example 2: variation of parameters

Now let's try an example that we could *not* solve using the method of undetermined coefficients, with a forcing term that involves hyperbolic cosine (cosh); recall that $\cosh(x) = \frac{e^x + e^{-x}}{2}$.
\begin{equation}
y^{\prime\prime} + 4 y^{\prime} + 4y = \cosh(x)
\end{equation}
First, we need to find the homogeneous solution:
\begin{align}
y^{\prime\prime} + 4 y^{\prime} + 4y &= 0 \\
\lambda^2 + 4 \lambda + 4 &= 0 \\
\rightarrow \lambda &= -2
\end{align}
So our homogeneous solution involves repeated roots:
\begin{equation}
y_H = c_1 e^{-2x} + c_2 x e^{-2x}
\end{equation}
where $y_1 = e^{-2x}$ and $y_2 = x e^{-2x}$.

Then, we need to find $u_1$ and $u_2$, so let's get the Wronksian and then solve
\begin{align}
W &= \begin{vmatrix} y_1 & y_2 \\ y_1^{\prime} & y_2^{\prime} \end{vmatrix} = e^{-2x} (e^{-2x}) (1 - 2x) - x e^{-2x}(-2 e^{-2x}) = e^{-4x} \\
%
u_1 &= - \int \frac{x e^{-2x} \cosh x}{e^{-4x}} dx + c_1 = -\int \frac{x \frac{1}{2}(e^x + e^{-x})}{e^{-2x}} dx + c_1 \\
 &= -\frac{1}{2} \int x (e^{3x} + e^x) dx + c_1 = -\frac{1}{2} \left[ \frac{1}{9} e^{3x}(3x-1) + e^x(x-1) \right] + c_1 \\
u_1 &= -\frac{1}{18} e^{3x}(3x-1) - \frac{1}{2} e^x (x-1) + c_1 \\
%
u_2 &= \int \frac{e^{-2x} \cosh x}{e^{-4x}} dx + c_2 = \frac{1}{2} \int e^{2x}(e^x + e^{-x}) dx + c_2 = \frac{1}{2} \int (e^{3x} + e^x) dx + c_2 \\
u_2 &= \frac{1}{6} e^{3x} + \frac{1}{2} e^x + c_2
\end{align}

Then, when we put these all together, we get the full (complicated) solution:
\begin{equation}
y(x) = \left[ -\frac{1}{18} e^{3x} (3x-1) - \frac{1}{2} e^x (x-1) + c_1 \right] e^{-2x} + \left( \frac{1}{6} e^{3x} + \frac{1}{2} e^x + c_2 \right) x e^{-2x}
\end{equation}

```matlab

```