Table of Contents

1  Transformations, Eigenvectors, and Eigenvalues

1.1  Linear Transformations

1.2  Transformations of Magnitude and Amplitude

1.2.1  Afine Transformations

1.3  Eigenvectors and Eigenvalues

1.4  Eigendecomposition

1.5  Rank of a Matrix

1.6  Inverse of a Square Full Rank Matrix

# Transformations, Eigenvectors, and Eigenvalues

Matrices and vectors are used together to manipulate spatial dimensions. This has a lot of applications, including the mathematical generation of 3D computer graphics, geometric modeling, and the training and optimization of machine learning algorithms. We're not going to cover the subject exhaustively here; but we'll focus on a few key concepts that are useful to know when you plan to work with machine learning.

## Linear Transformations
You can manipulate a vector by multiplying it with a matrix. The matrix acts a function that operates on an input vector to produce a vector output. Specifically, matrix multiplications of vectors are *linear transformations* that transform the input vector into the output vector.

For example, consider this matrix ***A*** and vector ***v***:

$$ A = \begin{bmatrix}2 & 3\\5 & 2\end{bmatrix} \;\;\;\; \vec{v} = \begin{bmatrix}1\\2\end{bmatrix}$$

We can define a transformation ***T*** like this:

$$ T(\vec{v}) = A\vec{v} $$

To perform this transformation, we simply calculate the dot product by applying the *RC* rule; multiplying each row of the matrix by the single column of the vector:

$$\begin{bmatrix}2 & 3\\5 & 2\end{bmatrix} \cdot  \begin{bmatrix}1\\2\end{bmatrix} = \begin{bmatrix}8\\9\end{bmatrix}$$

Here's the calculation in Python:

```python
import numpy as np

v = np.array([1,2])
A = np.array([[2,3],
              [5,2]])

t = A@v
print (t)
```

    [8 9]

In this case, both the input vector and the output vector have 2 components - in other words, the transformation takes a 2-dimensional vector and produces a new 2-dimensional vector; which we can indicate like this:

$$ T: \rm I\!R^{2} \to \rm I\!R^{2} $$

Note that the output vector may have a different number of dimensions from the input vector; so the matrix function might transform the vector from one space to another - or in notation, ${\rm I\!R}$n -> ${\rm I\!R}$m.

For example, let's redefine matrix ***A***, while retaining our original definition of vector ***v***:

$$ A = \begin{bmatrix}2 & 3\\5 & 2\\1 & 1\end{bmatrix} \;\;\;\; \vec{v} = \begin{bmatrix}1\\2\end{bmatrix}$$

Now if we once again define ***T*** like this:

$$ T(\vec{v}) = A\vec{v} $$

We apply the transformation like this:

$$\begin{bmatrix}2 & 3\\5 & 2\\1 & 1\end{bmatrix} \cdot  \begin{bmatrix}1\\2\end{bmatrix} = \begin{bmatrix}8\\9\\3\end{bmatrix}$$

So now, our transformation transforms the vector from 2-dimensional space to 3-dimensional space:

$$ T: \rm I\!R^{2} \to \rm I\!R^{3} $$

Here it is in Python:

```python
import numpy as np
v = np.array([1,2])
A = np.array([[2,3],
              [5,2],
              [1,1]])

t = A@v
print (t)
```

    [8 9 3]

```python
import numpy as np
v = np.array([1,2])
A = np.array([[1,2],
              [2,1]])

t = A@v
print (t)
```

    [5 4]

## Transformations of Magnitude and Amplitude

When you multiply a vector by a matrix, you transform it in at least one of the following two ways:
* Scale the length (*magnitude*) of the matrix to make it longer or shorter
* Change the direction (*amplitude*) of the matrix

For example consider the following matrix and vector:

$$ A = \begin{bmatrix}2 & 0\\0 & 2\end{bmatrix} \;\;\;\; \vec{v} = \begin{bmatrix}1\\0\end{bmatrix}$$

As before, we transform the vector ***v*** by multiplying it with the matrix ***A***:

\begin{equation}\begin{bmatrix}2 & 0\\0 & 2\end{bmatrix} \cdot  \begin{bmatrix}1\\0\end{bmatrix} = \begin{bmatrix}2\\0\end{bmatrix}\end{equation}

In this case, the resulting vector has changed in length (*magnitude*), but has not changed its direction (*amplitude*).

Let's visualize that in Python:

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

v = np.array([1,0])
A = np.array([[2,0],
              [0,2]])

t = A@v
print (t)

# Plot v and t
vecs = np.array([t,v])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['blue', 'orange'], scale=10)
plt.show()
```

The original vector ***v*** is shown in orange, and the transformed vector ***t*** is shown in blue - note that ***t*** has the same direction (*amplitude*) as ***v*** but a greater length (*magnitude*).

Now let's use a different matrix to transform the vector ***v***:
\begin{equation}\begin{bmatrix}0 & -1\\1 & 0\end{bmatrix} \cdot  \begin{bmatrix}1\\0\end{bmatrix} = \begin{bmatrix}0\\1\end{bmatrix}\end{equation}

This time, the resulting vector has been changed to a different amplitude, but has the same magnitude.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

v = np.array([1,0])
A = np.array([[0,-1],
              [1,0]])

t = A@v
print (t)

# Plot v and t
vecs = np.array([v,t])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['orange', 'blue'], scale=10)
plt.show()
```

Now let's see change the matrix one more time:
\begin{equation}\begin{bmatrix}2 & 1\\1 & 2\end{bmatrix} \cdot  \begin{bmatrix}1\\0\end{bmatrix} = \begin{bmatrix}2\\1\end{bmatrix}\end{equation}

Now our resulting vector has been transformed to a new amplitude *and* magnitude - the transformation has affected both direction and scale.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

v = np.array([1,0])
A = np.array([[2,1],
              [1,2]])

t = A@v
print (t)

# Plot v and t
vecs = np.array([v,t])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['orange', 'blue'], scale=10)
plt.show()
```

### Afine Transformations
An Afine transformation multiplies a vector by a matrix and adds an offset vector, sometimes referred to as *bias*; like this:

$$T(\vec{v}) = A\vec{v} + \vec{b}$$

For example:

\begin{equation}\begin{bmatrix}5 & 2\\3 & 1\end{bmatrix} \cdot  \begin{bmatrix}1\\1\end{bmatrix} + \begin{bmatrix}-2\\-6\end{bmatrix} = \begin{bmatrix}5\\-2\end{bmatrix}\end{equation}

This kind of transformation is actually the basis of linear regression, which is a core foundation for machine learning. The matrix defines the *features*, the first vector is the *coefficients*, and the bias vector is the *intercept*.

here's an example of an Afine transformation in Python:

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

v = np.array([1,1])
A = np.array([[5,2],
              [3,1]])
b = np.array([-2,-6])

t = A@v + b
print (t)

# Plot v and t
vecs = np.array([v,t])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['orange', 'blue'], scale=15)
plt.show()
```

## Eigenvectors and Eigenvalues
So we can see that when you transform a vector using a matrix, we change its direction, length, or both. When the transformation only affects scale (in other words, the output vector has a different magnitude but the same amplitude as the input vector), the matrix multiplication for the transformation is the equivalent operation as some scalar multiplication of the vector.

For example, earlier we examined the following transformation that dot-mulitplies a vector by a matrix:

$$\begin{bmatrix}2 & 0\\0 & 2\end{bmatrix} \cdot  \begin{bmatrix}1\\0\end{bmatrix} = \begin{bmatrix}2\\0\end{bmatrix}$$

You can achieve the same result by mulitplying the vector by the scalar value ***2***:

$$2 \times \begin{bmatrix}1\\0\end{bmatrix} = \begin{bmatrix}2\\0\end{bmatrix}$$

The following python performs both of these calculation and shows the results, which are identical.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

v = np.array([1,0])
A = np.array([[2,0],
              [0,2]])

t1 = A@v
print (t1)
t2 = 2*v
print (t2)

fig = plt.figure()
a=fig.add_subplot(1,1,1)
# Plot v and t1
vecs = np.array([t1,v])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['blue', 'orange'], scale=10)
plt.show()
a=fig.add_subplot(1,2,1)
# Plot v and t2
vecs = np.array([t2,v])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['blue', 'orange'], scale=10)
plt.show()
```

In cases like these, where a matrix transformation is the equivelent of a scalar-vector multiplication, the scalar-vector pairs that correspond to the matrix are known respectively as eigenvalues and eigenvectors. We generally indicate eigenvalues using the Greek letter lambda (λ), and the formula that defines eigenvalues and eigenvectors with respect to a transformation is:

$$ T(\vec{v}) = \lambda\vec{v}$$

Where the vector ***v*** is an eigenvector and the value ***λ*** is an eigenvalue for transformation ***T***.

When the transformation ***T*** is represented as a matrix multiplication, as in this case where the transformation is represented by matrix ***A***:

$$ T(\vec{v}) = A\vec{v} = \lambda\vec{v}$$

Then  ***v*** is an eigenvector and ***λ*** is an eigenvalue of ***A***.

A matrix can have multiple eigenvector-eigenvalue pairs, and you can calculate them manually. However, it's generally easier to use a tool or programming language. For example, in Python you can use the ***linalg.eig*** function, which returns an array of eigenvalues and a matrix of the corresponding eigenvectors for the specified matrix.

Here's an example that returns the eigenvalue and eigenvector pairs for the following matrix:

$$A=\begin{bmatrix}2 & 0\\0 & 3\end{bmatrix}$$

```python
import numpy as np
A = np.array([[2,0],
              [0,3]])
eVals, eVecs = np.linalg.eig(A)
print(eVals)
print(eVecs)
```

    [2. 3.]
    [[1. 0.]
     [0. 1.]]

So there are two eigenvalue-eigenvector pairs for this matrix, as shown here:

$$ \lambda_{1} = 2, \vec{v_{1}} = \begin{bmatrix}1 \\ 0\end{bmatrix}  \;\;\;\;\;\; \lambda_{2} = 3, \vec{v_{2}} = \begin{bmatrix}0 \\ 1\end{bmatrix} $$

Let's verify that multiplying each eigenvalue-eigenvector pair corresponds to the dot-product of the eigenvector and the matrix. Here's the first pair:

$$ 2 \times \begin{bmatrix}1 \\ 0\end{bmatrix} = \begin{bmatrix}2 \\ 0\end{bmatrix}  \;\;\;and\;\;\; \begin{bmatrix}2 & 0\\0 & 3\end{bmatrix} \cdot \begin{bmatrix}1 \\ 0\end{bmatrix} = \begin{bmatrix}2 \\ 0\end{bmatrix} $$

So far so good. Now let's check the second pair:

$$ 3 \times \begin{bmatrix}0 \\ 1\end{bmatrix} = \begin{bmatrix}0 \\ 3\end{bmatrix}  \;\;\;and\;\;\; \begin{bmatrix}2 & 0\\0 & 3\end{bmatrix} \cdot \begin{bmatrix}0 \\ 1\end{bmatrix} = \begin{bmatrix}0 \\ 3\end{bmatrix} $$

So our eigenvalue-eigenvector scalar multiplications do indeed correspond to our matrix-eigenvector dot-product transformations.

Here's the equivalent code in Python, using the ***eVals*** and ***eVecs*** variables you generated in the previous code cell:

```python
vec1 = eVecs[:,0]
lam1 = eVals[0]

print('Matrix A:')
print(A)
print('-------')

print('lam1: ' + str(lam1))
print ('v1: ' + str(vec1))
print ('Av1: ' + str(A@vec1))
print ('lam1 x v1: ' + str(lam1*vec1))

print('-------')

vec2 = eVecs[:,1]
lam2 = eVals[1]

print('lam2: ' + str(lam2))
print ('v2: ' + str(vec2))
print ('Av2: ' + str(A@vec2))
print ('lam2 x v2: ' + str(lam2*vec2))
```

    Matrix A:
    [[2 0]
     [0 3]]
    -------
    lam1: 2.0
    v1: [1. 0.]
    Av1: [2. 0.]
    lam1 x v1: [2. 0.]
    -------
    lam2: 3.0
    v2: [0. 1.]
    Av2: [0. 3.]
    lam2 x v2: [0. 3.]

You can use the following code to visualize these transformations:

```python
t1 = lam1*vec1
print (t1)
t2 = lam2*vec2
print (t2)

fig = plt.figure()
a=fig.add_subplot(1,1,1)
# Plot v and t1
vecs = np.array([t1,vec1])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['blue', 'orange'], scale=10)
plt.show()
a=fig.add_subplot(1,2,1)
# Plot v and t2
vecs = np.array([t2,vec2])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['blue', 'orange'], scale=10)
plt.show()
```

Similarly, earlier we examined the following matrix transformation:

$$\begin{bmatrix}2 & 0\\0 & 2\end{bmatrix} \cdot  \begin{bmatrix}1\\0\end{bmatrix} = \begin{bmatrix}2\\0\end{bmatrix}$$

And we saw that you can achieve the same result by mulitplying the vector by the scalar value ***2***:

$$2 \times \begin{bmatrix}1\\0\end{bmatrix} = \begin{bmatrix}2\\0\end{bmatrix}$$

This works because the scalar value 2 and the vector (1,0) are an eigenvalue-eigenvector pair for this matrix.

Let's use Python to determine the eigenvalue-eigenvector pairs for this matrix:

```python
import numpy as np
A = np.array([[2,0],
              [0,2]])
eVals, eVecs = np.linalg.eig(A)
print(eVals)
print(eVecs)
```

    [2. 2.]
    [[1. 0.]
     [0. 1.]]

So once again, there are two eigenvalue-eigenvector pairs for this matrix, as shown here:

$$ \lambda_{1} = 2, \vec{v_{1}} = \begin{bmatrix}1 \\ 0\end{bmatrix}  \;\;\;\;\;\; \lambda_{2} = 2, \vec{v_{2}} = \begin{bmatrix}0 \\ 1\end{bmatrix} $$

Let's verify that multiplying each eigenvalue-eigenvector pair corresponds to the dot-product of the eigenvector and the matrix. Here's the first pair:

$$ 2 \times \begin{bmatrix}1 \\ 0\end{bmatrix} = \begin{bmatrix}2 \\ 0\end{bmatrix}  \;\;\;and\;\;\; \begin{bmatrix}2 & 0\\0 & 2\end{bmatrix} \cdot \begin{bmatrix}1 \\ 0\end{bmatrix} = \begin{bmatrix}2 \\ 0\end{bmatrix} $$

Well, we already knew that. Now let's check the second pair:

$$ 2 \times \begin{bmatrix}0 \\ 1\end{bmatrix} = \begin{bmatrix}0 \\ 2\end{bmatrix}  \;\;\;and\;\;\; \begin{bmatrix}2 & 0\\0 & 2\end{bmatrix} \cdot \begin{bmatrix}0 \\ 1\end{bmatrix} = \begin{bmatrix}0 \\ 2\end{bmatrix} $$

Now let's use Pythonto verify and plot these transformations:

```python
vec1 = eVecs[:,0]
lam1 = eVals[0]

print('Matrix A:')
print(A)
print('-------')

print('lam1: ' + str(lam1))
print ('v1: ' + str(vec1))
print ('Av1: ' + str(A@vec1))
print ('lam1 x v1: ' + str(lam1*vec1))

print('-------')

vec2 = eVecs[:,1]
lam2 = eVals[1]

print('lam2: ' + str(lam2))
print ('v2: ' + str(vec2))
print ('Av2: ' + str(A@vec2))
print ('lam2 x v2: ' + str(lam2*vec2))

# Plot the resulting vectors
t1 = lam1*vec1
t2 = lam2*vec2

fig = plt.figure()
a=fig.add_subplot(1,1,1)
# Plot v and t1
vecs = np.array([t1,vec1])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['blue', 'orange'], scale=10)
plt.show()
a=fig.add_subplot(1,2,1)
# Plot v and t2
vecs = np.array([t2,vec2])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['blue', 'orange'], scale=10)
plt.show()
```

Let's take a look at one more, slightly more complex example. Here's our matrix:

$$\begin{bmatrix}2 & 1\\1 & 2\end{bmatrix}$$

Let's get the eigenvalue and eigenvector pairs:

```python
import numpy as np

A = np.array([[2,1],
              [1,2]])

eVals, eVecs = np.linalg.eig(A)
print(eVals)
print(eVecs)
```

    [3. 1.]
    [[ 0.70710678 -0.70710678]
     [ 0.70710678  0.70710678]]

This time the eigenvalue-eigenvector pairs are:

$$ \lambda_{1} = 3, \vec{v_{1}} = \begin{bmatrix}0.70710678 \\ 0.70710678\end{bmatrix}  \;\;\;\;\;\; \lambda_{2} = 1, \vec{v_{2}} = \begin{bmatrix}-0.70710678 \\ 0.70710678\end{bmatrix} $$

So let's check the first pair:

$$ 3 \times \begin{bmatrix}0.70710678 \\ 0.70710678\end{bmatrix} = \begin{bmatrix}2.12132034 \\ 2.12132034\end{bmatrix}  \;\;\;and\;\;\; \begin{bmatrix}2 & 1\\0 & 2\end{bmatrix} \cdot \begin{bmatrix}0.70710678 \\ 0.70710678\end{bmatrix} = \begin{bmatrix}2.12132034 \\ 2.12132034\end{bmatrix} $$

Now let's check the second pair:

$$ 1 \times \begin{bmatrix}-0.70710678 \\ 0.70710678\end{bmatrix} = \begin{bmatrix}-0.70710678\\0.70710678\end{bmatrix}  \;\;\;and\;\;\; \begin{bmatrix}2 & 1\\1 & 2\end{bmatrix} \cdot \begin{bmatrix}-0.70710678 \\ 0.70710678\end{bmatrix} = \begin{bmatrix}-0.70710678\\0.70710678\end{bmatrix} $$

With more complex examples like this, it's generally easier to do it with Python:

```python
vec1 = eVecs[:,0]
lam1 = eVals[0]

print('Matrix A:')
print(A)
print('-------')

print('lam1: ' + str(lam1))
print ('v1: ' + str(vec1))
print ('Av1: ' + str(A@vec1))
print ('lam1 x v1: ' + str(lam1*vec1))

print('-------')

vec2 = eVecs[:,1]
lam2 = eVals[1]

print('lam2: ' + str(lam2))
print ('v2: ' + str(vec2))
print ('Av2: ' + str(A@vec2))
print ('lam2 x v2: ' + str(lam2*vec2))

# Plot the results
t1 = lam1*vec1
t2 = lam2*vec2

fig = plt.figure()
a=fig.add_subplot(1,1,1)
# Plot v and t1
vecs = np.array([t1,vec1])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['blue', 'orange'], scale=10)
plt.show()
a=fig.add_subplot(1,2,1)
# Plot v and t2
vecs = np.array([t2,vec2])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['blue', 'orange'], scale=10)
plt.show()
```

## Eigendecomposition
So we've learned a little about eigenvalues and eigenvectors; but you may be wondering what use they are. Well, one use for them is to help decompose transformation matrices.

Recall that previously we found that a matrix transformation of a vector changes its magnitude, amplitude, or both. Without getting too technical about it, we need to remember that vectors can exist in any spatial orientation, or *basis*; and the same transformation can be applied in different *bases*.

We can decompose a matrix using the following formula:

$$A = Q \Lambda Q^{-1}$$

Where ***A*** is a trasformation that can be applied to a vector in its current base, ***Q*** is a matrix of eigenvectors that defines a change of basis, and ***Λ*** is a matrix with eigenvalues on the diagonal that defines the same linear transformation as ***A*** in the base defined by ***Q***.

Let's look at these in some more detail. Consider this matrix:

$$A=\begin{bmatrix}3 & 2\\1 & 0\end{bmatrix}$$

***Q*** is a matrix in which each column is an eigenvector of ***A***; which as we've seen previously, we can calculate using Python:

```python
import numpy as np

A = np.array([[3,2],
              [1,0]])

l, Q = np.linalg.eig(A)
print(Q)
```

    [[ 0.96276969 -0.48963374]
     [ 0.27032301  0.87192821]]

```python
importer numpy comme np

A = np.array ([[3,2],
              [1,0]])

l, Q = np.linalg.eig (A)
imprimer (Q)
```

So for matrix ***A***, ***Q*** is the following matrix:

$$Q=\begin{bmatrix}0.96276969 & -0.48963374\\0.27032301 & 0.87192821\end{bmatrix}$$

***Λ*** is a matrix that contains the eigenvalues for ***A*** on the diagonal, with zeros in all other elements; so for a 2x2 matrix, Λ will look like this:

$$\Lambda=\begin{bmatrix}\lambda_{1} & 0\\0 & \lambda_{2}\end{bmatrix}$$

In our Python code, we've already used the ***linalg.eig*** function to return the array of eigenvalues for ***A*** into the variable ***l***, so now we just need to format that as a matrix:

```python
L = np.diag(l)
print (L)
```

So ***Λ*** is the following matrix:

$$\Lambda=\begin{bmatrix}3.56155281 & 0\\0 & -0.56155281\end{bmatrix}$$

Now we just need to find ***Q-1***, which is the inverse of ***Q***:

```python
Qinv = np.linalg.inv(Q)
print(Qinv)
```

The inverse of ***Q*** then, is:

$$Q^{-1}=\begin{bmatrix}0.89720673 & 0.50382896\\-0.27816009 & 0.99068183\end{bmatrix}$$

So what does that mean? Well, it means that we can decompose the transformation of *any* vector multiplied by matrix ***A*** into the separate operations ***QΛQ-1***:

$$A\vec{v} = Q \Lambda Q^{-1}\vec{v}$$

To prove this, let's take vector ***v***:

$$\vec{v} = \begin{bmatrix}1\\3\end{bmatrix} $$

Our matrix transformation using ***A*** is:

$$\begin{bmatrix}3 & 2\\1 & 0\end{bmatrix} \cdot \begin{bmatrix}1\\3\end{bmatrix} $$

So let's show the results of that using Python:

```python
v = np.array([1,3])
t = A@v

print(t)

# Plot v and t
vecs = np.array([v,t])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['orange', 'b'], scale=20)
plt.show()
```

And now, let's do the same thing using the ***QΛQ-1*** sequence of operations:

```python
import math
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

t = (Q@(L@(Qinv)))@v

# Plot v and t
vecs = np.array([v,t])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['orange', 'b'], scale=20)
plt.show()
```

So ***A*** and ***QΛQ-1*** are equivalent.

If we view the intermediary stages of the decomposed transformation, you can see the transformation using ***A*** in the original base for ***v*** (orange to blue) and the transformation using ***Λ*** in the change of basis decribed by ***Q*** (red to magenta):

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

t1 = Qinv@v
t2 = L@t1
t3 = Q@t2

# Plot the transformations
vecs = np.array([v,t1, t2, t3])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['orange', 'red', 'magenta', 'blue'], scale=20)
plt.show()
```

So from this visualization, it should be apparent that the transformation ***Av*** can be performed by changing the basis for ***v*** using ***Q*** (from orange to red in the above plot) applying the equivalent linear transformation in that base using ***Λ*** (red to magenta), and switching back to the original base using ***Q-1*** (magenta to blue).

## Rank of a Matrix

The **rank** of a square matrix is the number of non-zero eigenvalues of the matrix. A **full rank** matrix has the same number of non-zero eigenvalues as the dimension of the matrix. A **rank-deficient** matrix has fewer non-zero eigenvalues as dimensions. The inverse of a rank deficient matrix is singular and so does not exist (this is why in a previous notebook we noted that some matrices have no inverse).

Consider the following matrix ***A***:

$$A=\begin{bmatrix}1 & 2\\4 & 3\end{bmatrix}$$

Let's find its eigenvalues (***Λ***):

```python
import numpy as np
A = np.array([[1,2],
              [4,3]])
l, Q = np.linalg.eig(A)
L = np.diag(l)
print(L)
```

$$\Lambda=\begin{bmatrix}-1 & 0\\0 & 5\end{bmatrix}$$

This matrix has full rank. The dimensions of the matrix is 2. There are two non-zero eigenvalues.

Now consider this matrix:

$$B=\begin{bmatrix}3 & -3 & 6\\2 & -2 & 4\\1 & -1 & 2\end{bmatrix}$$

Note that the second and third columns are just scalar multiples of the first column.

Let's examine it's eigenvalues:

```python
B = np.array([[3,-3,6],
              [2,-2,4],
              [1,-1,2]])
lb, Qb = np.linalg.eig(B)
Lb = np.diag(lb)
print(Lb)
```

$$\Lambda=\begin{bmatrix}3 & 0& 0\\0 & -6\times10^{-17} & 0\\0 & 0 & 3.6\times10^{-16}\end{bmatrix}$$

Note that matrix has only 1 non-zero eigenvalue. The other two eigenvalues are so extremely small as to be effectively zero. This is an example of a rank-deficient matrix; and as such, it has no inverse.

## Inverse of a Square Full Rank Matrix
You can calculate the inverse of a square full rank matrix by using the following formula:

$$A^{-1} = Q \Lambda^{-1} Q^{-1}$$

Let's apply this to matrix ***A***:

$$A=\begin{bmatrix}1 & 2\\4 & 3\end{bmatrix}$$

Let's find the matrices for ***Q***, ***Λ-1***, and ***Q-1***:

```python
import numpy as np
A = np.array([[1,2],
              [4,3]])

l, Q = np.linalg.eig(A)
L = np.diag(l)
print(Q)
Linv = np.linalg.inv(L)
Qinv = np.linalg.inv(Q)
print(Linv)
print(Qinv)
```

So:

$$A^{-1}=\begin{bmatrix}-0.70710678 & -0.4472136\\0.70710678 & -0.89442719\end{bmatrix}\cdot\begin{bmatrix}-1 & -0\\0 & 0.2\end{bmatrix}\cdot\begin{bmatrix}-0.94280904 & 0.47140452\\-0.74535599 & -0.74535599\end{bmatrix}$$

Let's calculate that in Python:

```python
Ainv = (Q@(Linv@(Qinv)))
print(Ainv)
```

That gives us the result:

$$A^{-1}=\begin{bmatrix}-0.6 & 0.4\\0.8 & -0.2\end{bmatrix}$$

We can apply the ***np.linalg.inv*** function directly to ***A*** to verify this:

```python
print(np.linalg.inv(A))
```

Vectores y Matrices/03-05-Transformations Eigenvectors and Eigenvalues.ipynb


```python
from sympy import *
init_printing()
import matplotlib.pyplot as plt
from IPython.display import display
```

```python
th,x,y = symbols('theta,x,y')
```

```python
Rth = Matrix([[cos(th),-sin(th)],[sin(th), cos(th)]])
simplify(Rth)
```

```python
X = Matrix([x,y])

Rth*X
```

```python
(X.T*Rth.T)
```

```python
simplify((X.T*Rth.T)*(Rth*X))
```

```python

```


# 演習課題（２）－２の計算例

中心から上の角度を計算

\begin{equation}
  \phi=\tan^{-1}\frac{0.5h}{3.0h}
\end{equation}

$\phi$を2倍して視野角$\theta$を計算

\begin{equation}
  \theta = 2\phi
\end{equation}

ラジアンを「分」に換算

\begin{equation}
  1 \mathrm{[rad]} = \frac{180\times 60}{\pi} \mathrm{[arc minute]}
\end{equation}

```
import math

phi = math.atan(0.5/3.0)
theta = 2*phi

print('theta = {} [分]'.format(theta/math.pi*180*60))
```

    theta = 1135.4786649630742 [分]


# Chapter 7 Solvers

```python
from sympy import *
x, y, z = symbols('x y z')
init_printing(use_unicode=True)
```

## 7.1 方程式についての注意

`Sympy`での方程式は`Eq`関数を使う

```python
Eq(x,y)
```

```python
solveset(Eq(x**2, 1), x) #Eq(左辺, 右辺)
```

```python
solveset(Eq(x**2 - 1, 0), x)
```

```python
solveset(x**2 - 1, x) #式 = 0　とした「式」が第一引数
```

--->`Sympy`では、式はすべて`=0`とみなされる. `Eq`関数を使わなくても、差で表現してやればよい.

## 7.2 代数的に方程式を解く

代数方程式を解く基本的な関数は`solveset(equation, variable=None, domain=S.Complexes)`.

他に関数`solve()`があるが、基本的には関数`solveset()`を用いる.

```python
solveset(x**2-x, x)
```

```python
solveset(x - x, x, domain=S.Reals)
```

```python
solveset(sin(x) - 1, x, domain=S.Reals)
```

--->勝手に`n`を導入して解いてくれる.

#### 解がないときは空集合を返す:

```python
solveset(exp(x),x)
```

```python
solveset(cos(x) - x, x)
```

#### 連立方程式: `linsolve関数`

```python
linsolve([x + y + z - 1, x + y + 2*z -3], (x, y, z))
```

```python
linsolve(Matrix((
    [1, 1, 1, 1],
    [1, 1, 2, 3])), (x, y, z)) #Matrix関数についてはChapter 8
```

```python
M = Matrix(((
1, 1, 1, 1),
 (1, 1, 2, 3)
))
```

```python
system = A, b = M[:,:-1], M[:,-1]
```

```python
linsolve(system, x, y, z)
```

ただし高次の式になると,

```python
solveset(x**3 - 6*x**2 + 9*x, x)
```

となって、解の重根の有無と縮退度がわからない. そこで

```python
roots(x**3 - 6*x**2 + 9*x, x)
```

とすれば、解`0`が1個、解`3`が2個出てくるとわかる.　実際

```python
factor(x**3 - 6*x**2 + 9*x)
```

**注意**: `solveset`の使えない方程式
    - 非線形多変数系
    - LambertWで解ける方程式

こういう場合は`solve()`関数で解ける:

```python
solve([x*y - 1, x -2], x, y)
```

オプション`dict ='True'`を付けると結果がリストの要素としての辞書型で得られる.

```python
solve([x*y - 1, x -2], x, y, dict ='True')
```

```python
solve(x*exp(x) - 1, x)
```

## 7.3 微分方程式を解く

関数の設定: SymPyオブジェクトへ。

```python
f, g = symbols('f g', cls=Function) #解きたいものが関数なので、cls=Function オプションを付ける.
```

```python
f(x)
```

☆導関数は、Chapter 6 でやったように、関数Derivative()を使ってもできるし、

```python
Derivative(f(x),x,1)
```

未知関数 `f(x)` に対するメソッド `diff` を用いても良い:

```python
f(x).diff(x)
```

**コメント**

ここでdiffメソッドが使えたのは、未知関数f(x)に対して用いているため。

==> 今、微分方程式$$f''(x)-2f'(x)+f(x)=\sin(x)$$を解くことを考える. これは

```python
diffeq = Eq(f(x).diff(x, 2) - 2*f(x).diff(x) + f(x),  sin(x))
```

```python
diffeq #微分方程式
```

微分方程式は`dsolve`関数で解ける！

```python
dsolve(diffeq, f(x)) #第二引数で解きたい関数を指定
```

```python
dsolve(f(x).diff(x)*(1-sin(f(x))),f(x))
```

次が最後！ [Chapter8 Matrices](https://hiroyuki827.github.io/SymPy_tutorial/Chapter8_Matrices.html)へ!


# Generalisation and overfitting

In this notebook we will explore the issue of overfitting and how we can measure how well the models we train generalise their predictions to unseen data. This will build upon the introduction to generalisation given in the [fourth lecture](http://www.inf.ed.ac.uk/teaching/courses/mlp/2016/mlp04-learn.pdf).

## Exercise: overfitting and model complexity in a 1D regression problem

As an exercise we will consider a regression problem. In particular we will attempt to use a multiple layer network model to learn to predict output values from inputs, given a fixed set of (noisy) observations of the underlying functional relationship between inputs and outputs. The aim of the exercise will be to visualise how increasing the complexity of the model we fit to the training data effects the ability of the model to make predictions across the input space.

### Function

To keep things simple we will consider a single input-output function defined by a fourth degree polynomial (quartic)

$$ f(x) = 10 x^4 - 17 x^3 + 8 x^2 - x $$

with the observed values being the function values plus zero-mean Gaussian noise

$$ y = f(x) + 0.01 \epsilon \qquad \epsilon \sim \mathcal{N}\left(\cdot;\,0,\,1\right) $$

The inputs will be drawn from the uniform distribution on $[0, 1]$.

First import the necessary modules and seed the random number generator by running the cell below.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('ggplot')
seed = 17102016
rng = np.random.RandomState(seed)
```

Write code in the cell below to calculate a polynomial function of one dimensional inputs.

If $\boldsymbol{c}$ is a length $P$ vector of coefficients corresponding to increasing powers in the polynomial (starting from the constant zero power term up to the $P-1^{\textrm{th}}$ power) the function should correspond to the following

\begin{equation}
  f_{\textrm{polynomial}}(x,\ \boldsymbol{c}) = \sum_{p=0}^{P-1} \left( c_p x^p \right)
\end{equation}

```python
def polynomial_function(inputs, coefficients):
    """Calculates polynomial with given coefficients of an array of inputs.

    Args:
        inputs: One-dimensional array of input values of shape (num_inputs,)
        coefficients: One-dimensional array of polynomial coefficient terms
           with `coefficients[0]` corresponding to the coefficient for the
           zero order term in the polynomial (constant) and `coefficients[-1]`
           corresponding to the highest order term.

    Returns:
        One dimensional array of output values of shape (num_inputs,)

    """
    return (inputs[:, None]**np.arange(coefficients.shape[0])).dot(coefficients)
```

Run the cell below to test your implementation.

```python
test_coefficients = np.array([-1., 3., 4.])
test_inputs = np.array([0., 0.5, 1., 2.])
test_outputs = np.array([-1., 1.5, 6., 21.])
assert polynomial_function(test_inputs, test_coefficients).shape == (4,), (
    'Function gives wrong shape output.'
)
assert np.allclose(polynomial_function(test_inputs, test_coefficients), test_outputs), (
    'Function gives incorrect output values.'
)
```

We now need to use the random number generator to sample input values and calculate the corresponding target outputs using your polynomial implementation with the relevant coefficients for our function. Do this by running the cell below.

```python
coefficients = np.array([0, -1., 8., -17., 10.])
input_dim, output_dim = 1, 1
noise_std = 0.01
num_data = 80
inputs = rng.uniform(size=(num_data, input_dim))
epsilons = rng.normal(size=num_data)
targets = (polynomial_function(inputs[:, 0], coefficients) +
           epsilons * noise_std)[:, None]
```

We will split the generated data points in to equal sized training and validation data sets and use these to create data provider objects which we can use to train models in our framework. As the dataset is small here we will use a batch size equal to the size of the data set. Run the cell below to split the data and set up the data provider objects.

```python
from mlp.data_providers import DataProvider
num_train = num_data // 2
batch_size = num_train
inputs_train, targets_train = inputs[:num_train], targets[:num_train]
inputs_valid, targets_valid = inputs[num_train:], targets[num_train:]
train_data = DataProvider(inputs_train, targets_train, batch_size=batch_size, rng=rng)
valid_data = DataProvider(inputs_valid, targets_valid, batch_size=batch_size, rng=rng)
```

We can now visualise the data we will be modelling. Run the cell below to plot the target outputs against inputs for both the training and validation sets. Note the clear underlying smooth functional relationship evident in the noisy data.

```python
fig = plt.figure(figsize=(8, 4))
ax = fig.add_subplot(111)
ax.plot(inputs_train[:, 0], targets_train[:, 0], '.', label='training data')
ax.plot(inputs_valid[:, 0], targets_valid[:, 0], '.', label='validation data')
ax.set_xlabel('Inputs $x$', fontsize=14)
ax.set_ylabel('Ouputs $y$', fontsize=14)
ax.legend(loc='best')
fig.tight_layout()
```

### Model

We will fit models with a varying number of parameters to the training data. As multi-layer logistic sigmoid models do not tend to perform well in regressions tasks like this we will instead use a [radial basis function (RBF) network](https://en.wikipedia.org/wiki/Radial_basis_function_network).

This model predicts the output as the weighted sum of basis functions (here Gaussian like bumps) tiled across the input space. The cell below generates a random set of weights and bias for a RBF network and plots the modelled input-output function across inputs $[0, 1]$. Run the cell below for several different number of weight parameters (specified with `num_weights` variable) to get a feel for the sort of predictions the RBF network models produce.

```python
num_weights = 15
weights_scale = 1.
bias_scale = 1.

def basis_function(x, centre, scale):
    return np.exp(-(x - centre)**2 / scale**2)

weights = rng.normal(size=num_weights) * weights_scale
bias = rng.normal() * bias_scale

centres = np.linspace(0, 1, weights.shape[0])
scale = 1. / weights.shape[0]

xs = np.linspace(0, 1, 200)
ys = np.zeros(xs.shape[0])

fig = plt.figure(figsize=(12, 4))
ax = fig.add_subplot(1, 1, 1)
for weight, centre in zip(weights, centres):
    ys += weight * basis_function(xs, centre, scale)
ys += bias
ax.plot(xs, ys)
ax.set_xlabel('Input', fontsize=14)
ax.set_ylabel('Output', fontsize=14)
```

You do not need to read in to the details of how to implement this model. All of the additional code you need to fit RBF networks is provided in the `RadialBasisFunctionLayer` in the `mlp.layers` module. The `RadialBasisFunctionLayer` class has the same interface as the layer classes we encountered in the previous lab, defining both `fprop` and `bprop` methods, and we can therefore include it as a layer in a `MultipleLayerModel` as with any other layer.

Here we will use the `RadialBasisFunctionLayer` as the first layer in a two layer model. This first layer calculates the basis function terms which are then be weighted and summed together in an `AffineLayer`, the second and final layer. This illustrates the advantage of using a modular modelling framework - we can reuse the code we previously implemented to train a quite different model architecture just by defining a new layer class.

Run the cell below to run some necessary setup code.

```python
from mlp.models import MultipleLayerModel
from mlp.layers import AffineLayer, RadialBasisFunctionLayer
from mlp.errors import SumOfSquaredDiffsError
from mlp.initialisers import ConstantInit, UniformInit
from mlp.learning_rules import GradientDescentLearningRule
from mlp.optimisers import Optimiser

# Regression problem therefore use sum of squared differences error
error = SumOfSquaredDiffsError()
# Use basic gradient descent learning rule with fixed learning rate
learning_rule = GradientDescentLearningRule(0.1)
# Initialise weights from uniform distribution and zero bias
weights_init = UniformInit(-0.1, 0.1)
biases_init = ConstantInit(0.)
# Train all models for 2000 epochs
num_epoch = 2000
```

The cell below defines RBF network models with varying number of weight parameters (equal to the number of basis functions) and fits each to the training set, recording the final training and validation set errors for the fitted models. Run it now to fit the models and calculate the error values.

```python
num_weight_list = [2, 5, 10, 25, 50, 100]
models = []
train_errors = []
valid_errors = []
for num_weight in num_weight_list:
    model = MultipleLayerModel([
        RadialBasisFunctionLayer(num_weight),
        AffineLayer(input_dim * num_weight, output_dim,
                    weights_init, biases_init)
    ])
    optimiser = Optimiser(model, error, learning_rule,
                          train_data, valid_data)
    print('-' * 80)
    print('Training model with {0} weights'.format(num_weight))
    print('-' * 80)
    _ = optimiser.train(num_epoch, -1)
    outputs_train = model.fprop(inputs_train)[-1]
    outputs_valid = model.fprop(inputs_valid)[-1]
    models.append(model)
    train_errors.append(error(outputs_train, targets_train))
    valid_errors.append(error(outputs_valid, targets_valid))
    print('  Final training set error: {0:.1e}'.format(train_errors[-1]))
    print('  Final validation set error: {0:.1e}'.format(valid_errors[-1]))
```

    --------------------------------------------------------------------------------
    Training model with 2 weights
    --------------------------------------------------------------------------------
      Final training set error: 2.0e-03
      Final validation set error: 1.1e-03
    --------------------------------------------------------------------------------
    Training model with 5 weights
    --------------------------------------------------------------------------------
      Final training set error: 4.5e-04
      Final validation set error: 3.0e-04
    --------------------------------------------------------------------------------
    Training model with 10 weights
    --------------------------------------------------------------------------------
      Final training set error: 5.1e-05
      Final validation set error: 8.3e-05
    --------------------------------------------------------------------------------
    Training model with 25 weights
    --------------------------------------------------------------------------------
      Final training set error: 3.9e-05
      Final validation set error: 9.5e-05
    --------------------------------------------------------------------------------
    Training model with 50 weights
    --------------------------------------------------------------------------------
      Final training set error: 1.5e-05
      Final validation set error: 1.6e-03
    --------------------------------------------------------------------------------
    Training model with 100 weights
    --------------------------------------------------------------------------------
      Final training set error: 1.0e-05
      Final validation set error: 4.2e-03

In the cell below write code to [plot bar charts](http://matplotlib.org/examples/api/barchart_demo.html) of the training and validation set errors for the different fitted models.

Some questions to think about from the plots:

  * Do the models with more free parameters fit the training data better or worse?
  * What does the validation set error value tell us about the models?
  * Of the models fitted here which would you say seems like it is most likely to generalise well to unseen data?
  * Do any of the models seem to be overfitting?

```python
fig = plt.figure(figsize=(12, 6))
ax1 = fig.add_subplot(1, 2, 1)
ax1.bar(0.1 + np.arange(len(num_weight_list)), train_errors)
ax1.set_xticks(0.5 + np.arange(len(num_weight_list)))
ax1.set_xticklabels(num_weight_list)
ax1.set_xlabel('Number of weight parameters')
ax1.set_ylabel('Error')
ax1.set_title('Training set')
ax2 = fig.add_subplot(1, 2, 2)
ax2.bar(0.1 + np.arange(len(num_weight_list)), valid_errors)
ax2.set_xticks(0.5 + np.arange(len(num_weight_list)))
ax2.set_xticklabels(num_weight_list)
ax2.set_xlabel('Number of weight parameters')
ax2.set_ylabel('Error')
ax2.set_title('Validation set')
fig.tight_layout()
```

Now let's visualise what the fitted model's predictions look like across the whole input space compared to the 'true' function we were trying to fit.

In the cell below, for each of the fitted models stored in the `models` list above:
  * Compute output predictions for the model across a linearly spaced series of 500 input points between 0 and 1 in the input space.
  * Plot the computed predicted outputs and true function values at the corresponding inputs as line plots on the same axis (use a new axis for each model).
  * On the same axis plot the training data sets input-target pairs as points.

```python
inputs_grid = np.linspace(0., 1., 500)
true_func_grid = polynomial_function(inputs_grid, coefficients)
for num_weight, model in zip(num_weight_list, models):
    fig = plt.figure(figsize=(8, 4))
    ax = fig.add_subplot(111)
    outputs_grid = model.fprop(inputs_grid)[-1]
    ax.plot(inputs_train, targets_train, '.', label='training data')
    ax.plot(inputs_grid, true_func_grid, label='true function')
    ax.plot(inputs_grid, outputs_grid, label='fitted function')
    ax.set_xlabel('Inputs $x$', fontsize=14)
    ax.set_ylabel('Ouputs $y$', fontsize=14)
    ax.set_title('{0} weight parameters'.format(num_weight))
    ax.legend()
```

You should be able to relate your answers to the questions above to what you see in these plots - ask a demonstrator if you are unsure what is going on. In particular for the models which appeared to be overfitting and generalising poorly you should now have an idea how this looks in terms of the model's predictions and how these relate to the training data points and true function values.


# Wrap LaTeX statements in dollar signs

`$a = 2$`

`$b = 7030$`

`$c = 4.375$`

$a = 2$

$b = 7030$

$c = 4.375$

# Other LaTeX Commands begin with a backslash

## Greek Letters:

`$\alpha, \alpha, \beta,  \gamma, \Gamma, \pi, \Pi, \phi, \varphi, \mu, \Phi$`

$\alpha, \alpha, \beta,  \gamma, \Gamma, \pi, \Pi, \phi, \varphi, \mu, \Phi$

## Set Notation:

`$\forall x \in X,  \exists p \leq 5$`

$\forall x \in X,  \exists p \leq 5$

## Limits:

`$\lim\limits_{x \to \infty} \exp(-x) = 0$`

$\lim\limits_{x \to \infty} \exp(-x) = 0$

## Powers using the caret "^"

`$x^2$`

$x^2$

## For complicated exponents/indices wrap them in curly braces

`$x^{2+i}$`

$x^{2+i}$

## Indices using the underscore "_"

`$B_{i}$`

$B_{i}$

> ### Combining powers and indices:

`$k_{n+1} = n^2 + k_n^2 - k_{n-1}$`

$k_{n+1} = n^2 + k_n^2 - k_{n-1}$

## Fractions

`$\frac{2}{3}$`

$\frac{2}{3}$

`$\frac{n!}{k!(n-k)!}$`

$\frac{n!}{k!(n-k)!}$

## Binomials

`$\binom{n}{k}$`

$\binom{n}{k}$

## Roots

`$\sqrt{2}$`

$\sqrt{2}$

`$\sqrt{a^2 + b^2}$`

$\sqrt{a^2 + b^2}$

## Sums

`$\sum_{i=1}^{10} t_i$`

$\sum_{i=1}^{10} t_i$

or

`$\displaystyle\sum_{i=1}^{10} t_i$`

$\displaystyle\sum_{i=1}^{10} t_i$

## Integrals

`$\int\limits_a^b$`

$\int\limits_a^b$

## Brackets and Delimiters

`$( a ), [ b ], \{ c \}, | d |, \| e \|,
\langle f \rangle, \lfloor g \rfloor,
\lceil h \rceil, \ulcorner i \urcorner$`

$( a ), [ b ], \{ c \}, | d |, \| e \|,
\langle f \rangle, \lfloor g \rfloor,
\lceil h \rceil, \ulcorner i \urcorner$

## Vectors

$\vec{a} =
  \begin{pmatrix}
           1\\
           2
  \end{pmatrix}$

## Matrices

 $B = \begin{bmatrix}
  a & b & c \\
  d & e & f \\
  g & h & i
 \end{bmatrix}$

## Whitespace

Often the most difficult part of LaTeX is getting the sizing and whitespace to cooperate.

Wrapping an entire equation in the \equation wrappers often helps keep it

Colab will recognize many of these commands as LaTeX without the dollar signs as well.

`$\begin{equation}
  x = a_0 + \cfrac{1}{a_1
          + \cfrac{1}{a_2
          + \cfrac{1}{a_3 + \cfrac{1}{a_4} } } }
\end{equation}$`

$\begin{equation}
  x = a_0 + \cfrac{1}{a_1
          + \cfrac{1}{a_2
          + \cfrac{1}{a_3 + \cfrac{1}{a_4} } } }
\end{equation}$

### Horizontal Spacing Options

<https://tex.stackexchange.com/questions/74353/what-commands-are-there-for-horizontal-spacing>

`\quad` is probably my favorite horizontal spacing option.

`$ a \quad b^2 \quad c_i \quad \pi $`

$ a \quad b^2 \quad c_i \quad \pi $

### Centering Equations

\begin{align}
c = -6.022\mathrm{e}{+23}
\qquad
d = \pi
\end{align}

## Normal Text in "Math Mode"

"Math Mode" is what it's called when you're inside of a statement that is wrapped in dollar signs.

`$ a+b = 12 ... \textrm{normal text} ... \frac{1}{2}$`

$ a+b = 12 ... \textrm{normal text} ... \frac{1}{2}$

# Lets try to recreate a few of these formulas together:

You can use the site: <http://latex2png.com/> To convert LaTeX into downloadable PNGs.

## Quadratic Formula

$x=\frac{-b\pm\sqrt{b^{2}-4ac}}{2a}$

## Standard Deviation

$\sigma = \sqrt{\frac{1}{N}\sum_{i=1}^{N}(x_i-\mu)^2}$

## General Matrix Notation

$A_{m,n} =
\begin{pmatrix}
 a_{1,1} & a_{1,2} & \cdots & a_{1,n} \\
 a_{2,1} & a_{2,2} & \cdots & a_{2,n} \\
\vdots & \vdots & \ddots & \vdots \\
 a_{m,1} & a_{m,2} & \cdots & a_{m,n}
\end{pmatrix}$

# Assignment:

## “What are your goals for data science?”

# Stretch Goal (1):

## Try to recreate these formulas on your own:

### Cauchy's Integral Formula

$f(a) = \frac{1}{2\pi i}\oint\frac{f(z)}{z-a}dz$

### Definition of Christoffel Symbols

$(\nabla_XY)^k = X^i(\nabla_iY)^k = X^i(\frac{\partial Y^k}{\partial X^i}+\Gamma_{im}^k Y^m)$

### Curl of a Vector Field

$\vec\nabla \times \vec F = (\frac{\partial F_z}{\partial y} - \frac{\partial F_y}{\partial z})\textrm{i} + (\frac{\partial F_x}{\partial z} - \frac{\partial F_z}{\partial x})\textrm{j} + (\frac{\partial F_y}{\partial x} - \frac{\partial F_x}{\partial y})\textrm{k}$

# Stretch Goal (2):

Visit: <https://projecteuler.net/>

Make an account and begin solving the challenges using Python. For each challenge that you solve, write an explanation of why your solution works using LaTeX.


# 10 ODE integrators: Verlet

```python
from importlib import reload

import numpy as np
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
matplotlib.style.use('ggplot')

import integrators2_solution as integrators2
```

```python
reload(integrators2)
```

    <module 'integrators2_solution' from '/Volumes/Data/oliver/Documents/Teaching/ASU/CompPhys_PHY494/2018/PHY494-resources/10_ODEs/integrators2_solution.py'>

## Velocity Verlet

Use expansion *forward* and *backward* (!) in time (Hamiltons (i.e. Newton without friction) equations are time symmetric)

\begin{align}
r(t + \Delta r) &\approx r(t) + \Delta t\, v(t) + \frac{1}{2m} \Delta t^2 F(t)\\
r(t) &\approx r(t + \Delta t) - \Delta t\, v(t + \Delta t) + \frac{1}{2m} \Delta t^2 F(t+\Delta t)
\end{align}

Solve for $v$:
\begin{align}
v(t+\Delta t) &\approx v(t) + \frac{1}{2m} \Delta t \big(F(t) + F(t+\Delta t)\big)
\end{align}

Complete **Velocity Verlet** integrator consists of the first and third equation.

In practice, split into three steps (calculate the velocity at the half time step):
\begin{align}
v(t+\frac{\Delta t}{2}) &= v(t) + \frac{\Delta t}{2} \frac{F(t)}{m} \\
r(t + \Delta r) &= r(t) + \Delta t\, v(t+\frac{\Delta t}{2})\\
v(t+\Delta t) &= v(t+\frac{\Delta t}{2}) + \frac{\Delta t}{2} \frac{F(t+\Delta t)}{m}
\end{align}

When writing production-level code, remember to re-use $F(t+\Delta t)$ als the "new" starting $F(t)$ in the next iteration (and don't recompute).

### Integration of planetary motion
Gravitational potential energy:
$$
U(r) = -\frac{GMm}{r}
$$
with $r$ the distance between the two masses $m$ and $M$.

#### Central forces
$$
U(\mathbf{r}) = f(r) = f(\sqrt{\mathbf{r}\cdot\mathbf{r}})\\
\mathbf{F} = -\nabla U(\mathbf{r}) = -\frac{\partial f(r)}{\partial r} \, \frac{\mathbf{r}}{r}
$$

#### Force of gravity
\begin{align}
\mathbf{F} &= -\frac{G m M}{r^2} \hat{\mathbf{r}}\\
\hat{\mathbf{r}} &= \frac{1}{\sqrt{x^2 + y^2}} \left(\begin{array}{c} x \\ y \end{array}\right)
\end{align}

#### Integrate simple planetary orbits
Set $M = 1$ (one solar mass) and $m = 3.003467×10^{-6}$ (one Earth mass in solar masses) and try initial conditions

\begin{alignat}{1}
x(0) &= 1,\quad  y(0) &= 0\\
v_x(0) &= 0,\quad  v_y(0) &= 6.179
\end{alignat}

Note that we use the following units:
* length in astronomical units (1 AU = 149,597,870,700 m )
* mass in solar masses (1 $M_☉ = 1.988435×10^{30}$ kg)
* time in years (1 year = 365.25 days, 1 day = 86400 seconds)

In these units, the gravitational constant is $G = 4\pi^2$ (in SI units $G = 6.674×10^{-11}\, \text{N}\cdot\text{m}^2\cdot\text{kg}^{-2}$).

```python
M_earth = 3.003467e-6
M_sun = 1.0
```

```python
G_grav = 4*np.pi**2

def F_gravity(r, m=M_earth, M=M_sun):
    rr = np.sum(r*r)
    rhat = r/np.sqrt(rr)
    return -G_grav*m*M/rr * rhat

def U_gravity(r, m=M_earth, M=M_sun):
    return -G_grav*m*M/np.sqrt(np.sum(r*r))
```

Let's now integrate the equations of motions under gravity with the **Velocity Verlet** algorithm:

```python
def planet_orbit(r0=np.array([1, 0]), v0=np.array([0, 6.179]), mass=M_earth, dt=0.001, t_max=1):
    """2D planetary motion with velocity verlet"""
    dim = len(r0)
    assert len(v0) == dim

    nsteps = int(t_max/dt)

    r = np.zeros((nsteps, dim))
    v = np.zeros_like(r)

    r[0, :] = r0
    v[0, :] = v0

    # start force evaluation for first step
    Ft = F_gravity(r[0], m=mass)
    for i in range(nsteps-1):
        vhalf = v[i] + 0.5*dt * Ft/mass
        r[i+1, :] = r[i] + dt * vhalf
        Ftdt = F_gravity(r[i+1], m=mass)
        v[i+1] = vhalf + 0.5*dt * Ftdt/mass
        # new force becomes old force
        Ft = Ftdt

    return r, v
```

```python
r, v = planet_orbit(dt=0.1, t_max=10)
```

```python
rx, ry = r.T
ax = plt.subplot(1,1,1)
ax.set_aspect(1)
ax.plot(rx, ry)
```

These are not closed orbits (as we would expect from a $1/r$ potential). But it gets much besser when stepsize is reduced to 0.01 (just rerun the code with `dt = 0.01` and replot):

```python
r, v = planet_orbit(dt=0.01, t_max=10)
```

```python
rx, ry = r.T
ax = plt.subplot(1,1,1)
ax.set_aspect(1)
ax.plot(rx, ry)
```

## Velocity Verlet vs RK4: Energy conservation
Assess the stability of `rk4` and `Velocity Verlet` by checking energy conservation over longer simulation times.

The file `integrators2.py` contains almost all code that you will need.

### Implement gravity force in `integrators2.py`
Add `F_gravity` to the `integrators2.py` module. Use the new function `unitvector()`.

### Planetary orbits with `integrators2.py`

```python
r0 = np.array([1, 0])
v0 = np.array([0, 6.179])
```

```python
from importlib import reload
reload(integrators2)
```

    <module 'integrators2_solution' from '/Volumes/Data/oliver/Documents/Teaching/ASU/CompPhys_PHY494/2018/PHY494-resources/10_ODEs/integrators2_solution.py'>

Use the new function `integrators2.integrate_newton_2d()` to integrate 2d coordinates.

#### RK4

```python
trk4, yrk4 = integrators2.integrate_newton_2d(x0=r0, v0=v0, t_max=30, mass=M_earth,
                                       h=0.01,
                                       force=integrators2.F_gravity,
                                       integrator=integrators2.rk4)
```

```python
rxrk4, ryrk4 = yrk4[:, 0, 0], yrk4[:, 0, 1]
ax = plt.subplot(1,1,1)
ax.set_aspect(1)
ax.plot(rxrk4, ryrk4)
```

```python
integrators2.analyze_energies(trk4, yrk4, integrators2.U_gravity, m=M_earth)
```

```python
print("Energy conservation RK4 for {} steps: {}".format(
        len(trk4),
        integrators2.energy_conservation(trk4, yrk4, integrators2.U_gravity, m=M_earth)))
```

    Energy conservation RK4 for 3000 steps: 3.5366039779498594e-06

#### Euler

```python
te, ye = integrators2.integrate_newton_2d(x0=r0, v0=v0, t_max=30, mass=M_earth,
                                         h=0.01,
                            force=F_gravity,
                            integrator=integrators2.euler)
rex, rey = ye[:, 0].T
```

```python
ax = plt.subplot(1,1,1)
ax.plot(rx, ry, label="RK4")
ax.plot(rex, rey, label="Euler")
ax.legend(loc="best")
ax.set_aspect(1)
```

```python
integrators2.analyze_energies(te, ye, integrators2.U_gravity, m=M_earth)
```

```python
print("Energy conservation Euler for {} steps: {}".format(
        len(te),
        integrators2.energy_conservation(te, ye, integrators2.U_gravity, m=M_earth)))
```

    Energy conservation Euler for 3000 steps: 0.6763156457080265

*Euler* is just awful... but we knew that already.

#### Velocity Verlet

```python
tv, yv = integrators2.integrate_newton_2d(x0=r0, v0=v0, t_max=30, mass=M_earth,
                                       h=0.01,
                                       force=F_gravity,
                                       integrator=integrators2.velocity_verlet)
```

```python
rxv, ryv = yv[:, 0].T
ax = plt.subplot(1,1,1)
ax.set_aspect(1)
ax.plot(rxv, ryv, label="velocity Verlet")
ax.plot(rxrk4, ryrk4, label="RK4")
ax.legend(loc="best")
```

```python
integrators2.analyze_energies(tv, yv, integrators2.U_gravity, m=M_earth)
```

```python
print("Energy conservation Velocity Verlet for {} steps: {}".format(
        len(tv),
        integrators2.energy_conservation(tv, yv, integrators2.U_gravity, m=M_earth)))
```

    Energy conservation Velocity Verlet for 3000 steps: 6.37822713158452e-05

*Velocity Verlet* only has moderate accuracy, especially when compared to *RK4*.

However, let's look at energy conservation over longer times:

#### Longer time scale stability
Run RK4 and Velocity Verlet for longer.

```python
tv2, yv2 = integrators2.integrate_newton_2d(x0=r0, v0=v0, t_max=1000, mass=M_earth,
                                       h=0.01,
                                       force=F_gravity,
                                       integrator=integrators2.velocity_verlet)
```

```python
print("Energy conservation Velocity Verlet for {} steps: {}".format(
        len(tv2),
        integrators2.energy_conservation(tv2, yv2, integrators2.U_gravity, m=M_earth)))
```

    Energy conservation Velocity Verlet for 100000 steps: 6.379584728912368e-05

```python
t4, y4 = integrators2.integrate_newton_2d(x0=r0, v0=v0, t_max=1000, mass=M_earth,
                                       h=0.01,
                                       force=F_gravity,
                                       integrator=integrators2.rk4)
```

```python
print("Energy conservation RK4 for {} steps: {}".format(
        len(t4),
        integrators2.energy_conservation(t4, y4, integrators2.U_gravity, m=M_earth)))
```

    Energy conservation RK4 for 100000 steps: 0.00011860288209883538

Velocity Verlet shows **good long-term stability** but relative low precision. On the other hand, RK4 has high precision but the accuracy decreases over time.

* Use a *Verlet* integrator when energy conservation is important and long term stability is required (e.g. molecular dynamics simulations). It is generally recommended to use an integrator that conserves some of the inherent symmetries and structures of the governing physical equations (e.g. for Hamilton's equations of motion, time reversal symmetry and the symplectic and area-preserving structure).
* Use *RK4* for high short-term accuracy (but may be difficult to know what "short term" should mean) or when solving general differential equations.

```python

```


# Using Auxiliary Variables in MCMC proposals

## Slice sampling

Slice sampling is a simple MCMC algorithm that introduces the idea of auxiliary variables. The motivation for slice sampling is that if we can sample uniformly from the region under the graph of the target distribution, we will have random samples from the target distribution. In the univariate case, the algorithm is as follows

- start with some $x$ where $p(x) \ne 0$
- repeat
    - sample $y$ (auxiliary variable) uniformly from 0 to $f(x)$
    - draw a horizontal line at $y$ within $p(x)$ (this may consist of multiple intervals)
    - sample $x$ from the horizontal segments

The auxiliary $y$ variable allows us to sample $(x, y)$ points that are in the region under the graph of the target distribution. Only the $x$ variable is used for the Monte Carlo samples - the $y$ variables are simply discarded. This works because the joint disribution is constructed so that it factors $p(x, y) = p(y \mid x) p(x)$ and so projecting leaves just $p(x)$. The slice sampler is a Markov Chain Monte Carlo method since the next $(x, y)$ position depends only on the current position. Like Gibbs sampling, there is no tuning process and all proposals are accepted. For slice sampling, you either need the inverse distribution function or some way to estimate it. Later we will see that Hamiltonian Monte Carlo also uses auxiliary variables to generate a new proposal in an analogous way.

A toy example illustrates the process - Suppose we want to draw random samples from the posterior distribution $\mathcal{N}(0, 1)$ using slice sampling

Start with some value $x$
- sample $y$ from $\mathcal{U}(0, f(x))$ - this is the horizontal "slice" that gives the method its name
- sample the next $x$ from $f^{-1}(y)$ - this is typically done numerically
- repeat

Will sketch picture in class to show what is going on.

## A simple slice sampler example

```python
import scipy.stats as stats
```

```python
dist = stats.norm(5, 3)
w = 0.5
x = dist.rvs()

niters = 1000
xs = []
while len(xs) < niters:
    y = np.random.uniform(0, dist.pdf(x))
    lb = x
    rb = x
    while y < dist.pdf(lb):
        lb -= w
    while y < dist.pdf(rb):
        rb += w
    x = np.random.uniform(lb, rb)
    if y > dist.pdf(x):
        if np.abs(x-lb) < np.abs(x-rb):
            lb = x
        else:
            lb = y
    else:
        xs.append(x)
```

```python
plt.hist(xs, 20)
pass
```

Notes on the slice sampler:

- the slice may consist of disjoint pieces for multimodal distributions
- the slice can be a rectangular hyperslab for multivariable posterior distributions
- sampling from the slice (i.e. finding the boundaries at level $y$) is non-trivial and may involve iterative rejection steps - see figure below (from Wikimedia) for a typical approach - the blue bars represent disjoint pieces of the true slice through a bimodal distribution and the black lines are the proposal distribution approximating the true slice

## Hamiltonian Monte Carlo (HMC)

HMC uses an auxiliary variable corresponding to the momentum of particles in a potential energy well to generate proposal distributions that can make use of gradient information in the posterior distribution. For reversibility to be maintained, the total energy of the particle has to be conserved - hence we are interested in Hamiltonian systems. The main attraction of HMC is that it works much better than other methods when variables of interest are highly correlated. Because we have to solve problems involving momentum, we need to understand how to numerically solve differential equations in a way that is both accurate (i.e. second order) and preserves total energy (necessary for a Hamiltonian system).

Example adapted from [MCMC: Hamiltonian Monte Carlo (a.k.a. Hybrid Monte Carlo)](https://theclevermachine.wordpress.com/2012/11/18/mcmc-hamiltonian-monte-carlo-a-k-a-hybrid-monte-carlo/)

## Hamiltonian systems

In a Hamiltonian system, we consider particles with position $x$ and momentum (or velocity if we assume unit mass) $v$. The total energy of the system $H(x, v) = K(v) + U(x)$, where $K$ is the kinetic energy and $U$ is the potential energy, is conserved. Such a system satisfies the following Hamiltonian equations

$$
\begin{align}
\frac{dx}{dt} &= & \frac{\delta H}{dv} \\
\frac{dv}{dt} &= & -\frac{\delta H}{dx}
\end{align}
$$

Since $K$ depends only on $v$ and $U$ depends only on $x$, we have
$$
\begin{align}
\frac{dx}{dt} &= & \frac{\delta K}{dv} \\
\frac{dv}{dt} &= & -\frac{\delta U}{dx}
\end{align}
$$

### Harmonic oscillator

We will consider solving a classical Hamiltonian system - that of a undamped spring governed by the second order differential equation

$$
x'' + x = 0
$$

We convert this to two first order ODEs by using a dummy variable $x' = v$ to get

$$
\begin{align}
x' &= v \\
v' &= -x
\end{align}
$$

From the Hamiltonian equations above, this is equivalent to a system with kinetic energy $K(v) = \frac{1}{2}v^2$ and potential energy $U(x) = \frac{1}{2}x^2$.

Writing in matrix form,

$$
A = \pmatrix{ x' \\ v' } = \pmatrix{0 & 1 \\ -1 & 0} \pmatrix{x \\ v}
$$

and in general, for the state vector $x$,

$$
x' = Ax
$$

We note that $A$ is anti- or skew-symmetric ($A^T = -A$), and hence has purely imaginary eigenvalues. Solving $|A - \lambda I = 0$, we see that the eigenvalues and eigenvectors are $i, \pmatrix{1\\i}$ and $-i, \pmatrix{1\\-i}$. Since the eigenvalues are pure imaginary, we see that the solution for the initial conditions $(x,v) = (1, 0)$ is $x(t) = e^{it}$ and the orbit just goes around a circle with a period of $2\pi$, neither growing nor decaying. Another weay of seeing this is that the Hamiltonian $H(u, v)$ or sum of potential ($U(x)) = \frac{1}{2}x^2$) and kinetic energy ($K(v) = \frac{1}{2}v^2$) is constant, i.e. in vector form, $(x^T x) = \text{constant}$.

## Finite difference methods

We want to find a finite difference approximation to $u' = Au$ that is **accurate** and **preserves total energy**. If total energy is not preserved, the orbit will either spiral in towards zero or outwards away from the unit circle. If the accuracy is poor, the orbit will not be close to its starting value after $t = 2\pi$. This gives us an easy way to visualize how good our numerical scheme is. We can also compare the numerical scheme to the Taylor series to evaluate its accuracy.

### Forward Euler

The simplest finite difference scheme for integrating ODEs is the forward Euler

$$
\frac{u_{n+1} - u_n}{\Delta t} = A u_n
$$

Rearranging terms, we get

$$
u_{n+1} = u_n + \Delta t A u_n = \left( I + \Delta t A \right) u_n
$$

Since the eigenvalues of $A$ are $\pm i$, we see that the eigenvalues of the forward Euler matrix are $1 \pm i$. Since the absolute value of the eigenvalues is greater than 1, we expect **growing** solutions - i.e. the solution will spiral away from the unit circle.

```python
import scipy.linalg as la
```

```python
def f_euler(A, u, N):
    orbit = np.zeros((N,2))

    dt = 2*np.pi/N
    for i in range(N):
        u = u + dt * A @ u
        orbit[i] = u
    return orbit
```

```python
A = np.array([[0,1],[-1,0]])
u = np.array([1.0,0.0])
N = 64
orbit = f_euler(A, u, N)
```

#### Accuracy

```python
la.norm(np.array([1.0,0.0]) - orbit[-1])
```

    0.3600318484671193

#### Conservation of energy

```python
plt.plot([p @ p for p in orbit])
pass
```

```python
ax = plt.subplot(111)
plt.plot(orbit[:, 0], orbit[:,1], 'o')
ax.axis('square')
plt.axis([-1.5, 1.5, -1.5, 1.5])
pass
```

#### Accuracy and conservation of energy

We can see that forward Euler is not very accurate and also does not preserve energy since the orbit spirals away from the unit circle.

### The trapezoidal method

The trapezoidal method uses the following scheme

$$
\frac{u_{n+1} - u_n}{\Delta t} = \frac{1}{2}  ( A u_{n+1} + A u_{n})
$$

This is an implicit scheme (because $u_{n+1}$ appears on the RHS) whose solution is

$$
u_{n+1} = \left(I - \frac{\Delta t}{2} A \right)^{-1} \left(I + \frac{\Delta t}{2} A \right) u_{n} = B u_n
$$

By inspection, we see that the eigenvalues are the complex conjugates of

$$
\frac{1 + \frac{\Delta t}{2} i}{1 - \frac{\Delta t}{2} i}
$$

whose absolute value is 1 - hence, energy is conserved. If we expand the matrix $B$ using the geometric series and compare with the Taylor expansion, we see that the trapezoidal method has local truncation error $O(h^3)$ and hence accuracy $O(h^2)$, where $h$ is the time step.

```python
def trapezoidal(A, u, N):
    p = len(u)
    orbit = np.zeros((N,p))

    dt = 2*np.pi/N
    for i in range(N):
        u = la.inv(np.eye(p) - dt/2 * A) @ (np.eye(p) + dt/2 * A) @ u
        orbit[i] = u
    return orbit
```

```python
A = np.array([[0,1],[-1,0]])
u = np.array([1.0,0.0])
N = 64
orbit = trapezoidal(A, u, N)
```

#### Accuracy

```python
la.norm(np.array([1.0,0.0]) - orbit[-1])
```

    0.005039305635733781

#### Conservation of energy

```python
plt.plot([p @ p for p in orbit])
pass
```

```python
ax = plt.subplot(111)
plt.plot(orbit[:, 0], orbit[:,1], 'o')
ax.axis('square')
plt.axis([-1.5, 1.5, -1.5, 1.5])
pass
```

### The leapfrog method

The leapfrog method uses a second order difference to update $u_n$. The algorithm simplifies to the following explicit scheme:

- First take one half-step for v
- Then take a full step for u
- Then take one final half step for v

It performs as well as the trapezoidal method, with the advantage of being an explicit scheme and cheaper to calculate, so the leapfrog method is used in HMC.

```python
def leapfrog(A, u, N):
    orbit = np.zeros((N,2))

    dt = 2*np.pi/N
    for i in range(N):
        u[1] = u[1] + dt/2 * A[1] @ u
        u[0] = u[0] + dt * A[0] @ u
        u[1] = u[1] + dt/2 * A[1] @ u
        orbit[i] = u
    return orbit
```

#### If we don't care about the intermediate steps, it is more efficient to just take 1/2 steps at the beginning and end

```python
def leapfrog2(A, u, N):
    dt = 2*np.pi/N

    u[1] = u[1] + dt/2 * A[1] @ u
    for i in range(N-1):
        u[0] = u[0] + dt * A[0] @ u
        u[1] = u[1] + dt * A[1] @ u

    u[0] = u[0] + dt * A[0] @ u
    u[1] = u[1] + dt/2 * A[1] @ u
    return u
```

```python
A = np.array([[0,1],[-1,0]])
u = np.array([1.0,0.0])
N = 64
```

```python
orbit = leapfrog(A, u, N)
```

#### Accuracy

```python
la.norm(np.array([1.0,0.0]) - orbit[-1])
```

    0.0025229913808033464

#### Conservation of energy

```python
plt.plot([p @ p for p in orbit])
pass
```

```python
ax = plt.subplot(111)
plt.plot(orbit[:, 0], orbit[:,1], 'o')
ax.axis('square')
plt.axis([-1.5, 1.5, -1.5, 1.5])
pass
```

## From Hamiltonians to probability distributions

The physical analogy considers the negative log likelihood of the target distribution $p(x)$ to correspond to a potential energy well, with a collection of particles moving on the surface of the well. The state of each particle is given only by its position and momentum (or velocity if we assume unit mass for each particle). In a Hamiltonian system, the total energy $H(x,, v) = U(x) + K(v)$ is conserved. From statistical mechanics, the probability of each state is related to the total energy of the system

$$
\begin{align}
p(x, v) & \propto e^{-H(x, v)} \\
&= e^{-U(x) - K(v)} \\
&= e^{-P(x)}e^{-K(v)} \\
& \propto p(x) \, p(v)
\end{align}
$$

Since the joint distribution factorizes $p(x, v) = p(x)\, p(v)$, we can select an initial random $v$ for a particle, numerically integrate using a finite difference method such as the leapfrog and then use the updated $x^*$ as the new proposal. The acceptance ratio for the new $x^*$ is

$$
\frac{ e^{ -U(x^*)-K(v^*) }} { e^{-U(x)-K(v)} } = e^{U(x)-U(x^*)+K(x)-K(x^*)}
$$

If our finite difference scheme was exact, the acceptance ration would be 1 since energy is conserved with Hamiltonian dynamics. However, as we have seen, the leapfrog method does not conserve energy perfectly and an accept/reject step is still needed.

### Example of HMC

We will explore how HMC works when the target distribution is bivariate normal centered at zero

$$
x \sim N(0, \Sigma)
$$

In practice of course, the target distribution will be the posterior distribution and depend on both data and distributional parameters.

The potential energy or negative log likelihood is proportional to
$$
U(x) = \frac{x^T\Sigma^{-1} x}{2}
$$

The kinetic energy is given by
$$
K(v) = \frac{v^T v}{2}
$$

where the initial $v_0$ is chosen at random from the unit normal at each step.

To find the time updates, we use the Hamiltonian equations and find the first derivatives of total energy with respect to $x$ and $v$

$$
\begin{align}
x' &= \frac{\delta K}{\delta v} &= v \\
v' &= -\frac{\delta U}{\delta x} &= -\Sigma^{-1} x \\
\end{align}
$$

giving us the block matrix

$$
A = \pmatrix{0 & 1 \\ -\Sigma^{-1} & 0}
$$

By using the first derivatives, we are making use of the gradient information on the log posterior to guide the proposal distribution.

#### This is what the target distribution should look like

```python
sigma = np.array([[1,0.8],[0.8,1]])
mu = np.zeros(2)
ys = np.random.multivariate_normal(mu, sigma, 1000)
sns.kdeplot(ys)
plt.axis([-3.5,3.5,-3.5,3.5])
pass
```

#### This is the HMC posterior

```python
def E(A, u0, v0, u, v):
    """Total energy."""
    return (u0 @ tau @ u0 + v0 @ v0) - (u @ tau@u + v @ v)
```

```python
def leapfrog(A, u, v, h, N):
    """Leapfrog finite difference scheme."""
    v = v - h/2 * A @ u
    for i in range(N-1):
        u = u + h * v
        v = v - h * A @ u

    u = u + h * v
    v = v - h/2 * A @ u

    return u, v
```

```python
niter = 100
h = 0.01
N = 100

tau = la.inv(sigma)

orbit = np.zeros((niter+1, 2))
u = np.array([-3,3])
orbit[0] = u
for k in range(niter):
    v0 = np.random.normal(0,1,2)
    u, v = leapfrog(tau, u, v0, h, N)

    # accept-reject
    u0 = orbit[k]
    a = np.exp(E(A, u0, v0, u, v))
    r = np.random.rand()

    if r < a:
        orbit[k+1] = u
    else:
        orbit[k+1] = u0
```

```python
sns.kdeplot(orbit)
plt.plot(orbit[:,0], orbit[:,1], alpha=0.2)
plt.scatter(orbit[:1,0], orbit[:1,1],  c='red', s=30)
plt.scatter(orbit[1:,0], orbit[1:,1],  c=np.arange(niter)[::-1], cmap='Reds')
plt.axis([-3.5,3.5,-3.5,3.5])
pass
```

```python

```


Euler Problem 123
=================

Let pn be the nth prime: 2, 3, 5, 7, 11, ..., and let r be the remainder when $(p_n-1)^n + (p_n+1)^n$ is divided by $p_n^2$.

For example, when n = 3, p3 = 5, and 43 + 63 = 280 ≡ 5 mod 25.

The least value of n for which the remainder first exceeds 109 is 7037.

Find the least value of n for which the remainder first exceeds 1010.

```python
from sympy import sieve
primes = list(sieve.primerange(1, 500000))[0::2]
for k, p in enumerate(primes):
    n = 2*k + 1
    remainder = (2*n*p) % (p*p)
    if remainder > 10**10:
        print(n)
        break
```

    21035

```python

```

Euler 123 - Prime square remainders.ipynb


# Simulating Poisson Networks Efficiently

When I first started my Ph.D. I learned about modeling large wireless network using point processes. They allow you to obtain tractable mathematical models that scale well with the size of the network. The most commonly used model in wireless networks is the homogeneous Poisson point proces (PPP), because it has a reasonable assumption about the network geometry and leads to tractable models. To briefly introduce the concept, say we have a set of two dimensional points $\phi = \{u_1, \cdots\}$ randomly distributed in $\mathbb{R}^2$, such that for every compact set $B \in \mathbb{R}^2$. The set of points $\phi$ is a HPPP if:
* The number of points of $\phi$ in $B$ (denoted by $N(B)$) are poisson distributed with rate $\lambda |B|$, where $\lambda$ is the intensity of the HPPP and $|B|$ is the are of $B$.
* If $B_1$,$B_2$, ..., $B_m$ are all disjoint sets in $\mathbb{R}^2$, then $N(B_1)$,$N(B_2)$, ..., $N(B_m)$ are independent random variables

For more details on the theory behind this I recommending reading the book [*Stochastic Geometry for Wireless Networks*](https://www.amazon.ca/Stochastic-Geometry-Wireless-Networks-Professor/dp/1107014697/ref=sr_1_1?ie=UTF8&qid=1535753229&sr=8-1&keywords=stochastic+geometry+for+wireless+networks), by Martin Haenggi.

In order to simulate a HPPP, we determine an area of interest, say the ball $b(0,R)$, and generate $N$ points inside, where $N$ is drawn from a poisson distribution with rate $\lambda \pi R^2$. An example of a function to generate a realization of a HPPP can be seem below:

```python
import numpy as np

def generate_PPP(lam, radius):
    # Number of points inside a ball
    N = np.random.poisson(lam * np.pi * radius**2)
    # Distance from the origin
    r = radius * np.sqrt(np.random.random(N))
    # Angle from the origin
    ang = 2 * np.pi * np.random.random(N)
    # Matrix of the PPP
    phi = np.vstack((r * np.cos(ang), r * np.sin(ang))).T
    return phi
```

Usually we run a simulation of a HPPP to validate the analytical results. When simulating Poisson networks, we are usually interested in averaging a performance metric, which depends on the distance distribution between the points of a HPPP and the origin or between two different HPPP's, through a Monte Carlo simulation. This means we have to calculate distances quite a few times.

## Naive Distance Calculation

When I first started simulating large networks, my simulations took a long time to run. Let's say you have to calculate the distances between two point processes $\phi_1$ and $\phi_2$ (e.g. a user point process and a base station point process), each with $N_1$ and $N_2$ points respectively. If you do it naively like I used to, you will have a nested for loop, looping through every point of each process, resulting in a complexity of $\mathcal{O}(N_1 N_2)$, as the function shown below

```python
def naive_distance(phi_1, phi_2):
    N1, _ = phi_1.shape
    N2, _ = phi_2.shape
    D = np.zeros((N1, N2))
    for n1 in range(N1):
        for n2 in range(N2):
            D[n1, n2] = np.sqrt((phi_1[n1, 0] - phi_2[n2, 0])**2 + (phi_1[n1, 1] - phi_2[n2, 1])**2)

    return D
```

## More Efficient Approach

Once I started simulating HPPP's with higher densities this simulations started taking hours. I did some research on how to speed them up and figured out we can calculate these distances more efficiently. The distance between points $\mathbf{x} \in \phi_1$ and $\mathbf{y} \in \phi_2$ is given by

\begin{equation}
D_{\mathbf{x}, \mathbf{y}} = \sqrt{(x_1 - y_1)^2 + (x_2 - y_2)^2} = \sqrt{x_1^2 + x_2^2 + y_1^2 + y_2^2 - 2 (x_1 y_1 + x_2 y_2)}
\end{equation}

We can obtain a matrix $\mathbf{D} \in \mathbb{R}^{N_1 \times N_2}$ where each coordinate $D_{i, j}$ is the distance between the $i$-th point in $\phi_1$ to the $j$-th point in $\phi_2$ by

\begin{equation}
D = (\mathbf{\Phi_1} \mathbf{1}_{2 \times 1})^T \mathbf{1}_{1 \times N_2} + (\mathbf{\Phi_2} \mathbf{1}_{2 \times 1})^T \mathbf{1}_{1 \times N_1} - 2 \mathbf{\Phi_1} \mathbf{\Phi_2}^T
\end{equation}

Where $\mathbf{\Phi_1}$ and $\mathbf{\Phi_2}$ are a $N_1 \times 2$ and a $N_2 \times 2$ matrix respectively, where each row is a point from $\phi_1$ and $\phi_2$. The vector/matrix $\mathbf{1}_{i \times j}$ is a $i \times j$ vector/matrix. This expression involves two matrix-vector multiplications, two inner products and one matrix-matrix multiplication. It might seem that the resulting complexity of this second algorithm is larger than the naive one, but processors and linear algebra libraries are optimized by experienced engineers to execute linear algebra operations really fast, which results in a smaller running time than the naive algorithm.

A sample function that calculates the distance by this method is shown below

```python
def efficient_distance(phi_1, phi_2):
    D = (np.sum(phi_1**2, axis=1).reshape((-1, 1))).dot(np.ones((1, phi_2.shape[0]))) + \
        (np.sum(phi_2**2, axis=1).reshape((-1, 1))).dot(np.ones((1, phi_1.shape[0]))).T - \
        2 * phi_1.dot(phi_2.T)
    D = np.sqrt(D)

    return D
```

## Comparison

Now let's compare the running time of both functions

```python
lam = 400
radius = 1
phi_1 = generate_PPP(lam, radius)
phi_2 = generate_PPP(lam, radius)
print("Size of phi_1: {0}".format(phi_1.shape))
print("Size of phi_2: {0}".format(phi_2.shape))
```

    Size of phi_1: (1250, 2)
    Size of phi_2: (1316, 2)

```python
%timeit d_naive = naive_distance(phi_1, phi_2)
```

    5.14 s ± 493 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

```python
%timeit d_eff = efficient_distance(phi_1, phi_2)
```

    46.4 ms ± 1.43 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

As you can see, on average, the the efficient approach was 110.78 times faster than the naive one.

You can download a Jupyter notebook with this tutorial [here](https://jvce92.github.io/files/simulating_poisson_networks.ipynb)


# Introduction to Deep Learning with PyTorch

In this notebook, you'll get introduced to [PyTorch](http://pytorch.org/), a framework for building and training neural networks. PyTorch in a lot of ways behaves like the arrays you love from Numpy. These Numpy arrays, after all, are just tensors. PyTorch takes these tensors and makes it simple to move them to GPUs for the faster processing needed when training neural networks. It also provides a module that automatically calculates gradients (for backpropagation!) and another module specifically for building neural networks. All together, PyTorch ends up being more coherent with Python and the Numpy/Scipy stack compared to TensorFlow and other frameworks.

## Neural Networks

Deep Learning is based on artificial neural networks which have been around in some form since the late 1950s. The networks are built from individual parts approximating neurons, typically called units or simply "neurons." Each unit has some number of weighted inputs. These weighted inputs are summed together (a linear combination) then passed through an activation function to get the unit's output.

Mathematically this looks like:

$$
\begin{align}
y &= f(w_1 x_1 + w_2 x_2 + b) \\
y &= f\left(\sum_i w_i x_i \right)
\end{align}
$$

With vectors this is the dot/inner product of two vectors:

$$
h = \begin{bmatrix}
x_1 \, x_2 \cdots  x_n
\end{bmatrix}
\cdot
\begin{bmatrix}
           w_1 \\
           w_2 \\
           \vdots \\
           w_n
\end{bmatrix}
$$

### Stack them up!

We can assemble these unit neurons into layers and stacks, into a network of neurons. The output of one layer of neurons becomes the input for the next layer. With multiple input units and output units, we now need to express the weights as a matrix.

We can express this mathematically with matrices again and use matrix multiplication to get linear combinations for each unit in one operation. For example, the hidden layer ($h_1$ and $h_2$ here) can be calculated

$$
\vec{h} = [h_1 \, h_2] =
\begin{bmatrix}
x_1 \, x_2 \cdots \, x_n
\end{bmatrix}
\cdot
\begin{bmatrix}
           w_{11} & w_{12} \\
           w_{21} &w_{22} \\
           \vdots &\vdots \\
           w_{n1} &w_{n2}
\end{bmatrix}
$$

The output for this small network is found by treating the hidden layer as inputs for the output unit. The network output is expressed simply

$$
y =  f_2 \! \left(\, f_1 \! \left(\vec{x} \, \mathbf{W_1}\right) \mathbf{W_2} \right)
$$

## Tensors

It turns out neural network computations are just a bunch of linear algebra operations on *tensors*, a generalization of matrices. A vector is a 1-dimensional tensor, a matrix is a 2-dimensional tensor, an array with three indices is a 3-dimensional tensor (RGB color images for example). The fundamental data structure for neural networks are tensors and PyTorch (as well as pretty much every other deep learning framework) is built around tensors.

With the basics covered, it's time to explore how we can use PyTorch to build a simple neural network.

```python
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import numpy as np
import torch

import helper
```

First, let's see how we work with PyTorch tensors. These are the fundamental data structures of neural networks and PyTorch, so it's imporatant to understand how these work.

```python
x = torch.rand(3, 2)
x
#torch.is_tensor(x)
```

    tensor([[0.7640, 0.1932],
            [0.0554, 0.8443],
            [0.0084, 0.7823]])

```python
y = torch.ones(x.size())
y
```

    tensor([[1., 1.],
            [1., 1.],
            [1., 1.]])

```python
z = x + y
z
```

    tensor([[1.7640, 1.1932],
            [1.0554, 1.8443],
            [1.0084, 1.7823]])

In general PyTorch tensors behave similar to Numpy arrays. They are zero indexed and support slicing.

```python
z[0]
```

    tensor([1.7640, 1.1932])

```python
z[:, 1:]
```

    tensor([[1.1932],
            [1.8443],
            [1.7823]])

Tensors typically have two forms of methods, one method that returns another tensor and another method that performs the operation in place. That is, the values in memory for that tensor are changed without creating a new tensor. In-place functions are always followed by an underscore, for example `z.add()` and `z.add_()`.

```python
# Return a new tensor z + 1
z.add(1)
```

    tensor([[2.7640, 2.1932],
            [2.0554, 2.8443],
            [2.0084, 2.7823]])

```python
# z tensor is unchanged
z
```

    tensor([[1.7640, 1.1932],
            [1.0554, 1.8443],
            [1.0084, 1.7823]])

```python
# Add 1 and update z tensor in-place
z.add_(1)
```

    tensor([[2.7640, 2.1932],
            [2.0554, 2.8443],
            [2.0084, 2.7823]])

```python
# z has been updated
z
```

    tensor([[2.7640, 2.1932],
            [2.0554, 2.8443],
            [2.0084, 2.7823]])

### Reshaping

Reshaping tensors is a really common operation. First to get the size and shape of a tensor use `.size()`. Then, to reshape a tensor, use `.resize_()`. Notice the underscore, reshaping is an in-place operation.

```python
z.size()
```

    torch.Size([3, 2])

```python
z.resize_(2, 3)
```

    tensor([[2.7640, 2.1932, 2.0554],
            [2.8443, 2.0084, 2.7823]])

```python
z
```

    tensor([[2.7640, 2.1932, 2.0554],
            [2.8443, 2.0084, 2.7823]])

## Numpy to Torch and back

Converting between Numpy arrays and Torch tensors is super simple and useful. To create a tensor from a Numpy array, use `torch.from_numpy()`. To convert a tensor to a Numpy array, use the `.numpy()` method.

```python
a = np.random.rand(4,3)
a
#torch.is_tensor(a)
```

    array([[ 0.12711662,  0.92413542,  0.42696275],
           [ 0.32314096,  0.68522034,  0.63669388],
           [ 0.85865113,  0.141158  ,  0.69359639],
           [ 0.1663766 ,  0.66859912,  0.6212608 ]])

```python
b = torch.from_numpy(a)
b
#torch.is_tensor(b)
```

    tensor([[0.1271, 0.9241, 0.4270],
            [0.3231, 0.6852, 0.6367],
            [0.8587, 0.1412, 0.6936],
            [0.1664, 0.6686, 0.6213]], dtype=torch.float64)

```python
c = b.numpy()
c
```

    array([[ 0.12711662,  0.92413542,  0.42696275],
           [ 0.32314096,  0.68522034,  0.63669388],
           [ 0.85865113,  0.141158  ,  0.69359639],
           [ 0.1663766 ,  0.66859912,  0.6212608 ]])

The memory is shared between the Numpy array and Torch tensor, so if you change the values in-place of one object, the other will change as well.

```python
# Multiply PyTorch Tensor by 2, in place
b.mul_(2)
```

    tensor([[0.5085, 3.6965, 1.7079],
            [1.2926, 2.7409, 2.5468],
            [3.4346, 0.5646, 2.7744],
            [0.6655, 2.6744, 2.4850]], dtype=torch.float64)

```python
# Numpy array matches new values from Tensor
a
```

    array([[ 0.50846648,  3.69654169,  1.70785102],
           [ 1.29256382,  2.74088136,  2.54677551],
           [ 3.4346045 ,  0.56463199,  2.77438555],
           [ 0.66550639,  2.67439646,  2.48504322]])

```python
c
```

    array([[ 0.50846648,  3.69654169,  1.70785102],
           [ 1.29256382,  2.74088136,  2.54677551],
           [ 3.4346045 ,  0.56463199,  2.77438555],
           [ 0.66550639,  2.67439646,  2.48504322]])

```python

```


# Identification of nonlinear systems

## Nonlinear estimation problem

Given a set of training data $\{(x_i, y_i)\}_{i\in I}$ with $x_i\in X$ and $y_i\in \mathbb{R}$, we seek to determine a function $g:X\to \mathbb{R}$ to "best" interpolate these data according to the model

$$
y = g(x).
$$

We will focus on functions in certain functional spaces. We shall, in particular, restrict our attention to a Hibert space $H$.

Our objective is to solve the following regularized least-squares problem over $H$ with a regularizer $R:H\to\mathbb{R}$ and regularization parameter $\gamma>0$

$$
\operatorname*{Minimize}_{g\in H} \sum_{i\in I} (y_i - g(x_i))^2 + \gamma R(g)
$$

A solution (minimizer) of this problem (when there exists one) is denoted by $g^\star$.

Good news: although this problem looks too abstract (the minimization is carried out over an abstract space $H$), it is possible to derive a closed-form solution for $g$ provided we endow $H$ with a certain structure. This is the structure of a reproducing kernel Hilbert space.

## Reproducing kernel Hilbert spaces

#### RKHS definition

A [reproducing kernel Hilbert space](https://en.wikipedia.org/wiki/Reproducing_kernel_Hilbert_space) (RKHS) is a Hilbert space $H$ of real-valued function over a nonempty set $X$ in which pointwise evaluation, defined as
$$
L_x(f) = f(x),\ (\forall f\in H),
$$
is a continuous mapping.

RHKS are fully characterised by a *positive semidefinite kernel* which is a symmetric mapping $K:X\times X\to\mathbb{R}$ such that
$$
\sum_{i,j=1}^{n}c_i c_j X(x_i, x_j) \geq 0,
$$
for all $p\in\mathbb{N}$, $c_1,\ldots, c_n\in\mathbb{R}$.

#### Moore-Aronszajn Theorem

According to the [Moore-Aronszajn Theorem](https://en.wikipedia.org/wiki/Reproducing_kernel_Hilbert_space#Moore%E2%80%93Aronszajn_theorem), RHKS are characterised by a positive semidefinite kernel and, conversely, positive semidefinite kernels induce RKHS.

To every RKHS $H$ corresponds a *unique* positive semidefinite kernel $K$ called the **reproducing kernel** which is such that
$$
g(x) = \langle g, K_x \rangle,\ (\forall x \in X)(\forall g \in H)
$$
Conversely, to a positive semidefinite kernel $K$ corresponds a unique induced RKHS whose reproducing kernel is $K$.

#### The norm of an RKHS

Let $K$ be a kernel and $H$ the corresponding RKHS.

The following functions are in $H$:

$$
g(x) = \sum_{i=1}^{p} a_i K(x_i, x),
$$

for some $a_i$ and $x_i$, $i=1,\ldots,p$.

The norm of $g$ is

$$
\|g\|_{H} = \sum_{i,j=1}^{p} a_i a_j K(x_i, x_j).
$$

#### The Gram matrix

Last step before we state a result of central importance in system identification, functional analysis and learning theory at large.

Let $Y = (y_i)_{i}\in\mathbb{R}^N$ be the obtained measurements and $x_i$ be the corresponding state vectors (typically past values of $y$ and $u$).

The **Gram matrix** or **kernel matrix** is the matrix $K\in\mathbb{R}^{N\times N}$ defined as
$$
K = (K(x_i, x_j))_{i,j}
$$

#### Representer theorem

Consider the identification problem:

$$
\mathbb{P}: \operatorname*{Minimize}_{g\in H} \sum_{i\in I} (y_i - g(x_i))^2 + \gamma \|g\|_H,
$$

where $\|{}\cdot{}\|_H$ is the norm of a RKHS $H$.

According to the [representer theorem](https://en.wikipedia.org/wiki/Representer_theorem), this problem has an **explicit solution** which is

$$
g^\star(x) {}={} \sum_{i=1}^{N} c_i^\star K(x_i, x)
$$

with $c^\star = (c_i^\star)_i$ given by

$$
c^\star = (K + \gamma I_N)^{-1}Y,
$$

where $I_N$ is the $N\times N$ identity matrix.

## Kernels

Let us have a look at a few examples of kerners:

#### Linear kernel

Let $P\in\mathbb{R}^{n\times n}$ be a symmetric positive semidefinite matrix and define

$$
K(x_1, x_2) = x_1'Px_2.
$$

According to the Moore-Aronszajn Theorem, the RHKS which is induced by this kernel is that of linear functions

$$
g(x) = K_{c}(x) = c'Px.
$$

If $P$ is full rank, then the RKHS is the space of all function $g(x) = \theta' x$ with $\theta\in\mathbb{R}^n$ and the norm on that space is

$$
\|g\|_H = \theta' P^{-1} \theta.
$$

#### Gaussian kernel

The [Gaussian kernel](https://en.wikipedia.org/wiki/Radial_basis_function_kernel) is the parametric kernel

$$
K(x_1, x_2) = \exp \left( -\rho \|x_1 - x_2\|^2 \right),
$$

with $\rho>0$.

#### Exponential kernel

The exponential kernel is very similar to the Gaussian kernel

$$
K(x_1, x_2) = \exp \left( -\rho \|x_1 - x_2\| \right),
$$

with $\rho>0$.

The only difference is that there is the norm $\|{}\cdot{}\|$ in lieu of the squared norm.

#### Sigmoid kernel

The sigmoid kernel is also known as the multilayer perceptron kernel. It is very popular in machine learning and, in particular, in SVM. It is defined as

$$
K(x_1, x_2) = \tanh (a x_1' x_2 + b).
$$

```python
# Necessary imports
import numpy as np
import scipy as sp
import cvxpy as cp
import matplotlib.pyplot as plt
```

## Kernel methods for identification in action

Let us give a simple nonlinear regression example.

Given a collection of data $\{(x_i, y_i)\}_{i=1}^{N}$ with $x_i\in\mathbb{R}^n$ and $y_i\in\mathbb{R}$, we aim at determining a function $g$ which solves $\mathbb{P}$.

```python
# KERNELS

class kernel:

    def __init__(self):
        pass

    def call(self, x, y):
        pass

class gaussian_kernel(kernel):

    def __init__(self, rho):
        super().__init__()
        self._rho = rho

    def call(self, x, y):
        r = np.exp( -self._rho* np.linalg.norm(x-y)**2 )
        return r

class exponential_kernel(kernel):

    def __init__(self, rho):
        super().__init__()
        self._rho = rho

    def call(self, x, y):
        r = np.exp( -self._rho* np.linalg.norm(x-y) )
        return r

class sigmoid_kernel(kernel):

    def __init__(self, a, b):
        super().__init__()
        self._a = a
        self._b = b

    def call(self, x, y):
        r = np.tanh(self._a * np.dot(x,y) + self._b)
        return r

# Example of use of a kernel:
sig_kernel = sigmoid_kernel(0.5,0.1)
kxy = sig_kernel.call(np.array([1,2]),np.array([4,3]))
```

### Exercise 1

The **cubic spline kernel** is a function $K:\mathbb{R}^n\times\mathbb{R}^n\to\mathbb{R}$ given by

$$
            K(x, y)
{}={}
            \prod_{i=1}^{n}
                1
                +
                x_i y_i
                +
                x_i y_i \min\{x_i, y_i\}
                -
                \frac{x_i + y_i}{2} \min\{x_i, y_i\}^2
                +
                \frac{\min\{x_i, y_i\}^3}{3}
$$

Implement the cubic spline kernel...

```python
# EXERCISE 1 - CODE
class cubic_spline_kernel(kernel):

    def __init__(self):
        super().__init__()

    def call(self, x, y):
        # YOUR CODE GOES HERE
        return 0

# Test your code
x = np.array([0.2, 0.5, 0.3])
y = np.array([0.3, 0.1, -0.1])
correct_cubic_spline_value = 1.08864
cub_spli_krnl = cubic_spline_kernel()
cub_spli_value = cub_spli_krnl.call(x,y)

# This will fail until you complete the implementation above
assert((np.abs(correct_cubic_spline_value - cub_spli_value) < 1e-5)), "wrong value"
```

```python
class rkhs_function:

    def __init__(self, X, c, krnl):
        if not isinstance(krnl, kernel):
            raise TypeError
        self._X = X
        shape_X = np.shape(X)
        self._num_data = shape_X[0]
        self._dim_data = shape_X[1]
        self._krnl = krnl
        self._c = c
        pass

    def _call(self, x):
        gx = 0.
        for i in range(0, self._num_data):
            gx += self._c[i] * self._krnl.call(self._X[i], x)
        return gx

    def call(self,x):
        shape_x = np.shape(x)
        if (x.ndim == 1):
            gx = self._call(x)
        else:
            n_data_points = shape_x[0]
            gx = np.zeros(n_data_points)
            for k in range(0, n_data_points):
                gx[k] = self._call(x[k])
        return gx

# Example of use
X = np.array([[1,2,5],[3,4,-1]])                 # data (centers of kernel functions)
c = np.array([0.1, 0.2])                         # coefficients
gauss_krnl = gaussian_kernel(0.5)                # choose a kernel
g_function = rkhs_function(X, c, gauss_krnl)     # create a function in the RHKS
gx1 = g_function.call(np.array([[1,1.9,4]]))
gx2 = g_function.call(np.array([[1,1.9,4], [1,2,3]]))        # use the function (on multiple vectors)

```

```python
# We now define the class `trainer` which will facilitate the training procedure
class trainer:

    def __init__(self, krnl, gamma_regularizer):
        self._krnl = krnl
        self._gamma_regularizer = gamma_regularizer
        pass

    def _construct_gram_matrix(self, x_data):
        # Construct the Gram matrix
        shape_data = np.shape(x_data)
        n_data = shape_data[0]
        gram_matrix = np.zeros([n_data, n_data])
        for i in range(0, n_data):
            for j in range(0, n_data):
                gram_matrix[i,j] = self._krnl.call(x_data[i], x_data[j])
        return gram_matrix

    def train(self, x_data, y_data):
        shape_data = np.shape(x_data)
        n_data = shape_data[0]
        gram_matrix = self._construct_gram_matrix(x_data)
        # Solve the least squares problem using the representer theorem
        c_star = np.linalg.solve(                                     \
                     self._gamma_regularizer * np.identity(n_data)    \
                      + gram_matrix, y_data)
        g_star = rkhs_function(x_data, c_star, self._krnl)
        return g_star
```

```python
# Construct some training data
n_training_data = 300
dim_x_data = 2
model_noise = 0.05

def hidden_function(x):
    return np.sin(0.4*x[0]) + np.sin(0.6*x[1])*x[0]*x[1] +  3*x[0]*(1-x[0]) +         \
             np.random.normal(0, model_noise, 1)

x_training_data  = np.zeros([n_training_data, dim_x_data])
y_training_data  = np.zeros([n_training_data, 1])
for i in range(0, n_training_data):
    x_training_data[i,0] = np.sin(0.2*i**1.3 + 1.)
    x_training_data[i,1] = np.sin(0.4*(i**1.2) + .5)
    y_training_data[i] = hidden_function(x_training_data[i])

# Choose a kernel
rho_gaussian = 0.5
gamma_reg = 0.05
krnl = gaussian_kernel(rho_gaussian)
trn = trainer(krnl, gamma_reg)
g_star = trn.train(x_training_data, y_training_data)

```

```python
# Validate against the training dataset
y_training_data_predicted = g_star.call(x_training_data)

# Create some arbitrary test data
def create_test_data(n_test_data):
    x_test_data  = np.zeros([n_test_data, dim_x_data])
    y_test_data  = np.zeros(n_test_data)
    y_test_data_predicted  = np.zeros(n_test_data)
    for i in range(0, n_test_data):
        x_test_data[i,0] = np.sin(0.1*i + 1.)
        x_test_data[i,1] = np.sin(2*(i**2) + 1.5)
        y_test_data[i] = hidden_function(x_test_data[i])
    return x_test_data, y_test_data

n_test_points = 40
x_test_data, y_test_data = create_test_data(n_test_points)
y_test_data_predicted = g_star.call(x_test_data)

# Plot actual-vs-predicted
plt.plot(y_training_data, y_training_data_predicted, marker='+', color='r', markersize=6, linestyle='None')
plt.plot(y_test_data, y_test_data_predicted, marker='x', color='g', markersize=8, linestyle='None')
miny = np.min([np.min(y_training_data), np.min(y_training_data_predicted)])
maxy = np.max([np.max(y_training_data), np.max(y_training_data_predicted)])
plt.plot( np.array([miny,maxy]), np.array([miny,maxy]), color='k' )
plt.ylim(miny,maxy)
plt.xlim(miny,maxy)
plt.grid()
plt.xlabel('y (actual)')
plt.ylabel('y (predicted)')
plt.legend(['training data','test data','perfect fit'])
plt.show()

error = y_test_data - y_test_data_predicted;
rmse= np.sqrt(np.linalg.norm(error)**2/n_test_points)
rmse
```

```python
# !  Let us now select appropriate values for the Gaussian     !
# !  kernel parameter `rho` and the regularization parameter   !
# !  so as to achieve good fit in the test data. We will       !
# !  measure the generalization capacity of the model using    !
# !  the RME of the residuals                                  !

points_rho   = 7
points_gamma = 5
rho_candidates = np.linspace(0.001, 1.5, points_rho)
gamma_reg_candidates = np.linspace(0.001, 2, points_gamma)

# The RMSE results will be registered in a matrix with 3 columns
# which correspond to `rho`, `gamma` and `rmse` respectively
rmse_results = np.zeros([points_rho*points_gamma, 3])

n_test_data = 80
x_test_data, y_test_data = create_test_data(n_test_data)
i = 0
for rho in rho_candidates:
    for gamma_reg in gamma_reg_candidates:
        krnl = gaussian_kernel(rho)                          # kernel
        trn = trainer(krnl, gamma_reg)                       # trainer w/t given kernel & regul.
        g_star = trn.train(x_training_data, y_training_data) # train with the same training data
        y_test_data_predicted = g_star.call(x_test_data)     # predict test data
        error = y_test_data - y_test_data_predicted
        rmse = np.sqrt(np.linalg.norm(error)**2/n_test_data)
        rmse_results[i,0] = rho                              # register results
        rmse_results[i,1] = gamma_reg
        rmse_results[i,2] = rmse
        i = i + 1

```

```python
from mpl_toolkits.mplot3d import Axes3D
fig = plt.figure(figsize=(11, 5))
ax = fig.add_subplot(111, projection='3d')
ax.scatter(rmse_results[:,0], rmse_results[:,1], rmse_results[:,2])
ax.set_xlabel('rho')
ax.set_ylabel('gamma')
ax.set_zlabel('RMSE')
plt.show()
```

### Exercise 2

- Repeat the above experiment using the sigmoid kernel and the cubic spline kernel
- Present your results with appropriate plots
- Determine the best model using the RMSE on test data

```python
# EXERCISE 2
# Your code goes here
```

## Identification of nonlinear dynamical systems

#### A few words about nonlinear systems

Consider a nonlinear discrete-time dynamical system of the form

$$
y(k) = F(y(k), \ldots, y(k-n), u(k), \ldots, u(k-m)),
$$

for $n,m\in\mathbb{N}$, typically with $n>m$ and $y(k)\in\mathbb{R}$, $u(k) \in \mathbb{R}$.

Therefore, $F$ is a mapping $F:\mathbb{R}^{n+m+2}\to\mathbb{R}$.

At time $k$ we may define the variable $x(k) = (y(k), \ldots, y(k-n), u(k), \ldots, u(k-m))$ which serves as the system state.

Then, the system dynamics becomes

$$
y(k) = F(x(k)).
$$

Suppose we have gathered input-output data from the above system and we have packed them in the form of a dataset $\{(x(t), y(t))\}_{t=1}^{N}$.

Then problem $\mathbb{P}$ on this data defines exactly the system identification problem:

$$
\operatorname*{Minimize}_{f\in H} \sum_{i\in I} (y(t) - f(x(t)))^2 + \gamma \|f\|_H
$$

#### Example: logistic map population dynamics

Consider the following nonlinear dynamical system

$$
\begin{align}
x(k+1) {}={}& F(x(k))\\
{}:={}& rx(k)(1-x(k)),
\end{align}
$$

with $r\in [0,4]$.

```python
# generate input signal
n_data = 50

# produce output data
y_init = .6                        # initial condition
y_logistic = np.zeros(n_data)      # output data
y_logistic[0] = y_init

r_logistic = 3.6
for t in range(1, n_data):
    y_logistic[t] = r_logistic*y_logistic[t-1]*(1-y_logistic[t-1])

# Define the system state
x_training = np.zeros([n_data-1, 1])
y_training = np.zeros(n_data)

for i in range(0, n_data-1):
    x_training[i,:] = y_logistic[i]

y_training = y_logistic[1:]

rho = 6
gamma = 1e-3
krnl = gaussian_kernel(rho)
trn = trainer(krnl, gamma)
f_star = trn.train(x_training, y_training)

y_init = .3
n_pred = 30
y_predicted = np.zeros(n_pred); y_actual = np.zeros(n_pred)
y_predicted[0] = y_init;y_actual[0] = y_init

for t in range(1,n_pred):
    y_predicted[t] = f_star.call(np.array([y_predicted[t-1]]))
    y_actual[t] = r_logistic*y_actual[t-1]*(1-y_actual[t-1])

plt.plot(range(0, n_pred), y_actual[:n_pred], marker='x')
plt.plot(range(0, n_pred), y_predicted,marker='+', color='r', markersize=6, linestyle=':')
plt.xlabel('time'); plt.ylabel('y (response)')
plt.grid()
plt.legend(['actual' , 'predicted'])
plt.show()
```