# MATH 452: Final Project

Remark:

Please upload your solutions for this project to Canvas with a file named "Final_Project_yourname.ipynb".

=================================================================================================================

## Problem 1 [20%]:

Consider the following linear system

\begin{equation}\label{matrix}
A\ast u =f,
\end{equation}
or equivalently $u=\arg\min \frac{1}{2} (A* v,v)_F-(f,v)_F$, where $(f,v)_F =\sum\limits_{i,j=1}^{n}f_{i,j}v_{i,j}$ is the Frobenius inner product.
Here $\ast$ represents a convolution with one channel, stride one and zero padding one. The convolution kernel $A$ is given by
$$
A=\begin{bmatrix} 0 & -1 & 0 \\ -1 & 4 & -1 \\ 0 & -1 & 0 \end{bmatrix},~~
$$
the solution $ u \in \mathbb{R}^{n\times n} $, and the RHS $ f\in \mathbb{R}^{n\times n}$ is given by $f_{i,j}=\dfrac{1}{(n+1)^2}.$

### Tasks:
Set $J=4$, $n=2^J-1$ and the number of iterations $M=100$. Use the gradient descent method and the multigrid method to solve the above problem with a random initial guess $u^0$. Let $u_{GD}$ and $u_{MG}$ denote the solutions obtained by gradient descent and multigrid respectively.

* [5%] Plot the surface of solution $u_{GD}$ and $u_{MG}$.

* [10%] Define error $e_{GD}^m = \|A * u^{m}_{GD}- f\|_F=\sqrt{\sum\limits_{i=1}^{n}\sum\limits_{j=1}^{n} |(A * u^{m}_{GD}- f)_{i,j}}|^2 $ for $m=0,1,2,3,...,M$. Similarly, we define the multigrid error $e_{MG}^m$. Plot the errors $e_{GD}^m$ and $e_{MG}^m$ as a function of the iteration $m$ (your x-axis is $m$ and your y-axis is the error). Put both plots together in the same figure.

* [5%] Find the minimal $m_1$ for which $e^{m_1}_{GD} <10^{-5}$ and the minimal $m_2$ for which $e^{m_2}_{MG} <10^{-5}$, and report the computational time for each method. Note that $m_1$ or $m_2$ may be greater than $M=100$, in this case you will have to run more iterations.

### Remark:

Below are examples of using gradient descent and multigrid iterations for M-times
* #### For gradient descent method with $\eta=\frac{1}{8}$, you need to write a code:

    Given initial guess $u^0$
$$
\begin{align}
&\text{for    }  m =  1,2,...,M\\
&~~~~\text{for    }  i,j = 1: n\\
&~~~~~~~~u_{i,j}^{m} = u_{i,j}^{m-1}-\eta(f_{i,j}-(A\ast u^{m-1})_{i,j})\\
&~~~~\text{endfor}\\
&\text{endfor}
\end{align}
$$

* #### For multigrid method, we have provided the framework code in F02_MultigridandMgNet.ipynb:

    Given initial guess $u^0$
$$
\begin{align}
&\text{for    }  m =  1,2,...,M\\
&~~~~u^{m} = MG1(u^{m-1},f, J, \nu)\\
&\text{endfor}
\end{align}
$$

=================================================================================================================

## Problem 2 [50%]:

Use SGD with momentum and weight decay to train MgNet on the Cifar10 dataset. Use 120 epochs, set the initial learning rate to 0.1, momentum to 0.9, weight decay to 0.0005, and divide the learning rate by 10 every 30 epochs. (The code to do this has been provided.) Let $b_i$ denote the test accuracy of the model after $i$ epochs, and let $b^*$ = $\max_i(b_i)$ be the best test accuracy attained during training.

### Tasks:
   * [30%] Train MgNet with the following three sets of hyper-parameters (As a reminder, the hyper-parameters of MgNet are $\nu$, the number of iterations of each layer, $c_u$, the number of channels for $u$, and $c_f$, the number of channels for $f$.):

    (1) $\nu=$[1,1,1,1], $c_u=c_f=64$.

    (2) $\nu=$[2,2,2,2], $c_u=c_f=64$.

    (3) $\nu=$[2,2,2,2], $c_u=c_f=64$, try to improve the test accuracy by implementing MgNet with $S^{l,i}$, which means different iterations in the same layer do not share the same $S^{l}$.

   * For each numerical experiment above, print the results with the following format:

       "Epoch: i, Learning rate: lr$_i$, Training accuracy: $a_i$, Test accuracy: $b_i$"

        where $i=1,2,3,...$ means the $i$-th epoch,  $a_i$ and $b_i$ are the training accuracy and test accuracy computed at the end of $i$-th epoch, and lr$_i$ is the learning rate of $i$-th epoch.

   * [10%] For each numerical experiment above, plot the test accuracy against the epoch count, i.e. the x-axis is the number of epochs $i$ and y-axis is the test accuracy $b_i$. An example plot is shown in the next cell.

   * [10%] Calculate the number of parameters that each of the above models has. Discuss why the number of parameters is different (or the same) for each of the models.

```python
from IPython.display import Image
Image(filename='plot_sample_code.png')

```

```python
# You can calculate the number of parameters of my_model by:
model_size = sum(param.numel() for param in my_model.parameters())

```

=================================================================================================================

## Problem 3 [25 %]:

Try to improve the MgNet Accuracy by increasing the number of channels. (We use the same notation as in the previous problem.) Double the number of channels to $c_u=c_f=128$ and try different $\nu$ to maximize the test accuracy.

### Tasks:
   * [20%] Report $b^{*}$, $\nu$ and the number of parameters of your model for each of the experiments you run.
   * [5%] For the best experiment, plot the test accuracy against the epoch count, i.e. the x-axis is the number of epochs $i$ and y-axis is the test accuracy $b_i$. (Same as for the previous problem.)

```python
# You can calculate the number of parameters of my_model by:
model_size = sum(param.numel() for param in my_model.parameters())

```

=================================================================================================================

## Problem 4 [5%]:

Continue testing larger MgNet models (i.e. increase the number of channels) to maximize the test accuracy. (Again, we use the same notation as in problem 2.)

### Tasks:

+  [5%] Try different training strategies and MgNet architectures with the goal of achieving $b^*>$ 95%. Hint: you can tune the number of epochs, the learning rate schedule, $c_u$, $c_f$, $\nu$, try different $S^{l,i}$ in the same layer $l$, etc...

=================================================================================================================


```python
import numpy as np
import sympy as sy
import matplotlib.pyplot as plt
from openmm import unit
```

# Harmonic well potential

The harmonic well potential is described by the following expression:

$$
f(x)=\frac{1}{2} k \left(x^2 + y^2 + z^2 \right)
$$ (harmonic-well-potential)

Where $k$ is the only parameter and represents the stiffness of the harmonic potential -or the stiffness of the harmonic spring described by the hookes' law-. Notice that the potential for potential dimensions $Y$ and $Z$, has the same shape. In this way we have a three dimensional harmonic well. But let's see here only the proyection over a single dimension $X$ since $Y$ and $Z$ are decorrelated, and there by will behave as $X$.

```python
def harmonic_well(x,k):
    return 0.5*k*x**2
```

```python
k=5.0 * unit.kilocalories_per_mole/ unit.nanometers**2 # stiffness of the harmonic potential

x_serie = np.arange(-5., 5., 0.05) * unit.nanometers

plt.plot(x_serie, harmonic_well(x_serie, k), 'r-')
plt.ylim(-1,5)
plt.xlim(-2,2)
plt.grid()
plt.xlabel("X ({})".format(unit.nanometers))
plt.ylabel("Energy ({})".format(unit.kilocalories_per_mole))
plt.title("Harmonic Well")
plt.show()
```

Different values of $k$ can be tested to graphically see how this parameter accounts for the openness of the well's arms. Or, as it is described below, the period of oscillations of a particle of mass $m$ in a newtonian dynamics.

The hooks' law describes the force suffered by a mass attached to an ideal spring as:

$$
F(x) = -k(x-x_{0})
$$ (hooks_law)

Where $k$ is the stiffness of the spring and $x_{0}$ is the equilibrium position. Now, since the force is minus the gradient of the potential energy $V(x)$,

$$
F(x) = -\frac{d V(x)}{dx},
$$ (grad_potential)

we can proof that the spring force is the result of the first harmonic potential derivative:

$$
V(x) = \frac{1}{2} k (x-x_{0})^{2}
$$ (x_potential)

And the angular frequency of oscillations of a spring, or a particle goberned by the former potential, is:

$$
\omega = \sqrt{\frac{k}{m}}
$$ (omega)

Where $m$ is the mass of the particle. This way the potential can also be written as:

$$
V(x) = \frac{1}{2} k (x-x_{0})^{2} = \frac{1}{2} m \omega^{2} (x-x_{0})^{2}
$$ (x_potential_2)

Finnally, the time period of these oscillations are immediately computed from the mass of the particle, $m$, and the stiffness parameter $k$. Given that by definition:

$$
T = 2\pi / \omega
$$ (period)

Then:

$$
T = 2\pi \sqrt{\frac{m}{k}}
$$ (period_2)


# Iterative Solvers 3 - The Conjugate Gradient Method

## Symmetric positive definite matrices

A very frequent type of matrices are symmetric positive definite matrices. Let $A\in\mathbb{R}^{n\times n}$ be a symmetric matrix (that is $A^T=A$). $A$ is called symmetric positive definite if

$$
x^TAx > 0, \forall x\neq 0.
$$

This is equivalent to the condition that all eigenvalues of $A$ are larger than zero (remember that symmetric matrices only have real eigenvalues).

One application of symmetric positive definite matrices are energy functionals. The expression $x^TAx$ arises when discretising functional involving kinetic energies (e.g. energies of the from $E = \frac{1}{2}m|\nabla f|^2$ for f a given function).

For linear systems involving symmetric positive definite matrices we can derive a special algorithm, namely the Method of Conjugate Gradients (CG).

## Lanczos - Arnoldi for symmetric matrices

Let us start with the Arnoldi recurrence relation

$$
AV_m = V_mH_m + h_{m+1,m}v_{m+1}e_m^T
$$

We know that $H_m$ is an upper Hessenberg matrix (i.e. the upper triangular part plus the first lower triangular diagonal can only be nonzero). Also, we know from the orthogonality of the $v_k$ vectors that

$$
V_m^TAV_m = H_m.
$$

Let $A$ now be symmetric. From the symmetry of $A$ an even nicer structure for $H_m$ arises. $H_m$ is upper Hessenberg, but now it is also symmetric. The only possible type of matrices to satisfy this condition are tridional matrices. These are matrices, where only the diagonal and the first upper and lower super/subdiagonals are nonzero.

Let us test this out. Below you find our simple implementation of Arnoldi's method. We then plot the resulting matrix $H_m$.

```python
import numpy as np

def arnoldi(A, r0, m):
    """Perform m-1 step of the Arnoldi method."""
    n = A.shape[0]

    V = np.empty((n, m + 1), dtype=np.float64)
    H = np.zeros((m+1, m), dtype=np.float64)

    V[:, 0] = r0 / np.linalg.norm(r0)

    for index in range(m):
        # Multiply the previous vector with A
        tmp = A @ V[:, index]
        # Now orthogonalise against the previous basis vectors
        h = V[:, :index + 1].T @ tmp # h contains all inner products against previous vectors
        H[:index + 1, index] = h
        w = tmp - V[:, :index + 1] @ h # Subtract the components in the directions of the previous vectors
        # Normalise and store
        H[index + 1, index] = np.linalg.norm(w)
        V[:, index + 1] = w[:] / H[index + 1, index]

    return V, H
```

The following code creates a random symmetric positive definite matrix.

```python
from numpy.random import RandomState

n = 500

rand = RandomState(0)
Q, _ = np.linalg.qr(rand.randn(n, n))
D = np.diag(rand.rand(n))
A = Q.T @ D @ Q
```

Now let's run Arnoldi's method and plot the matrix H. We are adding some artificial noise so as to ensure for the log-plot that all values are nonzero. The colorscale shows the logarithm of the magnitude of the entries.

```python
%matplotlib inline
from matplotlib import pyplot as plt

m = 50
r0 = rand.randn(n)
V, H = arnoldi(A, r0, m)

fig = plt.figure(figsize=(8, 8))
ax = fig.add_subplot(111)

im = ax.imshow(np.log10(1E-15 + np.abs(H)))
fig.colorbar(im)
```

It is clearly visible that only the main diagonal and the first upper and lower off-diagonal are nonzero, as expected. This hugely simplifies the Arnoldi iteration. Instead of orthogonalising $Av_m$ against all previous vectors we only need to orthogonalise against $v_m$ and $v_{m-1}$. All other inner products are already zero. Hence, the main orthogonalisation step now takes the form

$$
w = Av_m - (v_m^TAv_m)v_m - (v_{m-1}^TAv_m)v_{m-1}.
$$

Since the new vector $w$ is composed of only 3 vectors. This is also called a 3-term recurrence. The big advantage is that in addition to $Av_m$ we only need to keep $v_m$ and $v_{m-1}$ in memory. Hence, no matter how many iterations we do, the memory requirement remains constant, in contrast to Arnoldi for nonsymmetric matrices, where we need to keep all previous vectors in memory.

Arnoldi with a short recurrence relation for symmetric matrices has a special name. It is called **Lanczos method**.

## Solving linear systems of equations with Lanczos

We can now proceed exactly as in the Full orthogonalisation method and arrive at the linear system of equations

$$
T_my_m = \|r_0\|_2e_1,
$$

where $x_m = x_0 + V_my_m$ and $T_m = V_m^TAV_m$ is the tridiagonal matrix obtained from the Lanczos method.

The conjugate gradient method is an implementation of this approach. A very good derivation from Lanczos to CG is obtained in the beautiful book by Yousef Saad "[Iterative Methods for Sparse Linear Systems](https://www-users.cs.umn.edu/~saad/IterMethBook_2ndEd.pdf)", which is available online for free. Here, we will briefly motivate another approach to CG, which is a bit more intuitive and reveals more about the structure of the method, namely CG as an optimisation algorithm for a quadratic minimisation problem. One of the most beautiful summaries of this approach is contained in the paper [An introduction to the Conjugate Gradient Method Without the Agonizing Pain](https://www.cs.cmu.edu/~quake-papers/painless-conjugate-gradient.pdf) by Jonathan Shewchuk.

## A quadratic optimisation problem

We consider the quadratic minimisation problem

$$
\min_{x\in\mathbb{R}^n} f(x)
$$

with $f(x)=\frac{1}{2}x^TAx - b^Tx$. We have

$$
\nabla f(x) = Ax - b
$$

and hence the only stationary point is the solution of the linear system $Ax=b$. Furthermore, it is really a minimiser since $f''(x)$ is positive definite for all $x\in\mathbb{R}^n$ as $A$ is positive definite.

## The Method of Steepest Descent

Our first idea is the method of steepest descent. Remember that the negative gradient is a descent direction. Given a point $x_k$. We have

$$
-\nabla f(x_k) = b - Ax_k := r_k.
$$

The negative gradient is hence just the residual. Hence, we need to minimise along the direction of the residual, that is we will have
$x_{k+1} = x_k + \alpha_k r_k$ for some value $\alpha_k$. To compute $\alpha_k$ we just solve

$$
\frac{d}{d\alpha}f(x_k + \alpha r_k) = 0
$$

Since $\frac{d}{d\alpha}f(x_k + \alpha r_k) = r_{k+1}^Tr_k$ we just need to choose $\alpha_k$ such that $r_{k+1}$ is orthogonal to $r_k$. The solution is given by $\alpha_k = \frac{r_k^Tr_k}{r_k^TAr_k}$. This gives us a complete method consisting of three steps to get from $x_k$ to $x_{k+1}$.

$$
\begin{align}
r_k &= b - Ax_k\nonumber\\
\alpha_k &= \frac{r_k^Tr_k}{r_k^TAr_k}\nonumber\\
x_{k+1} &= x_k + \alpha_k r_k
\end{align}
$$

We are not going to derive the complete convergence analysis here but only state the final result. Let $\kappa := \frac{\lambda_{max}}{\lambda_{min}}$, where $\lambda_{max}$ and $\lambda_{min}$ are the largest, respectively smallest eigenvalue of $A$ (remember that all eigenvalues are positive since $A$ is symmetric positive definite). The number $\kappa$ is called the condition number of $A$. Let $e_k = x_k - x^*$ be the difference of the exact solution $x^*$ satisfying $Ax^*=b$ and our current iterate $x_k$. Note that $r_k = -Ae_k$.

We now have that

$$
\|e_k\|_A\leq \left(\frac{\kappa - 1}{\kappa + 1}\right)^k\|e_0\|_A,
$$

where $\|e_k\|_A := \left(e_k^TAe_k\right)^{1/2}$.

This is an extremely slow rate of convergence. Let $\kappa=10$, which is a fairly small number. Then the error reduces in each step only by a factor of $\frac{9}{11}\approx 0.81$ and we need 11 iterations for each digit of accuracy.

## The method of conjugate directions

The steepest descent approach was not bad. But we want to improve on it. The problem with the steepest descent method is that we have no guarantee that we are reducing the error $e_{k+1}$ as much as possible along our current direction $r_k$ when we minimize. But we can fix this.

Let us pick a set of directions $d_0, d_1, \dots, d_{n-1}$, which are mutually orthogonal, that is $d_i^Td_j =0$ for $i\neq j$. We now want to enforce the condition that

$$
e_{k+1}^Td_k = 0.
$$

This means that the remaining error is orthogonal to $d_k$ and hence is a linear combination of all the other search directions. We have therefore exhausted all the information from $d_k$. Let's play this through.

We know that $e_{k+1} = x_{k+1} - x^* = x_k -x^* + \alpha_k d_k = e_k + \alpha_kd_k$.

It follows that

$$
\begin{align}
e_{k+1}^Td_k &= d_k^T(e_k + \alpha_kd_k)\nonumber\\
             &= d_k^Te_k + \alpha_kd_k^Td_k = 0\nonumber
\end{align}
$$

and therefore $\alpha_k = -\frac{d_k^Te_k}{d_k^Td_k}$.

Unfortunately, this does not quite work in practice as we don't know $e_k$. But there is a solution. Remember that $r_k = -Ae_k$. We just need an $A$ in the right place. To achieve this we choose **conjugate directions**, that is we impose the condition that

$$
d_i^TAd_j = 0
$$

for $i\neq j$. We also impose the condition that $e_{k+1}^TAd_k = 0$. Writing this out we obtain

$$
\alpha_k = \frac{d_k^Tr_k}{d_k^TAd_k}.
$$

This expression is computable if we have a suitable set of conjugate directions $d_k$. Moreoever, it guarantees that the method converges in at most $n$ steps since in every iteration we are annihiliating the error in the direction of $d_k$ and there are only $n$ different directions.

## Conjugate Gradients - Mixing steepest descent with conjugate directions

The idea of conjugate gradients is to obtain the conjugate directions $d_i$ by taking the $r_i$ (the gradients) and to $A$-orthogonalise (conjugate) them against the previous directions. We are leaving out the details of the derivation and refer to the Shewchuk paper. But the final algorithm now takes the following form.

$$
\begin{align}
d_0 &= r_0 = b - Ax_0\nonumber\\
\alpha_i &= \frac{r_i^Tr_i}{d_i^TAd_i}\nonumber\\
x_{i+1} &=x_i + \alpha_id_i\nonumber\\
r_{i+1} &= r_i - \alpha_i Ad_i\nonumber\\
\beta_{i+1} &= \frac{r_{i+1}^Tr_{i+1}}{r_i^Tr_i}\nonumber\\
d_{i+1} &= r_{i+1} + \beta_{i+1}d_i\nonumber
\end{align}
$$

Conjugate Gradients has a much more favourable convergence bound than steepest descent. One can derive that

$$
\|e_i\|_A\leq 2\left(\frac{\sqrt{\kappa} - 1}{\sqrt{\kappa} + 1}\right)^i\|e_0\|_A.
$$

If we choose again the example that $\kappa=10$ we obtain

$$
\|e_i\|A\lessapprox 0.52^i\|e_0\|_A.
$$

Hence, we need around $4$ iterations for each digits of accuracy instead of 11 for the method of steepest descent.

## A numerical example

The following code creates a symmetric positive definite matrix.

```python
from scipy.sparse import diags

n = 10000

data = [2.1 * np.ones(n),
        -1. * np.ones(n - 1),
        -1. * np.ones(n - 1)]

offsets = [0, 1, -1]

A = diags(data, offsets=offsets, shape=(n, n), format='csr')
```

We now solve the associated linear system with CG and plot the convergence.

```python
from scipy.sparse.linalg import cg

b = rand.randn(n)
residuals = []

callback = lambda x: residuals.append(np.linalg.norm(b - A @ x) / np.linalg.norm(b))

sol, _ = cg(A, b, tol=1E-6, callback=callback, maxiter=1000)

fig = plt.figure(figsize=(8, 8))
ax = fig.add_subplot(111)
ax.semilogy(1 + np.arange(len(residuals)), residuals, 'k--')
ax.set_title('CG Convergence')
ax.set_xlabel('Iteration Step')
_ = ax.set_ylabel('$\|Ax-b\|_2 / \|b\|_2$')
```

```python

```


# Basis for grayscale images

## Introduction

Consider the set of real-valued matrices of size $M\times N$; we can turn this into a vector space by defining addition and scalar multiplication in the usual way:

\begin{align}
\mathbf{A} + \mathbf{B} &=
    \left[
        \begin{array}{ccc}
            a_{0,0} & \dots & a_{0,N-1} \\
            \vdots & & \vdots \\
            a_{M-1,0} & \dots & b_{M-1,N-1}
        \end{array}
    \right]
    +
    \left[
        \begin{array}{ccc}
            b_{0,0} & \dots & b_{0,N-1} \\
            \vdots & & \vdots \\
            b_{M-1,0} & \dots & b_{M-1,N-1}
        \end{array}
    \right]
    \\
    &=
    \left[
        \begin{array}{ccc}
            a_{0,0}+b_{0,0} & \dots & a_{0,N-1}+b_{0,N-1} \\
            \vdots & & \vdots \\
            a_{M-1,0}+b_{M-1,0} & \dots & a_{M-1,N-1}+b_{M-1,N-1}
        \end{array}
    \right]
    \\ \\ \\
\beta\mathbf{A} &=
    \left[
        \begin{array}{ccc}
            \beta a_{0,0} & \dots & \beta a_{0,N-1} \\
            \vdots & & \vdots \\
            \beta a_{M-1,0} & \dots & \beta a_{M-1,N-1}
        \end{array}
    \right]
\end{align}

As a matter of fact, the space of real-valued $M\times N$ matrices is completely equivalent to $\mathbb{R}^{MN}$ and we can always "unroll" a matrix into a vector. Assume we proceed column by column; then the matrix becomes

$$
    \mathbf{a} = \mathbf{A}[:] = [
        \begin{array}{ccccccc}
            a_{0,0} & \dots & a_{M-1,0} & a_{0,1} & \dots & a_{M-1,1} & \ldots & a_{0, N-1} & \dots & a_{M-1,N-1}
        \end{array}]^T
$$

Although the matrix and vector forms represent exactly the same data, the matrix form allows us to display the data in the form of an image. Assume each value in the matrix is a grayscale intensity, where zero is black and 255 is white; for example we can create a checkerboard pattern of any size with the following function:

```python
# usual pyton bookkeeping...
%pylab inline
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import IPython
from IPython.display import Image
import math
from __future__ import print_function

# ensure all images will be grayscale
gray();
```

    /opt/conda/envs/python2/lib/python2.7/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
      warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')

    Populating the interactive namespace from numpy and matplotlib

    <matplotlib.figure.Figure at 0x7f56f0a85f90>

```python
# let's create a checkerboard pattern
SIZE = 4
img = np.zeros((SIZE, SIZE))
for n in range(0, SIZE):
    for m in range(0, SIZE):
        if (n & 0x1) ^ (m & 0x1):
            img[n, m] = 255

# now display the matrix as an image
plt.matshow(img);
```

Given the equivalence between the space of $M\times N$ matrices and $\mathbb{R}^{MN}$ we can easily define the inner product between two matrices in the usual way:

$$
\langle \mathbf{A}, \mathbf{B} \rangle = \sum_{m=0}^{M-1} \sum_{n=0}^{N-1} a_{m,n} b_{m, n}
$$

(where we have neglected the conjugation since we'll only deal with real-valued matrices); in other words, we can take the inner product between two matrices as the standard inner product of their unrolled versions. The inner product allows us to define orthogonality between images and this is rather useful since we're going to explore a couple of bases for this space.

## Actual images

Conveniently, using IPython, we can read images from disk in any given format and convert them to numpy arrays; let's load and display for instance a JPEG image:

```python
img = np.array(plt.imread('cameraman.jpg'), dtype=int)
plt.matshow(img);
```

The image is a $64\times 64$ low-resolution version of the famous "cameraman" test picture. Out of curiosity, we can look at the first column of this image, which is is a $64×1$ vector:

```python
img[:,0]
```

    array([156, 157, 157, 152, 154, 155, 151, 157, 152, 155, 158, 159, 159,
           160, 160, 161, 155, 160, 161, 161, 164, 162, 160, 162, 158, 160,
           158, 157, 160, 160, 159, 158, 163, 162, 162, 157, 160, 114, 114,
           103,  88,  62, 109,  82, 108, 128, 138, 140, 136, 128, 122, 137,
           147, 114, 114, 144, 112, 115, 117, 131, 112, 141,  99,  97])

The values are integers between zero and 255, meaning that each pixel is encoded over 8 bits (or 256 gray levels).

## The canonical basis

The canonical basis for any matrix space $\mathbb{R}^{M\times N}$ is the set of "delta" matrices where only one element equals to one while all the others are 0. Let's call them $\mathbf{E}_n$ with $0 \leq n < MN$. Here is a function to create the canonical basis vector given its index:

```python
def canonical(n, M=5, N=10):
    e = np.zeros((M, N))
    e[(n % M), int(n / M)] = 1
    return e
```

Here are some basis vectors: look for the position of white pixel, which differentiates them and note that we enumerate pixels column-wise:

```python
plt.matshow(canonical(0));
plt.matshow(canonical(1));
plt.matshow(canonical(49));
```

## Transmitting images

Suppose we want to transmit the "cameraman" image over a communication channel. The intuitive way to do so is to send the  pixel values one by one, which corresponds to sending the coefficients of the decomposition of the image over the canonical basis. So far, nothing complicated: to send the cameraman image, for instance, we will send $64\times 64 = 4096$ coefficients in a row.

Now suppose that a communication failure takes place after the first half of the pixels have been sent. The received data will allow us to display an approximation of the original image only. If we replace the missing data with zeros, here is what we would see, which is not very pretty:

```python
# unrolling of the image for transmission (we go column by column, hence "F")
tx_img = np.ravel(img, "F")

# oops, we lose half the data
tx_img[int(len(tx_img)/2):] = 0

# rebuild matrix
rx_img = np.reshape(tx_img, (64, 64), "F")
plt.matshow(rx_img);
```

Can we come up with a trasmission scheme that is more robust in the face of channel loss? Interestingly, the answer is yes, and it involves a different, more versatile basis for the space of images. What we will do is the following:

* describe the Haar basis, a new basis for the image space
* project the image in the new basis
* transmit the projection coefficients
* rebuild the image using the basis vectors

We know a few things: if we choose an orthonormal basis, the analysis and synthesis formulas will be super easy (a simple inner product and a scalar multiplication respectively). The trick is to find a basis that will be robust to the loss of some coefficients.

One such basis is the **Haar basis**. We cannot go into too many details in this notebook but, for the curious, a good starting point is [here](https://chengtsolin.wordpress.com/2015/04/15/real-time-2d-discrete-wavelet-transform-using-opengl-compute-shader/). Mathematical formulas aside, the Haar basis works by encoding the information in a *hierarchical* way: the first basis vectors encode the broad information and the higher coefficients encode the detail. Let's have a look.

First of all, to keep things simple, we will remain in the space of square matrices whose size is a power of two. The code to generate the Haar basis matrices is the following: first we generate a 1D Haar vector and then we obtain the basis matrices by taking the outer product of all possible 1D vectors (don't worry if it's not clear, the results are what's important):

```python
def haar1D(n, SIZE):
    # check power of two
    if math.floor(math.log(SIZE) / math.log(2)) != math.log(SIZE) / math.log(2):
        print("Haar defined only for lengths that are a power of two")
        return None
    if n >= SIZE or n < 0:
        print("invalid Haar index")
        return None

    # zero basis vector
    if n == 0:
        return np.ones(SIZE)

    # express n > 1 as 2^p + q with p as large as possible;
    # then k = SIZE/2^p is the length of the support
    # and s = qk is the shift
    p = math.floor(math.log(n) / math.log(2))
    pp = int(pow(2, p))
    k = SIZE / pp
    s = (n - pp) * k

    h = np.zeros(SIZE)
    h[int(s):int(s+k/2)] = 1
    h[int(s+k/2):int(s+k)] = -1
    # these are not normalized
    return h

def haar2D(n, SIZE=8):
    # get horizontal and vertical indices
    hr = haar1D(n % SIZE, SIZE)
    hv = haar1D(int(n / SIZE), SIZE)
    # 2D Haar basis matrix is separable, so we can
    #  just take the column-row product
    H = np.outer(hr, hv)
    H = H / math.sqrt(np.sum(H * H))
    return H
```

First of all, let's look at a few basis matrices; note that the matrices have positive and negative values, so that the value of zero will be represented as gray:

```python
plt.matshow(haar2D(0));
plt.matshow(haar2D(1));
plt.matshow(haar2D(10));
plt.matshow(haar2D(63));
```

We can notice two key properties

* each basis matrix has positive and negative values in some symmetric patter: this means that the basis matrix will implicitly compute the difference between image areas
* low-index basis matrices take differences between large areas, while high-index ones take differences in smaller **localized** areas of the image

We can immediately verify that the Haar matrices are orthogonal:

```python
# let's use an 8x8 space; there will be 64 basis vectors
# compute all possible inner product and only print the nonzero results
for m in range(0,64):
    for n in range(0,64):
        r = np.sum(haar2D(m, 8) * haar2D(n, 8))
        if r != 0:
            print("[%dx%d -> %f] " % (m, n, r), end="")
```

    [0x0 -> 1.000000] [1x1 -> 1.000000] [2x2 -> 1.000000] [3x3 -> 1.000000] [4x4 -> 1.000000] [5x5 -> 1.000000] [6x6 -> 1.000000] [7x7 -> 1.000000] [8x8 -> 1.000000] [9x9 -> 1.000000] [10x10 -> 1.000000] [11x11 -> 1.000000] [12x12 -> 1.000000] [13x13 -> 1.000000] [14x14 -> 1.000000] [15x15 -> 1.000000] [16x16 -> 1.000000] [16x17 -> -0.000000] [17x16 -> -0.000000] [17x17 -> 1.000000] [18x18 -> 1.000000] [19x19 -> 1.000000] [20x20 -> 1.000000] [21x21 -> 1.000000] [22x22 -> 1.000000] [23x23 -> 1.000000] [24x24 -> 1.000000] [24x25 -> -0.000000] [25x24 -> -0.000000] [25x25 -> 1.000000] [26x26 -> 1.000000] [27x27 -> 1.000000] [28x28 -> 1.000000] [29x29 -> 1.000000] [30x30 -> 1.000000] [31x31 -> 1.000000] [32x32 -> 1.000000] [33x33 -> 1.000000] [34x34 -> 1.000000] [35x35 -> 1.000000] [36x36 -> 1.000000] [37x37 -> 1.000000] [38x38 -> 1.000000] [39x39 -> 1.000000] [40x40 -> 1.000000] [41x41 -> 1.000000] [42x42 -> 1.000000] [43x43 -> 1.000000] [44x44 -> 1.000000] [45x45 -> 1.000000] [46x46 -> 1.000000] [47x47 -> 1.000000] [48x48 -> 1.000000] [49x49 -> 1.000000] [50x50 -> 1.000000] [51x51 -> 1.000000] [52x52 -> 1.000000] [53x53 -> 1.000000] [54x54 -> 1.000000] [55x55 -> 1.000000] [56x56 -> 1.000000] [57x57 -> 1.000000] [58x58 -> 1.000000] [59x59 -> 1.000000] [60x60 -> 1.000000] [61x61 -> 1.000000] [62x62 -> 1.000000] [63x63 -> 1.000000]

OK! Everything's fine. Now let's transmit the "cameraman" image: first, let's verify that it works

```python
# project the image onto the Haar basis, obtaining a vector of 4096 coefficients
# this is simply the analysis formula for the vector space with an orthogonal basis
tx_img = np.zeros(64*64)
for k in range(0, (64*64)):
    tx_img[k] = np.sum(img * haar2D(k, 64))

# now rebuild the image with the synthesis formula; since the basis is orthonormal
#  we just need to scale the basis matrices by the projection coefficients
rx_img = np.zeros((64, 64))
for k in range(0, (64*64)):
    rx_img += tx_img[k] * haar2D(k, 64)

plt.matshow(rx_img);
```

Cool, it works! Now let's see what happens if we lose the second half of the coefficients:

```python
# oops, we lose half the data
lossy_img = np.copy(tx_img);
lossy_img[int(len(tx_img)/2):] = 0

# rebuild matrix
rx_img = np.zeros((64, 64))
for k in range(0, (64*64)):
    rx_img += lossy_img[k] * haar2D(k, 64)

plt.matshow(rx_img);
```

That's quite remarkable, no? We've lost the same amount of information as before but the image is still acceptable. This is because we lost the coefficients associated to the fine details of the image but we retained the "broad strokes" encoded by the first half.

Note that if we lose the first half of the coefficients the result is markedly different:

```python
lossy_img = np.copy(tx_img);
lossy_img[0:int(len(tx_img)/2)] = 0

rx_img = np.zeros((64, 64))
for k in range(0, (64*64)):
    rx_img += lossy_img[k] * haar2D(k, 64)

plt.matshow(rx_img);
```

In fact, schemes like this one are used in *progressive encoding*: send the most important information first and add details if the channel permits it. You may have experienced this while browsing the interned over a slow connection.

All in all, a great application of a change of basis!


# Demystifying `meshgrid()`

We use the Numpy [meshgrid()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.meshgrid.html) function to plot 2D arrays but it can be a bit mysterious as to what the "meshgrid" actually does or look like.

The docs say:

_Return coordinate matrices from coordinate vectors._

_Make N-D coordinate arrays for vectorized evaluations of N-D scalar/vector fields over N-D grids, given one-dimensional coordinate arrays x1, x2,..., xn._

## Visualization with a Gaussian
Testing with the initial condition, an isotropic 2D [Gaussian](http://mathworld.wolfram.com/GaussianFunction.html)

$$
g_2(x, y) = \frac{1}{2\pi\sigma^2} \exp\left[-\frac{(x-x_0)^2 + (y-y_0)^2}{2\sigma^2}\right]
$$

Note that the 2D Gaussian can be written as a product of two 1D Gaussians

\begin{align}
g_2(x, y) &= \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left[-\frac{(x-x_0)^2}{2\sigma^2}\right] \cdot
          \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left[-\frac{(y-y_0)^2}{2\sigma^2}\right]\\
        &= g_1(x) \cdot g_1(y)
\end{align}

Plot $g_2(x, y)$ by (1) constructing it from the product of $g_1$s and (2) from the explicit formula. Use [`np.meshgrid`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.meshgrid.html) to evaluate the function on a grid.

Use an *asymmetric* grid ($L_x=0.5, N_x=50$ and $L_y=1, N_y=100$) to clearly see the shape of the underlying arrays.

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
%matplotlib inline
```

```python
L = 0.5
Lx, Ly = 0.5, 1
Nx, Ny = 50, 100

x = np.linspace(0, Lx, Nx+1)  # need N+1!
y = np.linspace(0, Ly, Ny+1)
X, Y = np.meshgrid(x, y)

def gaussian2D(x, y, u0=0.05, x0=None, y0=None, sigma=0.1*L):
    x0 = np.mean(x) if x0 is None else x0
    y0 = np.mean(y) if y0 is None else y0
    return u0/(2*np.pi*sigma) * np.exp(-((x-x0)**2 + (y-y0)**2)/(2*sigma**2))

def gaussian(x, u0=0.05, x0=None, sigma=0.1*L):
    x0 = np.mean(x) if x0 is None else x0
    return u0/np.sqrt(2*np.pi*sigma) * np.exp(-(x-x0)**2 / (2*sigma**2))
```

#### Meshgrid

```python
print(x.shape, y.shape)
print(X.shape, Y.shape)
print(X)
print(Y)
```

    (51,) (101,)
    (101, 51) (101, 51)
    [[0.   0.01 0.02 ... 0.48 0.49 0.5 ]
     [0.   0.01 0.02 ... 0.48 0.49 0.5 ]
     [0.   0.01 0.02 ... 0.48 0.49 0.5 ]
     ...
     [0.   0.01 0.02 ... 0.48 0.49 0.5 ]
     [0.   0.01 0.02 ... 0.48 0.49 0.5 ]
     [0.   0.01 0.02 ... 0.48 0.49 0.5 ]]
    [[0.   0.   0.   ... 0.   0.   0.  ]
     [0.01 0.01 0.01 ... 0.01 0.01 0.01]
     [0.02 0.02 0.02 ... 0.02 0.02 0.02]
     ...
     [0.98 0.98 0.98 ... 0.98 0.98 0.98]
     [0.99 0.99 0.99 ... 0.99 0.99 0.99]
     [1.   1.   1.   ... 1.   1.   1.  ]]

#### Plot 1D Gaussians
Plot the 1D Gaussians using the meshgrid components.

```python
gaussian(X).shape
```

    (101, 51)

```python
fig = plt.figure()
ax = fig.add_subplot(111)
ax.contourf(X, Y, gaussian(X, sigma=0.1), 30, cmap=plt.cm.viridis)
ax.set_aspect(1)
plt.show()
```

```python
gaussian(Y).shape
```

    (101, 51)

```python
ax = plt.subplot(131)
ax.contourf(X, Y, gaussian(X, sigma=0.1), 30, cmap=plt.cm.viridis)
ax.set_aspect(1)
ax = plt.subplot(132)
ax.contourf(X, Y, gaussian(Y, sigma=0.1), 30, cmap=plt.cm.viridis)
ax.set_aspect(1)
ax = plt.subplot(133)
ax.contourf(X, Y, gaussian(X, sigma=0.1) * gaussian(Y, sigma=0.1), 30, cmap=plt.cm.viridis)
ax.set_aspect(1)
```

#### 2D gaussian

```python
gaussian2D(X, Y).shape
```

    (101, 51)

```python
ax = plt.subplot(111)
ax.contourf(X, Y, gaussian2D(X, Y, sigma=0.1), 30, cmap=plt.cm.viridis)
ax.set_aspect(1)
```

```python

```


---
author: Nathan Carter (ncarter@bentley.edu)
---

This answer assumes you have imported SymPy as follows.

```python
from sympy import *                   # load all math functions
init_printing( use_latex='mathjax' )  # use pretty math output
```

Let's create a simple example.  We'll be approximating $f(x)=\sin x$
centered at $a=0$ with a Taylor series of degree $n=5$.  We will be
applying our approximation at $x_0=1$.  What is the error bound?

```python
var( 'x' )
formula = sin(x)
a = 0
x_0 = 1
n = 5
```

We will not ask SymPy to compute the formula exactly, but will instead
have it sample a large number of $c$ values from the interval in question,
and compute the maximum over those samples.  (The exact solution can be too
hard for SymPy to compute.)

```python
# Get 1000 evenly-spaced c values:
cs = [ Min(x_0,a) + abs(x_0-a)*i/1000 for i in range(1001) ]
# Create the formula |f^(n+1)(x)|:
formula2 = abs( diff( formula, x, n+1 ) )
# Find the max of it on all the 1000 values:
m = Max( *[ formula2.subs(x,c) for c in cs ] )
# Compute the error bound:
N( abs(x_0-a)**(n+1) / factorial(n+1) * m )
```

$\displaystyle 0.00116870970112208$

The error is at most $0.00116871\ldots$.

database/tasks/How to compute the error bounds on a Taylor approximation/Python, using SymPy.ipynb


> This is one of the 100 recipes of the [IPython Cookbook](http://ipython-books.github.io/), the definitive guide to high-performance scientific computing and data science in Python.

# 15.5. A bit of number theory with SymPy

```python
from sympy import *
init_printing()
```

```python
import sympy.ntheory as nt
```

## Prime numbers

Test whether a number is prime.

```python
nt.isprime(2011)
```

Find the next prime after a given number.

```python
nt.nextprime(2011)
```

What is the 1000th prime number?

```python
nt.prime(1000)
```

How many primes less than 2011 are there?

```python
nt.primepi(2011)
```

We can plot $\pi(x)$, the prime-counting function (the number of prime numbers less than or equal to some number x). The famous *prime number theorem* states that this function is asymptotically equivalent to $x/\log(x)$. This expression approximately quantifies the distribution of the prime numbers among all integers.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
x = np.arange(2, 10000)
plt.plot(x, list(map(nt.primepi, x)), '-k', label='$\pi(x)$');
plt.plot(x, x / np.log(x), '--k', label='$x/\log(x)$');
plt.legend(loc=2);
```

Let's compute the integer factorization of some number.

```python
nt.factorint(1998)
```

```python
2 * 3**3 * 37
```

## Chinese Remainder Theorem

A lazy mathematician is counting his marbles. When they are arranged in three rows, the last column contains one marble. When they form four rows, there are two marbles in the last column, and there are three with five rows. The Chinese Remainer Theorem can give him the answer directly.

```python
from sympy.ntheory.modular import solve_congruence
```

```python
solve_congruence((1, 3), (2, 4), (3, 5))
```

There are infinitely many solutions: 58, and 58 plus any multiple of 60. Since 118 seems visually too high, 58 is the right answer.

> You'll find all the explanations, figures, references, and much more in the book (to be released later this summer).

> [IPython Cookbook](http://ipython-books.github.io/), by [Cyrille Rossant](http://cyrille.rossant.net), Packt Publishing, 2014 (500 pages).


# day06: Gradient Descent for Linear Regression

# Objectives

* Learn how to fit weight parameters of Linear Regression to a simple dataset via gradient descent
* Understand impact of step size
* Understand impact of initialization

# Outline
* [Part 1: Loss and Gradient for 1-dim. Linear Regression](#part1)
* [Part 2: Gradient Descent Algorithm in a few lines of Python](#part2)
* [Part 3: Debugging with Trace Plots](#part3)
* [Part 4: Selecting the step size](#part4)
* [Part 5: Selecting the initialization](#part5)
* [Part 6: Using SciPy's built-in routines](#part6)

# Takeaways

* Gradient descent is a simple algorithm that can be implemented in a few lines of Python
* * Practical issues include selecting step size and initialization
* Step size matters a lot
* * Need to select carefully for each problem

* Initialization of the parameters can matter too!

* scipy offers some useful tools for gradient-based optimization
* * scipy's toolbox cannot do scalable "stochastic" methods (requires a modest size dataset, not too big)
* * "L-BFGS-B" method is highly recommended if you have your loss and gradient functions available

```python
import numpy as np
```

```python
# import plotting libraries
import matplotlib
import matplotlib.pyplot as plt

%matplotlib inline
plt.style.use('seaborn') # pretty matplotlib plots

import seaborn as sns
sns.set('notebook', font_scale=1.25, style='whitegrid')
```

# Create simple dataset:   y = 1.234 * x + noise

We will *intentionally* create a toy dataset where we know that a good solution has slope near 1.234.

Naturally, the best slope for the finite dataset of N=100 examples we create won't be exactly 1.234 (because of the noise added plus the fact that our dataset size is limited).

```python
def create_dataset(N=100, slope=1.234, noise_stddev=0.1, random_state=0):
    random_state = np.random.RandomState(int(random_state))

    # input features
    x_N = np.linspace(-2, 2, N)

    # output features
    y_N = slope * x_N + random_state.randn(N) * noise_stddev

    return x_N, y_N
```

```python
x_N, y_N = create_dataset(N=50, noise_stddev=0.3)
```

```python
fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(5,5))
plt.plot(x_N, y_N, 'k.');
plt.xlabel('x');
plt.ylabel('y');
```

# Part 1: Gradient Descent for 1-dim. Linear Regression

## Define model

Consider the *simplest* linear regression model. A single weight parameter $w \in \mathbb{R}$ representing the slope of the prediction line. No bias/intercept.

To make predictions, we just compute the weight multiplied by the input feature
$$
\hat{y}(x) = w \cdot x
$$

## Define loss function

We want to minimize the total *squared error* across all N observed data examples (input features $x_n$, output responses $y_n$)

\begin{align}
    \min_{w \in \mathbb{R}} ~~ &\ell(w)
    \\
    \text{calc_loss}(w) = \ell(w) &= \sum_{n=1}^N (y_n - w x_n)^2
\end{align}

### Exercise 1A: Complete the code below

You should make it match the math expression above.

```python
def calc_loss(w):
    ''' Compute loss for slope-only least-squares linear regression

    Args
    ----
    w : float
        Value of slope parameter

    Returns
    -------
    loss : float
        Sum of squared error loss at provided w value
    '''
    yhat_N = x_N * w
    sum_squared_error = np.sum((y_N - yhat_N)**2) # todo compute the sum of squared error between y and yhat
    return sum_squared_error
```

# Define the gradient function

\begin{align}
\text{calc_grad}(w) = \ell'(w) &= \frac{\partial}{\partial w} [ \sum_{n=1}^N (y_n - w x_n)^2]
\\
&= \sum_{n=1}^N 2 (y_n - w x_n) (-x_n)
\\
&= 2 \sum_{n=1}^N (w x_n - y_n) (x_n)
\\
&= 2  w \left( \sum_{n=1}^N x_n^2 \right) - 2 \sum_{n=1}^N y_n x_n
\end{align}

Below, we've implemented the gradient calculation in code for you

```python
def calc_grad(w):
    ''' Compute gradient for slope-only least-squares linear regression

    Args
    ----
    w : float
        Value of slope parameter

    Returns
    -------
    g : float
        Value of derivative of loss function at provided w value
    '''
    g = 2.0 * w * np.sum(np.square(x_N)) - 2.0 * np.sum(x_N * y_N)
    return g
```

## Plot loss evaluated at each w from -3 to 8

We should see a "bowl" shape with one *global* minima, because our optimization problem is "convex"

```python
w_grid = np.linspace(-3, 8, 300) # create array of 300 values between -3 and 8
```

```python
loss_grid = np.asarray([calc_loss(w) for w in w_grid])
plt.plot(w_grid, loss_grid, 'b.-');
plt.xlabel('w');
plt.ylabel('loss(w)');
```

### Discussion 1b: Visually, at what value of $w$ does the loss function have a minima? Is it near where you would expect (hint: look above for the "true" slope value used to generate the data)

### Exercise 1c: Write NumPy code to identify which entry in the w_grid array corresponds to the lowest entry in the loss_grid array

Hint: use np.argmin

```python
# TODO write code here
index = np.argmin(loss_grid)
print(index)
print(w_grid[index])
```

    112
    1.120401337792643

## Sanity check: plot gradient evaluated at each w from -3 to 8

```python
grad_grid = np.asarray([calc_grad(w) for w in w_grid])
plt.plot(w_grid, grad_grid, 'b.-');
plt.xlabel('w');
plt.ylabel('grad(w)');
```

### Discussion 1d: Visually, at what value of $w$ does the gradient function cross zero? Is it the same place as the location of the minimum in the loss above?

TODO interpret the graph above and write your answer here, then discuss with your group

### Exercise 1d: Numerically, at which value of w does grad_grid cross zero?

We might try to estimate numerically where the gradient crosses zero.

We could do this in a few steps:

1) Compute the distance from each gradient in `grad_grid` to 0.0 (we could use just absolute distance)

2) Find the index of `grad_grid` with smallest distance (using `np.argmin`)

3) Plug that index into `w_grid` to get the $w$ value corresponding to that zero-crossing

```python
dist_from_zero_G = np.abs(grad_grid - 0.0)

zero_cross_index = 0 # TODO fix me for step 2 above

print("Zero crossing occurs at w = %.4f" % w_grid[0]) # TODO fix me for step 3 above
```

    Zero crossing occurs at w = -3.0000

## Part 2: Gradient Descent (GD) as an algorithm in Python

### Define minimize_via_grad_descent algorithm

Can you understand what each step of this algorithm does?

```python
def minimize_via_grad_descent(calc_loss, calc_grad, init_w=0.0, step_size=0.001, max_iters=100):
    ''' Perform minimization of provided loss function via gradient descent

    Args
    ----
    calc_loss : function
    calc_grad : function
    init_w : float
    step_size : float
    max_iters : positive int

    Return
    ----
    wopt: float
        array of optimized weights that approximately gives the least error
    info_dict : dict
        Contains information about the optimization procedure useful for debugging
        Entries include:
        * trace_loss_list : list of loss values
        * trace_grad_list : list of gradient values
    '''
    w = 1.0 * init_w
    grad = calc_grad(w)

    # Create some lists to track progress over time (for debugging)
    trace_loss_list = []
    trace_w_list = []
    trace_grad_list = []

    for iter_id in range(max_iters):
        if iter_id > 0:
            w = w - step_size * grad

        loss = calc_loss(w)
        grad = calc_grad(w)

        print("  iter %5d/%d | w  % 13.5f | loss % 13.4f | grad % 13.4f" % (
            iter_id, max_iters, w, loss, grad))

        trace_loss_list.append(loss)
        trace_w_list.append(w)
        trace_grad_list.append(grad)

    wopt = w
    info_dict = dict(
        trace_loss_list=trace_loss_list,
        trace_w_list=trace_w_list,
        trace_grad_list=trace_grad_list)

    return wopt, info_dict
```

### Discussion 2a: Which line of the above function does the *parameter update* happen?

Remember, in math, the parameter update of gradient descent is this:
$$
w \gets w - \alpha \nabla_w \ell(w)
$$

where $\alpha > 0$ is the step size.

In words, this math says *move* the parameter $w$ from its current value a *small step* in the "downhill" direction (indicated by gradient).

TODO write down here which line above *you* think it is, then discuss with your group

```python

```

### Try it! Run GD with step_size = 0.001

Running the cell below will have the following effects:

1) one line will be printed for every iteration, indicating the current w value and its associated loss

2) the "optimal" value of w will be stored in the variable named `wopt` returned by this function

3) a dictionary of information useful for debugging will be stored in the `info_dict` returned by this function

```python
wopt, info_dict = minimize_via_grad_descent(calc_loss, calc_grad, step_size=0.001);
```

      iter     0/100 | w        0.00000 | loss       93.3197 | grad     -156.5566
      iter     1/100 | w        0.15656 | loss       70.5104 | grad     -134.8304
      iter     2/100 | w        0.29139 | loss       53.5926 | grad     -116.1192
      iter     3/100 | w        0.40751 | loss       41.0446 | grad     -100.0047
      iter     4/100 | w        0.50751 | loss       31.7376 | grad      -86.1265
      iter     5/100 | w        0.59364 | loss       24.8345 | grad      -74.1743
      iter     6/100 | w        0.66781 | loss       19.7144 | grad      -63.8807
      iter     7/100 | w        0.73169 | loss       15.9168 | grad      -55.0156
      iter     8/100 | w        0.78671 | loss       13.1001 | grad      -47.3808
      iter     9/100 | w        0.83409 | loss       11.0110 | grad      -40.8055
      iter    10/100 | w        0.87489 | loss        9.4614 | grad      -35.1427
      iter    11/100 | w        0.91004 | loss        8.3121 | grad      -30.2657
      iter    12/100 | w        0.94030 | loss        7.4597 | grad      -26.0656
      iter    13/100 | w        0.96637 | loss        6.8274 | grad      -22.4483
      iter    14/100 | w        0.98882 | loss        6.3584 | grad      -19.3331
      iter    15/100 | w        1.00815 | loss        6.0106 | grad      -16.6501
      iter    16/100 | w        1.02480 | loss        5.7526 | grad      -14.3395
      iter    17/100 | w        1.03914 | loss        5.5612 | grad      -12.3495
      iter    18/100 | w        1.05149 | loss        5.4193 | grad      -10.6357
      iter    19/100 | w        1.06212 | loss        5.3140 | grad       -9.1597
      iter    20/100 | w        1.07128 | loss        5.2360 | grad       -7.8886
      iter    21/100 | w        1.07917 | loss        5.1781 | grad       -6.7938
      iter    22/100 | w        1.08597 | loss        5.1351 | grad       -5.8510
      iter    23/100 | w        1.09182 | loss        5.1032 | grad       -5.0390
      iter    24/100 | w        1.09686 | loss        5.0796 | grad       -4.3397
      iter    25/100 | w        1.10120 | loss        5.0621 | grad       -3.7375
      iter    26/100 | w        1.10493 | loss        5.0491 | grad       -3.2188
      iter    27/100 | w        1.10815 | loss        5.0394 | grad       -2.7721
      iter    28/100 | w        1.11092 | loss        5.0323 | grad       -2.3874
      iter    29/100 | w        1.11331 | loss        5.0270 | grad       -2.0561
      iter    30/100 | w        1.11537 | loss        5.0231 | grad       -1.7708
      iter    31/100 | w        1.11714 | loss        5.0201 | grad       -1.5250
      iter    32/100 | w        1.11866 | loss        5.0180 | grad       -1.3134
      iter    33/100 | w        1.11998 | loss        5.0164 | grad       -1.1311
      iter    34/100 | w        1.12111 | loss        5.0152 | grad       -0.9742
      iter    35/100 | w        1.12208 | loss        5.0143 | grad       -0.8390
      iter    36/100 | w        1.12292 | loss        5.0136 | grad       -0.7225
      iter    37/100 | w        1.12364 | loss        5.0132 | grad       -0.6223
      iter    38/100 | w        1.12427 | loss        5.0128 | grad       -0.5359
      iter    39/100 | w        1.12480 | loss        5.0125 | grad       -0.4615
      iter    40/100 | w        1.12526 | loss        5.0123 | grad       -0.3975
      iter    41/100 | w        1.12566 | loss        5.0122 | grad       -0.3423
      iter    42/100 | w        1.12600 | loss        5.0121 | grad       -0.2948
      iter    43/100 | w        1.12630 | loss        5.0120 | grad       -0.2539
      iter    44/100 | w        1.12655 | loss        5.0119 | grad       -0.2187
      iter    45/100 | w        1.12677 | loss        5.0119 | grad       -0.1883
      iter    46/100 | w        1.12696 | loss        5.0119 | grad       -0.1622
      iter    47/100 | w        1.12712 | loss        5.0118 | grad       -0.1397
      iter    48/100 | w        1.12726 | loss        5.0118 | grad       -0.1203
      iter    49/100 | w        1.12738 | loss        5.0118 | grad       -0.1036
      iter    50/100 | w        1.12749 | loss        5.0118 | grad       -0.0892
      iter    51/100 | w        1.12757 | loss        5.0118 | grad       -0.0768
      iter    52/100 | w        1.12765 | loss        5.0118 | grad       -0.0662
      iter    53/100 | w        1.12772 | loss        5.0118 | grad       -0.0570
      iter    54/100 | w        1.12777 | loss        5.0118 | grad       -0.0491
      iter    55/100 | w        1.12782 | loss        5.0118 | grad       -0.0423
      iter    56/100 | w        1.12787 | loss        5.0118 | grad       -0.0364
      iter    57/100 | w        1.12790 | loss        5.0118 | grad       -0.0314
      iter    58/100 | w        1.12793 | loss        5.0118 | grad       -0.0270
      iter    59/100 | w        1.12796 | loss        5.0118 | grad       -0.0233
      iter    60/100 | w        1.12798 | loss        5.0118 | grad       -0.0200
      iter    61/100 | w        1.12800 | loss        5.0118 | grad       -0.0172
      iter    62/100 | w        1.12802 | loss        5.0118 | grad       -0.0149
      iter    63/100 | w        1.12804 | loss        5.0118 | grad       -0.0128
      iter    64/100 | w        1.12805 | loss        5.0118 | grad       -0.0110
      iter    65/100 | w        1.12806 | loss        5.0118 | grad       -0.0095
      iter    66/100 | w        1.12807 | loss        5.0118 | grad       -0.0082
      iter    67/100 | w        1.12808 | loss        5.0118 | grad       -0.0070
      iter    68/100 | w        1.12808 | loss        5.0118 | grad       -0.0061
      iter    69/100 | w        1.12809 | loss        5.0118 | grad       -0.0052
      iter    70/100 | w        1.12810 | loss        5.0118 | grad       -0.0045
      iter    71/100 | w        1.12810 | loss        5.0118 | grad       -0.0039
      iter    72/100 | w        1.12810 | loss        5.0118 | grad       -0.0033
      iter    73/100 | w        1.12811 | loss        5.0118 | grad       -0.0029
      iter    74/100 | w        1.12811 | loss        5.0118 | grad       -0.0025
      iter    75/100 | w        1.12811 | loss        5.0118 | grad       -0.0021
      iter    76/100 | w        1.12812 | loss        5.0118 | grad       -0.0018
      iter    77/100 | w        1.12812 | loss        5.0118 | grad       -0.0016
      iter    78/100 | w        1.12812 | loss        5.0118 | grad       -0.0014
      iter    79/100 | w        1.12812 | loss        5.0118 | grad       -0.0012
      iter    80/100 | w        1.12812 | loss        5.0118 | grad       -0.0010
      iter    81/100 | w        1.12812 | loss        5.0118 | grad       -0.0009
      iter    82/100 | w        1.12812 | loss        5.0118 | grad       -0.0007
      iter    83/100 | w        1.12812 | loss        5.0118 | grad       -0.0006
      iter    84/100 | w        1.12812 | loss        5.0118 | grad       -0.0006
      iter    85/100 | w        1.12812 | loss        5.0118 | grad       -0.0005
      iter    86/100 | w        1.12813 | loss        5.0118 | grad       -0.0004
      iter    87/100 | w        1.12813 | loss        5.0118 | grad       -0.0004
      iter    88/100 | w        1.12813 | loss        5.0118 | grad       -0.0003
      iter    89/100 | w        1.12813 | loss        5.0118 | grad       -0.0003
      iter    90/100 | w        1.12813 | loss        5.0118 | grad       -0.0002
      iter    91/100 | w        1.12813 | loss        5.0118 | grad       -0.0002
      iter    92/100 | w        1.12813 | loss        5.0118 | grad       -0.0002
      iter    93/100 | w        1.12813 | loss        5.0118 | grad       -0.0001
      iter    94/100 | w        1.12813 | loss        5.0118 | grad       -0.0001
      iter    95/100 | w        1.12813 | loss        5.0118 | grad       -0.0001
      iter    96/100 | w        1.12813 | loss        5.0118 | grad       -0.0001
      iter    97/100 | w        1.12813 | loss        5.0118 | grad       -0.0001
      iter    98/100 | w        1.12813 | loss        5.0118 | grad       -0.0001
      iter    99/100 | w        1.12813 | loss        5.0118 | grad       -0.0001

### Discussion 2b: Does it appear from the *loss* values in trace above that the GD procedure converged?

### Discussion 2c: Does it appear from the *parameter* values in trace above that the GD procedure converged?

### Exercise 2d: What exactly is the gradient of the returned "optimal" value of w?

Use your `calc_grad` function to check the result. What is the gradient of the returned `wopt`?

Does this look totally converged? Can you find a $w$ value that would be even better?

```python
# TODO call calc_grad on the return value from above
```

## Part 3: Diagnostic plots for gradient descent

Let's look at some trace functions.

Whenever you run gradient descent, an *excellent* debugging strategy is the ability to plot the loss, the gradient magnitude, and the parameter of interest at every step of the algorithm.

```python
fig, axes = plt.subplots(nrows=1, ncols=3, sharex=True, sharey=False, figsize=(18,3.6))

axes[0].plot(info_dict['trace_loss_list']);
axes[0].set_title('loss');
axes[1].plot(info_dict['trace_grad_list']);
axes[1].set_title('grad');
axes[2].plot(info_dict['trace_w_list']);
axes[2].set_title('w');

plt.xlim([0, 100]);
```

### Discussion 3a: What value do we expect the *loss* to converge to? Should it always be zero?

### Discussion 3b: What value do we expect the *gradient* to converge to? Should it always be zero?

# Part 4: Larger step sizes

## Try with larger step_size = 0.014

```python
wopt, info_dict = minimize_via_grad_descent(calc_loss, calc_grad, step_size=0.014);
```

```python
fig, axes = plt.subplots(nrows=1, ncols=3, sharex=True, sharey=False, figsize=(12,3))

axes[0].plot(info_dict['trace_loss_list'], '.-');
axes[0].set_title('loss');
axes[1].plot(info_dict['trace_grad_list'], '.-');
axes[1].set_title('grad');
axes[2].plot(info_dict['trace_w_list'], '.-');
axes[2].set_title('w');
```

### Discussion 4a: What happens here? How is this step size different than in Part 3 above?

TODO discuss with your group

## Try with even larger step size 0.1

```python
wopt, info_dict = minimize_via_grad_descent(calc_loss, calc_grad, step_size=0.1, max_iters=25);
```

### Discussion 3b: What happens here with this even larger step size? Is it converging?

### Exercise 3c: What is the largest step size you can get to converge reasonably?

```python
# TODO try some other step sizes here
wopt, info_dict = minimize_via_grad_descent(calc_loss, calc_grad, step_size=0) # TODO fix step_size
```

# Part 5: Sensitivity to initial conditions

### Exercise 5a: Try to call the defined procedure with a different initial condition for $w$. What happens?

You could try $w = 5.0$ or something else.

```python
# TODO try some other initial condition for init_w
wopt2, info_dict2 = minimize_via_grad_descent(calc_loss, calc_grad, init_w=0, step_size=0.001, max_iters=10) # TODO fix step_size
```

### Exercise 5b: Try again with another initial value.

```python
# TODO try some other initial condition for init_w
wopt3, info_dict3 = minimize_via_grad_descent(calc_loss, calc_grad, init_w=0, step_size=0.001, max_iters=10) # TODO fix
```

### Exercise 5c: Make a trace plot

Make a trace plot showing convergence from multiple different starting values for $w$. What do you notice?

```python
# TODO
```

# Part 6: Using scipy's built-in gradient optimization tools

```python
import scipy.optimize
```

Take a look at SciPy's built in minimization toolbox

<https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html#scipy.optimize.minimize>

We'll use "L-BFGS", a second-order method that uses the function and its gradient.

This is a "quasi-newton" method, which you can get an intuition for here:

https://en.wikipedia.org/wiki/Newton%27s_method_in_optimization

```python
result = scipy.optimize.minimize(calc_loss, 0.0, jac=calc_grad, method='L-BFGS-B')

# Returns an object with several fields, let's print the result to get an idea
print(result)
```

```python
print(str(result.message))
```

```python
best_w = result.x
print(best_w)
```

```python

```


# In depth: Gradients

This tutorial shows how to supply gradient information about an objective to simplenlopt in SciPy or NLopt style. One example for modern automatic differentiation via the external package autograd is also included. The studied optimization problem is again the Rosenbrock function. Its objective and partial derivatives are given by

$$
\begin{align}
f(x, y) & =(1-x)^2+100(y-x^2)^2\\
\frac{\partial f}{\partial x} &=-2(1-x)-400x(y-x^2) \\
\frac{\partial f}{\partial y} &=200(y-x^2)
\end{align}
$$

## jac=callable

The easiest case which is also shown in the quickstart example. Objective and gradient are supplied as two individual functions:

```python
import numpy as np
import simplenlopt
from time import time

def rosenbrock(pos):

    x, y = pos
    return (1-x)**2 + 100 * (y - x**2)**2

def rosenbrock_grad(pos):

    x, y = pos
    dx = 2 * x -2 - 400 * x * (y-x**2)
    dy = 200 * (y-x**2)

    return np.array([dx, dy])

x0=np.array([-1.5, 2.25])
%timeit -n 1000 res = simplenlopt.minimize(rosenbrock, x0, jac = rosenbrock_grad)
```

    2.33 ms ± 51.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

## jac = True

Taking another look at the objective and its partial derivaties you can see that the expression in the brackets appear in both the objective and the partial derivatives. If both are calculated in individual functions, these terms are unnecessarily recomputed. This can be avoided by supplying both the objective and its gradient in one function. That the objective also contains the gradient information, is indicated by setting jac=True. Let's see how this works and how it affects the performance.

```python
def rosenbrock_incl_grad(pos):

    x, y = pos
    first_bracket = 1-x
    second_bracket = y-x*x

    obj = first_bracket*first_bracket+100*second_bracket*second_bracket
    dx = -2*first_bracket-400*x*second_bracket
    dy = 200 * second_bracket

    return obj, np.array([dx, dy])

%timeit -n 1000 res = simplenlopt.minimize(rosenbrock_incl_grad, x0, jac = True)
```

    2.25 ms ± 44.3 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

We see a small performance improvement. For more complicated objective functions, the performance gains can be massive though it repeated computations can be avoided.

## jac='nlopt'
This flag is mostly for former NLopt users. It indicates that the objective and its gradient are supplied in vanilla [NLopt style](https://nlopt.readthedocs.io/en/latest/NLopt_Tutorial/#example-in-python). NLopt requires another signature for the objective: ``f(x, grad)`` instead of ``f(x)``. The gradient given by grad is given by a NumPy array which must be replaced in-place. For the Rosenbrock example this looks like:

```python
def rosenbrock_nlopt_style(pos, grad):

    x, y = pos
    first_bracket = 1-x
    second_bracket = y-x*x

    obj = first_bracket*first_bracket+100*second_bracket*second_bracket

    if grad.size > 0:

        dx = -2*first_bracket-400*x*second_bracket
        dy = 200 * second_bracket
        grad[0] = dx
        grad[1] = dy

    return obj

%timeit -n 1000 res = simplenlopt.minimize(rosenbrock_nlopt_style, x0, jac = 'nlopt')
```

    2.14 ms ± 13.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

The in-place replacement led to another small performance gain. Side note: while the if statement might seem weird and unnecessary, it is required for some of the optimizers, so you are on the safe side if you include it in your objective function .

## jac = '3-point'/'2-point'
These flags tell simplenlopt which finite difference scheme to use. Finite differencing is borrowed from [SciPy](https://github.com/scipy/scipy/blob/v1.6.3/scipy/optimize/_numdiff.py). Note that '2-point' requires less function evaluations but is less precise and therefore more prone to cause optimization failures.

```python
%timeit -n 100 res = simplenlopt.minimize(rosenbrock, x0, jac = '3-point')
```

    7.39 ms ± 245 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

This example shows that finite differences are not competitive against analytical gradients. For simple cases such as low dimensional curve fitting they are often still useful. If possible, automatic differentiation represents a powerful alternative.

## Autodiff

In recent years, automatic differentiation (autodiff) has become one of the building blocks of machine learning. Many frameworks such as pytorch and tensorflow actually are centered around autodiff. Here, we will use the slightly older [autograd](https://github.com/hips/autograd) package to automatigally compute the gradient for us and feed it to simplenlopt.

```python
import autograd.numpy as anp
from autograd import value_and_grad

rosen_and_grad = value_and_grad(rosenbrock)
%timeit -n 10 res = simplenlopt.minimize(rosen_and_grad, x0, jac = True)
```

    17.3 ms ± 392 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

While autograd results in the worst performance for this example, autodiff shines when it comes to high dimensional problems where the inaccuracies of finite differences are much more severe. To circumvent autograd's performance issues, another candidate could be for example autograd's succesor [jax](https://github.com/google/jax) which additionally provides just-in-time compilation.


# Miscellaneous Modules
## SymPy
**SymPy** is a symbolic mathematics module for Python.

```
from __future__ import print_function, division
from sympy import *
x, y, z, t = symbols('x y z t')
k, m, n = symbols('k m n', integer=True)
f, g, h = symbols('f g h', cls=Function)
init_printing()

a = Integral(cos(x), x)
Eq(a, a.doit())
```

```
b = Integral(sqrt(1/x), x)
Eq(b, b.doit())
```

```
s = "x**2 + 3*x - 1/2"
c = sympify(s)
d = diff(c)
pretty_print(d)
print(d.subs(x, 2))
```

    2⋅x + 3
    7

```
print(integrate(exp(-x), (x, 0, oo)))
```

    1

```
integrate(exp(-x**2 - y**2), (x, -oo, oo), (y, -oo, oo))
```

## Vectorize Calculations
NumPy functions are optimized to work with arrays and are much faster than performing the computations in loops. This can be demonstrated by calculating the $\sin$ of an array of numbers using any **math** function inside a loop and using the corresponding NumPy function.

```
import math
import numpy as np

def f1(x):
    y = np.zeros(x.shape, dtype=float)
    for i in range(len(x)):
        y[i] = math.sin(x[i])
    return x

def f2(x):
    return np.sin(x)

x = np.linspace(0, 10, 1001)
%timeit f1(x)
%timeit f2(x)
```

    1000 loops, best of 3: 427 µs per loop
    10000 loops, best of 3: 24.8 µs per loop

## Numba

Numba provides a Just In Time compiler to generate optimized machine code to speed up computations in Python. It uses LLVM compiler infrastructure.

```
import numpy as np
from numba import f8, jit

def sum1d(x):
    s = 0.0
    for i in range(x.shape[0]):
        s += x[i]
    return s

@jit(f8(f8[:]))
def fast_sum1d(x):
    s = 0.0
    for i in range(x.shape[0]):
        s += x[i]
    return s

x = np.linspace(0, 100, 1001)

%timeit sum1d(x)
%timeit fast_sum1d(x)
```

    1000 loops, best of 3: 196 µs per loop
    1000000 loops, best of 3: 1.22 µs per loop

Let us compute the approximate value of $\pi$ using the following sum:

$$\pi \approx \sqrt{6 \sum_{k=1}^{n+1} \frac{1}{k^2}}$$

```
def approx_pi(n=10000):
    s = 0
    for k in range(1, n+1):
        s += 1.0 / k**2
    return (6 * s) ** 0.5

%timeit approx_pi(1000)
%timeit approx_pi(10000)

print(approx_pi(1000))
print(approx_pi(10000))
```

    10000 loops, best of 3: 174 µs per loop
    1000 loops, best of 3: 1.8 ms per loop
    3.14063805621
    3.14149716395

```
from numba.decorators import autojit

@autojit
def fast_approx_pi(n):
    s = 0
    for k in range(1, n+1):
        s += 1.0 / k**2
    return (6 * s) ** 0.5

n = 10000
%timeit approx_pi(n)
%timeit fast_approx_pi(n)
```

    100 loops, best of 3: 1.83 ms per loop
    10000 loops, best of 3: 44.8 µs per loop

## Other Modules

1. **Data frames, data munging:** pandas
2. **Machine learning:** scikit-learn
3. **Statistical modelling:** statsmodels
4. **Image processing:** scikits-image

# Image Processing using scikit-image
scikit-image is an image proessing module for SciPy.While SciPy has some image processing functions, scikit-image greatly enhances image processing capabilities of Python.

```
import matplotlib.pyplot as plt
from scipy import ndimage
from skimage import data, io, filter
%matplotlib inline

```

```
image = data.coins() # or any NumPy array!
plt.subplot(211)
io.imshow(image)
plt.subplot(212)
edges = filter.sobel(image)
io.imshow(edges)
plt.show()
```

```

```


# Initial Conditions
## Data

```python
m = 42.0; k = 32000.0; z = 0.02
x0 = 200/1000; v0 = -800/1000
P = 700; a = 1.3
```

## Particular Integral
The loading and the assumed particular integral, as well its derivatives
$$p(t) = P_0 \, \exp(-a t),$$
$$\xi(t)=C\exp(-a t),\quad\dot\xi=-aC\exp(-a t),\quad\ddot\xi=a^2C\exp(-a t).$$
Substituting $\xi$ into the EoM we can simplify and solve for $C$:
\begin{align*}
  C(k-ac+^2m)\exp(-a t) = P \exp(-at)\ \rightarrow\ C=\frac{P}{k-ac+a^2m}.
\end{align*}

```python
c = 2*z*np.sqrt(k*m)
C = P/(k-a*c+a*a*m)
```

```python
display(Math('C=%.2f\\,\\text{mm}'%(C*1000)))
```

$\displaystyle C=21.87\,\text{mm}$

## General Integral

$$x = \exp(-\zeta\omega_nt) (A\cos(\omega_Dt)+B\sin(\omega_Dt)) + C\exp(-a t),$$$$
  x(0) = A+C, \quad \dot x(0)= \omega_DB-\zeta\omega_nA-aC.$$

```python
wn2 = k/m
wn = np.sqrt(wn2)
wd = wn*np.sqrt(1-z*z)
# A+C=x0; wd B - z wn A -a C = v0
A = x0-C; B = (v0 + a*C + z*wn*A)/wd
```

```python
expw = f'\\exp(-{(z*wn):.3f}\\,t)' ; expa = f'\\exp(-{a:.3f}\\,t)'
cosd = f'\\cos({wd:.3f}\\,t)' ; sind = f'\\sin({wd:.3f}\\,t)'
lp = '\\big(' ; rp = '\\big)'
def f(v): return f'{v:+.4f}'
display(Latex(r'\begin{align*}x(t) &=' +
             expw+lp+f(A)+cosd+f(B)+sind+rp+f(A)+expa+r'\\' +
             r'\dot x(t) &=' +
             expw+lp+f(B*wd-z*wn*A)+cosd+f(-A*wd-z*wn*B)+sind+rp +
             f(-a*A)+expa + r'\end{align*}'))
```

\begin{align*}x(t) &=\exp(-0.552\,t)\big(+0.1781\cos(27.597\,t)-0.0244\sin(27.597\,t)\big)+0.1781\exp(-1.300\,t)\\\dot x(t) &=\exp(-0.552\,t)\big(-0.7716\cos(27.597\,t)-4.9025\sin(27.597\,t)\big)-0.2316\exp(-1.300\,t)\end{align*}

```python
t = np.linspace(0,2,1001)
def x(t): return np.exp(-z*wn*t)*(A*np.cos(wd*t)+B*np.sin(wd*t))+C*np.exp(-a*t)
plt.plot(t,100*x(t), label='$x(t)$')
plt.plot(t,100*P*np.exp(-a*t)/k, label='$\Delta_\\mathrm{stat}$')
plt.grid()
plt.legend()
plt.xlabel('$t/s$')
plt.ylabel('$x/cm$')
None
```

# Rayleigh Quotient

We need some symbols (`sp` is an alias for the `sympy` module, aka library, that provides symbolic math to Python)

```python
L, k, mu0, eta, x0, x1, omega = sp.symbols('L k mu_0 eta x_0 x_1 omega')
```

We write an expression for $\mu(\eta)$ and we integrate it to compute the total mass, $m=\int_0^L\mu(x)\,dx=\mu_0 L \int_0^1\mu(\eta)\,d\eta$

```python
mu = 2-eta
mass = sp.integrate(mu,(eta, 0, 1))
```

```python
display(Math('m='+sp.latex(mass)+r'\,\mu_0L'))
```

$\displaystyle m=\frac{3}{2}\,\mu_0L$

We write the expression for $V_\mathrm{max}$ (that's easy) and the one for $T_\mathrm{max}$, here we start computing the integral
$\int_0^L\mu(x)v^2(x)\,dx=L\int_0^1\mu(\eta)(x_0+(x_1-x_0)\eta)^2\,d\eta$ to finally adjust our expression collecting terms and multiplying by $\omega^2$

```python
V = (2*x0**2+3*x1**2)*k/2
temp = sp.integrate(mu*(x0+eta*(x1-x0))**2,(eta,0,1)).expand()*L
T = omega**2*sp.collect(temp,mu0)/2
```

```python
display(Math('T_\mathrm{max}=V_\mathrm{max}\Rightarrow'+sp.latex(sp.Eq(T,V))))
```

$\displaystyle T_\mathrm{max}=V_\mathrm{max}\Rightarrow\frac{L \omega^{2} \left(\frac{7 x_{0}^{2}}{12} + \frac{x_{0} x_{1}}{2} + \frac{5 x_{1}^{2}}{12}\right)}{2} = \frac{k \left(2 x_{0}^{2} + 3 x_{1}^{2}\right)}{2}$

Solving the previous equation we have $\omega^2=\omega^2(x_0, x_1)$,

```python
w2 = sp.solve(sp.Eq(T,V),omega)[1]**2
```

```python
display(sp.Eq(omega**2,w2))
```

Our next step is to remove all dependencies except $x_1$ and finally plot our function of the single variable $x_1$

```python
vals = {k:1, mu0:1, L:1, x0:1}
display(Math(r'\left.\frac{\omega^2}{\omega_0^2}\right|_{x_0=1} = '+sp.latex(w2.subs(vals))))

with rc_context(rc={'axes.grid':True}):
    plot = sp.plot(w2.subs(vals), (x1, 0,1), show=0)
    plot.ylabel = r'$\omega^2/\omega_0^2$'
    plot.show()
```

$\displaystyle \left.\frac{\omega^2}{\omega_0^2}\right|_{x_0=1} = \frac{12 \left(3 x_{1}^{2} + 2\right)}{5 x_{1}^{2} + 6 x_{1} + 7}$

It's apparent that the minimum is near the point $x_1=0.4$ but also $x_1=0.5$ should give us a very good approximation...

```python
w_1_2  = w2.subs(vals).subs(x1,sp.Rational(1,2))
w_4_10 = w2.subs(vals).subs(x1,sp.Rational(4,10))
display(Math(r'\omega^2='+sp.latex(w_1_2)+'\\frac{k}{\\mu_0L}'+
             '=%.6f\\frac{k}{\\mu_0L}'%w_1_2.evalf()+
             ',\\quad \\text{for }x_0=1,\ x_1=\\frac12'))
display(Math(r'\omega^2='+sp.latex(w_4_10)+'\\frac{k}{\\mu_0L}'+
             '=%.6f\\frac{k}{\\mu_0L}'%w_4_10.evalf()+
             ',\\quad \\text{for }x_0=1,\ x_1=0.4'))
```

$\displaystyle \omega^2=\frac{44}{15}\frac{k}{\mu_0L}=2.933333\frac{k}{\mu_0L},\quad \text{for }x_0=1,\ x_1=\frac12$

$\displaystyle \omega^2=\frac{248}{85}\frac{k}{\mu_0L}=2.917647\frac{k}{\mu_0L},\quad \text{for }x_0=1,\ x_1=0.4$

It's not difficult to find the location of the minimum (and hence the first eigenvector) and the corresponding eigenvalue.  First we derive $\omega^2(x_1)$ with respect to $x_1$ and find the roots of the numerator

```python
diffw = w2.subs(vals).diff(x1)
num, _ = sp.fraction(diffw.simplify())
display(sp.Eq(num, 0))
x1min = sp.solve(num)[0]
display(Math(r'\hat x_1 = '+ sp.latex(x1min)+'='+sp.latex(x1min.evalf(6))))
```

so we have an eigenvector, $\{1, \hat x_1\}$ and substituting $\hat x_1$ in $\omega(1,x_1)$ we have the corresponding eigenvalue

```python
display(Math(r'\omega^2(1,\ %.6f)=%.6f\,\omega_0^2'%
             (x1min,w2.subs(vals).subs(x1,x1min))))
```

$\displaystyle \omega^2(1,\ 0.408753)=2.917486\,\omega_0^2$

# Structural Testing

We can estimate the stiffness and the damping ratio.

The stiffness is easy

```python
P = 48*1000; x0 = 12/1000; x4 = 9.81/1000
k = P/x0
```

```python
display(Math(r'k = %.1f\,\text{kN/m}'%(k/1000)))
```

$\displaystyle k = 4000.0\,\text{kN/m}$

and the damping ratio is easy too, mind that we don't need the period of vibration to apply our formula
$$\zeta_{n+1} = \frac{\log\frac{x_0}{x_4}}{4\cdot2\pi}\,\sqrt{1-\zeta_n^2}.$$

Here we start with $\zeta_0=0$ and stop at $\zeta_2$ because we have a meaningful value.

```python
d = np.log(x0/x4); m = 4
z0 = 0
z1 = d/(2*m*np.pi)*np.sqrt(1-z0**2)
z2 = d/(2*m*np.pi)*np.sqrt(1-z1**2)
```

```python
display(Math(r'\zeta_0=%d\,\%%,\ \zeta_1=%.6f\,\%%,\ \zeta_2=%.6f\,\%%.'%(z0*100,z1*100,z2*100)))
```

$\displaystyle \zeta_0=0\,\%,\ \zeta_1=0.801760\,\%,\ \zeta_2=0.801735\,\%.$

Just for fun, we can plot the decrement after $m$ cycles, $x_m/x_0$, as a function of $\zeta$.  On the plot also a horizontal line for our $x_4/x_0=0.8175$,  the intersection with the red curve gives the estimated value of the damping ratio.

```python
z = np.linspace(0,0.03,31)
for n in range(1,6):
    plt.plot(z*100, np.exp(-z*n*2*np.pi/np.sqrt(1-z*z)), label='m=%d'%n)
plt.plot(z*100, z**0*x4/x0, '--', color='xkcd:green', label='$x_4/x_0$ in test')
plt.grid()
plt.legend()
plt.xlabel('$\zeta\,\%$')
plt.title(r'$x_m/x_0$ for varying $\zeta$.')
None
```

# Vibration Isolation & Numerical Integration

## Data, TR

```python
m = 18E3; P0 = 1E3; t0 = 6; w0 = 2*np.pi*10; Pmax = 300
z00 = 0.00; z01 = 0.01; z12 = 0.12
TR = Pmax/P0
```

## Plot the phase,
the angular velocity, the angular acceleration and the unbalanced load.

In the graph of the angular velocity I have put in evidence the natural frequency of vibration of an undamped suspension system (dashed line), a horizontal band of angular velocities around such frequency (in red) that are expected to excite the system in near-resonance and a vertical band (in blue) that highlights the time interval in which this near-resonance excitation is expected to take place.

The same vertical band is drawn on the plot of the unbalanced load and, especially, will be drawn on the plots of the transmitted force, to highlight the dynamic amplification of the response corresponding to a load that is just a fraction of the steady state load.

```python
t = np.linspace(0,8,1001)
a = t/t0
phi0 = w0*t0*np.where(t<t0, a**2*(1-a/3), a-1/3)
phi1 = w0*np.where(t<t0, 2*a-a*a, 1)
phi2 = w0*np.where(t<t0, 2-2*a, 0)/t0
P = P0*(phi1**2*np.sin(phi0)-phi2*np.cos(phi0))/w0**2
```

```python
B200 = (1+TR)/TR
k00 = m * w0**2 / B200
wn2 = k00/m
wn = np.sqrt(wn2)
fn = wn/2/np.pi
flow = fn/1.3
fhig = fn*1.3
# 20 a - 10 a*a = f → 10 a*a -20 a + f = 0
#                   → a = 1 - sqrt(100-10f)/10 → t = t0*a
tlow = t0*(1-np.sqrt(100-10*flow)/10)
thig = t0*(1-np.sqrt(100-10*fhig)/10)

fig = plt.figure(figsize=(8,4))
axes = [plt.subplot2grid((2,3),(0,n)) for n in (0,1,2)]
for extra, ax, f, label in zip((0,1,0),
                               axes,
                              (phi0,phi1,phi2),
                              ('$\phi$', '$\dot\phi$', '$\ddot\phi$')):
    ax.plot(t, f/2/np.pi, label=label)
    ax.xaxis.set_major_locator(plt.MultipleLocator(2))
    ax.set_xlabel('$t/s$')
    ax.set_ylabel(label[:-1]+r'/2\pi$')
    ax.legend()
    ax.grid()
    if extra:
        ax.axhline(y=fn, alpha=0.6, linestyle='--')
        ax.axhspan(flow, fhig, alpha=0.12, color='red')
        ax.axvspan(tlow, thig, alpha=0.12)
ax =  plt.subplot2grid((2,3),(1,0), colspan=3)
ax.plot(t, P/1000, label='$P(t)$')
ax.legend()
ax.grid()
ax.axvspan(tlow, thig, alpha=0.15)
ax.set_xlabel('$t/s$')
ax.set_ylabel('$P$/kN')
fig.tight_layout()
None
```

## Design the Suspension System
for the two different values of the damping ratio $\zeta$ defined at the beginning of this section.

From $$\frac{\sqrt{1+(2\zeta\beta)^2}}{\sqrt{(1-\beta^2)^2+(2\zeta\beta)^2}}=\mathrm{TR}$$ we have
$$ \mathrm{TR}^2((1-\beta^2)^2+(2\zeta\beta)^2)-1-(2\zeta\beta)^2=0$$
that, expanded, is
$$\mathrm{TR}^2 \beta^4 + ((4 \zeta^2 - 2) \mathrm{TR}^2 - 4 \zeta^2)\beta^2 + \mathrm{TR}^2 - 1=0.$$
Substituting $\mathrm{TR}=0.3$ and rearranging we have
$$9 \beta^4 - (364 z^2 + 18)\beta^2 - 91 = 0$$
whose positive root is
$$ \beta^2 = \frac{364 \zeta^2 + \sqrt{(364 \zeta^2 + 18)^2 + 3276}}{18} + 1.$$

```python
B201 = (364*z01**2 + np.sqrt((364*z01**2 + 18)**2 + 3276)) / 18 + 1
k01 = m * w0**2 / B201
c01 = 2*z01*np.sqrt(m*k01)

B212 = (364*z12**2 + np.sqrt((364*z12**2 + 18)**2 + 3276)) / 18 + 1
k12 = m * w0**2 / B212
c12 = 2*z12*np.sqrt(m*k12)
```

```python
print('            mass [kg]  damper [kN·s/m]    spring [MN/m]')
for z_, k_, c_ in ((z00,k00,0), (z01,k01,c01), (z12,k12,c12)):
    print('z= %4.1f%%'%(z_*100),'     '.join('%12.3f'%x for x in (m,c_/1E3,k_/1E6)))
```

                mass [kg]  damper [kN·s/m]    spring [MN/m]
    z=  0.0%    18000.000            0.000           16.399
    z=  1.0%    18000.000           10.863           16.389
    z= 12.0%    18000.000          124.896           15.045

## Numerical Integration
### A function factory

```python
def new_integrator(m,c,k,h):
    c0 = 3*c + 6*m/h
    m0 = 3*m + c*h/2
    k0 = k + 3*c/h + 6*m/h/h
    def integrator(x0, v0, p0, p1):
        a0 = (p0-c*v0-k*x0)/m
        dp = p1 - p0 + a0*m0 + v0*c0
        dx = dp/k0
        dv = 3*(dx/h-v0)-a0*h/2
        return x0+dx, v0+dv
    return integrator
```

### Compute the response for $\zeta=1\,\%$

```python
# Instantiate an integrator
next_xv = new_integrator(m, c01, k01, 8/1000)
# Set the initial conditions, x0 = 0, v0 = 0
R01 = [(0, 0)]
for p0, p1 in zip(P, P[1:]):
    R01.append(next_xv(*R01[-1], p0, p1))
```

```python
R01 = np.array(R01)
plt.figure(figsize=(9,3))
plt.plot(t,k01*R01[:,0]+c01*R01[:,1])
plt.grid()
plt.xlabel('$t$/s')
plt.ylabel('$f$/N')
plt.axvspan(tlow, thig, alpha=0.15)
plt.title(r'Transmitted force $f=f(t)$ for $\zeta=%.2f\,\%%$'%(z01*100))
plt.gca().yaxis.set_major_locator(plt.MultipleLocator(600));
```

Notice how much the transmitted force has been amplified during the near-resonance interval,
notice also that the response, well after this interval, has still a dominant frequency component
corresponding to the natural frequency of vibration of the suspended mass, that superposed
to the steady-state component of the response leads to a large value of the transmitted force.

### Compute the response for $\zeta=12\,\%$

```python
next_xv = new_integrator(m, c12, k12, 8/1000)
R12 = [(0, 0)]
for p0, p1 in zip(P, P[1:]):
    R12.append(next_xv(*R12[-1], p0, p1))
```

```python
R12 = np.array(R12)
plt.figure(figsize=(9,3))
plt.plot(t,k12*R12[:,0]+c12*R12[:,1])
plt.grid()
plt.xlabel('$t$/s')
plt.ylabel('$f$/N')
plt.axvspan(tlow, thig, alpha=0.15)
plt.title(r'Transmitted force $f=f(t)$ for $\zeta=%.2f\,\%%$'%(z12*100))
plt.gca().yaxis.set_major_locator(plt.MultipleLocator(300));
```

With a largish value of the damping ratio we have a controlled response during the near-resonant excitation interval and the transient response is quickly damped out, leading in a short time to a transmitted force that has exactly (numerical integration works, doesn't it?) the target amplitude of $300\,{}$N.

# Initialization Cells

```python
%matplotlib inline
%config InlineBackend.figure_formats = ['svg']
```

```python
import numpy as np
import sympy as sp

import matplotlib.pyplot as plt
from matplotlib import rc_context

from IPython.display import Math, Latex

sp.init_printing(use_latex=True)
```

```python
%%html
<style>
div.prompt{display:none}
h1 {
    color: #ffffff!important;
    background-color: #606090;
    text-align:center;
    margin-top: 2cm;
    margin-bottom: 1cm;
    padding: 15px;
    font-size: x-large!important ;
}
</style>
```

<style>
div.prompt{display:none}
h1 {
    color: #ffffff!important;
    background-color: #606090;
    text-align:center;
    margin-top: 2cm;
    margin-bottom: 1cm;
    padding: 15px;
    font-size: x-large!important ;
}
</style>


```python
from ipywidgets import interactive, interact
import matplotlib.pyplot as plt
import numpy as np
import ipywidgets as widgets
import sympy as sym
import seaborn as sns
import plotly.graph_objects as go
from plotly.offline import init_notebook_mode, iplot
from numba import jit

init_notebook_mode(connected=True)
jit(nopython=True, parallel=True)
sns.set()

```

```python

class plot():

    def __init__(self, preWidgetN):

        self.N = preWidgetN

        x,y,n ,k = sym.symbols('x, y,n,k', real=True)
        X=np.linspace(0, 10, 100)

        f = sym.Sum((-1)**k*(x**(2*k+1))/(sym.factorial(2*k+1)),(k,0, n))
        #f = sym.Sum((-1)**k*(x**(2*k))/(sym.factorial(2*k)),(k,0, n))
        #print(sym.latex(f))
        f = f.subs(n, self.N.value)
        f = sym.lambdify(x, f)
        self.trace1 = go.Scatter(x=X, y=np.sin(X),
                            mode='lines+markers',
                            name='sin'
                               )
        self.trace2 = go.Scatter(x=X, y=f(X),
                            mode='lines',
                            name=r'$\sum_{k=0}^{%s} \frac{\left(-1\right)^{k} x^{2 k + 1}}{\left(2 k + 1\right)!}$' %(self.N.value)
                               )

        layout = go.Layout(template='plotly_dark')

        self.fig = go.FigureWidget(data=[self.trace1, self.trace2],
                                   layout = layout,
                                   layout_yaxis_range=[-3 , 3]
                                  )

    def sineSeries(self, change):

        x,y,n ,k = sym.symbols('x, y,n,k', real=True)
        X=np.linspace(0, 10, 100)
        f = sym.Sum((-1)**k*(x**(2*k+1))/(sym.factorial(2*k+1)),(k,0, n))
        #f = sym.Sum((-1)**k*(x**(2*k))/(sym.factorial(2*k)),(k,0, n))
        f = f.subs(n, self.N.value)
        f = sym.lambdify(x, f)

        with self.fig.batch_update():
            self.fig.data[1].x = X
            self.fig.data[1].y = f(X)
            self.fig.data[1].name = r'$\sum_{k=0}^{%s} \frac{\left(-1\right)^{k} x^{2 k + 1}}{\left(2 k + 1\right)!}$' %(self.N.value)

        return

    def show(self):
        self.N.observe(self.sineSeries, names='value')
        display(self.N, self.fig)
        return
```

```python
N = widgets.IntSlider(min=0, max=20, step=1, value=0, description='partial sum order')

p = plot(N)

p.show()
```

    IntSlider(value=0, description='partial sum order', max=20)

    FigureWidget({
        'data': [{'mode': 'lines+markers',
                  'name': 'sin',
                  'type': 'scat…

```python

```


# Example #1: Neural Network for $y = \sin(x)$

Same example as yesterday, a sine-curve with 10 points as training values:

```
import numpy as np
import matplotlib.pyplot as plt

x = np.arange(0,6.6, 0.6)
y = np.sin(x)

xplot = np.arange(0, 6.6, 0.01)
yplot = np.sin(xplot)

plt.scatter(x,y, color="b", label="Training")
plt.plot(xplot, yplot, color="g", label="sin(x)")

plt.legend()
plt.show()
```

## Defining the architecture of our neural network:

Fully connected with 1 input node, 1 hidden layer, 1 output node.

Layer connections:
\begin{equation}
y = b+\sum_i x_i w_i
\end{equation}

**Question:** "How many weights are there in the above example?"

### Defining the Activation function (sigmoid):
\begin{equation}
\sigma\left(x\right) = \frac{1}{1 + \exp\left(-x\right)}
\end{equation}
Popular because the derivative of the sigmoid function is simple:
\begin{equation}
\frac{\mathrm{d}}{\mathrm{d}x}\sigma\left(x\right) = \sigma\left(x\right)\left(1 - \sigma\left(x\right)\right)
\end{equation}

```
def activation(val):

  sigmoid = 1.0 / (1.0 + np.exp(-val))
  return sigmoid
```

### Defining the architecture (i.e. the layers):

*   `input_value` - Input value
*   `w_ih` - Weights that connect input layer with hidden layer
*   `w_io` - Weights that connect hidden layer with output layer

```
def model(input_value, w_ih, w_ho):

  hidden_layer = activation(input_value * w_ih)
  output_value = np.sum(hidden_layer*w_ho)

  return output_value
```

Let's start by testing the neural network with random weights:

```
np.random.seed(1000)
random_weights_ih = np.random.random(10)
random_weights_ho = np.random.random(10)

print(random_weights_ih)
print(random_weights_ho)
print()

val = 2.0
sinx_predicted = model(val, random_weights_ih, random_weights_ho)

print("Predicted:", sinx_predicted)
print("True:     ", np.sin(2.0))
```

Setting our Model parameters:

```
# The number of nodes in the hidden layer
HIDDEN_LAYER_SIZE = 40

# L2-norm regularization
L2REG = 0.01
```

## Optimizing the weights:

We want to find the best set of weights $\mathbf{w}$ that minimizes some loss function. For example we can minimize the squared error (like we did in least squares fitting):

\begin{equation}
L\left(\mathbf{w}\right) = \sum_i \left(y_i^\mathrm{true} -  y_i^\mathrm{predicted}(\mathbf{w}) \right)^{2}
\end{equation}
Or with L2-regularization:
\begin{equation}
L\left(\mathbf{w}\right) = \sum_i \left(y_i^\mathrm{true} -  y_i^\mathrm{predicted}(\mathbf{w}) \right)^{2} + \lambda\sum_j w_j^{2}
\end{equation}
Just like in the numerics lectures and exercises, we can use a function from SciPy to do this minimization: `scipy.optimize.minimize()`.

```
def loss_function(parameters):

  w_ih = parameters[:HIDDEN_LAYER_SIZE]
  w_ho = parameters[HIDDEN_LAYER_SIZE:]

  squared_error = 0.0

  for i in range(len(x)):

    # Predict y for x[i]
    y_predicted = model(x[i], w_ih, w_ho)

    # Without # Regularization
    squared_error = squared_error + (y[i] - y_predicted)**2

    # With regularization
    # rmse += (z - y[i])**2 + np.linalg.norm(parameters) * L2REG

  return squared_error
```

## Running the minimization with `scipy.optimize.minimize()`:

Documentation: https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html

Since we haven't implemented the gradient of the neural network, we can't use optimizers that require the gradient. One algorithm we can use is the Nelder-Mead optimizer.

```
from scipy.optimize import minimize

# Define random initial weights
np.random.seed(666)
p = np.random.random(size=2*HIDDEN_LAYER_SIZE)

# Minimize the loss function with parameters p
result = minimize(loss_function, p, method="Nelder-Mead",
                  options={"maxiter": 100000, "disp": True})

wfinal_in = result.x[:HIDDEN_LAYER_SIZE]
wfinal_hl = result.x[HIDDEN_LAYER_SIZE:]

print(wfinal_in)
print(wfinal_hl)
```

```

# Print sin(2.5) and model(2.5)
val = 2.5
sinx_predicted = model(val, wfinal_in, wfinal_hl)

print("Predicted:", sinx_predicted)
print("True:     ", np.sin(val))
```

Lets make a plot with pyplot!

```
xplot = np.arange(0,6.6, 0.01)
yplot = np.sin(xplot)

ypred = np.array([model(val, wfinal_in, wfinal_hl) for val in xplot])

import matplotlib.pyplot as plt

plt.plot(xplot,yplot, color="g", label="sin(x)")
plt.scatter(x, y, color="b", label="Training")
plt.plot(xplot, ypred, color="r", label="Predicted")

plt.ylim([-2,2])
plt.show()

```

## What to do about "crazy" behaviour?
*   Regularization
*   Adjust hyperparameters (hidden layer size)


# Lecture 23

# Beta distribution, Bayes' Billiards

## Beta Distribution

The Beta distribution is characterized as $Beta(\alpha, \beta)$ where $\alpha \gt 0 \text{, } \beta \gt 0$.

The Beta distribution PDF is given by:

\begin{align}
  f(x) &= c \, x^{\alpha-1} \, (1 - x)^{\beta-1} \quad \text{where } 0 \lt x \lt 1
\end{align}

We will put aside the normalization constant $c$ for now (wait until lecture 25!).

$Beta(\alpha ,\beta)$ distribution

* is a flexible family of continuous distributions over $(0,1)$ (see graph below for some examples)
* often used as a _prior_ for a parameter in range $(0,1)$
* _conjugate prior_ for $Binomial$ distribution

```python
import matplotlib
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import beta
%matplotlib inline

plt.xkcd()

x = np.linspace(0, 1.0, 500)

alphas = [0.5, 5, 1, 1, 2, 3]
betas  = [0.5, 3, 2, 1, 1, 5]
params = map(list, zip(alphas, betas))

_, ax = plt.subplots(figsize=(12,8))

for a,b in params:
    y = beta.pdf(x, a, b)
    ax.plot(x, y, label=r'$\alpha$={}, $\beta$={}'.format(a,b))
ax.grid(True, which='both')
ax.axhline(y=0, color='k')
ax.axvline(x=0, color='k')

legend = ax.legend()
for label in legend.get_texts():
    label.set_fontsize('large')
for label in legend.get_lines():
    label.set_linewidth(1.5)

plt.title(r'Beta Distribution $f(x) = \frac{x^{\alpha-1} \, (1-x)^{\beta-1}}{B(\alpha,\beta)}$')
plt.xlabel(r'x')
plt.ylabel(r'f(x)')
plt.xlim((0,1.0))
plt.ylim((0,5.0))

plt.show()
```

### Conjugate prior to the Binomial

Recall Laplace's Rule of Sucession dealing with the problem of the sun rising: there, we probability $p$ that the sun will rise on any given day $X_k$, given a consecutive string of days $X_1,X_2, \dots, X_{k-1}$, to be i.i.d. $Bern(p)$.

We made an assumption that $p \sim Unif(0,1)$.

$Beta(1,1)$ is the same as $Unif(0,1)$, and so we will show how to generalize using the $Beta$ distribution.

Given $X|p \sim Bin(n,p)$. We get to observe $X$, but we do not know the true value of $p$.

In such a case, we can assume that the _prior_ $p \sim Beta(\alpha, \beta)$. After observing $n$ further trials, where perhaps $k$ are successes and $n-k$ are failures, we can use this information to update our beliefs on the nature of $p$ using Bayes Theorem.

So we what we want is the _posterior_ $p|X$, since we will get to observe more values of $X$ and want to update our understanding of $p$.

\begin{align}
  f(p|X=k) &= \frac{P(X=k|p) \, f(p)}{P(X=k)} \\
  &= \frac{\binom{n}{k} \, p^k \, (1-p)^{n-k} \, c \, p^{\alpha-1}(1-p)^{\beta-1}}{P(X=k)} \\
  &\propto p^{\alpha + k - 1} (1-p)^{\beta + n - k - 1} \quad \text{since }\binom{n}{k} \text{, }c \text{ and }P(X=k) \text{ do not depend on }p \\
  \\
  \Rightarrow p|X &\sim Beta(\alpha + X, \beta + n - X)
\end{align}

_Conjugate_ refers to the fact that we are looking at an entire _family_ of $Beta$ distributions as the _prior_. We started off with $Beta(\alpha, \beta)$, and after an additional $n$ more observations of $X$ we end up with $Beta(\alpha + X, \beta + n - X)$.

### Bayes' Billiards

The $Beta(\alpha, \beta)$ distribution has PDF:

\begin{align}
  f(x) &= c \, x^{\alpha-1} \, (1 - x)^{\beta-1}
\end{align}

Let's try to find the normalizing constant $c$ for the case where $\alpha \gt 0 \text{, } \beta \gt 0$, and $\alpha,\beta \in \mathbb{Z}$

In order to do that, we need to find out

\begin{align}
  \int_0^1 x^k \, (1 - x)^{n-k} \, dx \\
  \\
  \rightarrow \int_0^1 \binom{n}{k} x^k \, (1 - x)^{n-k} \, dx
\end{align}

#### Story 1
We have $n+1$ white billiard balls, and we paint one of them pink. Now we throw them down on the number line from 0 to 1.

#### Story 2
We throw our $n+1$ white billiard balls down on the number line from 0 to 1, and _then_ randomly select one of them to paint in pink.

_Note that the image above could have resulted from either of the stories, so both stories are actually equivalent._

#### Doing calculus without using calculus

At this point, we know exactly how to the above integral _without using any calculus._

Let $X = \text{# balls to left of pink}$, so $X$ ranges from $0$ to $n$. If we condition on $p$ (pink billiard ball), we can consider this to be a binomial distribution problem, where "success" means being to the left of the pink ball.

\begin{align}
  P(X = k) &= \int_0^1 P(X=k|p) \, f(p) \, dp &\quad \text{conditioning on } p \\
  &= \int_0^1 \binom{n}{k} p^k \, (1-p)^{n-k} \, dp &\quad \text{since } f(p) \sim Unif(0,1) \\
  \\
  & &\quad \text{but from Story 2, } k \text{ could be any value in } \{0,n\} \\
  \\
  &= \boxed{ \frac{1}{n+1} }
\end{align}

And so now we have the normalizing constant when $\alpha, \beta$ are positive integers.