```
# default_exp twoDim
```

```
#hide
import matplotlib.pyplot as plt
import seaborn as sns
import matplotlib.cm as cm
import matplotlib.colors as colors
import matplotlib.colorbar as colorbar
import matplotlib.tri as tri
plt.rcParams['figure.figsize'] = (10,6)
import sympy; sympy.init_printing()
# code for displaying matrices nicely
def display_matrix(m):
    display(sympy.Matrix(m))
```

# twoDim
> Code for a 2-D problem.

```
#hide
from nbdev.showdoc import *
```

# 2 dimensional case (PDE)

We consider the following 2-D problem:

$$\nabla\cdot\left(\kappa(x)\nabla u(x)\right)=f(x) \quad\forall x\in D=[0,1]^{2}$$
$$u(x)=0\quad\forall x\in\partial D$$
where here $f$ is again a random forcing term, assumed to be a GP in this work.

## Variational formulation

The variational formulation is given by:

$$a(u,v)=L(v)$$

where:

$$a(u,v)=\int_{D}\nabla u\cdot\left(\kappa\nabla u\right)dx$$

and

$$L(v)=\int_{D}fvdx$$

We will make the following choices for $\kappa,f$:

$$\kappa(x)=1$$

$$f\sim\mathcal{G}\mathcal{P}(\bar{f},k_{f})$$

$$\bar{f}(x)=1$$

$$ k_{f}(x,y) = \sigma_f^{2}\exp\left(-\frac{\|x-y\|^2}{2l_f^2}\right)$$

$$ \sigma_{f} = 0.1$$

$$ l_f = 0.4 $$

where $\|\cdot\|$ is the usual Euclidean norm.

Since we do not have access to a suitable Green's function for this problem, we will have to estimate the rate of convergence of the statFEM prior and posterior by comparing them on a sequence of refined meshes. More details on this will follow later. Thus, we need similar code as for the 1-D problem.

## statFEM prior mean

We will again utilise FEniCS to obtain the statFEM prior mean. For this purpose, we create a function `mean_assembler` which will assemble the mean for the statFEM prior.

```
#export
from dolfin import *
import numpy as np
from scipy import integrate
from scipy.spatial.distance import cdist
from scipy.linalg import sqrtm
from scipy.sparse import csr_matrix
from scipy.sparse.linalg import spsolve
from scipy.interpolate import interp1d
from joblib import Parallel, delayed
import multiprocessing

# code to assemble the mean for a given mesh size
def mean_assembler(h,f_bar):
    "This function assembles the mean for the statFEM prior for our 2-D problem"
    # get size of the grid
    J = int(np.round(1/h))

    # set up the mesh and function space for FEM
    mesh = UnitSquareMesh(J,J)
    V = FunctionSpace(mesh,'Lagrange',1)

    # set up boundary condition
    def boundary(x, on_boundary):
        return on_boundary

    bc = DirichletBC(V, 0.0, boundary)
    # set up the functions κ and f
    κ = Constant(1.0)
    f = f_bar

    # set up the bilinear form for the variational problem
    u = TrialFunction(V)
    v = TestFunction(V)
    a = inner(κ*grad(u),grad(v))*dx

    # set up the linear form
    L = f*v*dx

    # solve the variational problem
    μ = Function(V)
    solve(a == L, μ, bc)

    return μ
```

`mean_assembler` takes in the mesh size `h` and the mean function `f_bar` for the forcing and computes the mean of the approximate statFEM prior, returning this as a FEniCS function.

> Important: `mean_assembler` requires `f_bar` to be represented as a FEniCS function/expression/constant.

Let's check that this is working:

```
h = 0.1
f_bar = Constant(1.0)
μ = mean_assembler(h,f_bar)
μ
```

```
# check the type of μ
assert type(μ) == function.function.Function
```

Let's plot $\mu$:

```
#hide_input
# use FEniCS to plot μ
plot(μ)
plt.xlabel(r'$x$')
plt.ylabel(r'$y$')
plt.title(r'Plot of statFEM mean for $h=%.2f$'%h)
plt.show()
```

## statFEM prior covariance

We will also utilise FEniCS again to obtain an approximation of our statFEM covariance function.

The statFEM covariance can be approximated as follows:

$$c_u^{\text{FEM}}(x,y)\approx\sum_{i,j=1}^{J}\varphi_{i}(x)Q_{ij}\varphi_{j}(y)$$

where $Q=A^{-1}MC_{f}M^{T}A^{-T}$ and where the $\{\varphi_{i}\}_{i=1}^{J}$ are the FE basis functions corresponding to the interior nodes of our domain.

with $C_f$ being the kernel matrix of $f$ (evaluated on the FEM grid).

As we will be comparing the statFEM covariance functions for finer and finer FE mesh sizes we will need to be able to assemble the statFEM covariance function on a grid. As discussed in oneDim, we can assemble such covariance matrices in a very efficient manner. The code remains largely the same as in the 1-D case and so we do not go into as much detail here.

We start by creating a function `kernMat` which assembles the covariance matrix corresponding to a covariance function `k` on a grid `grid`.

```
#export
def kernMat(k,grid,parallel=True,translation_inv=False):
    "Function to compute the covariance matrix $K$ corresponding to the covariance kernel $k$ on a grid. This matrix has $ij$-th entry $K_{ij}=k(x_i,x_j)$ where $x_i$ is the $i$-th point of the grid."
    # get the length of the grid
    n = grid.shape[0]
    # preallocate an n x n array of zeros to hold the cov matrix
    K = np.zeros((n,n))

    # check if the cov matrix should be computed in parallel
    if parallel:
        # compute the cov matrix in parallel by computing the upper triangular part column by column
        # set up function to compute the ith column of the upper triangular part:
        def processInput(i):
            return np.array([k(grid[i,:],grid[j,:]) for j in range(i,n)])

        # get the number of cpu cores present and compute the upper triangular columns in parallel
        num_cores = multiprocessing.cpu_count()
        results = Parallel(n_jobs=num_cores)(delayed(processInput)(i) for i in range(n))

        # store the results in the appropriate positions in K
        #for (i,v) in enumerate(results[0:n-1]):
        for (i,v) in enumerate(results): # is this correct???
            K[i,i:] = v

        # only the upper triangular part has been formed, so use the symmetry of the cov mat to get full K:
        K = K + K.T - np.diag(K.diagonal())
        return K
    elif translation_inv:
        # reshape grid so that it has correct dimensions
        grid = grid.reshape(n,-1)

        # compute the distance matrix D
        D = cdist(grid,grid)

        # evaluate the kernel function using D
        K = k(D)
        return K
    else:
        # compute the cov mat using a nested for loop
        for i in range(n):
            for j in range(i,n):
                K[i,j] = k(grid[i,:],grid[j,:])
        K = K + K.T - np.diag(K.diagonal())
        return K
```

> Note: This function takes in two optional boolean arguments `parallel` and `translation_inv`. The first of these specifies whether or not the cov matrix should be computed in parallel and the second specifies whether or not the cov kernel is translation invariant. If it is, the covariance matrix is computed more efficiently using the `cdist` function from scipy.

Let's quickly test if this function is working, by computing the cov matrix for white noise, which has kernel function $k(x,y)=\delta(x-y)$. For a grid of length $N$ this should be the $N\times N$ identity matrix.

```
# set up the kernel function
# set up tolerance for comparison
tol = 1e-16
def k(x,y):
    if (np.abs(x-y) < tol).all():
        # x == y within the tolerance
        return 1.0
    else:
        # x != y within the tolerance
        return 0.0

# set up grid
n = 21
x_range = np.linspace(0,1,n)
grid = np.array([[x,y] for x in x_range for y in x_range])
N = len(grid) # get length of grid (N=n^2)
K = kernMat(k,grid,True,False) # parallel mode

# check that this is the N x N identity matrix
assert (K == np.eye(N)).all()
```

We now create a function `BigPhiMat` to utilise FEniCS to efficiently compute the matrix $\boldsymbol{\Phi}$ defined above.

```
#export
def BigPhiMat(J,grid):
    "Function to compute the $\Phi$ matrix."
    # create the FE mesh and function space
    mesh = UnitSquareMesh(J,J)
    V = FunctionSpace(mesh,'Lagrange',1)
    # get the tree for the mesh
    tree = mesh.bounding_box_tree()
    # set up a function to compute the ith column of Phi corresponding to the ith grid point
    def Φ(i):
        x = grid[i]
        cell_index = tree.compute_first_entity_collision(Point(*x))
        cell = Cell(mesh,cell_index)
        cell_global_dofs = V.dofmap().cell_dofs(cell_index)
        vertex_coordinates = cell.get_vertex_coordinates()
        cell_orientation = cell.orientation()
        data = V.element().evaluate_basis_all(x,vertex_coordinates,cell_orientation)
        return (data,cell_global_dofs,i*np.ones_like(cell_global_dofs))
    # compute all the columns of Phi using the function above
    res = [Φ(i) for i in range(len(grid))]
    # assemble the sparse matrix Phi using the results
    data = np.hstack([res[i][0] for i in range(len(grid))])
    row = np.hstack([res[i][1] for i in range(len(grid))])
    col = np.hstack([res[i][2] for i in range(len(grid))])
    return csr_matrix((data,(row,col)),shape=(V.dim(),len(grid)))
```

`BigPhiMat` takes in two arguments: `J`, which controls the FE mesh size ($h=1/J^{2}$), and `grid` which is the grid in the definition of $\boldsymbol{\Phi}$. `BigPhiMat` returns $\boldsymbol{\Phi}$ as a sparse `csr_matrix` for memory efficiency.

> Note: Note that since FEniCS works with the FE functions corresponding to all the FE dofs and our statFEM cov matix only uses the FE functions corresponding to non-boundary dofs we need to account for this in the code. See the source code for `BigPhiMat` to see how this is done.

We now create a function `cov_asssembler` which assembles the approximate FEM covariance matrix on the grid.

```
#export
# function to assemble the fem covariance
def cov_assembler(J,k_f,grid,parallel,translation_inv):
    "Function to assemble the approximate FEM covariance matrix on the reference grid."

    # set up mesh and function space
    mesh = UnitSquareMesh(J,J)
    V = FunctionSpace(mesh,'Lagrange',1)

    # set up FE grid
    x_grid = V.tabulate_dof_coordinates()

    # set up boundary condition
    def boundary(x, on_boundary):
        return on_boundary

    bc = DirichletBC(V, 0.0, boundary)

    # get the boundary and interior dofs
    bc_dofs = bc.get_boundary_values().keys()
    first, last = V.dofmap().ownership_range()
    all_dofs = range(last - first)
    interior_dofs = list(set(all_dofs) - set(bc_dofs))
    bc_dofs = list(set(bc_dofs))

    # set up the function κ
    κ = Constant(1.0)

    # get the mass and stiffness matrices as sparse csr_matrices
    u = TrialFunction(V)
    v = TestFunction(V)

    mass_form = u*v*dx
    a = inner(κ*grad(u),grad(v))*dx

    M = assemble(mass_form)
    A = assemble(a)
    M = as_backend_type(M).mat()
    A = as_backend_type(A).mat()
    M = csr_matrix(M.getValuesCSR()[::-1],shape=M.size)
    A = csr_matrix(A.getValuesCSR()[::-1],shape=A.size)

    # extract the submatrices corresponding to the interior dofs
    M = M[interior_dofs,:][:,interior_dofs]
    A = A[interior_dofs,:][:,interior_dofs]

    # get the forcing cov matrix on the interior nodes of the grid
    Σ_int = kernMat(k_f,x_grid[interior_dofs],parallel,translation_inv)

    # form the matrix Q in the defintion of the approximate FEM cov mat
    # Note: overwrite Σ_int for memory efficiency.
#     Σ_int = M @ Σ_int @ M.T
    Σ_int = Σ_int @ M.T
    Σ_int = M @ Σ_int

    Σ_int = spsolve(A,Σ_int)
    Σ_int = spsolve(A,Σ_int.T).T

    # ensure Σ_int is symmetric
    Σ_int = 0.5*(Σ_int + Σ_int.T)

    # get big phi matrix on the grid (extracting only the rows corresponding to the
    # interior dofs)
    Phi = BigPhiMat(J,grid)[interior_dofs,:]
    #print("Computed Phi")

    # assemble cov mat on grid using Phi and Σ_int
    Σ = Phi.T @ Σ_int @ Phi

    # ensure Σ is symmetric and return
    Σ = 0.5*(Σ + Σ.T)
    return Σ
```

`cov_assembler` takes in several arguments which are explained below:

- `J`: controls the FE mesh size ($h=1/J^{2})$
- `k_f`: the covariance function for the forcing $f$
- `grid`: the reference grid where the FEM cov matrix should be computed on
- `parallel`: boolean argument indicating whether the intermediate computation of $C_f$ should be done in parallel
- `translation_inv`: boolean argument indicating whether the intermediate computation of $C_f$ should be computed assuming `k_f` is translation invariant or not

As a quick demonstration that the code is working, we will the statFEM cov matrix for a relatively coarse grid.

```
# set up kernel function for forcing
f_bar = Constant(1.0)

l_f = 0.4
σ_f = 0.1

def k_f(x):
    return (σ_f**2)*np.exp(-(x**2)/(2*(l_f**2)))

# set up grid
n = 21
x_range = np.linspace(0,1,n)
grid = np.array([[x,y] for x in x_range for y in x_range])

# get the statFEM grid for a particular choice of J
J = 10
Σ = cov_assembler(J,k_f,grid,False,True)
```

Let's plot a heatmap of the statFEM cov matrix:

```
#hide_input
sns.heatmap(Σ,cbar=True,
              annot=False,
              xticklabels=False,
              yticklabels=False,
              cmap=cm.viridis)
plt.title('Heat map of statFEM covariance matrix')
plt.show()
```

> Note: that the banded structure in the above statFEM covariance matrix is due to the internal ordering of the FE grid in FEniCS.

## statFEM posterior mean

The statFEM posterior from incorporating sensor readings has the same form as given in oneDim. We will thus require very similar code as to the 1-D case. We start by creating a function `m_post` which evaluates the posterior mean at a given point.

```
#export
def m_post(x,m,c,v,Y,B):
    "This function evaluates the posterior mean at the point $x$."
    m_vect = np.array([m(y_i) for y_i in Y]).flatten()
    c_vect = c(x).flatten()

    # compute the update term
    update = c_vect @ np.linalg.solve(B,m_vect-v)

    # return m_post
    return (m(x) - update)
```

`m_post` takes in several arguments which are explained below:

- `x`: point where the posterior mean will be evaluated
- `m`: function which computes the prior mean at a given point y
- `c`: function which returns the vector (c(x,y)) for y in Y (note: c is the prior covariance function)
- `v`: vector of noisy sensor readings
- `Y`: vector of sensor locations
- `B`: the matrix $\epsilon^{2}I+C_Y$ to be inverted in order to obtain the posterior

We now require code to generate samples from a GP with mean $m$ and cov function $k$ on a grid. We write the function `sample_gp` for this purpose.

```
#export
def sample_gp(n_sim,m,k,grid,par,trans,tol=1e-9):
    "Function to sample a GP with mean $m$ and cov $k$ on a grid."

    # get length of grid
    d = len(grid)

    # construct mean vector
    μ = np.array([m(x) for x in grid]).reshape(d,1)

    # construct covariance matrix
    Σ = kernMat(k,grid,parallel=par,translation_inv=trans)

    # construct the cholesky decomposition Σ = GG^T
    # we add a small diagonal perturbation to Σ to ensure it
    # strictly positive definite
    G = np.linalg.cholesky(Σ + tol * np.eye(d))

    # draw iid standard normal random vectors
    Z = np.random.normal(size=(d,n_sim))

    # construct samples from GP(m,k)
    Y = G@Z + np.tile(μ,n_sim)

    # return the sampled fields
    return Y
```

`sample_gp` takes in several arguments which are explained below:

- `n_sim`: number of trajectories to be sampled
- `m`: mean function for the GP
- `k`: cov function for the GP
- `grid`: grid of points on which to sample the GP
- `par`: boolean argument indicating whether the computation of the cov matrix should be done in parallel
- `trans`: boolean argument indicating whether the computation of the cov matrix should be computed assuming `k` is translation invariant or not
- `tol`: controls the size of the tiny diagonal perturbation added to cov matrix to ensure it is strictly positive definite (defaults to `1e-9`)

As a quick demonstration that the code is working lets generate 2 realisations of white noise, using the kernel `k` from one of the previous tests and plot a heatmap of these random fields side-by-side.

```
#hide_input
n = 41
x_range = np.linspace(0,1,n)
grid = np.array([[x,y] for x in x_range for y in x_range])

# set up mean
def m(x):
    return 0.0

np.random.seed(23534)
samples = sample_gp(2,m,k,grid,True,False)

sample_1 = samples[:,0].flatten()
sample_2 = samples[:,1].flatten()

vmin = min(sample_1.min(),sample_2.min())
vmax = max(sample_1.max(),sample_2.max())
cmap = cm.jet
norm = colors.Normalize(vmin=vmin,vmax=vmax)

x = grid[:,0].flatten()
y = grid[:,1].flatten()
triang = tri.Triangulation(x,y)

plt.rcParams['figure.figsize'] = (12,6)
fig, axs = plt.subplots(ncols=3, gridspec_kw=dict(width_ratios=[4,4,0.2]))
axs[0].tricontourf(triang,sample_1.flatten(),cmap=cmap)
axs[1].tricontourf(triang,sample_2.flatten(),cmap=cmap)
cb = colorbar.ColorbarBase(axs[2],cmap=cmap,norm=norm)
fig.suptitle('Realisations of white-noise fields')
plt.show()
```

Let's also quickly generate 2 realisations for the kernel `k_f` above:

```
#hide_input
np.random.seed(534)
samples = sample_gp(2,m,k_f,grid,False,True)

sample_1 = samples[:,0].flatten()
sample_2 = samples[:,1].flatten()

vmin = min(sample_1.min(),sample_2.min())
vmax = max(sample_1.max(),sample_2.max())
cmap = cm.jet
norm = colors.Normalize(vmin=vmin,vmax=vmax)

x = grid[:,0].flatten()
y = grid[:,1].flatten()
triang = tri.Triangulation(x,y)

plt.rcParams['figure.figsize'] = (12,6)
fig, axs = plt.subplots(ncols=3, gridspec_kw=dict(width_ratios=[4,4,0.2]))
axs[0].tricontourf(triang,sample_1.flatten(),cmap=cmap)
axs[1].tricontourf(triang,sample_2.flatten(),cmap=cmap)
cb = colorbar.ColorbarBase(axs[2],cmap=cmap,norm=norm)
fig.suptitle(r'Realisations of random fields with covariance $k_f$')
plt.show()
```

The next bit of code we require is code to generate noisy sensor readings from our system. We write the function `gen_sensor` for this purpose.

```
#export
def gen_sensor(ϵ,m,k,Y,J,par,trans,tol=1e-9,require=False):
    "Function to generate noisy sensor readings of the solution u on a sensor grid Y."

    # get number of sensors from the sensor grid Y
    s = len(Y)

    # create FEM space and grid
    mesh = UnitSquareMesh(J,J)
    V = FunctionSpace(mesh,'Lagrange',1)
    grid = V.tabulate_dof_coordinates()

    # sample a single f on the grid
    f_sim = sample_gp(1,m,k,grid,par=par,trans=trans,tol=tol)

    # set up a FEM function for this realisation
    f = Function(V)
    f.vector().set_local(f_sim.flatten())

    # use FENICS to find the corresponding solution u
    # set up boundary condition
    def boundary(x, on_boundary):
        return on_boundary

    bc = DirichletBC(V, 0.0, boundary)

    # set up the function κ
    κ = Constant(1.0)

    # set up the bilinear form for the variational problem
    u = TrialFunction(V)
    v = TestFunction(V)
    a = inner(κ*grad(u),grad(v))*dx

    # set up the linear form
    L = f*v*dx

    # solve the variational problem
    u_sol = Function(V)
    solve(a == L, u_sol, bc)

    # get solution on grid Y:
    u_Y = np.array([u_sol(y_i) for y_i in Y])

    # add N(0,ϵ^2) to each evaluation point
    u_S = u_Y + ϵ*np.random.normal(size=s)

    if require:
        return u_S, f_sim, u_sol
    else:
        return u_S
```

`gen_sensor` takes in several arguments which are explained below:

- `ϵ`: controls the amount of sensor noise
- `m`: mean function for the forcing f
- `k`: cov function for the forcing f
- `Y`: vector of sensor locations
- `J`: controls the FE mesh size ($h=1/J^{2}$)
- `par`: boolean argument indicating whether the computation of the forcing cov matrix should be done in parallel
- `trans`: boolean argument indicating whether the computation of the forcing cov matrix should be computed assuming `k` is translation invariant or not
- `tol`: controls the size of the tiny diagonal perturbation added to forcing cov matrix to ensure it is strictly positive definite (defaults to `1e-9`)
- `require` : boolean argument indicating whether or not to also return the realisation of the forcing `f_sim` and the FEniCS solution `u_sol` (defaults to `False`)

> Warning: Since we do not have access to the true solution we must use FEniCS to get the solution for our system. Thus, one must choose a small enough `J` in `gen_sensor` above to ensure we get realistic noisy sensor readings.

Let's demonstrate that this code is working, by generating $s=25$ sensor observations with the sensors equally space in the domain $D$.

```
# set up mean function for forcing
def m_f(x):
    return 1.0

# set up sensor grid and sensor noise level
ϵ = 0.2
s = 25
s_sqrt = int(np.round(np.sqrt(s)))
Y_range = np.linspace(0.01,0.99,s_sqrt)
Y = np.array([[x,y] for x in Y_range for y in Y_range])
J_fine = 100 # FE mesh size to compute solution on

# generate the sensor observations
np.random.seed(235)
v_dat = gen_sensor(ϵ,m_f,k_f,Y,J_fine,False,True)
```

```
#export
class MyExpression(UserExpression):
    "Class to allow users to user their own functions to create a FEniCS UserExpression."
    def eval(self, value, x):
        value[0] = self.f(x)
    def value_shape(self):
        return ()
```

```
show_doc(MyExpression,title_level=4)
```

class MyExpression[source]

> MyExpression(**\*`args`**, **\*\*`kwargs`**) :: `UserExpression`

Class to allow users to user their own functions to create a FEniCS UserExpression.

We now require code which will create the matrix $C_Y,h$ and the function $\mathbf{c}^{(h)}$ required for the statFEM posterior mean. We will create the function `fem_cov_assembler_post` for this purpose.

```
#export
def fem_cov_assembler_post(J,k_f,Y,parallel,translation_inv):
    "Function to create the matrix $C_{Y,h}$ and the vector function $c^{(h)}$ required for the statFEM posterior mean."

    # set up mesh and function space
    mesh = UnitSquareMesh(J,J)
    V = FunctionSpace(mesh,'Lagrange',1)
    tree = mesh.bounding_box_tree()

    # set up grid
    x_grid = V.tabulate_dof_coordinates()

    # set up boundary condition
    def boundary(x, on_boundary):
        return on_boundary

    bc = DirichletBC(V, 0.0, boundary)

    # get the boundary and interior dofs
    bc_dofs = bc.get_boundary_values().keys()
    first, last = V.dofmap().ownership_range()
    all_dofs = range(last - first)
    interior_dofs = list(set(all_dofs) - set(bc_dofs))
    bc_dofs = list(set(bc_dofs))

    # set up the function κ
    κ = Constant(1.0)

    # get the mass and stiffness matrices
    u = TrialFunction(V)
    v = TestFunction(V)

    mass_form = u*v*dx
    a = inner(κ*grad(u),grad(v))*dx

    M = assemble(mass_form)
    A = assemble(a)
    M = as_backend_type(M).mat()
    A = as_backend_type(A).mat()
    M = csr_matrix(M.getValuesCSR()[::-1],shape=M.size)
    A = csr_matrix(A.getValuesCSR()[::-1],shape=A.size)

    # extract the submatrices corresponding to the interior dofs
    M = M[interior_dofs,:][:,interior_dofs]
    A = A[interior_dofs,:][:,interior_dofs]

    # get the forcing cov matrix on the interior nodes of the grid
    Σ_int = kernMat(k_f,x_grid[interior_dofs],parallel,translation_inv)

    # form the matrix Q in the defintion of the approximate FEM cov mat
    # Note: overwrite Σ_int for memory efficiency
    Σ_int = M @ Σ_int @ M.T

    Σ_int = spsolve(A,Σ_int)
    Σ_int = spsolve(A,Σ_int.T).T

    # ensure Σ_int is symmetric
    Σ_int = 0.5*(Σ_int + Σ_int.T)

    # get big phi matrix on the sensor grid (only need the interior dofs)
    Phi = BigPhiMat(J,Y)[interior_dofs,:]

    # assemble the FEM cov mat on the sensor grid and ensure it is symmetric
    Σ_s = Phi.T @ Σ_int @ Phi
    Σ_s = 0.5*(Σ_s + Σ_s.T)

    # set up function to yield the vector (c(x,y)) for y in Y
    def Φ(x):
        cell_index = tree.compute_first_entity_collision(Point(*x))
        cell_global_dofs = V.dofmap().cell_dofs(cell_index)
        cell = Cell(mesh, cell_index)
        vertex_coordinates = cell.get_vertex_coordinates()
        cell_orientation = cell.orientation()
        data = V.element().evaluate_basis_all(x,vertex_coordinates,cell_orientation)
        col = np.zeros_like(cell_global_dofs)
        res = csr_matrix((data,(cell_global_dofs,col)),shape=(V.dim(),1))[interior_dofs,:]
        return res

    def c_fem(x):
        return Φ(x).T @ Σ_int @ Phi

    #return Σ_s and c_fem
    return Σ_s, c_fem
```

`fem_cov_assembler_post` takes in several arguments which are explained below:

- `J`: controls the FE mesh size ($h=1/J^2$)
- `k_f`: the covariance function for the forcing $f$
- `Y`: vector of sensor locations
- `parallel`: boolean argument indicating whether the computation of the forcing cov mat should be done in parallel
- `translation_inv`: boolean argument indicating whether the computation of the forcing cov mat should be computed assuming `k_f` is translation invariant or not

With all of this code in place we can now finally write the function `m_post_fem_assmebler` which will assemble the statFEM posterior mean function.

```
#export
def m_post_fem_assembler(J,f_bar,k_f,ϵ,Y,v_dat,par=False,trans=True):
    "Function to assemble the statFEM posterior mean function."

    # get number of sensors
    s = len(Y)

    # set up mesh and function space
    mesh = UnitSquareMesh(J,J)
    V = FunctionSpace(mesh,'Lagrange',1)

    # set up boundary condition
    def boundary(x, on_boundary):
        return on_boundary

    bc = DirichletBC(V, 0.0, boundary)

    # set up the functions κ and f
    κ = Constant(1.0)
    f = f_bar

    # set up the bilinear form for the variational problem
    u = TrialFunction(V)
    v = TestFunction(V)
    a = inner(κ*grad(u),grad(v))*dx

    # set up linear form
    L = f*v*dx

    # solve the variational problem
    μ_fem = Function(V)
    solve(a == L, μ_fem, bc)

    # use fem_cov_assembler_post to obtain cov mat on sensor grid and function to compute vector
    # (c(x,y)) for y in Y
    C_fem_s, c_fem = fem_cov_assembler_post(J,k_f,Y,parallel=par,translation_inv=trans)

    # form B_fem_s by adding noise contribution
    C_fem_s += (ϵ**2)*np.eye(s)

    # assemble function to compute posterior mean and return
    def m_post_fem(x):
        return m_post(x,μ_fem,c_fem,v_dat,Y,C_fem_s)

    return m_post_fem
```

`m_post_fem_assembler` takes in several arguments which are explained below:

- `J`: controls the FE mesh size ($h=1/J^{2}$)
- `f_bar`: the mean function for the forcing $f$
- `k_f`: the covariance function for the forcing $f$
- `ϵ`: controls the amount of sensor noise
- `Y`: vector of sensor locations
- `v_dat`: vector of noisy sensor observations
- `par`: boolean argument passed to `fem_cov_assembler_post`'s argument `parallel` (defaults to `False`)
- `trans`: boolean argument passed to `fem_cov_assembler_post`'s argument `translation_inv` (defaults to `True`)

Let's quickly check that this function is working.

```
J = 20
f_bar = Constant(1.0)
m_post_fem = m_post_fem_assembler(J,f_bar,k_f,ϵ,Y,v_dat)
# compute posterior mean at a location x in D
x = np.array([0.3,0.1])
m_post_fem(x)
```

## statFEM posterior covariance

The form of the statFEM posterior covariance remains the same as given in oneDim. Thus, we require very similar code as to the 1-D case. We start by creating a function `c_post` which evaluates the posterior covariance at a given point.

```
#export
def c_post(x,y,c,Y,B):
    "This function evaluates the posterior covariance at $(x,y)$"

    # compute vectors c_x and c_y:
    c_x = np.array([c(x,y_i) for y_i in Y])
    c_y = np.array([c(y_i,y) for y_i in Y])

    # compute update term
    update = c_x @ np.linalg.solve(B,c_y)

    # return c_post
    return (c(x,y) - update)
```

`c_post` takes in several arguments which are explained below:

- `x`,`y`: points to evaluate the covariance at
- `c`: function which returns the prior covariance at any given pair $(x,y)$
- `Y`: vector of sensor locations
- `B`: the matrix $\epsilon^{2}I+C_{Y}$ to be inverted in order to obtain the posterior

To compare the statFEM covariance matrices for finer and finer FE mesh sizes we will require some more code. First we create a function `post_fem_cov_assembler` which helps us to quickly assemble the statFEM posterior covariance matrix as explained in oneDim.

```
#export
def post_fem_cov_assembler(J,k_f,grid,Y,parallel,translation_inv):
    "Function which assembles the matrices $Σ_X$ ,$Σ_{XY}$, and $Σ_Y$ required for the statFEM posterior covariance."

    # set up mesh and function space
    mesh = UnitSquareMesh(J,J)
    V = FunctionSpace(mesh,'Lagrange',1)

    # set up grid
    x_grid = V.tabulate_dof_coordinates()

    # set up boundary condition
    def boundary(x, on_boundary):
        return on_boundary

    bc = DirichletBC(V, 0.0, boundary)

    # get the boundary and interior dofs
    bc_dofs = bc.get_boundary_values().keys()
    first, last = V.dofmap().ownership_range()
    all_dofs = range(last - first)
    interior_dofs = list(set(all_dofs) - set(bc_dofs))
    bc_dofs = list(set(bc_dofs))

    # set up the function κ
    κ = Constant(1.0)

    # get the mass and stiffness matrices
    u = TrialFunction(V)
    v = TestFunction(V)

    mass_form = u*v*dx
    a = inner(κ*grad(u),grad(v))*dx

    M = assemble(mass_form)
    A = assemble(a)
    M = as_backend_type(M).mat()
    A = as_backend_type(A).mat()
    M = csr_matrix(M.getValuesCSR()[::-1],shape=M.size)
    A = csr_matrix(A.getValuesCSR()[::-1],shape=A.size)

    # extract the submatrices corresponding to the interior dofs
    M = M[interior_dofs,:][:,interior_dofs]
    A = A[interior_dofs,:][:,interior_dofs]

    # get the forcing cov matrix on the interior nodes of the grid
    Σ_int = kernMat(k_f,x_grid[interior_dofs],parallel,translation_inv)

    # form the matrix Q in the defintion of the approximate FEM cov mat
    # Note: overwrite Σ_int for memory efficiency
    Σ_int = M @ Σ_int @ M.T

    Σ_int = spsolve(A,Σ_int)
    Σ_int = spsolve(A,Σ_int.T).T

    # ensure Σ_int is symmetric
    Σ_int = 0.5*(Σ_int + Σ_int.T)

    # get big phi matrix on the grid (only need the interior nodes)
    Phi_grid = BigPhiMat(J,grid)[interior_dofs,:]

    # get big phi matrix on the sensor grid (only need the interior nodes)
    Phi_Y = BigPhiMat(J,Y)[interior_dofs,:]

    # assemble the FEM cov mat on the sensor grid using Σ_int and Phi_Y
    Σ_Y = Phi_Y.T @ Σ_int @ Phi_Y

    # assemble the FEM cov mat on the grid using Σ_int and Phi_grid
    Σ_X = Phi_grid.T @ Σ_int @ Phi_grid

    # assemble cross term matrix (with ijth entry c(x_i,y_j))
    Σ_XY = Phi_grid.T @ Σ_int @ Phi_Y

    # return these sigma matrices
    return Σ_Y, Σ_X, Σ_XY
```

`post_fem_cov_assembler` takes in several arguments which are explained below:

- `J`: controls the FE mesh size ($h=1/J^2$)
- `k_f`: the covariance function for the forcing $f$
- `grid`: the fixed reference grid $\{x_{i}\}_{i=1}^{N}$ on which to assemble the posterior cov mat
- `Y`: vector of sensor locations.
- `parallel`: boolean argument indicating whether the computation of the forcing cov mat should be done in parallel
- `translation_inv`: boolean argument indicating whether the computation of the forcing cov mat should be computed assuming `k_f` is translation invariant or not

Finally, we create the function `c_post_fem_assembler` which assembles the statFEM posterior cov mat on the reference grid using the matrices `post_fem_cov_assembler` returns.

```
#export
def c_post_fem_assembler(J,k_f,grid,Y,ϵ,par,trans):
    "Function to assemble the statFEM posterior cov mat on a reference grid specified by grid."
    # use post_fem_cov_assembler to get the sigma matrices needed for posterior cov mat
    Σ_Y, Σ_X, Σ_XY = post_fem_cov_assembler(J,k_f,grid,Y,parallel=par,translation_inv=trans)

    # create the matrix B (store in Σ_Y for memory efficiency)
    s = len(Y) # number of sensor points
    Σ_Y += (ϵ**2)*np.eye(s)

    #form the posterior cov matrix (store in Σ_X for memory efficiency)
    Σ_X -= Σ_XY @ np.linalg.solve(Σ_Y,Σ_XY.T)

    return Σ_X
```

```
#hide
from nbdev.export import notebook2script; notebook2script()
```

    Converted 00_oneDim.ipynb.
    Converted 01_twoDim.ipynb.
    Converted index.ipynb.
    Converted oneDim_prior_results.ipynb.


# Abstraction

We have seen **duck typing** as a convient abstraction tool:
* As a **user**, we don't have to care about how a specific type, say an `Array`, is implemented. By *being an array* (i.e. a subtype of `AbstractArray`) it behaves like we expect and we can just use it.
* As a **developer**, as long as we make our objects behave like, say, and `AbstractArray`, we can implement it in whatever way we deem appropriate and it will work with all kinds of algorithms.

Building upon this principle, we also want to formulate our **algorithms** in an abstract way such that it works will all kinds of data types irrespective of their precise implementation (or even meaning). This is generally known as **generic programming**.

From [Wikipedia](https://en.wikipedia.org/wiki/Generic_programming):
> **Generic programming** is a style of computer programming in which algorithms are written in terms of types *to-be-specified-later* that are then *instantiated* when needed for specific types provided as parameters.

## Example: Vandermonde matrix

[Vandermonde matrix:](https://en.wikipedia.org/wiki/Vandermonde_matrix)
\begin{align}V=\begin{bmatrix}1&\alpha _{1}&\alpha _{1}^{2}&\dots &\alpha _{1}^{n-1}\\1&\alpha _{2}&\alpha _{2}^{2}&\dots &\alpha _{2}^{n-1}\\1&\alpha _{3}&\alpha _{3}^{2}&\dots &\alpha _{3}^{n-1}\\\vdots &\vdots &\vdots &\ddots &\vdots \\1&\alpha _{m}&\alpha _{m}^{2}&\dots &\alpha _{m}^{n-1}\end{bmatrix}\end{align}

```julia
function vander_naive(x::Vector)
    m = length(x)
    V = Matrix{Float64}(undef, m, m)
    for j = 1:m
        V[j,1] = 1.0
    end
    for i= 2:m
        for j = 1:m
            V[j,i] = x[j] * V[j,i-1]
            end
        end
    return V
end
```

```julia
vander_naive(rand(3))
```

```julia
vander_naive(1:3)
```

```julia
vander_naive([1,2,3])
```

Why is the result a matrix of floating point numbers....?

Even worse:

```julia
vander_naive(rand(ComplexF64, 3))
```

Our algorithm `vander_naive` is not generic! It assumes too much about the input type (it only takes `Vector`s) and also special cases `Float64`. Instead we should try to
* keep function argument types generic if possible
* avoid explicit typing (which is rarely necessary)

```julia
function vander_generic(x::AbstractVector{T}) where T
    m = length(x)
    V = Matrix{T}(undef, m, m)
    for j = 1:m
        V[j,1] = one(x[j])
    end
    for i= 2:m
        for j = 1:m
            V[j,i] = x[j] * V[j,i-1]
            end
        end
    return V
end
```

Not only does our code now work in the obvious cases above:

```julia
vander_generic(1:3)
```

```julia
vander_generic(rand(ComplexF64, 3))
```

It also extends to data types that we probably didn't even have in mind when writing down the algorithm:

```julia
vander_generic([3, "Stadt", 4 + 5im])
```

(Note that `*` is string concatenation and `one(String) == ""`, representing the identity element under multiplication.)

### And all of this genericity comes at no performance penality

```julia
using BenchmarkTools
```

```julia
x = rand(Float64, 10);
@btime vander_naive($x);
@btime vander_generic($x);
```

Actually, for this specific example **our generic code is faster** in a few cases inasmuch as type conversions are unnecessary.

```julia
x = rand(Bool, 10);
@btime vander_naive($x);
@btime vander_generic($x);
```

```julia
x = rand(Float32, 10);
@btime vander_naive($x);
@btime vander_generic($x);
```

# Generic Programming + Multiple Dispatch

The possibility to write generic algorithms that compile to fast machine code (a point we will investigate later) in combination with multiple dispatch leads to an ([unreasonable](https://www.youtube.com/watch?v=kc9HwsxE1OY)) amount of code reuse. This sharing of code comes in two forms:
1. **Sharing types**
 * Examples: DataStructures.jl, OrderedCollections.jl, StaticArrays.jl, Colors.jl, Measurements.jl
 * **You can define methods on types after the type is defined!**

2. **Sharing generic algorithms**
 * Examples: StatsBase.jl, SortingAlgorithms.jl, GenericLinearAlgebra.jl,
 * **Methods are selected based on all argument types**

As of the time of this writing, **~ 800 packages** depend on the data types provided in [DataStructures.jl](https://juliacollections.github.io/DataStructures.jl/latest/).

**~ 900 packages** reuse type implementations in [OrderedCollections.jl](https://github.com/JuliaCollections/OrderedCollections.jl).

**~ 550 packages** reuse [SortingAlgorithms.jl](https://github.com/JuliaCollections/SortingAlgorithms.jl).

**~ 300 packages** reuse [DiffRules.jl](https://github.com/JuliaDiff/DiffRules.jl).

**~ 50 packages** reuse [GenericSVD.jl](GenericSVD.jl).

(~ 5000 packages overall)

# Fancy example of generic programming

```julia
using Interact
```

```julia
@manipulate for n in 1:20
    [i*j for i in 1:n, j in 1:n]
end
```

```julia
# generic algorithm
function insert_block(A, i, j, blockvalue)
    B = copy(A)
    B[i:i+2, j:j+2] .= blockvalue
    return B
end
```

```julia
# generic algorithm
function insert_block(A::AbstractMatrix{T}, i::Integer, j::Integer, blockvalue::T) where T
    B = copy(A)
    B[i:i+2, j:j+2] .= blockvalue
    return B
end
```

```julia
A = fill(0, 9, 9)
```

```julia
insert_block(A, 3, 5, 1)
```

```julia
A = fill(0, 10, 10)
n = size(A, 1)

@manipulate for i in 1:n-2, j in 1:n-2
    insert_block(A, i, j, 1)
end
```

### Let's add some color!

Our function `insert_block` is generic. We don't care about the element type of `A`. Pretty much every type is fine!

```julia
using Colors
```

```julia
colors = distinguishable_colors(10)
```

```julia
basecolor = colors[1]
```

```julia
accentcolor = colors[4]
```

```julia
A = fill(basecolor, 10, 10)
```

```julia
A = fill(basecolor, 10, 10)
n = size(A, 1)

@manipulate for i in 1:n-2, j in 1:n-2
    insert_block(A, i, j, accentcolor)
end
```

## Emergent features: Example DifferentiaEquations.jl

$$\frac{du(t)}{dt} = -cu(t)$$

```julia
using OrdinaryDiffEq

#Half-life of Carbon-14 is 5730 years.
c = 5.730

#Setup
u0 = 1.0
tspan = (0.0, 1.0)

#Define the problem
radioactivedecay(u,p,t) = -c*u

#Pass to solver
prob = ODEProblem(radioactivedecay,u0,tspan)
```

```julia
sol = solve(prob, Tsit5(), reltol=1e-8, abstol=1e-8);
```

```julia
using Plots

plot(sol.t, sol.u, ylabel="u(t)", xlabel="t", lw=2, legend=false)
```

### Arbitrary precision

```julia
using OrdinaryDiffEq, Measurements, Plots

#Half-life of Carbon-14 is 5730 years.
c = big(5.730) # now a BigFloat

#Setup
u0 = big(1.0) # now a BigFloat
tspan = (0.0, 1.0)

#Define the problem
radioactivedecay(u,p,t) = -c*u

#Pass to solver
prob = ODEProblem(radioactivedecay,u0,tspan)
sol = solve(prob, Tsit5(), reltol=1e-8, abstol=1e-8);

plot(sol.t, sol.u, ylabel="u(t)", xlabel="t", lw=2, legend=false)
```

```julia
sol.u
```

### With uncertainties: Measurements.jl

`Measurement` type from [Measurements.jl]() and differential equation solver from [OrdinaryDiffEq.jl](https://github.com/JuliaDiffEq/OrdinaryDiffEq.jl) (i.e. [DifferentialEquations.jl](https://github.com/JuliaDiffEq/DifferentialEquations.jl))

```julia
using Measurements
```

```julia
1 ± 0.1
```

```julia
(1 ± 0.1) * (2 ± 0.05)
```

```julia
using OrdinaryDiffEq, Measurements, Plots

#Half-life of Carbon-14 is 5730 years.
c = 5.730 ± 2

#Setup
u0 = 1.0 ± 0.1
tspan = (0.0, 1.0)

#Define the problem
radioactivedecay(u,p,t) = -c*u

#Pass to solver
prob = ODEProblem(radioactivedecay,u0,tspan)
sol = solve(prob, Tsit5(), reltol=1e-8, abstol=1e-8);

plot(sol.t, sol.u, ylabel="u(t)", xlabel="t", lw=2, legend=false)
```

Note that, in some sense, **Julia implemented that feature by itself**.

The authors of Measurements.jl and DifferentialEquations.jl never had any collabration on this.

It **just works**.

## Symbolic computations

Let's compute the Vandermonde matrix symbolically.

\begin{align}V=\begin{bmatrix}1&\alpha _{1}&\alpha _{1}^{2}&\dots &\alpha _{1}^{n-1}\\1&\alpha _{2}&\alpha _{2}^{2}&\dots &\alpha _{2}^{n-1}\\1&\alpha _{3}&\alpha _{3}^{2}&\dots &\alpha _{3}^{n-1}\\\vdots &\vdots &\vdots &\ddots &\vdots \\1&\alpha _{m}&\alpha _{m}^{2}&\dots &\alpha _{m}^{n-1}\end{bmatrix}\end{align}

```julia
using SymPy
```

```julia
@vars a b c d e
```

```julia
sin(a)sin(b)+cos(a)*cos(b)
```

```julia
simplify(ans)
```

```julia
vander_generic([a,b,c,d,e])
```

(See [Symbolics.jl](https://github.com/JuliaSymbolics/Symbolics.jl) for a pure-Julia SymPy replacement.)

# Core messages of this Notebook

* It is simple to write **type-generic code** in Julia and you should do it.
* Generally, **generic code is just as fast as specific code**.
* Generic Programming + Multiple Dispatch = **lots of code sharing and emergent features**


# Differential Drive Robotics Platform

## Problem Setup

A common robot platform is that of the [differential drive](http://en.wikipedia.org/wiki/Differential_wheeled_robot). The robot is propelled using two wheels, each with their own motor. Usually a caster is used as a third wheel to provide stability. By varying the speeds of the two motors, the robot can change direction.

Differential drive platforms are highly maneuverable, as they have a very small turn radius. They are also extremely simple to design. In this example, the dynamics of a differential drive platform will be explored. The resulting model is highly accurate, but makes a few assumptions:

**Assumptions**

- Wheels roll without slipping
- Body is symmetric

**The setup is as follows:**

```python
from IPython.display import SVG
SVG(filename='Robot.svg')
```

## Equations of Motion

We’ll start by generating the equations of motion for the robot with `sympy.physics.mechanics`. The `mechanics` module provides a clean, intuitive workflow for deriving the equations of motion for multibody dynamic systems using either `KanesMethod` or `LagrangesMethod` objects. In this example we'll make use of the `LagrangesMethod` object, which generates the equations of motion using [Lagrangian Mechanics](http://en.wikipedia.org/wiki/Lagrangian_mechanics) with constraints and generalized forces.

First we import the necessary functionality from SymPy.

```python
from sympy.physics.mechanics import *
from sympy import sin, cos, symbols, Matrix, solve
```

```python
init_vprinting()
```

Next we need to define an inertial reference frame, as well as a coordinate origin. All coordinates and reference frames defined later will be in reference to the world coordinate frame.

```python
# Inertial Reference Frame
N = ReferenceFrame('N')

# Define a world coordinate origin
O = Point('O')
O.set_vel(N, 0)
```

Lagrange's Method requires a state vector $\vec{q}$ that contains all the generalized coordinates required to define the system in space. In the case of the robot:

$$
\vec{q} = [\matrix{x && y && \theta && \phi_1 && \phi_2}]^T
$$

As these are all time varying variables, they need to be created as `dynamicsymbols`:

```python
theta = dynamicsymbols('theta')                      # Rotation about N.z
x, y = dynamicsymbols('x, y')                        # Coordinates of robot in World Frame
phi1, phi2 = dynamicsymbols('phi_1, phi_2')          # Angular displacement of wheel

# Create q and dq vectors
q = Matrix([x, y, theta, phi1, phi2])
dq = q.diff()
```

We'll also define some constant parameters that define the specific robot:

```python
# Constants for the wheels
r = symbols('r')                                     # Radius of wheel
m_w = symbols('m_w')                                 # Mass of the wheel
t_w = symbols('t_w')                                 # Thickness of the wheel

# Constants for the Robot Body
w = symbols('w')                                     # 2*w is the width of the wheel base
d = symbols('d')                                     # Distance between axel and center of mass
m_b = symbols('m_b')                                 # Mass of the body
Ixx, Iyy, Izz = symbols('Ixx, Iyy, Izz')             # Moments of inertia of body
```

Next we need to define a reference frame relative to the robot, and a point located at the center of the wheel axis. This will define the robot's location in the world coordinate frame. The velocity of this point is just the derivative of its x and y coordinates.

```python
# Robot Reference Frame
R = N.orientnew('R', 'Axis', [theta, N.z])

# Center of wheel base
Cw = O.locatenew('Cw', x*N.x + y*N.y)

# Set the velocity of point Cw
Cw.set_vel(N, x.diff()*N.x + y.diff()*N.y)
```

Coordinates of the 2 wheel hubs then need to be specified. To do this we can use the `locatenew` method. The hubs 1 and 2 are located at $-w\vec{R_y}$ and $w\vec{R_y}$ away from the center of the wheel base $C_w$ respectively.

The velocity of the hubs is then found using the `v2pt_theory` method, which calculates the velocity of a point based off the velocity of another point in the same frame, and the angular velocity of that frame ([reference](http://en.wikipedia.org/wiki/Rotating_reference_frame#Relation_between_velocities_in_the_two_frames)).

```python
# Points at wheel hubs
H1 = Cw.locatenew('H1', -w*R.y)
H2 = Cw.locatenew('H2', w*R.y)

# Set the velocity of points H1 and H2
H1.v2pt_theory(Cw, N, R)
H2.v2pt_theory(Cw, N, R);
```

Rotating reference frames fixed to each wheel are then created. These rotate about the $\vec{R_y}$ axis with the angular position of the wheels, $\phi_1$ and $\phi_2$.

```python
# Create reference frames for wheels 1 and 2
W1 = R.orientnew('W1', 'Axis', [phi1, R.y])
W2 = R.orientnew('W2', 'Axis', [phi2, R.y])
```

`sympy.physics.mechanics` provides classes for rigid bodies. These make calculating the lagrangian of a multibody system easier, as each body can be handled separately. To do this, each body needs the following things specified:

- A name
- A center of mass
- A reference frame
- A mass
- A tuple of (Inertia, Point of rotation)

Modeling the wheels as solid cylinders, the inertia can be calculated about the hubs, and `RigidBody` objects can be created for each wheel.

```python
# Calculate inertia of the wheel
Iw = inertia(R, m_w*(3*r**2 + t_w**2)/12, m_w*r**2/2, m_w*(3*r**2 + t_w**2)/12)

# Create rigid bodies for wheels
Wheel1 = RigidBody('Wheel1', H1, W1, m_w, (Iw, H1))
Wheel2 = RigidBody('Wheel2', H2, W2, m_w, (Iw, H2))
```

The same can be done for the body of the robot. In this case we model the body as a symmetric object with inertias $I_{xx}$, $I_{yy}$, and $I_{zz}$. The center of mass of the body is located a distance $d$ in front of the wheel axel.

```python
# Calculate inertia of body
Ib = inertia(R, Ixx, Iyy, Izz)

# Center of mass of body
Cm = Cw.locatenew('Cm', d*R.x)
Cm.v2pt_theory(Cw, N, R)

# Create a rigid body object for body
Body = RigidBody('Body', Cm, R, m_b, (Ib, Cm))
```

## Nonholonomic Constraints
At this point, the Lagrangian of the system can be calculated. However, the system still isn't fully defined; the constraints still need to be specificed.

At any instance in time, the velocity of the ground contact points must have a velocity of zero. To specify this in the Lagrange equations, a constraint matrix needs to be created.

First, two points are created where the wheels contact the ground, and their velocities calculated.

```python
# Create two points, where the wheels contact the ground
C1 = H1.locatenew('C1', -r*R.z)
C2 = H2.locatenew('C2', -r*R.z)
# Calculate velocity of points
C1.v2pt_theory(H1, N, W1)
C2.v2pt_theory(H2, N, W2);
```

Using these velocities, a system of equations can be created. As the velocities must be zero, this expression = 0.

```python
# Express the velocity of points in the inertial frame
con1 = C1.vel(N).express(N).args[0][0]
con2 = C2.vel(N).express(N).args[0][0]
# Create a matrix of constraints
constraints = con1.col_join(con2)
constraints
```

Since there are 6 equations with 3 unknowns ($\dot{x}$, $\dot{y}$, and $\dot{\theta}$), there are duplicate equations. Using solve, and reconfiguring the expression, this can be reduced to the required set:

```python
# Solve for dx, dy, and dtheta in terms of dphi1 and dphi2
sol = solve(constraints, dq[:3])

# Split the resulting dict into a rhs and lhs, that are equivalent
sol_rhs = Matrix(list(sol.values()))
sol_lhs = Matrix(list(sol.keys()))

# Since sol_rhs = sol_lhs --> sol_rhs - sol_lhs = 0
# This forms the basis of our constraint matrix.
# Combining, and solving for a linear representation:
c = (sol_rhs - sol_lhs).jacobian(dq[:5])
c
```

This is our constraint matrix. The `LagrangesMethod` class requires that the constraints be specified as

`coneqs` = $ C\dot{\vec{q}} = 0 $

Thus:

```python
# Constraint Equations
coneqs = (c*dq)
coneqs
```

## Generalized Forces

The inputs of system are the torques at the wheels. Using the right-hand-rule, it can be seen that a positive torque will exert force on the system in the $R_x$ direction. With a wheel radius of $r$, the force on the wheel hubs is:

$\vec{F_{hub}} = \tau*\vec{R_x}$

```python
# Define forces on system:
T1, T2 = symbols('tau_1, tau_2')              # Torques from the wheels
fl = [(H1, r*T1*R.x),
      (H2, r*T2*R.x)]
```

## Solving the System Dynamics

We are now ready to solve the for the equations of motion. First, calculate the Lagrangian of the system $L = T - V$:

```python
Lag = Lagrangian(N, Wheel1, Wheel2, Body)
```

Next, a `LagrangesMethod` object is created. This takes the following parameters:

- The system Lagrangian
- The generalized coordinate vector $\vec{q}$
- The constraint equations
- A list force tuples: (Point, Force at point)
- An inertial reference frame

```python
lm = LagrangesMethod(Lag, q, nonhol_coneqs=coneqs, forcelist=fl, frame=N)
```

The equations of motion can then be found:

```python
le = lm.form_lagranges_equations()
le
```

## Simulation

To gain insight into the system dynamics, numerical simulation will be used. First, we need to import a few more functions:

```python
from sympy import lambdify, Dummy
from sympy.physics.mechanics import msubs
from scipy import array, hstack, linspace, ones
from scipy import random, interp, vstack
from scipy.integrate import odeint
import matplotlib.pyplot as plt
%matplotlib inline
```

The vector $\ddot{\vec{q}}$ can be calculated in a closed form expression using the `rhs` method. This method takes quite a long time to solve entirely symbolically (it took over an hour on my machine), so a simplifying assumption will be made: $d = 0$. This is actually pretty reasonable, as the motors and battery will be the heaviest part, and should be situated around the axle.

```python
# Solve for the rhs:
rhs = lm.rhs()

# Substitute in d = 0
rhs = msubs(rhs, {d: 0})
```

We're almost done, all we need to do is model the dynamics of the motor. In general, we can model the current dynamics of the motor as follows:

$$
\dot{i} = -K/L \dot{\phi} - R/L i + V/L
$$

$$
\tau = K i
$$

where $i$ and $V$ are the current and voltage through the motor, $K$ is the motor constant, $R$ is the coil resistance, and $L$ is the coil inductance. Creating this model in sympy is fairly straightforward:

```python
# Create dynamic symbols for current and voltage
i_1, i_2 = dynamicsymbols('i_1, i_2')        # Currents through motor 1 and 2
V_1, V_2 = symbols('V_1, V_2')               # Voltages across the motor terminals

# Define some motor constants.
# Assuming motor 1 and 2 are the same:
R = symbols('R')                             # Coil resistance
L = symbols('L')                             # Coil inductance
K1, K2 = symbols('K1, K2')                   # Motor constant

# Define the motor dynamics
di = Matrix([[-K1/L*phi1.diff() - R/L*i_1 + V_1/L],
             [-K2/L*phi2.diff() - R/L*i_2 + V_2/L]])
```

Now we just need to combine the motor model with the model for the robot:

```python
# Define consts:
params = [Izz,  t_w,  m_w,    r,  m_b,   w,    R,      L,   K1,  K2]
values = [  5, 0.15,  2.0, 0.15, 50.0, 0.6, 0.05, 0.0001,  1.0, 1.0]       # Values of constants

# Create a list of dynamic symbols for simulation
dynamics = q.T.tolist()[0] + dq.T.tolist()[0] + lm.lam_vec.T.tolist()[0] + [i_1, i_2]

# Set the inputs to be the motor voltages
inputs = [V_1, V_2]

# Add the motor model to the rhs equations
aug_rhs = rhs.col_join(di)

# Substitute in K*i for T in the rhs equations
aug_rhs = aug_rhs.subs({T1: K1*i_1, T2: K2*i_2})
```

Currently the codegen functionality provided by `PyDy` doesn't support equations with Lagrange Multipliers (this is planned for future releases). For now `lambdify` can be used to convert the expression into a function to solve the rhs based on the current state and inputs.

```python
# Create a list of dynamic symbols for simulation
dummys = [Dummy() for i in dynamics]
dummydict = dict(zip(dynamics, dummys))
# Sub in the dummy symbols
rhs_dummy = msubs(aug_rhs, dummydict)
# Lambdify function
rhs_func = lambdify(dummys + inputs + params, rhs_dummy, modules='numpy')
```

```python
# Create a function in the required format for odeint
def right_hand_side(x, t, ts, us, values):
    """Calculate the rhs of the integral, at
    time t, state x.
    ts, us are used to get the current input
    values are constants in the integral"""

    # Interp is needed to get u for timestep
    u1 = interp(t, ts, us[:,0])
    u2 = interp(t, ts, us[:,1])

    arguments = hstack((x, u1, u2, values))

    # Need call to array and reshape, as odeint
    # requires state vector to be 1d
    dx = array(rhs_func(*arguments))
    return dx.reshape(dx.size)
```

Now we have everything we need to simulate the motion. First, let's see what happens if we input a constant, unbalanced voltage signal. Motor 1 gets 24 volts, Motor 2 gets 20 volts. Intuitively, this should cause the robot to drive in a circle.

```python
ts = linspace(0, 30, 3000)
us = vstack((24*ones(len(ts)), 20*ones(len(ts)))).T
# Run the simulation
x0 = [0,] * 15                         # Start out at origin, unmoving
xs = odeint(right_hand_side, x0, ts, args=(ts, us, values))

plt.plot(xs[:,5], xs[:,6])
plt.title('Robot Position')
plt.xlabel('x (m)')
plt.ylabel('y (m)');
```

As seen by the simulation, the results line up with our intuition. The reason it doesn't immediately start out in a circle is due to the time it takes for it to accelerate from rest.

If we were designing such a platform, we might want to see the effects different design parameters have on the motion. Because the solution is symbolic, it's easy to experiment. Here we'll see what having a heavier body and larger inertia do to the motion:

```python
ts = linspace(0, 30, 1000)
us = vstack((24*ones(len(ts)), 20*ones(len(ts)))).T
# Body is twice as heavy, with twice as much rotational inertia
values = [  10, 0.15,  2.0, 0.15, 100.0, 0.6, 0.05, 0.0001,  1.0, 1.0]       # Values of constants
# Run the simulation
x0 = [0,] * 15                         # Start out at origin, unmoving
xs = odeint(right_hand_side, x0, ts, args=(ts, us, values))
plt.plot(xs[:,0], xs[:,1])
plt.title('Robot Position')
plt.xlabel('x (m)')
plt.ylabel('y (m)');
```


# Day4_Eigenvalues and SVD

```python
import scipy
from scipy import linalg, matrix
import numpy as np
```

```python
from sympy.solvers import solve
from sympy import Symbol
```

```python
import matplotlib.pyplot as plt
%matplotlib inline
```

## Equation Solver

```python
x = Symbol('x')
solve(x**2 - 1, x)
```

    [-1, 1]

## Eigenvalues with Equation

```python
A = matrix([[1, 2], [3, -4]])
A
```

    matrix([[ 1,  2],
            [ 3, -4]])

```python
lam = Symbol('lam')
```

```python
A_lam = A - lam*np.asmatrix(np.identity(2))
A_lam
```

    matrix([[-1.0*lam + 1, 2],
            [3, -1.0*lam - 4]], dtype=object)

```python
equation = A_lam[0,0]*A_lam[1,1] - A_lam[0,1]*A_lam[1,0]
equation
```

    (-1.0*lam - 4)*(-1.0*lam + 1) - 6

```python
solve(equation, lam)
```

    [-5.00000000000000, 2.00000000000000]

## Eigenvalues and Eigenvectors with Package

```python
eigenvalue, eigenvector = linalg.eig(A)
```

```python
eigenvalue
```

    array([ 2.+0.j, -5.+0.j])

```python
eigenvector
```

    array([[ 0.89442719, -0.31622777],
           [ 0.4472136 ,  0.9486833 ]])

## Eigen Value Decomposition

```python
eigenvalue, eigenvector = linalg.eig(A)
```

```python
eigenvalue.shape[0]
```

    2

```python
L = np.identity(eigenvalue.shape[0])
for i in range(eigenvalue.shape[0]) :
    L[i, i] = eigenvalue[i]
L
```

    C:\Users\slcf\Anaconda3\lib\site-packages\ipykernel_launcher.py:3: ComplexWarning: Casting complex values to real discards the imaginary part
      This is separate from the ipykernel package so we can avoid doing imports until

    array([[ 2.,  0.],
           [ 0., -5.]])

```python
S= np.asmatrix(eigenvector)
S
```

    matrix([[ 0.89442719, -0.31622777],
            [ 0.4472136 ,  0.9486833 ]])

```python
A*S
```

    matrix([[ 1.78885438,  1.58113883],
            [ 0.89442719, -4.74341649]])

```python
S*L
```

    matrix([[ 1.78885438,  1.58113883],
            [ 0.89442719, -4.74341649]])

```python
A*S==S*L
```

    matrix([[ True, False],
            [False,  True]], dtype=bool)

```python
np.allclose(A*S, S*L)
```

    True

## SVD

```python
A = matrix([[3, 1, 1], [-1, 3, 1]])
A
```

    matrix([[ 3,  1,  1],
            [-1,  3,  1]])

```python
U, s, V = linalg.svd(A, full_matrices=True)
```

```python
U = np.asmatrix(U)
U
```

    matrix([[-0.70710678, -0.70710678],
            [-0.70710678,  0.70710678]])

```python
s = np.asmatrix(s)
s
```

    matrix([[ 3.46410162,  3.16227766]])

```python
V = np.asmatrix(V)
V
```

    matrix([[ -4.08248290e-01,  -8.16496581e-01,  -4.08248290e-01],
            [ -8.94427191e-01,   4.47213595e-01,   5.27355937e-16],
            [ -1.82574186e-01,  -3.65148372e-01,   9.12870929e-01]])

```python
list(A.shape)
```

    [2, 3]

```python
np.min(list(A.shape))
```

    2

```python
S = np.zeros((A.shape))
for i in range(np.min(list(A.shape))) :
    S[i, i] = s[0,i]
S
```

    array([[ 3.46410162,  0.        ,  0.        ],
           [ 0.        ,  3.16227766,  0.        ]])

```python
U*S*V
```

    matrix([[ 3.,  1.,  1.],
            [-1.,  3.,  1.]])

## Image Compression with SVD

https://github.com/rameshputalapattu/jupyterexplore/blob/master/jupyter_interactive_environment_exploration.ipynb

```python
import matplotlib.image as mpimg

img = mpimg.imread('sample.png')
```

```python
plt.imshow(img)
```

```python
from skimage.color import rgb2gray
from skimage import img_as_ubyte, img_as_float

gray_images = {
    "Pierrot":rgb2gray(img_as_float(img))
}
```

```python
def compress_svd(image, k):
    U, s, V = linalg.svd(image,full_matrices=False)
    reconst_matrix = np.dot(U[:,:k],np.dot(np.diag(s[:k]),V[:k,:]))

    return reconst_matrix, s
```

```python
reconst_matrix, s = compress_svd(rgb2gray(img_as_float(img)),50)
```

```python
s[:5]
```

    array([ 266.53154921,   68.28481384,   57.90747286,   39.15993506,
             36.53256079])

```python
plt.plot(s[:5])
```

```python
def compress_show_gray_images(img_name,k):

    image=gray_images[img_name]

    original_shape = image.shape
    reconst_img,s = compress_svd(image, k)

    fig,axes = plt.subplots(1,2,figsize=(8,5))

    axes[0].plot(s)

    compression_ratio =100.0* (k*(original_shape[0] + original_shape[1])+k)/(original_shape[0]*original_shape[1])

    axes[1].set_title("compression ratio={:.2f}".format(compression_ratio)+"%")
    axes[1].imshow(reconst_img,cmap='gray')
    axes[1].axis('off')

    fig.tight_layout()
```

```python
from ipywidgets import interact,interactive,interact_manual
interact(compress_show_gray_images,img_name=list(gray_images.keys()),k=(1,100));
```

    A Jupyter Widget

## Applications

Data projection - pca

Data quantization - spectral clustering methods

Feature selection - apply svd keep high singular value dimensions

https://github.com/bwcho75/dataanalyticsandML/blob/master/Clustering/1.%20KMeans%20clustering-IRIS%202%20feature.ipynb

@ Iris Data

Data Set Information:

This is perhaps the best known database to be found in the pattern recognition literature. Fisher's paper is a classic in the field and is referenced frequently to this day. (See Duda & Hart, for example.) The data set contains 3 classes of 50 instances each, where each class refers to a type of iris plant. One class is linearly separable from the other 2; the latter are NOT linearly separable from each other.

Predicted attribute: class of iris plant.

This is an exceedingly simple domain.

This data differs from the data presented in Fishers article (identified by Steve Chadwick, spchadwick '@' espeedaz.net ). The 35th sample should be: 4.9,3.1,1.5,0.2,"Iris-setosa" where the error is in the fourth feature. The 38th sample: 4.9,3.6,1.4,0.1,"Iris-setosa" where the errors are in the second and third features.

Attribute Information:

sepal length in cm
sepal width in cm
petal length in cm
petal width in cm
class: -- Iris Setosa -- Iris Versicolour -- Iris Virginica

```python
from sklearn import datasets
import pandas as pd
iris = datasets.load_iris()

labels = pd.DataFrame(iris.target)
labels.columns=['labels']

data = pd.DataFrame(iris.data)
data.columns=['Sepal length','Sepal width','Petal length','Petal width']

data = pd.concat([data,labels],axis=1)

data.head()
```

      Sepal length
      Sepal width
      Petal length
      Petal width
      labels

      0
      5.1
      3.5
      1.4
      0.2
      0

      1
      4.9
      3.0
      1.4
      0.2
      0

      2
      4.7
      3.2
      1.3
      0.2
      0

      3
      4.6
      3.1
      1.5
      0.2
      0

      4
      5.0
      3.6
      1.4
      0.2
      0

```python
feature = data[['Sepal length','Sepal width']]
feature.head()
```

      Sepal length
      Sepal width

      0
      5.1
      3.5

      1
      4.9
      3.0

      2
      4.7
      3.2

      3
      4.6
      3.1

      4
      5.0
      3.6

```python
from sklearn.cluster import SpectralClustering
import matplotlib.pyplot  as plt
import seaborn as sns

model = SpectralClustering(n_clusters=3)
model.fit(feature)

predict = pd.DataFrame(model.fit_predict(feature))
predict.columns=['predict']

r = pd.concat([feature,predict],axis=1)
```

```python
plt.scatter(r['Sepal length'],r['Sepal width'],c=data['labels'],alpha=0.7)
plt.title("Real")
plt.show()
```

```python
plt.scatter(r['Sepal length'],r['Sepal width'],c=r['predict'],alpha=0.7)
plt.title("Predict")
plt.show()
```


# 数式微分のPython実装：sympy

数式自分を行うpythonモジュールにsympyがある．
これを使って数式微分を行う．

```python
# sympyのインポート
import sympy as sp
sp.init_printing() # LaTeX形式で表示する

from sympy import diff, integrate, Derivative
```

## sympyでの変数・関数と表示

```python
# sympyで用いる変数をsymbols()で宣言する
x, m, sigma = sp.symbols('x m sigma')
```

```python
# 宣言した変数で数式を作る
gauss = 1 / sp.sqrt(2 * sp.pi * sigma**2) * sp.exp( -(x - m)**2 / (2 * sigma**2))
print(gauss)
print(type(gauss))
```

    sqrt(2)*exp(-(-m + x)**2/(2*sigma**2))/(2*sqrt(pi)*sqrt(sigma**2))
    <class 'sympy.core.mul.Mul'>

```python
gauss # そのまま評価すると，init_printing()のあとならLaTeX形式で表示される
```

```python
# 注意：すべての関数・変数はsympyのオブジェクトでなければならない

print("sympy: ", type(sp.exp)) # ok, sympy version
print(sp.exp(x)) # ok

import numpy as np
print("numpy: ", type(np.exp)) # no, this is numpy version
# print(np.exp(x)) # doesn't work !

import math
print("builtin: ", type(math.exp)) # no, this is built-in
# print(math.exp(x)) # doesn't work !
```

    sympy:  <class 'sympy.core.function.FunctionClass'>
    exp(x)
    numpy:  <class 'numpy.ufunc'>
    builtin:  <class 'builtin_function_or_method'>

```python
# 補足：init_printing()を使わずにLaTeX形式で出力する方法

from sympy import latex

# let's make the latex string
gauss_str = latex(gauss, mode='inline')
print(gauss_str)

# use this Latex() for displaying the latex string
from IPython.display import Latex
Latex(gauss_str)
```

    $\frac{\sqrt{2} e^{- \frac{\left(- m + x\right)^{2}}{2 \sigma^{2}}}}{2 \sqrt{\pi} \sqrt{\sigma^{2}}}$

$\frac{\sqrt{2} e^{- \frac{\left(- m + x\right)^{2}}{2 \sigma^{2}}}}{2 \sqrt{\pi} \sqrt{\sigma^{2}}}$

# 数式の微積分

以下は数式の微分と積分を計算する簡単な例である．

## 微分

微分はsympyオブジェクトの関数のメソッド`diff`を用いる．
微分したい変数を引数に与える．

### 2次関数

```python
x, y, a, b, c, d, e = sp.symbols('x y a b c d e')
```

```python
f = a * x**2 + b * x + c
```

```python
f
```

```python
f.diff(x) # 1階微分
```

```python
f.diff(x).diff(x) # 2階微分
```

```python
g = a * x**2 + b * x * y + c * x + d * y + e
```

```python
g
```

```python
g.diff(x) # 1階微分
```

```python
g.diff(x).diff(y) # 2階微分
```

### ガウス関数

```python
x, m, sigma = sp.symbols('x m sigma')
```

```python
# gauss関数をxで微分する
gauss.diff(x)
```

```python
# 再度xで微分する（2階微分）
gauss.diff(x).diff(x)
```

```python
# sigmaで微分することも可能
gauss.diff(sigma)
```

## 積分

### 2次関数の積分

```python
x, y, a, b, c, d, e = sp.symbols('x y a b c d e')
```

```python
f = a * x**2 + b * x + c
```

```python
f, f.integrate(x), f.integrate(x).integrate(x)
```

```python
g = a * x**2 + b * x * y + c * x + d * y + e
```

```python
g, g.integrate(x), g.integrate(x).integrate(y)
```

### ガウス関数の積分

```python
x, m, sigma = sp.symbols('x m sigma')
```

```python
# gaussのxによる不定積分
integrate(gauss, x)
```

```python
# aからｂまでの定積分
a, b = sp.symbols('a b')
gauss.integrate((x, a, b))
```

# 行列計算

## 固有値計算の例

```python
# ランダムな整数行列を生成
a = np.random.randint(1, 10, size=(2,2))
a # numpyオブジェクト
```

    array([[1, 4],
           [8, 8]])

```python
# numpyを用いて数値的に固有値を計算
np.linalg.eigvals(a)
```

    array([-2.15206735, 11.15206735])

```python
# いちどlistにしてからsympyのMatrixオブジェクトへ変換する
a = sp.Matrix(a.tolist())
a # sympyオブジェクト
```

```python
# sympyを用いて解析的に固有値を計算
a.eigenvals()
```

```python
# 実数値行列でもできないことはない
b = sp.Matrix(np.random.randn(2,2))
b.eigenvals()
```

## 行列演算

```python
a + a # 行列の和は+でよい
```

```python
a * a # 行列の積は*でよい（要素ごとの積ではない）
```

```python
a, a.transpose()
```

## ヤコビ行列

$\boldsymbol{f} = R(\theta_1) R(\theta_2) \boldsymbol{x}$を考える．

ここで
$R(\theta) = \begin{pmatrix}
	\sin\theta & \cos\theta \\
	-\cos\theta & \sin\theta
	\end{pmatrix}
$

```python
theta1, theta2, x1, x2 = sp.symbols('theta1 theta2 x1 x2')

R1 = sp.Matrix([[ sp.sin(theta1), sp.cos(theta1)],
                 [-sp.cos(theta1), sp.sin(theta1)]])
R2 = sp.Matrix([[ sp.sin(theta2), sp.cos(theta2)],
                 [-sp.cos(theta2), sp.sin(theta2)]])

f = R1 * R2 * sp.Matrix([[x1], [x2]])
```

```python
f # 関数fの評価
```

```python
f.diff(theta1) # theta1による偏微分
```

```python
f.diff(theta2) # theta2による偏微分
```

ヤコビ行列を計算するには，ベクトル値関数`f`のメソッド`jacobian`に，
微分する変数のリストを与える．

```python
f.jacobian([theta1, theta2])
```

# Task

## 座標変換のヤコビ行列

ある座標系における点$\boldsymbol{x}$を
$
\boldsymbol{y} = R_1 \boldsymbol{x} + \boldsymbol{t}_1
$
と座標変換し，点$\boldsymbol{y}$とする．
これをさらに
$
\boldsymbol{z} = R_2 \boldsymbol{y} + \boldsymbol{t}_2
$
と座標変換して点$\boldsymbol{z}$とする．
ここで$R_i$は回転行列
$$
R_i = \begin{pmatrix}
\sin\theta_i & \cos\theta_i \\
-\cos\theta_i & \sin\theta_i
\end{pmatrix}
$$
である．

sympyでヤコビ行列
$\dfrac{\partial \boldsymbol{z}}{\partial \boldsymbol{\theta}}$
を求めよ．

## 交差エントロピーの微分

$N=5$とする．
`a1, ..., a5`と`y1, ..., y5`
をsympy変数として定義する．
以下の関数を定義する．

$f(\boldsymbol{a}) = -\sum_{i=1}^{N} y_i \log f_i(\boldsymbol{a})$

ただし

$f_i(\boldsymbol{a}) = \operatorname{softmax}_i(\boldsymbol{a}) = \dfrac{e^{a_i}}{\sum_{j=1}^N e^{a_j}}$

sympyで勾配ベクトル
$\dfrac{\partial f}{\partial \boldsymbol{a}}$
を求めよ．

```python

```


# Poisson-Zero Hidden Markov Model

## Summary

The following exposition uses [`pymc3-hmm`](https://github.com/AmpersandTV/pymc3-hmm) to both simulate and estimate a hidden Markov model (HMM) with emissions consisting of a Poisson random variable and the point at zero.

```python
import numpy as np
import pandas as pd

import theano
import theano.tensor as tt

import pymc3 as pm

from pymc3_hmm.utils import compute_steady_state
from pymc3_hmm.distributions import Constant, SwitchingProcess, DiscreteMarkovChain
from pymc3_hmm.step_methods import FFBSStep, TransMatConjugateStep
```

## Introduction

Our observation model can be described as follows:

\begin{align}
  \label{eq:pois-zero-model}
  Y_t &= S_t \epsilon_t,\quad
  \epsilon_t \sim \operatorname{Pois}\left( \mu \right)
  \\
  S_t &\sim \operatorname{Bern}\left( \pi_t \right)
  \;,
\end{align}
where $y_t \sim Y_t$ are the observed values sampled from the observation distribution, $Y_t$, spanning $t \in \left\{0, \dots, T \right\}$.

The "hidden" state sequence, $\{S_t\}$, is driven by the following Markov relationship:

\begin{equation*}
    \operatorname{P}\left(S_t \mid S_{t-1}\right) \triangleq \Gamma_{t,t-1} \in \mathbb{R}^{2 \times 2}_{[0, 1]}
\end{equation*}

The marginal state probability, $\pi_t$, is then given by

\begin{equation*}
  \begin{aligned}
    \operatorname{P}\left( S_t \right)
    &= \int_{S_{t-1}} \operatorname{P}\left(S_t \mid S_{t-1}\right)
    \operatorname{dP}\left(S_{t-1}\right)
    \\
    &=
    \begin{pmatrix}
      \Gamma^{(0, 0)}_{t,t-1} & \Gamma^{(0, 1)}_{t,t-1}
      \\
      \Gamma^{(1, 0)}_{t,t-1} & \Gamma^{(1, 1)}_{t,t-1}
    \end{pmatrix}^\top
    \begin{pmatrix}
      \pi_{t-1}
      \\
      1 - \pi_{t-1}
    \end{pmatrix}
    \\
    &=
      \begin{pmatrix}
      \pi_{t}
      \\
      1 - \pi_{t}
    \end{pmatrix}
  \;.
  \end{aligned}
\end{equation*}

Our model is effectively a zero-inflated Poisson with a simple first-order Markov process as the mixing distribution.

In the remainder of this exposition, we will assume Dirichlet priors for each row of $\Gamma_{t, t-1}$, the conjugate Gamma prior for $\mu$, and the steady-state value of $\Gamma_{t, t-1}$ for $\pi_0$.

## Simulation

```python
def create_poisson_zero_hmm(mu, observed, p_0_a=np.r_[5, 1], p_1_a=np.r_[3, 5]):

    p_0_rv = pm.Dirichlet("p_0", p_0_a)
    p_1_rv = pm.Dirichlet("p_1", p_1_a)

    P_tt = tt.stack([p_0_rv, p_1_rv])
    P_rv = pm.Deterministic("P_t", tt.shape_padleft(P_tt))

    pi_0_tt = compute_steady_state(P_rv)

    S_rv = DiscreteMarkovChain("S_t", P_rv, pi_0_tt, shape=np.shape(observed)[-1])
    S_rv.tag.test_value = np.array(observed > 0, dtype=np.int32)

    Y_rv = SwitchingProcess(
        "Y_t", [Constant.dist(np.array(0, dtype=np.int64)), pm.Poisson.dist(mu)], S_rv, observed=observed
    )

    return Y_rv

```

```python
np.random.seed(2032)

mu_true = 5000

with pm.Model() as sim_model:
    _ = create_poisson_zero_hmm(mu_true, np.zeros(10000))

sim_point = pm.sample_prior_predictive(samples=1, model=sim_model)
sim_point["Y_t"] = sim_point["Y_t"].squeeze()

y_t = sim_point["Y_t"]
```

## Estimation

We will use the "true" data-generating observation model to estimate the parameters $\mu$ and $\Gamma_{t, t-1}$ (the latter as rows denoted by `p_0` and `p_1`).  For demonstration purposes, we choose hyper-parameters for the $\mu$ prior that are "far" from the true $\mu$ value.

The sampling steps for $S_t$ are performed using forward-filtering backward-sampling (FFBS).  The steps for $\Gamma_{t, t-1}$ are conjugate samples conditional on the sampled state sequence $\{S_t\}$ provided by FFBS.  Both of these step methods are available in `pymc3-hmm`.

While $\mu$ could also be sampled exactly using a conjugate step—conditional on $\{S_t\}$—we instead sample this parameter using PyMC3's built-in NUTS.

```python
with pm.Model() as test_model:
    E_mu, Var_mu = 1000.0, 10000.0
    mu_rv = pm.Gamma("mu", E_mu ** 2 / Var_mu, E_mu / Var_mu)

    _ = create_poisson_zero_hmm(mu_rv, y_t, p_0_a=np.r_[1, 1], p_1_a=np.r_[1, 1])

with test_model:
    transmat_step = TransMatConjugateStep(test_model.P_t)
    states_step = FFBSStep([test_model.S_t])
    mu_step = pm.NUTS([test_model.mu])

    posterior_trace = pm.sample(
        step=[transmat_step, states_step, mu_step],
        return_inferencedata=True,
        chains=1,
        progressbar=True,
    )
```

    Sequential sampling (1 chains in 1 job)
    CompoundStep
    >TransMatConjugateStep: [p_0, p_1]
    >FFBSStep: [S_t]
    >NUTS: [mu]

  <progress value='2000' class='' max='2000' style='width:300px; height:20px; vertical-align: middle;'></progress>
  100.00% [2000/2000 09:34<00:00 Sampling chain 0, 0 divergences]

    Sampling 1 chain for 1_000 tune and 1_000 draw iterations (1_000 + 1_000 draws total) took 574 seconds.
    Only one chain was sampled, this makes it impossible to run some convergence checks

## Posterior Samples

```python
import matplotlib.pyplot as plt

import arviz as az

ax = az.plot_trace(
    posterior_trace.posterior,
    var_names=[
        "mu",
        "p_0",
        "p_1",
    ],
    lines=[
        ("mu", {}, [mu_true]),
        ("p_0", {}, [sim_point["p_0"]]),
        ("p_1", {}, [sim_point["p_1"]]),
    ],
    compact=True,
    figsize=(15, 15),
)

plt.tight_layout()
```

## Posterior Predictive Samples

```python
with test_model:
    posterior_pred_trace = pm.sample_posterior_predictive(
        posterior_trace.posterior, var_names=["S_t", "Y_t"]
    )
```

  <progress value='1000' class='' max='1000' style='width:300px; height:20px; vertical-align: middle;'></progress>
  100.00% [1000/1000 00:33<00:00]

```python
from matplotlib.ticker import MaxNLocator

from pymc3_hmm.utils import plot_split_timeseries_histograms

columns = pd.MultiIndex.from_tuples(
    [("Y_t", i) for i in range(posterior_pred_trace["Y_t"].shape[0])],
    names=("var", "sample"),
)

plot_data = pd.DataFrame(
    posterior_pred_trace["Y_t"].T,
    index=pd.date_range(
        start="1/1/2020", periods=posterior_pred_trace["Y_t"].shape[1], freq="H"
    ),
    columns=columns,
)
plot_data["E[S_t]"] = posterior_pred_trace["S_t"].mean(0)
plot_data["E[Y_t]"] = posterior_pred_trace["Y_t"].mean(0)
plot_data["y_t"] = y_t
```

```python
def plot_fn(ax, data, **kwargs):
    ax.plot(
        data["y_t"],
        label="y_t",
        alpha=0.7,
        color="red",
        linewidth=0.8,
        drawstyle="steps",
    )

    ax.plot(
        data["E[Y_t]"],
        label="E[Y_t]",
        alpha=0.3,
        color="blue",
        linewidth=0.8,
        drawstyle="steps",
    )

axes_split_data = plot_split_timeseries_histograms(
    plot_data,
    sample_col="Y_t",
    plot_fn=plot_fn,
    split_max=10,
    figsize=(15, 30),
    twin_column_name="E[S_t]",
    twin_plot_kwargs={
        "color": "green",
        "drawstyle": "steps",
        "linestyle": "--",
        "alpha": 0.4,
    },
)

for (_, twin_ax), _ in axes_split_data:
    _ = twin_ax.set_ylabel(
        "E[S_t]", color=twin_ax.get_lines()[0].get_color()
    )
    _ = twin_ax.yaxis.set_major_locator(MaxNLocator(integer=True))

plt.tight_layout()
```

## Out-of-sample Posterior Predictives

```python
with test_model, theano.config.change_flags(compute_test_value="off"):

    test_model.S_t.distribution.shape = (24 * 7,)

    out_of_sample_pp = pm.sample_posterior_predictive(
        posterior_trace.posterior.drop_vars("S_t"),
        var_names=["S_t", "Y_t"],
        model=test_model,
    )
```

  <progress value='1000' class='' max='1000' style='width:300px; height:20px; vertical-align: middle;'></progress>
  100.00% [1000/1000 01:04<00:00]

```python
from pymc3_hmm.utils import plot_timeseries_histograms

columns = pd.MultiIndex.from_tuples(
    [("Y_t", i) for i in range(out_of_sample_pp["Y_t"].shape[0])],
    names=("var", "sample"),
)

out_of_sample_data = pd.DataFrame(
    out_of_sample_pp["Y_t"].T,
    index=pd.date_range(
        start="1/1/2021", periods=out_of_sample_pp["Y_t"].shape[1], freq="H"
    ),
    columns=columns
)
out_of_sample_data[("mean", 0)] = out_of_sample_pp["Y_t"].mean(0)

ax = plot_timeseries_histograms(axes=None, data=out_of_sample_data["Y_t"])
```


# Function

### Projectile

\begin{equation}
R=\frac{u^2\sin 2\theta}{g}
\end{equation}

\begin{equation}
TF=\frac{2u\sin \theta}{g}
\end{equation}

\begin{equation}
H=\frac{u^2\sin^2\theta}{2g}
\end{equation}

```python
import numpy as np                     # from numpy import* not required to write np
import pandas as pd                    # from pandas import*
import matplotlib.pyplot as pltfrom    # from matplotlib import*
%matplotlib inline
```

```python
def projectile(u,theta):
    g=9.8                                        # acceleration due to gravity
    R = u**2*np.sin(2*np.pi*theta/180)/g         # range
    H= u**2*np.sin(np.pi*theta/180)**2/(2*g)     # max. height
    TF = 2*u*np.sin(np.pi*theta/180)/g           # time of flight
    return [R,H,TF]
```

```python
p=projectile(100,60)
p
```

    [883.699391616774, 382.6530612244897, 17.67398783233548]

```python
Angle=[]  #list for angle
R=[]      #list for range
H=[]      # list for height
TF=[]      #list for time of flight
for angle in range(1,91):
    a=projectile(100,angle)
    Angle.append(angle)
    R.append(a[0])  # added element  in list R
    H.append(a[1])  # added element  in list H
    TF.append(a[2]) # added element  in list TF
```

```python
plt.plot(Angle,R,'g-.',label='Range')
plt.plot(Angle,H,'b^',label='Max.height')
plt.xlabel('Angle(degree)')
plt.ylabel('Distance(m)')
plt.title('Projectile')
plt.legend()
plt.show()
```

```python
plt.plot(Angle,TF,'k*')
plt.xlabel('Angle(degree)')
plt.ylabel('Time of flight(sec)')
plt.title('Projectile')
plt.savefig('projective.eps')
plt.show()
```

```python
data={}    # dictionary
data.update({"Angle":Angle,"Range":R ,"Time of flight":TF,"Max.Height": H})
```

```python
DF = pd.DataFrame(data)
DF
```

```python
DF.to_csv("projectile.csv")  #save data in csv format in excel
```

```python

```


# Going from linear regression parameters to correlation

```python
import numpy as np
from scipy.stats import pearsonr
import matplotlib.pyplot as plt
%matplotlib inline
```

Let's generate some data.

```python
N = 100
X = np.random.normal(0, 2, (N, 1))
X = np.hstack((np.ones((N, 1)), X))
y = np.random.normal(0, 1, N)

b = np.linalg.lstsq(X, y, rcond=None)[0]
```

What is the correlation between the predictor and the target?

```python
corr, p = pearsonr(X[:, 1], y)
print("Correlation: %.5f (p = %.3f)" % (corr, p))
```

    Correlation: -0.15013 (p = 0.136)

Now, we can also calculate the correlation from the linear regression parameter ($\beta_{j}$) directly:

\begin{align}
r_{X_{j}, y} = \hat{\beta}_{j} \cdot \frac{\sigma_{X_{j}}}{\sigma_{y}}
\end{align}

which holds only for bivariate regression.

```python
corr_from_beta = b[1] * (X[:, 1].std() / y.std())
print("Correlation calculated from parameter: %.5f" % corr_from_beta)
```

    Correlation calculated from parameter: -0.15013

```python
def semipartial_pearsonr(X, y, idx=0):
    other_idx = np.ones(X.shape[1]).astype(bool)
    other_idx[idx] = False
    other_X = X[:, other_idx]
    this_X = X[:, idx]
    resid_y = y - other_X.dot(np.linalg.lstsq(other_X, y, rcond=None)[0])
    return pearsonr(resid_y, this_X)[0]
    #resid_X = this_X - other_X.dot(np.linalg.lstsq(other_X, this_X, rcond=None)[0])
    #return pearsonr(resid_X, y)[0]

def partial_pearsonr(X, y, idx=0):
    other_idx = np.ones(X.shape[1]).astype(bool)
    other_idx[idx] = False
    other_X = X[:, other_idx]
    this_X = X[:, idx]
    resid_y = y - other_X.dot(np.linalg.lstsq(other_X, y, rcond=None)[0])
    resid_X = this_X - other_X.dot(np.linalg.lstsq(other_X, this_X, rcond=None)[0])
    return pearsonr(resid_y, resid_X)[0]
```

```python
X = np.random.normal(0, 2, (N, 2))
X = np.hstack((np.ones((N, 1)), X))
X[:, 1:] = (X[:, 1:] - X[:, 1:].mean(axis=0)) / X[:, 1:].std(axis=0)
y = np.random.normal(0, 1, N)

print(semipartial_pearsonr(X, y, idx=1))
print(partial_pearsonr(X, y, idx=1))
print(pearsonr(X[:, 1], y)[0])
b = np.linalg.lstsq(X, y, rcond=None)[0]
print(b[1])
```

    0.20251130419172506
    0.20267002428413428
    0.20506437871773314
    0.1770180139750527


# Gradient descent

Gradient descent is an **optimization algorithm** used to **minimize some function** by **iteratively moving in the direction of steepest descent** as defined by the **negative of the gradient**. In machine learning, we use gradient descent to **update the parameters of our model**. **Parameters** refer to **coefficients** in **Linear Regression** and **weights** in **neural networks**.

This section aims to provide you an explanation of gradient descent and **intuitions** towards the **behaviour of different algorithms for optimizing it**. These explanations will help you put them to use.

We are first going to introduce the gradient descent, solve it for a regression problem and look at its different variants. Then, we will then briefly summarize challenges during training. Finally, we will introduce the **most common optimization algorithms** by showing their motivation to resolve these challenges and list some advices for facilitate the algorithm choice.

## Introduction

Consider the 3-dimensional graph below in **the context of a cost function**. Our **goal** is to **move from the mountain in the top right corner (high cost) to the dark blue sea in the bottom left (low cost)**. The **arrows** represent the **direction** of steepest descent (negative gradient) from any given point–the direction that **decreases the cost function** as quickly as possible

 Gradient descent intuition.

Starting at the top of the mountain, we take our first step **downhill** in the **direction specified by the negative gradient**. Next we **recalculate the negative gradient** (passing in the coordinates of our new point) and take another step in the direction it specifies. We continue this **process iteratively until** we get to the **bottom of our graph**, or to a **point where we can no longer move downhill–a local minimum**.

### Learning rate

The **size of these steps** is called the **learning rate**. With a **high learning rate** we can cover more ground each step, but we **risk overshooting the lowest point** since the slope of the hill is constantly changing. With a **very low learning rate**, we can **confidently move in the direction of the negative gradient** since we are **recalculating it so frequently**. A **low learning rate is more precise**, but calculating the gradient is **time-consuming**, so it will take us a very long time to get to the bottom.

 impacts of learning rate choice.

### Cost function

A **Loss Function (Error function)**  tells us **“how good”** our **model** is at making predictions for a **given set of parameters**. The cost function has its own curve and its own gradients. The **slope of this curve** tells us how to **update our parameters** to make the model more **accurate**.

## Numerical solution for gradient descent

Let's run gradient descent using a **linear regression cost function**.

There are **two parameters** in our cost function we can control:
-  $ \beta_0$ : **(the bias)**
-  $\beta_1 $ : **(weight or coefficient)**

Since we need to consider the **impact each one** has on the final prediction, we need to use **partial derivatives**. We calculate the **partial derivatives of the cost function with respect to each parameter and store the results in a gradient**.

### Solution

**Given the cost function**

$$f(\beta_0,\beta_1) =  \frac{1}{2}\frac{\partial MSE}{\partial\beta} = \frac{1}{2N} \sum_{i=1}^{n} (y_i - (\beta_1 x_i + \beta_0))^2 =  \frac{1}{2N} \sum_{i=1}^{n} ((\beta_1 x_i + \beta_0) - y_i)^2$$

**The gradient can be calculated as**

$$
\begin{split}f'(\beta_0,\beta_1) =
   \begin{bmatrix}
     \frac{\partial f}{\partial \beta_0}\\
     \frac{\partial f}{\partial \beta_1}\\
    \end{bmatrix}
=
   \begin{bmatrix}
     \frac{1}{2N} \sum -2((\beta_1x_i + \beta_0) - y_i ) \\
     \frac{1}{2N} \sum -2x_i((\beta_1x_i + \beta_0) - y_i) \\
    \end{bmatrix}
=
   \begin{bmatrix}
     \frac{-1}{N} \sum ((\beta_1x_i + \beta_0) - y_i ) \\
     \frac{-1}{N} \sum  x_i((\beta_1x_i + \beta_0) - y_i) \\
    \end{bmatrix}
    \end{split}
$$

To solve for the gradient, we **iterate** through our **data points** using our **$\beta_1$ and $\beta_0$ values** and compute the **partial derivatives**. This **new gradient** tells us the **slope of our cost function at our current position** (current parameter values) and the **direction we should move to update our parameters**. The **size of our update** is **controlled by the learning rate**.

**Pseudocode of this algorithm**

``` python

Function  gradient_descent(X, Y, learning_rate, number_iterations):

    m : 1
    b : 1
    m_deriv : 0
    b_deriv : 0
    data_length : length(X)
    loop i : 1  --> number_iterations:
        loop i : 1  ->  data_length  :
            m_deriv : m_deriv -X[i] * ((m*X[i] + b) - Y[i])
            b_deriv : b_deriv - ((m*X[i] + b) - Y[i])
        m : m - (m_deriv / data_length) * learning_rate
        b : b - (b_deriv / data_length) * learning_rate

    return m, b

```

## Gradient descent variants

There are **three variants of gradient descent**, which **differ in how much data we use to compute the gradient** of the objective function. Depending on the **amount of data**, we make a **trade-off between the accuracy of the parameter update and the time it takes to perform an update**.

### Batch gradient descent

Batch gradient descent, known also as Vanilla gradient descent, computes the gradient of the cost function with respect to the parameters $\theta$  for the **entire training dataset** :

$$
\theta = \theta - \eta \cdot \nabla_\theta J( \theta)
$$

As we need to calculate the gradients for the **whole dataset** to **perform just one update**, **batch** gradient descent can be **very slow** and **is intractable for datasets that don't fit in memory**. Batch gradient descent also **doesn't allow us to update our model online**.

### Stochastic gradient descent

Stochastic gradient descent (SGD) in contrast performs a parameter update for each training example $x^{(i)}$ and label $y^{(i)}$

- Choose an initial vector of parameters $w$ and learning rate $\eta$.
- Repeat until an approximate minimum is obtained:
    - Randomly shuffle examples in the training set.
    - For $i \in 1, \dots, n$
        - $\theta = \theta - \eta \cdot \nabla_\theta J( \theta; x^{(i)}; y^{(i)})$

**Batch gradient descent** performs **redundant computations for large datasets**, as it **recomputes** gradients for **similar examples** before each **parameter update**. SGD does away with **this redundancy by performing one update at a time**. It is therefore usually **much faster and can also be used to learn online**.
**SGD** performs **frequent updates with a high variance that cause the objective function to fluctuate heavily as in the image  below**.

 SGD fluctuation.

While batch gradient descent **converges to the minimum of the basin** the parameters are placed in, SGD's fluctuation, on the one hand, **enables it to jump to new and potentially better local minima**. On the other hand, **this ultimately complicates convergence to the exact minimum**, as **SGD will keep overshooting**. However, it **has been shown** that when we **slowly decrease the learning rate**, SGD shows the same **convergence behaviour as batch gradient descent**, almost certainly **converging to a local or the global minimum for non-convex and convex optimization respectively**.

### Mini-batch gradient descent

Mini-batch gradient descent finally takes the best of both worlds and performs an update for every mini-batch of
n training examples:

$$
\theta = \theta - \eta \cdot \nabla_\theta J( \theta; x^{(i:i+n)}; y^{(i:i+n)})
$$

This way, it :

- **reduces the variance of the parameter updates**, which can lead to **more stable convergence**.
- can make use of **highly optimized matrix optimizations** common to state-of-the-art deep learning libraries that make computing the gradient very efficient. **Common mini-batch sizes range between 50 and 256**, but can vary for different applications.

**Mini-batch** gradient descent is **typically the algorithm of choice when training a neural network**.

## Gradient Descent challenges

Vanilla mini-batch gradient descent, however, does not guarantee good convergence, but offers a few challenges that need to be addressed:

- Choosing a proper **learning rate** can be difficult. A learning rate that is **too small** leads to **painfully slow convergence**, while a learning rate that is **too large** can hinder convergence and cause the loss function to **fluctuate** around the minimum or even to **diverge**.

- **Learning rate schedules** try to **adjust the learning rate during training** by e.g. annealing, i.e. reducing the learning rate according to a pre-defined schedule or when the change in objective between epochs falls below a threshold. These schedules and thresholds, however, have to be defined in advance and are thus unable to adapt to a dataset's characteristics.

- Additionally, the same learning rate applies to all parameter updates. **If our data is sparse and our features have very different frequencies**, we might not want to update all of them to the same extent, but perform **a larger update for rarely occurring features**.

- Another key challenge of **minimizing highly non-convex error** functions common for neural networks is **avoiding** getting **trapped in their numerous suboptimal local minima**. These **saddle points (local minimas)** are usually surrounded by a plateau of the same error, which makes it **notoriously hard for SGD to escape**, as the gradient is close to zero in all dimensions.

## Gradient descent optimization algorithms

In the following, we will outline some **algorithms** that are **widely** used by the **deep learning community** to deal with the aforementioned **challenges**.

### Momentum

**SGD** has trouble **navigating ravines (areas where the surface curves much more steeply in one dimension than in another)**, which are common **around local optima**. In these scenarios, **SGD oscillates across the slopes of the ravine while only making hesitant progress** along the bottom towards the local optimum as in the image below.

 SGD and momentum.

[Source](https://distill.pub/2017/momentum/)

No momentum: moving toward local largest gradient create oscillations.

With momentum: accumulate velocity to avoid oscillations.

**Momentum** is a method that helps **accelerate SGD** in the **relevant direction and dampens oscillations** as can be seen in image above. It does this by **adding a fraction $\gamma$ of the update vector** of the **past time step to the current update vector**

$$
\begin{align}
\begin{split}
v_t &= \rho v_{t-1} + \nabla_\theta J( \theta) \\
\theta &= \theta - v_t
\end{split}
\end{align}
$$

```python
vx = 0
while True:
    dx = gradient(J, x)
    vx = rho * vx + dx
    x -= learning_rate * vx
```

**Note**: The momentum term **$\rho$** is usually set to 0.9 or a similar value.

Essentially, when using momentum, we push **a ball down a hill**. The **ball accumulates momentum as it rolls downhill**, becoming faster and faster on the way **(until it reaches its terminal velocity if there is air resistance, i.e.
$\rho$ <1 )**.

The same thing happens to our parameter updates: **The momentum term increases** for **dimensions whose gradients point in the same directions and reduces updates for dimensions whose gradients change directions**. As a result, we **gain faster convergence** and **reduced oscillation**.

### AdaGrad: adaptive learning rates

- Added element-wise scaling of the gradient based on the historical sum of squares in each dimension.

- "Per-parameter learning rates" or "adaptive learning rates"

```python
grad_squared = 0
while True:
    dx = gradient(J, x)
    grad_squared += dx * dx
    x -= learning_rate * dx / (np.sqrt(grad_squared) + 1e-7)
```

- Progress along “steep” directions is damped.
- Progress along “flat” directions is accelerated.
- Problem: step size over long time => Decays to zero.

### RMSProp: “Leaky AdaGrad”

```python
grad_squared = 0
while True:
    dx = gradient(J, x)
    grad_squared += decay_rate * grad_squared + (1 - decay_rate) * dx * dx
    x -= learning_rate * dx / (np.sqrt(grad_squared) + 1e-7)
```

- `decay_rate = 1`: gradient descent
- `decay_rate = 0`: AdaGrad

### Nesterov accelerated gradient

However, a ball that rolls down a hill, blindly following the slope, is highly **unsatisfactory**. We'd like to have a smarter ball, a ball that has **a notion of where it is going** so that it **knows to slow down before the hill slopes up again**.
Nesterov accelerated gradient (NAG) is a way to give **our momentum term this kind of prescience**. We know that we will use our momentum term $\gamma v_{t-1}$ to move the parameters  $\theta$.

Computing $\theta - \gamma v_{t-1}$ thus gives us **an approximation of the next position of the parameters** (the gradient is missing for the full update), a rough idea where our parameters are going to be.

We can now effectively look ahead by calculating the gradient not w.r.t. to our current parameters $\theta$ but w.r.t. the approximate future position of our parameters:

\begin{align}
\begin{split}
v_t &= \gamma v_{t-1} + \eta \nabla_\theta J( \theta - \gamma v_{t-1} ) \\
\theta &= \theta - v_t
\end{split}
\end{align}

Again, we set the momentum term $\gamma$ to a value of around 0.9. While **Momentum first computes the current gradient and then takes a big jump in the direction of the updated accumulated gradient** , **NAG** first **makes a big jump in the direction of the previous accumulated gradient, measures the gradient and then makes a correction, which results in the complete NAG update**. This anticipatory update **prevents** us from **going too fast** and results in **increased responsiveness**, which has significantly **increased the performance of RNNs** on a number of tasks

### Adam

**Adaptive Moment Estimation (Adam)** is a method that computes **adaptive learning rates** for each parameter. In addition to storing an **exponentially decaying average of past squared gradients $v_t$**, Adam also keeps an **exponentially decaying average of past gradients $m_t$, similar to momentum**. Whereas momentum can be seen as a ball running down a slope, Adam behaves like a **heavy ball with friction**, which thus prefers **flat minima in the error surface**. We compute the decaying averages of past and past squared gradients $m_t$ and $v_t$ respectively as follows:

\begin{align}
\begin{split}
m_t &= \beta_1 m_{t-1} + (1 - \beta_1) \nabla_\theta J( \theta) \\
v_t &= \beta_2 v_{t-1} + (1 - \beta_2) \nabla_\theta J( \theta)^2
\end{split}
\end{align}

$m_{t}$ and $v_{t}$ are estimates of the first moment (the mean) and the second moment (the uncentered variance) of the gradients respectively, hence the name of the method.
Adam (almost)

```python
first_moment = 0
second_moment = 0
while True:
    dx = gradient(J, x)
    # Momentum:
    first_moment = beta1 * first_moment + (1 - beta1) * dx
    # AdaGrad/RMSProp
    second_moment = beta2 * second_moment + (1 - beta2) * dx * dx
    x -= learning_rate * first_moment / (np.sqrt(second_moment) + 1e-7)
```

As $m_{t}$ and $v_{t}$ are initialized as vectors of 0's, the authors of Adam observe that they are biased towards zero, especially during the initial time steps, and especially when the decay rates are small (i.e. $\beta_{1}$ and $\beta_{2}$ are close to 1).
They counteract these biases by computing bias-corrected first and second moment estimates:

\begin{align}
\hat{m}_t &= \dfrac{m_t}{1 - \beta^t_1} \\
\hat{v}_t &= \dfrac{v_t}{1 - \beta^t_2}
\end{align}

They then use these to update the parameters (Adam update rule):

$$
\theta_{t+1} = \theta_{t} - \dfrac{\eta}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t
$$

- $\hat{m}_t$ Accumulate gradient: velocity.
- $\hat{v}_t$ Element-wise scaling of the gradient based on the historical sum of squares in each dimension.

- Choose Adam as default optimizer
- Default values of 0.9 for $\beta_1$, 0.999 for $\beta_2$, and $10^{-7}$ for $\epsilon$.
- learning rate in a range between $1e-3$ and $5e-4$

```python

```