# Demo - Lid driven cavity

**Mikael Mortensen** (email: `mikaem@math.uio.no`), Department of Mathematics, University of Oslo.

Date: **May 6, 2019**

**Summary.** The lid driven cavity is a classical benchmark for Navier Stokes solvers.
This is a demonstration of how the Python module [shenfun](https://github.com/spectralDNS/shenfun) can be used to solve the lid
driven cavity problem with full spectral accuracy using a mixed (coupled) basis
in a 2D tensor product domain. The demo also shows how to use mixed
tensor product spaces for vector valued equations. Note that the regular
lid driven cavity, where the top wall has constant velocity and the
remaining three walls are stationary, has a singularity at the two
upper corners, where the velocity is discontinuous.
Due to their global nature, spectral methods
are usually not very good at handling problems with discontinuities, and
for this reason we will also look at a regularized lid driven cavity,
where the top lid moves according to $(1-x)^2(1+x)^2$, thus removing
the corner discontinuities.

Velocity vectors for the lid driven cavity at Reynolds number 100.

## Navier Stokes equations

The nonlinear steady Navier Stokes equations are given in strong form as

$$
\begin{align*}
\nu \nabla^2 \boldsymbol{u} - \nabla p &= \nabla \cdot \boldsymbol{u} \boldsymbol{u} \quad \text{in }  \Omega , \\
\nabla \cdot \boldsymbol{u} &= 0 \quad \text{in } \Omega  \\
\int_{\Omega} p dx &= 0 \\
\boldsymbol{u}(x, y=1) = (1, 0) \, &\text{ or }\, \boldsymbol{u}(x, y=1) = ((1-x)^2(1+x)^2, 0) \\
\boldsymbol{u}(x, y=-1) &= (0, 0) \\
\boldsymbol{u}(x=\pm 1, y) &= (0, 0)
\end{align*}
$$

where $\boldsymbol{u}, p$ and $\nu$ are, respectively, the
fluid velocity vector, pressure and kinematic viscosity. The domain
$\Omega = [-1, 1]^2$ and the nonlinear term $\boldsymbol{u} \boldsymbol{u}$ is the
outer product of vector $\boldsymbol{u}$ with itself. Note that the final
$\int_{\Omega} p dx = 0$ is there because there is no Dirichlet boundary
condition on the pressure and the system of equations would otherwise be
ill conditioned.

We want to solve these steady nonlinear Navier Stokes equations with the Galerkin
method, using the [shenfun](https://github.com/spectralDNS/shenfun) Python
package. The first thing we need to do then is to import all of shenfun's
functionality

```
%matplotlib inline

import matplotlib.pyplot as plt
from shenfun import *
```

Note that MPI for Python ([mpi4py](https://bitbucket.org/mpi4py/mpi4py))
is a requirement for shenfun, but the current solver cannot be used with more
than one processor.

## Tensor product spaces

With the Galerkin method we need function spaces for both velocity and
pressure, as well as for the
nonlinear right hand side. A Dirichlet space will be used for velocity,
whereas there is no boundary restriction on the pressure space. For both
two-dimensional spaces we will use one basis function for the $x$-direction,
$\mathcal{X}_k(x)$, and one for the $y$-direction, $\mathcal{Y}_l(y)$. And
then we create two-dimensional basis functions like

$$
\begin{equation}
v_{kl}(x, y) = \mathcal{X}_k(x) \mathcal{Y}_l(y),  \label{eq:nstestfunction} \tag{1}
\end{equation}
$$

and solutions (trial functions) as

$$
\begin{equation}
    u(x, y) = \sum_{k}\sum_{l} \hat{u}_{kl} v_{kl}(x, y). \label{eq:nstrialfunction} \tag{2}
\end{equation}
$$

For the homogeneous Dirichlet boundary condition the basis functions
$\mathcal{X}_k(x)$ and $\mathcal{Y}_l(y)$ are chosen as composite
Legendre polynomials (we could also use Chebyshev):

$$
\begin{equation}
\mathcal{X}_k(x) = L_k(x) - L_{k+2}(x), \quad \forall \, k \in \boldsymbol{k}^{N_0-2}, \label{eq:D0} \tag{3}
\end{equation}
$$

$$
\begin{equation}
\mathcal{Y}_l(y) = L_l(y) - L_{l+2}(y), \quad \forall \, l \in \boldsymbol{l}^{N_1-2}, \label{eq:D1} \tag{4}
\end{equation}
$$

where $\boldsymbol{k}^{N_0-2} = (0, 1, \ldots, N_0-3)$, $\boldsymbol{l}^{N_1-2} = (0, 1, \ldots, N_1-3)$
and $N = (N_0, N_1)$ is the number
of quadrature points in each direction. Note that $N_0$ and $N_1$ do not need
to be the same. The basis funciton ([3](#eq:D0)) satisfies
the homogeneous Dirichlet boundary conditions at $x=\pm 1$ and ([4](#eq:D1)) the same
at $y=\pm 1$. As such, the basis function $v_{kl}(x, y)$ satisfies the homogeneous Dirichlet boundary
condition for the entire domain.

With shenfun we create these homogeneous spaces, $D_0^{N_0}(x)=\text{span}\{L_k-L_{k+2}\}_{k=0}^{N_0-2}$ and
$D_0^{N_1}(y)=\text{span}\{L_l-L_{l+2}\}_{l=0}^{N_1-2}$ as

```
N = (45, 45)
family = 'Legendre' # or use 'Chebyshev'
quad = 'GL'         # for Chebyshev use 'GC' or 'GL'
D0X = FunctionSpace(N[0], family, quad=quad, bc=(0, 0))
D0Y = FunctionSpace(N[1], family, quad=quad, bc=(0, 0))
```

The spaces are here the same, but we will use `D0X` in the $x$-direction and
`D0Y` in the $y$-direction. But before we use these bases in
tensor product spaces, they remain identical as long as $N_0 = N_1$.

Special attention is required by the moving lid. To get a solution
with nonzero boundary condition at $y=1$ we need to add one more basis function
that satisfies that solution. In general, a nonzero boundary condition
can be added on both sides of the domain using the following basis

$$
\begin{equation}
\mathcal{Y}_l(y) = L_l(y) - L_{l+2}(y), \quad \forall \, l \in \boldsymbol{l}^{N_1-2}.
\label{_auto1} \tag{5}
\end{equation}
$$

$$
\begin{equation}
\mathcal{Y}_{N_1-2}(y) = (L_0+L_1)/2 \quad \left(=(1+y)/2\right),
\label{_auto2} \tag{6}
\end{equation}
$$

$$
\begin{equation}
\mathcal{Y}_{N_1-1}(y) = (L_0-L_1)/2 \quad \left(=(1-y)/2\right).
\label{_auto3} \tag{7}
\end{equation}
$$

And then the unknown component $N_1-2$ decides the value at $y=1$, whereas
the unknown at $N_1-1$ decides the value at $y=-1$. Here we only need to
add the $N_1-2$ component, but for generality this is implemented in shenfun
using both additional basis functions. We create the space
$D_1^{N_1}(y)=\text{span}\{\mathcal{Y}_l(y)\}_{l=0}^{N_1-1}$ as

```
D1Y = FunctionSpace(N[1], family, quad=quad, bc=(0, 1))
```

where `bc=(1, 0)` fixes the values for $y=1$ and $y=-1$, respectively.
For a regularized lid driven cavity the velocity of the top lid is
$(1-x)^2(1+x)^2$ and not unity. To implement this boundary condition
instead, we can make use of [sympy](https://www.sympy.org) and
quite straight forward do

```
import sympy
x = sympy.symbols('x')
#D1Y = FunctionSpace(N[1], family, quad=quad, bc=(0, (1-x)**2*(1+x)**2))
```

Uncomment the last line to run the regularized boundary conditions.
Otherwise, there is no difference at all between the regular and the
regularized lid driven cavity implementations.

The pressure basis that comes with no restrictions for the boundary is a
little trickier. The reason for this has to do with
inf-sup stability. The obvious choice of basis functions are the
regular Legendre polynomials $L_k(x)$ in $x$ and $L_l(y)$ in the
$y$-directions. The problem is that for the natural choice of
$(k, l) \in \boldsymbol{k}^{N_0} \times \boldsymbol{l}^{N_1}$
there are nullspaces and the problem is not well-defined. It turns out
that the proper choice for the pressure basis is simply the regular
Legendre basis functions, but for
$(k, l) \in \boldsymbol{k}^{N_0-2} \times \boldsymbol{l}^{N_1-2}$.
The bases $P^{N_0}(x)=\text{span}\{L_k(x)\}_{k=0}^{N_0-3}$ and
$P^{N_1}(y)=\text{span}\{L_l(y)\}_{l=0}^{N_1-3}$ are created as

```
PX = FunctionSpace(N[0], family, quad=quad)
PY = FunctionSpace(N[1], family, quad=quad)
PX.slice = lambda: slice(0, N[0]-2)
PY.slice = lambda: slice(0, N[1]-2)
```

Note that we still use these spaces with the same $N_0 \cdot N_1$
quadrature points in real space, but the two highest frequencies have
been set to zero.

We have now created all relevant function spaces for the problem at hand.
It remains to combine these spaces into tensor product spaces, and to
combine tensor product spaces into mixed (coupled) tensor product
spaces. From the Dirichlet bases we create two different tensor
product spaces, whereas one is enough for the pressure

$$
\begin{equation}
V_{1}^{\boldsymbol{N}}(\boldsymbol{x}) = D_0^{N_0}(x) \otimes D_1^{N_1}(y),
\label{_auto4} \tag{8}
\end{equation}
$$

$$
\begin{equation}
V_{0}^{\boldsymbol{N}}(\boldsymbol{x}) = D_0^{N_0}(x) \otimes D_0^{N_1}(y),
\label{_auto5} \tag{9}
\end{equation}
$$

$$
\begin{equation}
P^{\boldsymbol{N}}(\boldsymbol{x}) = P^{N_0}(x) \otimes P^{N_1}(y).
\label{_auto6} \tag{10}
\end{equation}
$$

With shenfun the tensor product spaces are created as

```
V1 = TensorProductSpace(comm, (D0X, D1Y))
V0 = TensorProductSpace(comm, (D0X, D0Y))
P = TensorProductSpace(comm, (PX, PY), modify_spaces_inplace=True)
```

These tensor product spaces are all scalar valued.
The velocity is a vector, and a vector requires a mixed vector basis like
$W_1^{\boldsymbol{N}} = V_1^{\boldsymbol{N}} \times V_0^{\boldsymbol{N}}$. The vector basis is created
in shenfun as

```
W1 = VectorSpace([V1, V0])
W0 = VectorSpace([V0, V0])
```

Note that the second vector basis, $W_0^{\boldsymbol{N}} = V_0^{\boldsymbol{N}} \times V_0^{\boldsymbol{N}}$, uses
homogeneous boundary conditions throughout.

## Mixed variational form

We now formulate a variational problem using the
Galerkin method: Find
$\boldsymbol{u} \in W_1^{\boldsymbol{N}}$ and $p \in P^{\boldsymbol{N}}$ such that

$$
\begin{equation}
\int_{\Omega} (\nu \nabla^2 \boldsymbol{u} - \nabla p ) \cdot \boldsymbol{v} \, dxdy = \int_{\Omega} (\nabla \cdot \boldsymbol{u}\boldsymbol{u}) \cdot \boldsymbol{v}\, dxdy \quad\forall \boldsymbol{v} \, \in \, W_0^{\boldsymbol{N}}, \label{eq:nsvarform} \tag{11}
\end{equation}
$$

$$
\begin{equation}
\int_{\Omega} \nabla \cdot \boldsymbol{u} \, q \, dxdy = 0 \quad\forall q \, \in \, P^{\boldsymbol{N}}.
\label{_auto7} \tag{12}
\end{equation}
$$

Note that we are using test functions $\boldsymbol{v}$ with homogeneous
boundary conditions.

The first obvious issue with Eq ([11](#eq:nsvarform)) is the nonlinearity.
In other words we will
need to linearize and iterate to be able to solve these equations with
the Galerkin method. To this end we will introduce the solution on
iteration $k \in [0, 1, \ldots]$ as $\boldsymbol{u}^k$ and compute the nonlinearity
using only known solutions
$\int_{\Omega} (\nabla \cdot \boldsymbol{u}^k\boldsymbol{u}^k) \cdot \boldsymbol{v}\, dxdy$.
Using further integration by parts we end up with the equations to solve
for iteration number $k+1$ (using $\boldsymbol{u} = \boldsymbol{u}^{k+1}$ and $p=p^{k+1}$
for simplicity)

$$
\begin{equation}
-\int_{\Omega} \nu \nabla \boldsymbol{u} \, \colon \nabla \boldsymbol{v} \, dxdy + \int_{\Omega} p \nabla \cdot \boldsymbol{v} \, dxdy = \int_{\Omega} (\nabla \cdot \boldsymbol{u}^k\boldsymbol{u}^k) \cdot \boldsymbol{v}\, dxdy \quad\forall \boldsymbol{v} \, \in \, W_0^{\boldsymbol{N}}, \label{eq:nsvarform2} \tag{13}
\end{equation}
$$

$$
\begin{equation}
\int_{\Omega} \nabla \cdot \boldsymbol{u} \, q \, dxdy = 0 \quad\forall q \, \in \, P^{\boldsymbol{N}}.
\label{_auto8} \tag{14}
\end{equation}
$$

Note that the nonlinear term may also be integrated by parts and
evaluated as $\int_{\Omega}-\boldsymbol{u}^k\boldsymbol{u}^k  \, \colon \nabla \boldsymbol{v} \, dxdy$. All
boundary integrals disappear since we are using test functions with
homogeneous boundary conditions.

Since we are to solve for $\boldsymbol{u}$ and $p$ at the same time, we formulate a
mixed (coupled) problem: find $(\boldsymbol{u}, p) \in W_1^{\boldsymbol{N}} \times P^{\boldsymbol{N}}$
such that

$$
\begin{equation}
a((\boldsymbol{u}, p), (\boldsymbol{v}, q)) = L((\boldsymbol{v}, q)) \quad \forall (\boldsymbol{v}, q) \in W_0^{\boldsymbol{N}} \times P^{\boldsymbol{N}},
\label{_auto9} \tag{15}
\end{equation}
$$

where bilinear ($a$) and linear ($L$) forms are given as

$$
\begin{equation}
    a((\boldsymbol{u}, p), (\boldsymbol{v}, q)) = -\int_{\Omega} \nu \nabla \boldsymbol{u} \, \colon \nabla \boldsymbol{v} \, dxdy + \int_{\Omega} p \nabla \cdot \boldsymbol{v} \, dxdy + \int_{\Omega} \nabla \cdot \boldsymbol{u} \, q \, dxdy,
\label{_auto10} \tag{16}
\end{equation}
$$

$$
\begin{equation}
    L((\boldsymbol{v}, q); \boldsymbol{u}^{k}) = \int_{\Omega} (\nabla \cdot \boldsymbol{u}^{k}\boldsymbol{u}^{k}) \cdot \boldsymbol{v}\, dxdy.
\label{_auto11} \tag{17}
\end{equation}
$$

Note that the bilinear form will assemble to a block matrix, whereas the right hand side
linear form will assemble to a block vector. The bilinear form does not change
with the solution and as such it does not need to be reassembled inside
an iteration loop.

The algorithm used to solve the equations are:

  * Set $k = 0$

  * Guess $\boldsymbol{u}^0 = (0, 0)$

  * while not converged:

    * assemble $L((\boldsymbol{v}, q); \boldsymbol{u}^{k})$

    * solve $a((\boldsymbol{u}, p), (\boldsymbol{v}, q)) = L((\boldsymbol{v}, q); \boldsymbol{u}^{k})$ for $\boldsymbol{u}^{k+1}, p^{k+1}$

    * compute error = $\int_{\Omega} (\boldsymbol{u}^{k+1}-\boldsymbol{u}^{k})^2 \, dxdy$

    * if error $<$ some tolerance then converged = True

    * $k$ += $1$

## Implementation of solver

We will now implement the coupled variational problem described in previous
sections. First of all, since we want to solve for the velocity and pressure
in a coupled solver, we have to
create a mixed tensor product space $VQ = W_1^{\boldsymbol{N}} \times P^{\boldsymbol{N}}$ that
couples velocity and pressure

```
VQ = CompositeSpace([W1, P])    # Coupling velocity and pressure
```

We can now create test- and trialfunctions for the coupled space $VQ$,
and then split them up into components afterwards:

```
up = TrialFunction(VQ)
vq = TestFunction(VQ)
u, p = up
v, q = vq
```

**Notice.**

The test function `v` is using homogeneous Dirichlet boundary conditions even
though it is derived from `VQ`, which contains `W1`. It is currently not (and will
probably never be) possible to use test functions with inhomogeneous
boundary conditions.

With the basisfunctions in place we may assemble the different blocks of the
final coefficient matrix. For this we also need to specify the kinematic
viscosity, which is given here in terms of the Reynolds number:

```
Re = 100.
nu = 2./Re
if family.lower() == 'legendre':
    A = inner(grad(v), -nu*grad(u))
    G = inner(div(v), p)
else:
    A = inner(v, nu*div(grad(u)))
    G = inner(v, -grad(p))
D = inner(q, div(u))
```

The assembled subsystems `A, G` and `D` are lists containg the different blocks of
the complete, coupled, coefficient matrix. `A` actually contains 4
tensor product matrices of type [TPMatrix](https://shenfun.readthedocs.io/en/latest/shenfun.html#shenfun.matrixbase.TPMatrix). The first two
matrices are for vector component zero of the test function `v[0]` and
trial function `u[0]`, the
matrices 2 and 3 are for components 1. The first two matrices are as such for

          A[0:2] = inner(grad(v[0]), -nu*grad(u[0]))

Breaking it down the inner product is mathematically

$$
\begin{equation}
\label{eq:partialeq1} \tag{18}
\int_{\Omega}-\nu \left(\frac{\partial \boldsymbol{v}[0]}{\partial x}, \frac{\partial \boldsymbol{v}[0]}{\partial y}\right) \cdot \left(\frac{\partial \boldsymbol{u}[0]}{\partial x}, \frac{\partial \boldsymbol{u}[0]}{\partial y}\right) dx dy .
\end{equation}
$$

We can now insert for test function $\boldsymbol{v}[0]$

$$
\begin{equation}
\boldsymbol{v}[0]_{kl} = \mathcal{X}_k \mathcal{Y}_l, \quad (k, l) \in \boldsymbol{k}^{N_0-2} \times \boldsymbol{l}^{N_1-2}
\label{_auto12} \tag{19}
\end{equation}
$$

and trialfunction

$$
\begin{equation}
\boldsymbol{u}[0]_{mn} = \sum_{m=0}^{N_0-3} \sum_{n=0}^{N_1-1} \hat{\boldsymbol{u}}[0]_{mn} \mathcal{X}_m \mathcal{Y}_n,
\label{_auto13} \tag{20}
\end{equation}
$$

where $\hat{\boldsymbol{u}}$ are the unknown degrees of freedom for the velocity vector.
Notice that the sum over the second
index runs all the way to $N_1-1$, whereas the other indices runs to either
$N_0-3$ or $N_1-3$. This is because of the additional basis functions required
for the inhomogeneous boundary condition.

Inserting for these basis functions into ([18](#eq:partialeq1)), we obtain after a few trivial
manipulations

$$
\begin{equation}
 -\sum_{m=0}^{N_0-3} \sum_{n=0}^{N_1-1} \nu \Big( \underbrace{\int_{-1}^{1} \frac{\partial \mathcal{X}_k}{\partial x} \frac{\partial \mathcal{X}_m}{\partial x} dx \int_{-1}^{1} \mathcal{Y}_l \mathcal{Y}_n dy}_{A[0]} +  \underbrace{\int_{-1}^{1} \mathcal{X}_k X_m dx \int_{-1}^{1} \frac{\partial \mathcal{Y}_l}{\partial y} \frac{\partial \mathcal{Y}_n}{\partial y} dy}_{A[1]}  \Big) \hat{\boldsymbol{u}}[0]_{mn}.
\label{_auto14} \tag{21}
\end{equation}
$$

We see that each tensor product matrix (both A[0] and A[1]) is composed as
outer products of two smaller matrices, one for each dimension.
The first tensor product matrix, A[0], is

$$
\begin{equation}
    \underbrace{\int_{-1}^{1} \frac{\partial \mathcal{X}_k}{\partial x} \frac{\partial \mathcal{X}_m}{\partial x} dx}_{c_{km}} \underbrace{\int_{-1}^{1} \mathcal{Y}_l \mathcal{Y}_n dy}_{f_{ln}}
\label{_auto15} \tag{22}
\end{equation}
$$

where $C\in \mathbb{R}^{N_0-2 \times N_1-2}$ and $F \in \mathbb{R}^{N_0-2 \times N_1}$.
Note that due to the inhomogeneous boundary conditions this last matrix $F$
is actually not square. However, remember that all contributions from the two highest
degrees of freedom ($\hat{\boldsymbol{u}}[0]_{m,N_1-2}$ and $\hat{\boldsymbol{u}}[0]_{m,N_1-1}$) are already
known and they can, as such, be  moved directly over to the right hand side of the
linear algebra system that is to be solved. More precisely, we can split the
tensor product matrix into two contributions and obtain

$$
\sum_{m=0}^{N_0-3}\sum_{n=0}^{N_1-1} c_{km}f_{ln} \hat{\boldsymbol{u}}[0]_{m, n} = \sum_{m=0}^{N_0-3}\sum_{n=0}^{N_1-3}c_{km}f_{ln}\hat{\boldsymbol{u}}[0]_{m, n} + \sum_{m=0}^{N_0-3}\sum_{n=N_1-2}^{N_1-1}c_{km}f_{ln}\hat{\boldsymbol{u}}[0]_{m, n}, \quad \forall (k, l) \in \boldsymbol{k}^{N_0-2} \times \boldsymbol{l}^{N_1-2},
$$

where the first term on the right hand side is square and the second term is known and
can be moved to the right hand side of the linear algebra equation system.

At this point all matrices, both regular and boundary matrices, are
contained within the three lists A, G and D. We can now create a solver
for block matrices that incorporates these boundary conditions
automatically

```
sol = la.BlockMatrixSolver(A+G+D)
```

In the solver `sol` there is now a regular block matrix found in
`sol.mat`, which is the symmetric

$$
\begin{bmatrix}
      A[0]+A[1] & 0 & G[0] \\
      0 & A[2]+A[3] & G[1] \\
      D[0] & D[1] & 0
  \end{bmatrix}
$$

The boundary matrices are similarly collected in a boundary block matrix
in `sol.bc_mat`. This matrix is used under the hood to modify the
right hand side.

We now have all the matrices we need in order to solve the Navier Stokes equations.
However, we also need some work arrays for iterations

```
# Create Function to hold solution. Use set_boundary_dofs to fix the degrees
# of freedom in uh_hat that determines the boundary conditions.
uh_hat = Function(VQ).set_boundary_dofs()
ui_hat = uh_hat[0]

# New solution (iterative)
uh_new = Function(VQ).set_boundary_dofs()
ui_new = uh_new[0]
```

The nonlinear right hand side also requires some additional attention.
Nonlinear terms are usually computed in physical space before transforming
to spectral. For this we need to evaluate the velocity vector on the
quadrature mesh. We also need a rank 2 Array to hold the outer
product $\boldsymbol{u}\boldsymbol{u}$. The required arrays and spaces are
created as

```
bh_hat = Function(VQ)

# Create arrays to hold velocity vector solution
ui = Array(W1)

# Create work arrays for nonlinear part
QT = CompositeSpace([W1, W0])  # for uiuj
uiuj = Array(QT)
uiuj_hat = Function(QT)
```

The right hand side $L((\boldsymbol{v}, q);\boldsymbol{u}^{k});$ is computed in its
own function `compute_rhs` as

```
def compute_rhs(ui_hat, bh_hat):
    global ui, uiuj, uiuj_hat, V1, bh_hat0
    bh_hat.fill(0)
    ui = W1.backward(ui_hat, ui)
    uiuj = outer(ui, ui, uiuj)
    uiuj_hat = uiuj.forward(uiuj_hat)
    bi_hat = bh_hat[0]
    bi_hat = inner(v, div(uiuj_hat), output_array=bi_hat)
    #bi_hat = inner(grad(v), -uiuj_hat, output_array=bi_hat)
    return bh_hat
```

Here [outer()](https://shenfun.readthedocs.io/en/latest/shenfun.utilities.html#shenfun.utilities.outer) is a shenfun function that computes the
outer product of two vectors and returns the product in a rank two
array (here `uiuj`). With `uiuj` forward transformed to `uiuj_hat`
we can assemble the linear form either as `inner(v, div(uiuj_hat)` or
`inner(grad(v), -uiuj_hat)`.

Now all that remains is to guess an initial solution and solve
iteratively until convergence. For initial solution we simply set the
velocity and pressure to zero. With an initial solution we are ready
to start iterating.
However, for convergence it is necessary to add some underrelaxation $\alpha$,
and update the solution each time step as

$$
\begin{align*}
\hat{\boldsymbol{u}}^{k+1} &= \alpha \hat{\boldsymbol{u}}^* + (1-\alpha)\hat{\boldsymbol{u}}^{k},\\
\hat{p}^{k+1} &= \alpha \hat{p}^* + (1-\alpha)\hat{p}^{k},
\end{align*}
$$

where $\hat{\boldsymbol{u}}^*$ and $\hat{p}^*$ are the newly computed velocity
and pressure returned from `M.solve`. Without underrelaxation the solution
will quickly blow up. The iteration loop goes as follows

```
converged = False
count = 0
alfa = 0.5
while not converged:
    count += 1
    bh_hat = compute_rhs(ui_hat, bh_hat)
    uh_new = sol(bh_hat, u=uh_new, constraints=((2, 0, 0),))
    error = np.linalg.norm(ui_hat-ui_new)
    uh_hat[:] = alfa*uh_new + (1-alfa)*uh_hat
    converged = abs(error) < 1e-8 or count >= 100
    print('Iteration %d Error %2.4e' %(count, error))

up = uh_hat.backward()
u, p = up

X = V0.local_mesh(True)
plt.figure()
plt.quiver(X[0], X[1], u[0], u[1])
```

Note that the `constraints=((2, 0, 0),)` keyword argument
ensures that the pressure integrates to zero, i.e., $\int_{\Omega} p \omega dxdy=0$.
Here the number 2 tells us that block component 2 in the mixed space
(the pressure) should be integrated, dof 0 should be fixed, and it
should be fixed to 0.

The last three lines plots velocity vectors, like also seen in the [figure](#fig:drivencavity)
in the top of this demo. The solution is apparently nice
and smooth, but hidden underneath are Gibbs oscillations from the
corner discontinuities. This is painfully obvious when switching from
Legendre to Chebyshev polynomials. With Chebyshev the same plot looks
like the [Figure](#fig:drivencavitycheb) below. However, choosing instead the
regularized lid, with no discontinuities, the solutions will be nice and
smooth, both for Legendre and Chebyshev polynomials.

Velocity vectors for Re=100 using Chebyshev.

## Complete solver

A complete solver can be found in demo [NavierStokesDrivenCavity.py](https://github.com/spectralDNS/shenfun/blob/master/demo/NavierStokesDrivenCavity.py).


#  Step-by-step NMO correction

Devito is equally useful as a framework for other stencil computations in general; for example, computations where all array indices are affine functions of loop variables. The Devito compiler is also capable of generating
arbitrarily nested, possibly irregular, loops. This key feature is needed to support many complex algorithms that are used in engineering and scientific practice, including applications from image processing, cellular automata, and machine-learning. This tutorial, a step-by-step NMO correction, is an example of it.

In reflection seismology, normal moveout (NMO) describes the effect that the distance between a seismic source and a receiver (the offset) has on the arrival time of a reflection in the form of an increase of time with offset. The relationship between arrival time and offset is hyperbolic.

Based on the field geometry information, each individual trace is assigned to the midpoint between the shot and receiver locations associated with that trace. Those traces with the same midpoint location are grouped together, making up a common midpoint gather (CMP).

Consider a reflection event on a CMP gather. The difference between the two-way time at a given offset and the two-way zero-offset time is called normal moveout (NMO). Reflection traveltimes must be corrected for NMO prior to summing the traces in the CMP gather along the offset axis. The normal moveout depends on velocity above the reflector, offset, two-way zero-offset time associated with the reflection event, dip of the reflector, the source-receiver azimuth with respect to the true-dip direction, and the degree of complexity of the near-surface and the medium above the reflector.

# Seismic modelling with devito

Before the NMO corretion we will describe a setup of seismic modelling with Devito in a simple 2D case. We will create a physical model of our domain and define a multiple source and an according set of receivers to model for the forward model. But first, we initialize some basic utilities.

```python
import numpy as np
import sympy as sp

from devito import *
```

We will create a simple velocity model here by hand for demonstration purposes. This model essentially consists of three layers, each with a different velocity: 1.5km/s in the top layer, 2.5km/s in the middle layer and 4.5 km/s in the bottom layer.

```python
#NBVAL_IGNORE_OUTPUT
from examples.seismic import Model, plot_velocity

shape = (301, 501)  # Number of grid point (nx, ny, nz)
spacing = (10., 10)  # Grid spacing in m. The domain size is now 3km by 5km
origin = (0., 0)  # What is the location of the top left corner.

# Define a velocity profile. The velocity is in km/s
v = np.empty(shape, dtype=np.float32)
v[:,:100] = 1.5
v[:,100:350] = 2.5
v[:,350:] = 4.5

# With the velocity and model size defined, we can create the seismic model that
# encapsulates these properties. We also define the size of the absorbing layer as 10 grid points
model = Model(vp=v, origin=origin, shape=shape, spacing=spacing, space_order=4, nbl=40)

plot_velocity(model)
```

Next we define the positioning and the wave signal of our source, as well as the location of our receivers. To generate the wavelet for our sources we require the discretized values of time that we are going to use to model a multiple "shot", which depends on the grid spacing used in our model. We will use one source and eleven receivers. The source is located in the position (550, 20). The receivers start at (550, 20) with an even horizontal spacing of 100m at consistent depth.

```python
from examples.seismic import TimeAxis

t0 = 0.  # Simulation starts a t=0
tn = 2400.  # Simulation last 2.4 second (2400 ms)
dt = model.critical_dt  # Time step from model grid spacing

time_range = TimeAxis(start=t0, stop=tn, step=dt)

nrcv = 250 # Number of Receivers
```

```python
#NBVAL_IGNORE_OUTPUT
from examples.seismic import RickerSource

f0 = 0.010  # Source peak frequency is 10Hz (0.010 kHz)
src = RickerSource(name='src', grid=model.grid, f0=f0,
                   npoint=1, time_range=time_range)

# Define the wavefield with the size of the model and the time dimension
u = TimeFunction(name="u", grid=model.grid, time_order=2, space_order=4)

# We can now write the PDE
pde = model.m * u.dt2 - u.laplace + model.damp * u.dt
stencil = Eq(u.forward, solve(pde, u.forward))

src.coordinates.data[:, 0] = 400 # Source coordinates
src.coordinates.data[:, -1] = 20.  # Depth is 20m
```

```python
#NBVAL_IGNORE_OUTPUT
from examples.seismic import Receiver

rec = Receiver(name='rec', grid=model.grid, npoint=nrcv, time_range=time_range)
rec.coordinates.data[:,0] = np.linspace(src.coordinates.data[0, 0], model.domain_size[0], num=nrcv)
rec.coordinates.data[:,-1] = 20.  # Depth is 20m

# Finally we define the source injection and receiver read function to generate the corresponding code
src_term = src.inject(field=u.forward, expr=src * dt**2 / model.m)

# Create interpolation expression for receivers
rec_term = rec.interpolate(expr=u.forward)

op = Operator([stencil] + src_term + rec_term, subs=model.spacing_map)
op(time=time_range.num-1, dt=model.critical_dt)
```

    Operator `Kernel` run in 1.25 s

How we are modelling a horizontal layers, we will group this traces and made a NMO correction using this set traces.

```python
offset = []
data = []
for i, coord in enumerate(rec.coordinates.data):
    off = (src.coordinates.data[0, 0] - coord[0])
    offset.append(off)
    data.append(rec.data[:,i])
```

Auxiliary function for plotting traces:

```python
#NBVAL_IGNORE_OUTPUT
import matplotlib as mpl
import matplotlib.pyplot as plt
from matplotlib import cm
from mpl_toolkits.axes_grid1 import make_axes_locatable

mpl.rc('font', size=16)
mpl.rc('figure', figsize=(8, 6))

def plot_traces(rec, xb, xe, t0, tn, colorbar=True):
    scale = np.max(rec)/100
    extent = [xb, xe, 1e-3*tn, t0]
    plot = plt.imshow(rec, cmap=cm.gray, vmin=-scale, vmax=scale, extent=extent)
    plt.xlabel('X position (km)')
    plt.ylabel('Time (s)')

    # Create aligned colorbar on the right
    if colorbar:
        ax = plt.gca()
        divider = make_axes_locatable(ax)
        cax = divider.append_axes("right", size="5%", pad=0.05)
        plt.colorbar(plot, cax=cax)
    plt.show()
```

# Common Midpoint Gather

At this point, we have a dataset composed of the receivers. "If our model wasn't purely horizontal, we would have to sort these traces by common midpoints prior to NMO correction."

```python
plot_traces(np.transpose(data), rec.coordinates.data[0][0]/1000, rec.coordinates.data[nrcv-1][0]/1000, t0, tn)
```

# NMO Correction

We can correct the measured traveltime of a reflected wave $t$ at a given offset $x$ to obtain the traveltime at normal incidence $t_0$ by applying the following equation:

\begin{equation*}
t = \sqrt{t_0^2 + \frac{x^2}{V_{nmo}^2}}
\end{equation*}

in which $V_{nmo}$ is the NMO velocity. This equation results from the Pythagorean theorem, and is only valid for horizontal reflectors. There are variants of this equation with different degrees of accuracy, but we'll use this one for simplicity.

For the NMO Correction we use a grid of size samples x traces.

```python
ns = time_range.num # Number of samples in each trace
grid = Grid(shape=(ns, nrcv)) # Construction of grid with samples X traces dimension
```

In this example we will use a constant velocity guide. The guide will be arranged in a SparseFunction with the number of points equal to number of samples in the traces.

```python
vnmo = 1500
vguide = SparseFunction(name='v', grid=grid, npoint=ns)
vguide.data[:] = vnmo
```

The computed offset for each trace will be arraged in another SparseFunction with number of points equal to number of traces.

```python
off = SparseFunction(name='off', grid=grid, npoint=nrcv)
off.data[:] = offset
```

The previous modelled traces will be arranged in a SparseFunction with the same dimensions as the grid.

```python
amps = SparseFunction(name='amps', grid=grid, npoint=ns*nrcv, dimensions=grid.dimensions, shape=grid.shape)
amps.data[:] = np.transpose(data)
```

Now, we define SparseFunctions with the same dimensions as the grid, describing the NMO traveltime equation. The $t_0$ SparseFunction isn't offset dependent, so the number of points is equal to the number of samples.

```python
sample, trace = grid.dimensions

t_0 = SparseFunction(name='t0', grid=grid, npoint=ns, dimensions=[sample], shape=[grid.shape[0]])
tt = SparseFunction(name='tt', grid=grid, npoint=ns*nrcv, dimensions=grid.dimensions, shape=grid.shape)
snmo = SparseFunction(name='snmo', grid=grid, npoint=ns*nrcv, dimensions=grid.dimensions, shape=grid.shape)
s = SparseFunction(name='s', grid=grid, dtype=np.intc, npoint=ns*nrcv, dimensions=grid.dimensions,
                   shape=grid.shape)
```

The Equation relates traveltimes: the one we can measure ($t_0$) and the one we want to know (t). But the data in our CMP gather are actually a matrix of amplitudes measured as a function of time ($t_0$) and offset. Our NMO-corrected gather will also be a matrix of amplitudes as a function of time (t) and offset. So what we really have to do is transform one matrix of amplitudes into the other.

With Equations we describe the NMO traveltime equation, and use the Operator to compute the traveltime and the samples for each trace.

```python
#NBVAL_IGNORE_OUTPUT

dtms = model.critical_dt/1000 # Time discretization in ms
E1 = Eq(t_0, sample*dtms)
E2 = Eq(tt, sp.sqrt(t_0**2 + (off[trace]**2)/(vguide[sample]**2) ))
E3 = Eq(s, sp.floor(tt/dtms))
op1 = Operator([E1, E2, E3])
op1()
```

    Operator `Kernel` run in 0.01 s

With the computed samples, we remove all that are out of the samples range, and shift the amplitude for the correct sample.

```python
#NBVAL_IGNORE_OUTPUT

s.data[s.data >= time_range.num] = 0
E4 = Eq(snmo, amps[s[sample, trace], trace])

op2 = Operator([E4])
op2()

stack = snmo.data.sum(axis=1) # We can stack traces and create a ZO section!!!

plot_traces(snmo.data, rec.coordinates.data[0][0]/1000, rec.coordinates.data[nrcv-1][0]/1000, t0, tn)
```

# References:

    https://library.seg.org/doi/full/10.1190/tle36020179.1
    https://wiki.seg.org/wiki/Normal_moveout
    https://en.wikipedia.org/wiki/Normal_moveout


# Vibration modes of a membrane in parabolic coordinates

```python
%matplotlib notebook
```

```python
import numpy as np
from scipy.linalg import eigh
from sympy import (symbols, lambdify, init_printing,
                   expand, Matrix, diff, integrate)
from sympy.utilities.lambdify import lambdify
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
```

```python
x, y, r, s= symbols('x y r s')
init_printing()
```

## Boundary conditions

The boundary conditions are satisfied multiplying by $b(x, y)$.

```python
b = lambda x, y: (2*y - x**2 + 1)*(2*y - 1 + x**2)
bound = b(x, y)
```

```python
b_num = lambdify((x,y), bound, "numpy")
X, Y = np.mgrid[-1:1:200j, -0.5:0.5:200j]
bound_num = b_num(X,Y)
```

```python
fig = plt.figure(figsize=(6, 3))
plt.contourf(X, Y, bound_num, 12, cmap="RdYlBu", vmin=-1, vmax=1)
plt.contour(X, Y, bound_num, [0], colors="black")
plt.axis("image");
```

    <IPython.core.display.Javascript object>

## Approximating functions

```python
def w_fun(x, y, m, n):
    """ Trial function. """
    c = symbols('c:%d' % (m*n)) # This is the way of define the coefficients c_i
    w = []
    for i in range(0, m):
        for j in range(0, n):
            w.append(x**i * y**j)

    return w, c

def u_fun(x, y, m, n):
    """ Complete function. Contains the boundary and trial functions. """
    w, c = w_fun(x, y, m, n)
    return [b(x, y) * phi for phi in w ], c

m = 10
n = 9
u, c = u_fun(x, y, m, n)
```

## Matrices and solution

```python
dudx = [diff(u[k], x) for k in range(len(c))]
dudy = [diff(u[k], y) for k in range(len(c))]
```

```python
Kaux = Matrix(m*n, m*n, lambda ii, jj: dudx[ii]*dudx[jj] + dudy[ii]*dudy[jj])
Maux = Matrix(m*n, m*n, lambda ii, jj: u[ii]*u[jj])
K = Matrix(m*n, m*n, lambda i,j: 0)
M = Matrix(m*n, m*n, lambda i,j: 0)
```

The integrals should be of the form

$$B_{ij} = \int\limits_{-1}^1\int\limits_{\frac{1}{2}(x^2 - 1)}^{\frac{1}{2}(1 - x^2)}
  A_{ij} \mathrm{d}y\, \mathrm{d}x\, ,$$

```python
for row in range(m*n):
    for col in range(row, m*n):
        K_inte = Kaux[row, col]
        M_inte = Maux[row, col]
        K_inte = integrate(K_inte, (y, (x**2 - 1)/2, (1 - x**2)/2), (x, -1, 1))
        M_inte = integrate(M_inte, (y, (x**2 - 1)/2, (1 - x**2)/2), (x, -1, 1))
        K[row, col] += K_inte
        M[row, col] += M_inte
        if row != col:
            K[col, row] += K_inte
            M[col, row] += M_inte
```

```python
Kn = np.array(K).astype(np.float64)
Mn = np.array(M).astype(np.float64)
```

```python
vals, vecs = eigh(Kn, Mn, eigvals=(0,8))
vals
```

    array([ 16.10095349,  30.93334613,  50.33837571,  50.33838703,
            74.38718728,  74.38735248, 102.69815051, 102.75135697,
           105.00010187])

## Visualization of the modes

```python
mask = np.ones_like(X)
mask[bound_num > 0] = np.nan
```

```python
plt.figure(figsize=(8, 8))
for i in range(8):
    U = sum(vecs[j, i]*u[j] for j in range(m*n))
    vecU = lambdify((x,y), U, "numpy")
    Z = vecU(X,Y)*mask
    Z_max = Z.max()
    Z_max = max (Z_max, -Z.min())
    plt.subplot(4, 2, i + 1)
    plt.title(r"$k^2=%.2f$" % vals[i], size=12);
    plt.contour(X, Y, bound_num, [0], colors="black")
    plt.contourf(X, Y, Z, 12, cmap="RdYlBu", vmin=-1.2, vmax=1.2)
    plt.axis("image")
    plt.axis(False)
plt.savefig("../../../Documents/%dx%d.png" % (m, n), dpi=300)
```

    <IPython.core.display.Javascript object>

## Analytic mass matrix

The mass matrix can be integrated analytically.

```python
from sympy import gamma

def mass_coeff(j, k, m, n):
    coeff = (k + n + 2)*(k + n + 4)*(1 + (-1)**(j + m))*(1 + (-1)**(k + n))
    coeff *= gamma(n + k + 1)*gamma((1 + m + j)/2)/gamma((13 + m + j + 2*n + 2*k)/2)
    coeff /= 2**(k + n - 1)
    return coeff
```

```python
mass_coeff(0, 0, 0, 0)
```

```python
vals
```

    array([ 16.10095349,  30.93334613,  50.33837571,  50.33838703,
            74.38718728,  74.38735248, 102.69815051, 102.75135697,
           105.00010187])

```python

```


# 2. Approximate Retrieval

Quickly find nearest neighbors in (very) high dimensions.

Examples:
 * Image search and image completion
 * Song search

## Distance functions

 * $d : S \times S \rightarrow \mathbb{R}$ is a **distance function** iff
     - $\forall s, t \in S : d(s, t) \ge 0$
     - $\forall s : d(s, s) = 0$
     - $\forall s, t \in S : d(s, t) = d(t, s)$
     - $\forall s, t, r \in S: d(s, t) + d(t, r) \ge d(s, r)$ (triangle inequality)
     - if $\forall s, t \in S: d(s, t) = 0 \implies s = t$, then d is a "stronger" function called a **metric**
 * We make use of this by representing **objects as vectors**
     - images become feature vectors (see Computer Vision course)
     - documents become bag-of-words or tf-idf representations
 * Many types of distances
     - $\ell_p$, such as the Euclidean distance ($\ell_2 = \| x' - x \|_2 = \sqrt{\sum_{i = 1}^{D}{(x'_i - x_i)^2}}$)
     - cosine distance (used a lot in text search), $d_{\text{cosine}}(x, x') = \arccos \frac{x^T x'}{\|x\|_2 \|x'\|_2} = \theta$
     - edit distance (expensive)
     - Jaccard-distance (for sets)

## Curse of dimensionality
In very large dimensions, the minimum distance between any two points gets very close to the maximum distance between any points.

$ \lim_{D \rightarrow \infty} P[d_{max} \le (1 + \epsilon)d_{min}] = 1 $

## Approximate retrieval
### Input
A data set $S$ and a distance function $d$.

### Problem 1: Nearest neighbor
Given $q$, find $s* = \text{argmin}_{s \in S} d(q, s)$

### Problem 2: Near-duplicate detection
Find all $s$, $s'$ in $S$, with distance at most $\epsilon$.

* Use **(word) shingling** and **Jaccard distance** as a similarity measure.
* Can even hash shingles to save space
* Jaccard similarity: $JSim(A, B) = \frac{|A \cap B|}{|A \cup B|}$
* Jaccard distance: $d(A, B) = 1 - JSim(A, B)$

```python
def jaccard_sim(a, b):
    """The Jaccard similarity of the two sets a and b."""
    return len(a & b) * 1.0 / len(a | b)

def jaccard_distance(a, b):
    return 1 - jaccard_sim(a, b)

x = {1, 5, 6, 10}
y = {2, 5, 6, 20}
print("Similarity: %.2f" % jaccard_sim(x, y))
print("Distance:   %.2f" % jaccard_distance(x, y))
```

    Similarity: 0.33
    Distance:   0.67

* Scale remains problematic; we can't just do a double loop over all $N$ elements...
* Hashing works well for exact duplicates, can it work with near duplicates?
* **Yes**, we have **locality sensitive hashing** (LSH)

## Min-hashing
Reorder shingle matrix rows with random permutation $\pi$

$\operatorname{hash}(C) =$ minimum row number in which permuted column contains a one (C represents a column, i.e. a document in shingle form)

$\operatorname{hash}(C) = h_\pi(C) = \min_{i:C(i)=1}\pi(i)$ (hence the **min** in min-hashing)

Turns out that the probability of two documents sharing a hash is equal to their Jaccard similarity: $P[h(C_1) = h(C_2)] = Sim(C_1, C_2)$ (trivial but interesting proof; see slides).

This means we can **use many hash functions**, see how often they clash for a pair of documents, and we have a decent estimate for the documents' Jaccard similarity.

An alternative to min-hashing is sim-hashing (see Information Retrieval)

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
shingle_matrix = np.array([
    [1, 0, 1, 0],
    [1, 0, 0, 1],
    [0, 1, 0, 1],
    [0, 1, 0, 1],
    [0, 1, 0, 1],
    [1, 0, 1, 0],
    [1, 0, 1, 0]
])
```

* Want to find all duplicates with > 90% similarity
* Apply min-hash to all documents and look for candidate pairs (documents hashed to same bucket)
    - we find 90% of 90%-duplicates (since the probability of a single hash function colliding is 90% by default, since Pr = Sim)
    - we miss 10% of 90%-duplicates
    - $P(\text{miss with 1 function }) = 1 - s$ ($s$ = similarity)
    - $P(\text{miss with k functions}) = (1 - s)^k$
    - multiple hash functions $\implies$ exponentially fewer misses!

```python
def min_hash(matrix, permutation):
    signature = [None for col in matrix[0]]
    for index, row in enumerate(permutation):
        for col, byte in enumerate(signature):
            if byte is None and matrix[row][col] == 1:
                signature[col] = index

    return signature

def array_sim(a1, a2):
    in_common = len(a1[a1 == a2])
    total = len(a1)
#     print("In common: %d" % in_common)
#     print("Total:     %d" % total)
    return in_common * 1.0 / total

def col_col_sim(matrix, c1_index, c2_index):
    tr = matrix.T
    c1 = tr[c1_index]
    c2 = tr[c2_index]
    return array_sim(c1, c2)

def sig_sig_sim(matrix, c1_index, c2_index, hash_fns):
    sig_matrix = np.array([min_hash(matrix, hf) for hf in hash_fns])
    tr = sig_matrix.T
    c1 = tr[c1_index]
    c2 = tr[c2_index]
    return array_sim(c1, c2)

```

```python
# Same permutations as in slides, just 0-indexed and reversed
p1 = [0, 4, 1, 6, 5, 3, 2]
p2 = [2, 1, 3, 0, 6, 4, 5]
p3 = [4, 5, 0, 1, 6, 3, 2]
print(min_hash(shingle_matrix, p1))
print(min_hash(shingle_matrix, p2))
print(min_hash(shingle_matrix, p3))
```

    [0, 1, 0, 1]
    [1, 0, 3, 0]
    [1, 0, 1, 0]

```python
hash_fns = [p1, p2, p3]
pairs = [(0, 2), (1, 3), (0, 1), (2, 3)]
for c1, c2 in pairs:
    print("Columns %d and %d:" % (c1 + 1, c2 + 1))
    print("Col/col similarity: %.2f" % col_col_sim(shingle_matrix, c1, c2))
    print("Sig/sig similarity: %.2f" % sig_sig_sim(shingle_matrix, c1, c2, hash_fns))
```

    Columns 1 and 3:
    Col/col similarity: 0.86
    Sig/sig similarity: 0.67
    Columns 2 and 4:
    Col/col similarity: 0.86
    Sig/sig similarity: 1.00
    Columns 1 and 2:
    Col/col similarity: 0.00
    Sig/sig similarity: 0.00
    Columns 3 and 4:
    Col/col similarity: 0.00
    Sig/sig similarity: 0.00

## Scaling up min-hash
While the above method works well, it's quite problematic to actually permute a huge dataset

We therefore want to implement the **permutations as linear hash funtions** (see your local linear algebra course for more info about why this works!).

\begin{equation}
    \pi(r) = a \cdot r + b \operatorname{mod} n =: h_{a,b}(r), \> a, b \> \text{random}
\end{equation}

*This is huge!* It allows us to iteratively process any shingle matrix (row-wise), without having to actually move (huge) data around (necessary for performing actual permutations).

Moreover, $h_{a,b}$ is efficient to store.

Note: on slide 17, try to really figure out the prof's corrections, if time allows it.

## Increasing the accuracy
 * More hash functions, any collision $\implies$ similar leads to **fewer false negatives**, but **more false positives**

```python
index = np.arange(0, 1, 0.01)
curves = {'k = %d' % k: 1 - (1 - index) ** k for k in range(1, 20, 2)}
fr = pd.DataFrame(data=curves, index=pd.Index(index))
```

```python
ax = fr.plot()
plt.xlabel("Similarity")
plt.ylabel("P(hit)")
```

 * At this point, we would still need to do pairwise comparison of columns in the signature matrix and see whether they are close enough. Even though we would typically use far fewer hash functions than we would have shingles, we're still talking about $\mathcal{O}(N^2)$ operations.
 * Let's hash some more!
 * Hash entire columns (i.e. sets of many min-hashes of the same document) and bucketize; only evaluate similarity of the documents which share a bucket.
 * Split matrix **rows into $b$ bands** of $r$ rows. So each band contains a few ($r$) hashes for **all** documents.
 * We do a bucketization in each band.
 * (Just use a linear hash function to hash the partial signature for every band, for every document.)
 * $h_{a_i, b}(s) = \sum\limits_{i=1}^{r}a_is_i + b \operatorname{mod} n$, with $a_1, \dots, a_r$ and $b$ random
 * Each band is independent (i.e. its own hash table)
 * If there's a collision between two documents in at least one band, treat the documents as candidates. Why is this banding useful?

## Analysis of partitioning
 * So why do the banding at all?
 * $C_i = [B_{i, 1}, \dots, B_{i, b}]$, a document represented as a bunch of bands (each containing $r$ signatures, so $r$ numbers, essentially)
 * $P\big(h(C_{1,j}) = h(C_{2, j})\big) = s^r$ (pow(similarity, band length)), probability of collision in fixed band $j$; (basic probability theory, since every band has $r$ elements, and the probability of an individual collision is equal to our target similarity, as explained earlier in this course)
 * $P(\text{no collision on band j}) = 1 - s^r$
 * $P(\text{no collision on any band}) = (1 - s^r)^b$ (basic probability theory)
 * $P(C_1 \> \text{and} \> C_2 \> \text{are candidate pair}) = P(\text{collision on any band}) = 1 - (1 - s^r)^b$
 * This is a versatile function; mess around with $r$ and $b$ to produce your optimal curve! (see slides for pretty graphs)
 * Prefer false positives, and perform more detailed checks when necessary.

 * Key insight: use a hash function to hash similar items to same bucket.
 * Use multiple hash functions in clever ways (e.g. using banding) to **boost** the gap between similar and non-similar pairs.
 * See formal definition and $(d_1, d_2, p_1, p_2)$-sensitivity

 * How to generalize what we did above for Jaccard distance?
 * AND/OR construction
 * r-way AND $\implies$ decrease false positives (want ALL match) $(d_1, d_2, p_1, p_2) \rightarrow (d_1, d_2, p_1^r, p_2^r)$
 * b-way OR $\implies$ decrease false negatives (want ANY match) $(d1, d2, p_1, p_2) \rightarrow (d_1, d_2, 1 - (1 - p_1)^b, 1 - (1 - p_2)^b)$
 * Combine in either order. For our Jaccard-similarity example above, this meant r-way AND first (want identical match between all band components, to consider bands as candidate pairs), then a b-way OR (we cared about at least one band matching between two documents)

* Can do even more complex combindations.
* Total number of hash functions is the product of all $r$ and $b$ values. A $(4, 4)$ OR-AND followed by a $(4, 4)$ AND-OR uses $4 \cdot 4
\cdot 4 \cdot 4$ hash functions (think bands within bands).

TODO: Numeric example from slides maybe.

## LSH for other hash functions
 * Can use with cosine distance by fixing a hyperplane (by its unit-length normal $w \in \mathbb{R}^d$) and just seeing on which side of it a feature vector is.
 * Namely: $h_w(v) = \operatorname{sign}(w^Tv)$
 * But more on this in a future lecture (Active Learning)

```python

```


# Simple UWB only model

This model was developped when I realized that the simple model had two important issues:

1. First, it supposed that we knew the rotation between the world frame and the earth magnetic frame, as the beacons' coordinates are in world frame, but the attitude of the robot is in earth magnetic frame.
2. The model had a lot of different variances to tune, making the tuning hard to do.

This model removes those issues by relying only on the UWB radio ranging for measurement, and no prediction step.

## Model
The state contains only the position of the robot:

\begin{equation}
\mathbf{x} = \begin{pmatrix}
    x\\
    y
    \end{pmatrix}
\end{equation}

There is no prediction, therefore the prediction step is simply:

\begin{equation}
\mathbf{x}_{k+1} = \mathbf{x}_{k}
\end{equation}

For the measurement, the UWB system gives us the distance $d$ to a beacon.
The beacon's position $\mathbf{b}$ is known and assumed to be fixed.
Therefore the measurement model becomes:

\begin{equation}
h(\mathbf{x}, \mathbf{b}) = \sqrt{(x - b_x)^2 + (y - b_y)^2}
\end{equation}

## Calibrating the variance

To compute the variance of the model, we estimate that the robot is moving at a constant speed of maximum $V_{max}$ between two measurements updates (which occurs at $f_{UWB}$).
Therefore, the maximum distance that a robot can do is given by:

\begin{equation}
d = \frac{V_{max}}{f}
\end{equation}

If we assume that $d = 2 \sigma$, this means that our hypothesis is valid 97.5% of the time.
Therefore, the variance is given by

\begin{equation}
\sigma^2 = \left( \frac{V_{max}}{2 f} \right)^2
\end{equation}

In our example below, if we take $f = 10$ and $V_{max} = 1$, this means that sigma is:

```python
vmax = 1
f = 200
f_uwb = 10
variance = (vmax / (2 * f_uwb))**2
print("Variance = {:.5f}".format(variance))
```

    Variance = 0.00250

For the measurement variance, we measured a few hundred points and got a standard deviation of about 3 cm.

## Simulation

```python
import sympy as sp
import numpy as np
import matplotlib.pyplot as plt
import trajectories
import ekf
import functools

%matplotlib inline
sp.init_printing()
```

```python
# defines a few variables
x, y = sp.symbols('x y')
state = sp.Matrix([x, y])
dt = sp.symbols('Delta_t')
b_x, b_y = sp.symbols('b_x b_y') # beacon position
```

```python
g = state
h_b = sp.Matrix([sp.sqrt((x - b_x)**2 + (y - b_y)**2)])

BEACON_POS = [
    (-1.5, 0),
    (1.5, 1),
    (1.5, -1),
]
```

```python
def simulate_trajectory(process_variance, measurement_variance, f=200, N=10000):
    # Empty control input
    u = sp.Matrix([])

    # Prepares the numerical version of the update functions
    g_num = sp.lambdify([state, u], g, 'numpy')
    G_num = sp.lambdify([state, u], g.jacobian(state), 'numpy')
    h_num = sp.lambdify([[b_x, b_y], state], h_b, 'numpy')
    H_num = sp.lambdify([[b_x, b_y], state], h_b.jacobian(state), 'numpy')

    # Sample covariance for the model
    Q_b = np.array([[measurement_variance]])
    R = np.eye(2) * process_variance

    # Prepare the kalman filter
    predictor = ekf.Predictor(g_num, G_num, R)
    correctors = [
        ekf.Corrector(functools.partial(h_num, pos),
                      functools.partial(H_num, pos),
                      Q_b
                     )
    for pos in BEACON_POS]

    mu = np.array([0, 0]) # initial state
    sigma = np.diag([1, 1])

    x, xhat = [], []
    y, yhat = [], []
    ts = []

    for i, p in zip(range(N), trajectories.generate_circular_traj(1, np.deg2rad(10), 1/f)):
        # feeds the input into Kalman
        filter_input = np.array([])
        mu, sigma = predictor(mu, sigma, filter_input)

        if i % (f / f_uwb) == 0:
            for corrector in correctors:
                z = corrector.h(np.array([p.pos[0], p.pos[1]]))
                z += np.random.normal(0, 0.03)
                mu, sigma = corrector(mu, sigma, z)

        # Saves the data
        ts.append(p.timestamp)
        x.append(p.pos[0])
        xhat.append(mu[0])
        y.append(p.pos[1])
        yhat.append(mu[1])

    return ts, x, y, xhat, yhat
```

```python
ts, x, y, xhat, yhat = simulate_trajectory(process_variance=0.00250, measurement_variance=0.03**2)
```

```python
plt.plot(x, y)
plt.plot(xhat, yhat)
plt.plot([x for x, y in BEACON_POS],[y for x, y in BEACON_POS], 'x')
plt.legend(('Simulated trajectory', 'EKF output', 'anchors'))
plt.title('trajectory')
plt.xlabel('x [m]')
plt.ylabel('y [m]')
plt.gcf().savefig('uwb_only_trajectory.pdf')
plt.show()

error = [np.sqrt((x-xh)**2+(y-yh)**2) for x,xh,y,yh in zip(x, xhat,y,yhat)]
plt.plot(ts, error)
plt.xlabel('time [s]')
plt.ylabel('error [m]')
plt.ylim(0, 0.1)
plt.title('Position error (RMS = {:.3f} m)'.format(np.mean(error)))
plt.gcf().savefig('uwb_only_error.pdf')
plt.show()
```

```python
def cost(variances):
    process_variance = variances[0]
    measurement_variance = variances[1]
    _, x, xh, y, yh = simulate_trajectory(process_variance, measurement_variance, N=100)

    return sum(np.sqrt((x-xh)**2+(y-yh)**2) for x,xh,y,yh in zip(x, xhat,y,yhat))
```

```python
x0 = np.array([0.1**2, 0.03**2])
cost(x0)
```

    array([ 96.17540632])

```python
# We tried to optimize it but its not very efficient
# scipy.optimize.basinhopping(cost, x0, niter=10)
```

```python

```


# Support Vector Machines

    Notebook version: 1.0 (Oct 28, 2015)
                      1.1 (Oct 27, 2016)
                      2.0 (Nov 2, 2017)

    Authors: Jesús Cid Sueiro (jcid@tsc.uc3m.es)
             Jerónimo Arenas García (jarenas@tsc.uc3m.es)

    Changes:
    v.1.0 - First version
    v.1.1 - Typo correction and illustrative figures for linear SVM
    v.2.0 - Compatibility with Python 3 (backcompatible with Python 2.7)

This notebook is a compilation of material taken from several sources:

- The  sklearn documentation
- A  notebook  by  Jake Vanderplas
-  Wikipedia

```python
from __future__ import print_function

# To visualize plots in the notebook
%matplotlib inline

# Imported libraries
#import csv
#import random
#import matplotlib
import matplotlib.pyplot as plt
from mpl_toolkits import mplot3d
#import pylab

import numpy as np
#from sklearn.preprocessing import PolynomialFeatures
from sklearn import svm
from sklearn.datasets.samples_generator import make_blobs
from sklearn.datasets.samples_generator import make_circles

from ipywidgets import interact
```

## 1. Introduction

  [Source:  sklearn documentation ]

Support vector machines (SVMs) are a set of supervised learning methods used for classification, regression and outliers detection.

The advantages of support vector machines are:

- Effective in high dimensional spaces.
- Still effective in cases where number of dimensions is greater than the number of samples.
- Uses a subset of training points in the decision function (called support vectors), so it is also memory efficient.
- Versatile: different Kernel functions can be specified for the decision function.

The disadvantages of support vector machines include:

- SVMs do not directly provide probability estimates.

## 2. Motivating Support Vector Machines

  [Source:  notebook  by  Jake Vanderplas]

Support Vector Machines (SVMs) are a kind of ***discriminative*** classifiers: that is, they draw a boundary between clusters of data without making any explicit assumption about the probability model underlying the data generation process.

Let's show a quick example of support vector classification. First we need to create a dataset:

```python
X, y = make_blobs(n_samples=50, centers=2, random_state=0, cluster_std=0.60)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='copper')
plt.xlabel("$x_0$", fontsize=14)
plt.ylabel("$x_1$", fontsize=14)
plt.axis('equal')
plt.show()
```

A discriminative classifier attempts to draw a line between the two sets of data. Immediately we see a problem: such a line is ill-posed! For example, we could come up with several possibilities which perfectly discriminate between the classes in this example:

```python
xfit = np.linspace(-1, 3.5)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='copper')

for m, b in [(1, 0.65), (0.5, 1.6), (-0.2, 2.9)]:
    plt.plot(xfit, m * xfit + b, '-k')

plt.xlim(-1, 3.5);
plt.xlabel("$x_0$", fontsize=14)
plt.ylabel("$x_1$", fontsize=14)
plt.axis('equal')
plt.show()
```

These are three very different separators which perfectly discriminate between these samples. Depending on which you choose, a new data point will be classified almost entirely differently! How can we improve on this?

Support Vector Machines (SVM) select the boundary decision maximizing the ***margin***. The margin of a classifier is defined as twice the maximum signed distance between the decision boundary and the training data. By *signed* we mean that the distance to misclassified samples is counted negatively. Thus, if the classification problem is "separable" (i.d. if there exist a decision boundary with zero errors in the training set), the SVM will choose the zero-error decision boundary that is "as far as possible" from the training data.

In summary, what a SVM does is to not only draw a line, but consider the "sample free" region about the line. Here's an example of what it might look like:

```python
xfit = np.linspace(-1, 3.5)
plt.scatter(X[:,0], X[:,1], c=y, s=50, cmap='copper')

for m, b, d in [(1, 0.65, 0.33), (0.5, 1.6, 0.55), (-0.2, 2.9, 0.2)]:
    yfit = m * xfit + b
    plt.plot(xfit, yfit, '-k')
    plt.fill_between(xfit, yfit-d, yfit+d, edgecolor='none',
                     color='#AAAAAA', alpha=0.4)

plt.xlim(-1, 3.5)
plt.xlabel("$x_0$", fontsize=14)
plt.ylabel("$x_1$", fontsize=14)
plt.axis('equal')
plt.show()
```

Notice here that if we want to maximize this width, the middle fit is clearly the best. This is the intuition of the SVM, which optimizes a linear discriminant model in conjunction with a margin representing the perpendicular distance between the datasets.

## 3. Linear SVM

  [Source: adapted from  Wikipedia ]

In order to present the SVM in a formal way, consider a training dataset $\mathcal{S} = \left\{ (\mathbf{x}^{(k)}, y^{(k)}) \mid \mathbf{x}^{(k)}\in \Re^N,\, y^{(k)} \in \{-1,1\}, k=1,\ldots, K\right\}$, where the binary symmetric label $y^{(k)}\in \{-1,1\}$ indicates the class to which the point $\mathbf{x}^{(k)}$ belongs. Each $\mathbf{x}^{(k)}$ is a $p$-dimensional real vector. We want to find the maximum-margin hyperplane that divides the points having $y^{(k)}=1$ from those having $y^{(k)}=-1$.

Any hyperplane can be written as the set of points $\mathbf{x}$ satisfying

$$
\mathbf{w}^\intercal \mathbf{x} - b=0,
$$

where ${\mathbf{w}}$ denotes the (not necessarily normalized) normal vector to the hyperplane. The parameter $\tfrac{b}{\|\mathbf{w}\|}$ determines the offset of the hyperplane from the origin along the normal vector ${\mathbf{w}}$.

If the training data are linearly separable, we can select two parallel hyperplanes in a way that they separate the data and there are no points between them, and then try to maximize their distance. The region bounded by them is called "the margin". These hyperplanes can be described by the equations

$$\mathbf{w}^\intercal \mathbf{x} - b=1$$
and
$$\mathbf{w}^\intercal \mathbf{x} - b=-1.$$

Note that the two equations above can represent any two parallel hyperplanes in $\Re^N$. Essentially, the direction of vector $\mathbf{w}$ determines the orientation of the hyperplanes, whereas parameter $b$ and the norm of $\mathbf{w}$ can be used to select their exact location.

To compute the distance between the hyperplanes, we can obtain the projection of vector ${\mathbf x}_1 - {\mathbf x}_2$, where ${\mathbf x}_1$ and ${\mathbf x}_2$ are points from each of the hyperplanes, onto a unitary vector ortonormal to the hyperplanes:

$$\text{Distance between hyperplanes} = \left[\frac{\mathbf{w}}{\|\mathbf{w}\|}\right]^\intercal ({\mathbf x}_1 - {\mathbf x}_2) = \frac{\mathbf{w}^\intercal {\mathbf x}_1 - \mathbf{w}^\intercal {\mathbf x}_2}{\|\mathbf{w}\|} = \frac{2}{\|\mathbf{w}\|}.$$

Therefore, to maximize the distance between the planes we want to minimize $\|\mathbf{w}\|$.

As we also have to prevent data points from falling into the margin, we add the following constraints: for each $k$ either

\begin{align}
\mathbf{w}^\intercal \mathbf{x}^{(k)} - b &\ge +1, \qquad\text{ if } y^{(k)}=1, \qquad \text{or} \\
\mathbf{w}^\intercal \mathbf{x}^{(k)} - b &\le -1, \qquad\text{ if } y^{(k)}=-1.
\end{align}

This can be rewritten as:

$$
y^{(k)}(\mathbf{w}^\intercal \mathbf{x}^{(k)} - b) \ge 1, \quad \text{ for all } 1 \le k \le K.
$$

We can put this together to get the optimization problem:

$$
(\mathbf{w}^*,b^*) = \arg\min_{(\mathbf{w},b)} \|\mathbf{w}\| \\
\text{subject to: }
y^{(k)}(\mathbf{w}^\intercal \mathbf{x}^{(k)} - b) \ge 1, \, \text{ for any } k = 1, \dots, K
$$

This optimization problem is difficult to solve because it depends on $\|\mathbf{w}\|$, the norm of $\mathbf{w}$, which involves a square root. Fortunately it is possible to alter the minimization objective $\|\mathbf{w}\|$ by substituting it with $\tfrac{1}{2}\|\mathbf{w}\|^2$ (the factor of $\frac{1}{2}$ being used for mathematical convenience) without changing the solution (the minimum of the original and the modified equation have the same $\mathbf{w}$ and $b$):

$$
(\mathbf{w}^*,b^*) = \arg\min_{(\mathbf{w},b)} \frac{1}{2}\|\mathbf{w}\|^2 \\
\text{subject to: }
y^{(k)}(\mathbf{w}^\intercal \mathbf{x}^{(k)} - b) \ge 1, \, \text{ for any } k = 1, \dots, K
$$

This is a particular case of a *quadratic programming* problem.

### 3.1. Primal form

The optimization problem stated in the preceding section can be solved by means of a generalization of the Lagrange method of multipliers for inequality constraints, using the so called Karush–Kuhn–Tucker (KKT) multipliers $\boldsymbol{\alpha}$. According to it, the constrained problem can be expressed as

$$(\mathbf{w}^*,b^*, \boldsymbol{\alpha}^*) = \arg\min_{\mathbf{w},b } \max_{\boldsymbol{\alpha}\geq 0 } \left\{ \frac{1}{2}\|\mathbf{w}\|^2 - \sum_{k=1}^{K}{\alpha^{(k)}[y^{(k)}(\mathbf{w}^\intercal \mathbf{x}^{(k)} - b)-1]} \right\}
$$

that is, we look for a *saddle point*.

A key result in convex optimization theory is that, for the kind of optimization problems discussed here (see  here, for instance), the *max* and *min* operators are interchangeable, so that
$$
(\mathbf{w}^*,b^*, \boldsymbol{\alpha}^*) = \arg\max_{\boldsymbol{\alpha}\geq 0 } \min_{\mathbf{w},b } \left\{ \frac{1}{2}\|\mathbf{w}\|^2 - \sum_{k=1}^{K}{\alpha^{(k)}[y^{(k)}(\mathbf{w}^\intercal \mathbf{x}^{(k)} - b)-1]} \right\}
$$

Note that the inner minimization problem is now quadratic in $\mathbf{w}$ and, thus, the minimum can be found by differentiation:
$$
\mathbf{w}^* = \sum_{k=1}^K{\alpha^{(k)} y^{(k)}\mathbf{x}^{(k)}}.
$$

#### 3.1.1. Support Vectors

In view of the optimization problem, we can check that all the points which can be separated as $y^{(k)}(\mathbf{w}^\intercal \mathbf{x}^{(k)} - b) - 1 > 0 $ do not matter since we must set the corresponding $\alpha^{(k)}$ to zero. Therefore, only a few $\alpha^{(k)}$ will be greater than zero. The corresponding $\mathbf{x}^{(k)}$ are known as `support vectors`.

It can be seen that the optimum parameter vector $\mathbf{w}^\ast$ can be expressed in terms of the support vectors only:

$$
\mathbf{w}^* = \sum_{k\in {\cal{S}}_{SV}}{\alpha^{(k)} y^{(k)}\mathbf{x}^{(k)}}.
$$

where ${\cal{S}}_{SV}$ is the set of indexes associated to support vectors.

#### 3.1.2. The computation of $b$

Support vectors lie on the margin and satisfy $y^{(k)}(\mathbf{w}^\intercal \mathbf{x}^{(k)} - b) = 1$. From this condition, we can obtain the value of $b$, since for any support vector:

$$\mathbf{w}^\intercal\mathbf{x}^{(k)} - b = \frac{1}{y^{(k)}} = y^{(k)} \iff b = \mathbf{w}^\intercal\mathbf{x}^{(k)} - y^{(k)}
$$

This estimate of $b$, the centerpoint of the division, depends only on a single pair $y^{(k)}$ and $x^{(k)}$.  We may get a more robust estimate of the center by averaging over all of the $N_{SV}$ support vectors, if we believe the population mean is a good estimate of the midpoint, so in practice, $b$ is often computed as:

$$
b = \frac{1}{N_{SV}} \sum_{\mathbf{x}^{(k)}\in {\cal{S}}_{SV}}{(\mathbf{w}^\intercal\mathbf{x}^{(k)} - y^{(k)})}
$$

### 3.2. Dual form

Writing the classification rule in its unconstrained dual form reveals that the *maximum-margin hyperplane* and therefore the classification task is only a function of the *support vectors*, the subset of the training data that lie on the margin.

Using the fact that $\|\mathbf{w}\|^2 = \mathbf{w}^\intercal \mathbf{w}$ and substituting $\mathbf{w} = \sum_{k=1}^K{\alpha^{(k)} y^{(k)}\mathbf{x}^{(k)}}$, we obtain

\begin{align}
(b^*, \boldsymbol{\alpha}^*)
   &= \arg\max_{\boldsymbol{\alpha}\geq 0 }
   \min_b \left\{
   \sum_{k=1}^{K}\alpha^{(k)} -
   \frac{1}{2}
   \sum_{k=1}^K \sum_{j=1}^K {\alpha^{(k)} \alpha^{(j)} y^{(k)} y^{(j)} (\mathbf{x}^{(k)})^\intercal\mathbf{x}^{(j)}}
   + b \sum_{k=1}^{K}\alpha^{(k)}y^{(k)}
\right\}
\end{align}

Noting that, if $\sum_{k=1}^{K}\alpha^{(k)}y^{(k)} \neq 0$ the optimal value  of $b$ is $+\infty$ of $-\infty$, and

\begin{align}
\min_b \left\{
\sum_{k=1}^{K}\alpha^{(k)} -
\frac{1}{2}
\sum_{k=1}^K \sum_{j=1}^K {\alpha^{(k)} \alpha^{(j)} y^{(k)} y^{(j)} (\mathbf{x}^{(k)})^\intercal\mathbf{x}^{(j)}}
+ b \sum_{k=1}^{K}\alpha^{(k)}y^{(k)}
\right\} = -\infty.
\end{align}

Therefore, any $\boldsymbol{\alpha}$ satifying $\sum_{k=1}^{K}\alpha^{(k)}y^{(k)} \neq 0$ is suboptimal, so that the optimal multipliers must satisfy the condition  $\sum_{k=1}^{K}\alpha^{(k)}y^{(k)} = 0$.

Summarizing, the dual formulation of the optimization problem is

$$
\boldsymbol{\alpha}^* = \arg\max_{\boldsymbol{\alpha}\geq 0} \sum_{k=1}^K \alpha^{(k)} -
       \frac12 \sum_{k,j} \alpha^{(k)} \alpha^{(j)} y^{(k)} y^{(j)} k(\mathbf{x}^{(k)}, \mathbf{x}^{(j)})  \\
\text{subject to: } \qquad \sum_{k=1}^K \alpha^{(k)} y^{(k)} = 0.
$$

where the *kernel* $k(\cdot)$ is defined by $k(\mathbf{x}^{(k)},\mathbf{x}^{(j)})=(\mathbf{x}^{(k)})^\intercal\mathbf{x}^{(j)}$.

Many implementations of the SVM use this dual formulation. They proceed in three steps:

1. Solve the dual problem to obtain $\boldsymbol{\alpha}^*$. Usually, only a small number of $\alpha^{*(k)}$ are nonzero. The corresponding values of ${\bf x}^{(k)}$ are called the *support vectors*.
2. Compute $\mathbf{w}^* = \sum_{k=1}^K{\alpha^{*(k)} y^{(k)}\mathbf{x}^{(k)}}$
3. Compute $b^* = \frac{1}{N_{SV}} \sum_{\alpha^{*(k)}\neq 0}{(\mathbf{w}^{*\intercal}\mathbf{x}^{(k)} - y^{(k)})}
$

## 4. Fitting a Support Vector Machine

  [Source:  notebook  by  Jake Vanderplas]

Now we'll fit a Support Vector Machine Classifier to these points.

```python
clf = svm.SVC(kernel='linear')
clf.fit(X, y)
```

    SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
      decision_function_shape=None, degree=3, gamma='auto', kernel='linear',
      max_iter=-1, probability=False, random_state=None, shrinking=True,
      tol=0.001, verbose=False)

To better visualize what's happening here, let's create a quick convenience function that will plot SVM decision boundaries for us:

```python
def plot_svc_decision_function(clf, ax=None):
    """Plot the decision function for a 2D SVC"""
    if ax is None:
        ax = plt.gca()
    x = np.linspace(plt.xlim()[0], plt.xlim()[1], 30)
    y = np.linspace(plt.ylim()[0], plt.ylim()[1], 30)
    Y, X = np.meshgrid(y, x)
    P = np.zeros_like(X)
    for i, xi in enumerate(x):
        for j, yj in enumerate(y):
            P[i, j] = clf.decision_function(np.array([xi, yj]).reshape(1,-1))
    # plot the margins
    ax.contour(X, Y, P, colors='k',
               levels=[-1, 0, 1], alpha=0.5,
               linestyles=['--', '-', '--'])

plt.scatter(X[:, 0:1], X[:, 1:2], c=y, s=50, cmap='copper')
plot_svc_decision_function(clf);
```

Notice that the dashed lines touch a couple of the points: these points are the *support vectors*. In scikit-learn, these are stored in the ``support_vectors_`` attribute of the classifier:

```python
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='copper')
plot_svc_decision_function(clf)
plt.scatter(clf.support_vectors_[:, 0], clf.support_vectors_[:, 1],
            s=200, facecolors='none');
```

Let's use IPython's interact functionality to explore how the distribution of points affects the support vectors and the discriminative fit. (This is only available in IPython 2.0+, and will not work in a static view)

```python
def plot_svm(N=10):
    X, y = make_blobs(n_samples=200, centers=2,
                      random_state=0, cluster_std=0.60)
    X = X[:N]
    y = y[:N]
    clf = svm.SVC(kernel='linear')
    clf.fit(X, y)
    plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='copper')
    plt.xlim(-1, 4)
    plt.ylim(-1, 6)
    plot_svc_decision_function(clf, plt.gca())
    plt.scatter(clf.support_vectors_[:, 0], clf.support_vectors_[:, 1],
                s=200, facecolors='none')

interact(plot_svm, N=[10, 200], kernel='linear')
```

Notice the unique thing about SVM is that only the support vectors matter: that is, if you moved any of the other points without letting them cross the decision boundaries, they would have no effect on the classification results!

## 5. Non-separable problems.

  [Source: adapted from  Wikipedia ]

In 1995, Corinna Cortes and Vladimir N. Vapnik suggested a modified maximum margin idea that allows for mislabeled examples. If there exists no hyperplane that can split the `positive` and `negative` samples, the `Soft Margin`
 method will choose a hyperplane that splits the examples as cleanly as possible, while still maximizing the distance to the nearest cleanly split examples. The method introduces non-negative slack variables, $\xi^{(k)}$, which measure the degree of misclassification of the data $\mathbf{x}^{(k)}$

$$y^{(k)}(\mathbf{w}^\intercal\mathbf{x}^{(k)} - b) \ge 1 - \xi^{(k)} \quad 1 \le k \le K.
$$

The objective function is then increased by a function which penalizes non-zero $\xi^{(k)}$, and the optimization becomes a trade off between a large margin and a small error penalty. If the penalty function is linear, the optimization problem becomes:

$$(\mathbf{w}^*,\mathbf{\xi}^*, b^*) = \arg\min_{\mathbf{w},\mathbf{\xi}, b } \left\{\frac{1}{2} \|\mathbf{w}\|^2 + C \sum_{k=1}^K \xi^{(k)} \right\} \\
\text{subject to: } \quad y^{(k)}(\mathbf{w}\cdot\mathbf{x_i} - b) \ge 1 - \xi^{(k)}, \quad \xi^{(k)} \ge 0, \quad  k=1,\ldots, K.
$$

This constraint along with the objective of minimizing $\|\mathbf{w}\|$ can be solved using KKT multipliers as done above. One then has to solve the following problem:

$$
\arg\min_{\mathbf{w}, \mathbf{\xi}, b } \max_{\boldsymbol{\alpha}, \boldsymbol{\beta} }
\left\{ \frac{1}{2}\|\mathbf{w}\|^2
+ C \sum_{k=1}^K \xi^{(k)}
- \sum_{k=1}^K {\alpha^{(k)}[y^{(k)}(\mathbf{w}^\intercal \mathbf{x}^{(k)} - b) -1 + \xi^{(k)}]}
- \sum_{k=1}^K \beta^{(k)} \xi^{(k)} \right \}\\
\text{subject to: } \quad
\alpha^{(k)}, \beta^{(k)} \ge 0.
$$

A similar analysis to that in the separable case can be applied to show that the dual formulation of the optimization problem is

$$
        \boldsymbol{\alpha}^* = \arg\max_{0 \leq \alpha^{(k)} \leq C , \, k=1,\ldots,K} \sum_{k=1}^K \alpha^{(k)} -
       \frac12 \sum_{k,j} \alpha^{(k)} \alpha^{(j)} y^{(k)} y^{(j)} k(\mathbf{x}^{(k)}, \mathbf{x}^{(j)})  \\
\text{subject to: } \qquad \sum_{k=1}^K \alpha^{(k)} y^{(k)} = 0.
$$
Note that the only difference with the separable case is given by the constraints $\alpha^{(k)} \leq C$.

## 6. Nonlinear classification

  [Source: adapted from  Wikipedia ]

The original optimal hyperplane algorithm proposed by Vapnik in 1963 was a linear classifier. However, in 1992, Bernhard E. Boser, Isabelle M. Guyon and Vladimir N. Vapnik suggested a way to create nonlinear classifiers by applying the *kernel trick* to maximum-margin hyperplanes. The resulting algorithm is formally similar, except that every dot product is replaced by a nonlinear kernel function. This allows the algorithm to fit the maximum-margin hyperplane in a transformed feature space. The transformation may be nonlinear and the transformed space high dimensional; thus though the classifier is a hyperplane in the high-dimensional feature space, it may be nonlinear in the original input space.

The kernel is related to the transform $\phi(\mathbf{x})$ by the equation $k(\mathbf{x}, \mathbf{x}') = \phi(\mathbf{x})^\intercal \phi(\mathbf{x}')$. However, note that we do not need to explicitly compute $\phi(\mathbf{x})$, as long as we can express all necessary calculations in terms of the kernel function only, as it is the case for the optimization problem in the dual case.

The predictions of the SVM classifier can also be expressed in terms of kernels only, so that we never need to explicitely compute $\phi(\mathbf{x})$.

$$\begin{align}\hat y({\mathbf{x}}) & = {\mathbf {w^\ast}}^\intercal {\mathbf{x}} + b^\ast \\ \\
& = \left[\sum_{k \in {\cal{S}}_{SV}} \alpha^{(k)^*} y^{(k)} \phi(\mathbf{x}^{(k)})\right]^\intercal {\mathbf{x}} + b^\ast  \\ \\
& = b^\ast + \sum_{k \in {\cal{S}}_{SV}} \alpha^{(k)^*} y^{(k)} k(\mathbf{x}^{(k)}, {\mathbf{x}})
\end{align}$$

Some common kernels include:

* **Gaussian**: $k(\mathbf{x},\mathbf{x}')=\exp(-\gamma \|\mathbf{x} - \mathbf{x}'\|^2)$, for $\gamma > 0$. Sometimes parametrized using $\gamma=1/{2 \sigma^2}$. This is by far the most widely used kernel.
* Polynomial (homogeneous): $k(\mathbf{x},\mathbf{x}')=(\mathbf{x}^\intercal \mathbf{x}')^d$
* Polynomial (inhomogeneous): $k(\mathbf{x},\mathbf{x}') = (\mathbf{x}^\intercal \mathbf{x}' + 1)^d$
* Hyperbolic tangent: $k(\mathbf{x},\mathbf{x}') = \tanh(\kappa \mathbf{x}^\intercal \mathbf{x}'+c)$, for some (not every) $\kappa > 0$ and $c < 0$.

### 6.1. Example.

  [Source:  notebook  by  Jake Vanderplas]

Where SVM gets incredibly exciting is when it is used in conjunction with *kernels*.
To motivate the need for kernels, let's look at some data which is not linearly separable:

```python
X, y = make_circles(100, factor=.1, noise=.1)

clf = svm.SVC(kernel='linear').fit(X, y)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='copper')
plot_svc_decision_function(clf);
```

Clearly, no linear discrimination will ever separate these data.
One way we can adjust this is to apply a **kernel**, which is some functional transformation of the input data.

For example, one simple model we could use is a **radial basis function**

```python
r = np.exp(-(X[:, 0] ** 2 + X[:, 1] ** 2))
```

If we plot this along with our data, we can see the effect of it:

```python
def plot_3D(elev=30, azim=30):
    ax = plt.subplot(projection='3d')
    ax.scatter3D(X[:, 0], X[:, 1], r, c=y, s=50, cmap='spring')
    ax.view_init(elev=elev, azim=azim)
    ax.set_xlabel('x')
    ax.set_ylabel('y')
    ax.set_zlabel('r')

interact(plot_3D, elev=[-90, 90], azip=(-180, 180));
```

We can see that with this additional dimension, the data becomes trivially linearly separable!
This is a relatively simple kernel; SVM has a more sophisticated version of this kernel built-in to the process. This is accomplished by using the Gaussian kernel (``kernel='rbf'``), short for *radial basis function*:

```python
clf = svm.SVC(kernel='rbf')
clf.fit(X, y)

plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='copper')
plot_svc_decision_function(clf)
plt.scatter(clf.support_vectors_[:, 0], clf.support_vectors_[:, 1],
            s=200, facecolors='none');
```

Here there are effectively $N$ basis functions: one centered at each point! Through a clever mathematical trick, this computation proceeds very efficiently using the "Kernel Trick", without actually constructing the matrix of kernel evaluations.

** Exercise**: Apply the linear SVM and the SVM with Gaussian kernel to the discrimination of classes `Versicolor` and `Virginica` in the Iris Dataset, using atributes $x_0$ and $x_1$ only. Plot the corresponding decision boundaries, the support vectors.

## 7. Hyperparameters

Note that the SVM formulation has several free parameters (hyperparameters) that must be selected out of the optimization problem:

- The free parameter $C$ used to solve non-separable problems.
- The kernel parameters (for instance, parameter $\gamma$ from the Gaussian kernel.

These parameters are usually adjusted using a cross-validation procedure.


```python
import numpy as np
%matplotlib inline
from mpl_toolkits import mplot3d
import matplotlib.pyplot as plt
import pickle
```

# 8-7. Batch Normalizations

Just like we normalize the input of the network, it turns out that normalizing values before the activation each layer is beneficial for training neural networks. Usually, a batch norm layer is inserted before an activation layer.

The benefits of batch norm are:

1. Faster training via higher learning rate
1. Robustness for initial weight
1. Slight regularization effect

Paper: Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift [arXiv:1502.03167](https://arxiv.org/abs/1502.03167)

With the learnable parameter $\gamma$ and $\beta$ (initially 1 and 0), the forward pass of batch norm is defined as followings:

\begin{align}
\mu      &\leftarrow \frac{1}{m} \sum_{i=1}^{m} x_i \\
\sigma^2 &\leftarrow \frac{1}{m} \sum_{i=1}^{m} (x_i - \mu)^2 \\
\hat{x_i}&\leftarrow \frac{x_i - \mu}{\sqrt{\sigma^2 + \epsilon}} \\
y_i      &\leftarrow \gamma \hat{x_i} + \beta
\end{align}

The backward pass can be derived using computational graph. [Understanding the backward pass through Batch Normalization Layer](https://kratzert.github.io/2016/02/12/understanding-the-gradient-flow-through-the-batch-normalization-layer.html) by Frederik Kratzert has a detailed explanation.

## Five layer net w/ or wo/ batch normalization

* Five layers
* Around 40 neurons
* ReLU
* Random initialization ($10$–$10^{-4}$)
* With or without batch normalization

```python
plt.rcParams['figure.figsize'] = [16, 16]
```

```python
fig, axes = plt.subplots(4, 4)

for row in range(4):
    for col in range(4):
        index = row * 4 + col

        with_log = pickle.load(open('code/08_techniques/08_7_batch_norm/experiment_{}_True.pkl'.format(index), "rb"))
        without_log = pickle.load(open('code/08_techniques/08_7_batch_norm/experiment_{}_False.pkl'.format(index), "rb"))

        current = axes[row, col]
        current.plot(with_log['accuracy_test_itr'], with_log['accuracy_test'], 'b', label='with bn')
        current.plot(without_log['accuracy_test_itr'], without_log['accuracy_test'], 'r', label='without bn')
        current.legend()
        current.set_title('stddev %.5f' % with_log['weight_stddev'])

plt.show()
```