# Problem set 7: Solving the consumer problem with income risk

```python
import numpy as np
import scipy as sp
from scipy import linalg
from scipy import optimize
from scipy import interpolate
import sympy as sm

%matplotlib inline
import matplotlib.pyplot as plt
plt.style.use('seaborn-whitegrid')
from matplotlib import cm
from mpl_toolkits.mplot3d import Axes3D
```

# Tasks

```python
sm.init_printing(use_unicode=True)
```

## Optimization problem I

Consider the function

$$
f(\boldsymbol{x}) = f(x_1,x_2) = (x_1^2 - x_1x_2 + x_2^2)^2
$$

Define it in **sympy** by:

```python
x1 = sm.symbols('x_1')
x2 = sm.symbols('x_2')
f = (x1**2 - x1*x2 + x2**2)**2
f
```

The **Jacobian** is

```python
f1 = sm.diff(f,x1)
f2 = sm.diff(f,x2)
sm.Matrix([f1,f2])
```

The **Hessian**  is

```python
f11 = sm.diff(f,x1,x1)
f12 = sm.diff(f,x1,x2)
f21 = sm.diff(f,x2,x1)
f22 = sm.diff(f,x2,x2)
sm.Matrix([[f11,f12],[f21,f22]])
```

**Question A:** Create a 3D plot and a contour plot of $f(x_1,x_2)$ such as those in the answer below.

```python
_f = sm.lambdify((x1,x2),f)

# write your code here
```

**Answer:**

```python
# a. grids
x1_vec = np.linspace(-2,2,500)
x2_vec = np.linspace(-2,2,500)
x1_grid,x2_grid = np.meshgrid(x1_vec,x2_vec,indexing='ij')
f_grid = _f(x1_grid,x2_grid)

# b. main
fig = plt.figure()
ax = fig.add_subplot(1,1,1,projection='3d')
cs = ax.plot_surface(x1_grid,x2_grid,f_grid,cmap=cm.jet)

# c. add labels
ax.set_xlabel('$x_1$')
ax.set_ylabel('$x_2$')
ax.set_zlabel('$f$')

# d. invert xaxis
ax.invert_xaxis()

# e. add colorbar
fig.colorbar(cs);
```

```python
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
levels = np.sort([j*10**(-i) for i in [-1,0,1,2,3,4] for j in [0.5,1,1.5]])
cs = ax.contour(x1_grid,x2_grid,f_grid,levels=levels,cmap=cm.jet)
fig.colorbar(cs);
```

**Question B:** Construct python functions for the jacobian and the hessian.

```python
f_python = lambda x: _f(x[0],x[1])

# write your code here
```

**Answer:**

```python
_f1 = sm.lambdify((x1,x2),f1)
_f2 = sm.lambdify((x1,x2),f2)
_f11 = sm.lambdify((x1,x2),f11)
_f12 = sm.lambdify((x1,x2),f12)
_f21 = sm.lambdify((x1,x2),f21)
_f22 = sm.lambdify((x1,x2),f22)

def f_jac(x):
    return np.array([_f1(x[0],x[1]),_f2(x[0],x[1])])

def f_hess(x):
    row1 = [_f11(x[0],x[1]),_f12(x[0],x[1])]
    row2 = [_f21(x[0],x[1]),_f22(x[0],x[1])]
    return np.array([row1,row2])

```

**Question C:** Minimize $f(x_1,x_2)$ using respectively

1. Nelder-Mead,
* BFGS without analytical jacobian,
* BFGS with analytical jacobian, and
* Newton-CG with analytical jacobian and hessian

Compare the results and discuss which optimizer you prefer.

**Optional:** If you wish, you can use the functions defined in the hidden cells below to also track how the optimizers converges to the solution.

```python
def collect(x):

    # globals used to keep track across iterations
    global evals # set evals = 0 before calling optimizer
    global x0
    global x1s
    global x2s
    global fs

    # a. initialize list
    if evals == 0:
        x1s = [x0[0]]
        x2s = [x0[1]]
        fs = [f_python(x0)]

    # b. append trial values
    x1s.append(x[0])
    x2s.append(x[1])
    fs.append(f_python(x))

    # c. increment number of evaluations
    evals += 1
```

```python
def contour():

    global evals
    global x1s
    global x2s
    global fs

    # a. contour plot
    fig = plt.figure(figsize=(10,4))
    ax = fig.add_subplot(1,2,1)
    levels = np.sort([j*10**(-i) for i in [-1,0,1,2,3,4] for j in [0.5,1,1.5]])
    cs = ax.contour(x1_grid,x2_grid,f_grid,levels=levels,cmap=cm.jet)
    fig.colorbar(cs)
    ax.plot(x1s,x2s,'-o',ms=4,color='black')
    ax.set_xlabel('$x_1$')
    ax.set_ylabel('$x_2$')

    # b. function value
    ax = fig.add_subplot(1,2,2)
    ax.plot(np.arange(evals+1),fs,'-o',ms=4,color='black')
    ax.set_xlabel('iteration')
    ax.set_ylabel('function value')
```

```python
x0 = [-2,-1] # suggested initial guess

# write your code here
```

**Answer:**

```python
print('Nelder-Mead:')
evals = 0
result = optimize.minimize(f_python,x0,method='Nelder-Mead',callback=collect,options={'disp':True})
contour()
```

```python
print('BFGS without analytical gradient:')

evals = 0
result = optimize.minimize(f_python,x0,method='BFGS',callback=collect,options={'disp':True})
contour()
```

```python
print('BFGS with analytical gradient:')

evals = 0
result = optimize.minimize(f_python,x0,jac=f_jac,method='BFGS',callback=collect,options={'disp':True})
contour()
```

```python
print('Newton-CG with analytical gradient and hessian:')

evals = 0
result = optimize.minimize(f_python,x0,jac=f_jac,hess=f_hess,method='Newton-CG',callback=collect,options={'disp':True})
contour()
```

## Optimization problem II

Consider the function

$$
f(x_1,x_2) = (4-2.1x_1^2 + \frac{x_1^4}{3})x_1^2 + x_1x_2 + (4x_2^2 - 4)x_2^2)
$$

Define it in **sympy** by:

```python
x1 = sm.symbols('x_1')
x2 = sm.symbols('x_2')
f = (4-2.1*x1**2 + (x1**4)/3)*x1**2 + x1*x2 + (4*x2**2 - 4)*x2**2
_f = sm.lambdify((x1,x2),f)
f
```

Create **3D plot**:

```python
# a. grids
x1_vec = np.linspace(-2,2,500)
x2_vec = np.linspace(-1,1,500)
x1_grid,x2_grid = np.meshgrid(x1_vec,x2_vec,indexing='ij')
f_grid = _f(x1_grid,x2_grid)

# b. main
fig = plt.figure()
ax = fig.add_subplot(1,1,1,projection='3d')
cs = ax.plot_surface(x1_grid,x2_grid,f_grid,cmap=cm.jet)

# c. add labels
ax.set_xlabel('$x_1$')
ax.set_ylabel('$x_2$')
ax.set_zlabel('$f$')

# d. invert xaxis
ax.invert_xaxis()

# e. remove background
ax.xaxis.pane.fill = False
ax.yaxis.pane.fill = False
ax.zaxis.pane.fill = False

# f. add colorbar
fig.colorbar(cs);
```

**Question A:** Find the minimum of the function starting from each of the suggested initial values below. Print the first 20 solutions, and all solutions aftwards, which is the best yet to be found. Save the solutions and associated function values in `xs` and `fs`.

```python
# a. python function for f
f_python = lambda x: _f(x[0],x[1])

# b. initial guesses
np.random.seed(1986)
K = 1000
x0s = np.empty((K,2))
x0s[:,0] = -2 + 4*np.random.uniform(size=K)
x0s[:,1] = -1 + 2*np.random.uniform(size=K)

# c. solutions and associated values
xs = np.empty((K,2))
fs = np.empty(K)

# write your code here
```

**Answer:**

```python
fopt = np.inf
xopt = np.nan
for i,x0 in enumerate(x0s):

    # a. optimize
    result = optimize.minimize(f_python,x0,method='BFGS')
    xs[i,:] = result.x
    fs[i] = result.fun

    # b. print first 20 or if better than seen yet
    if i < 20 or fs[i] < fopt: # plot 20 first or if improving
        if fs[i] < fopt:
            xopt = xs[i,:]
            fopt = fs[i]
        print(f'{i:4d}: x0 = ({x0[0]:6.2f},{x0[0]:6.2f})',end='')
        print(f' -> converged at ({xs[i][0]:6.2f},{xs[i][1]:6.2f}) with f = {fs[i]:.12f}')

# best solution
print(f'\nbest solution:\n x = ({xopt[0]:6.2f},{xopt[1]:6.2f}) -> f = {fopt:.12f}')
```

       0: x0 = (  0.28,  0.28) -> converged at (  0.09, -0.71) with f = -1.031628453485
       1: x0 = ( -1.69, -1.69) -> converged at ( -1.70,  0.80) with f = -0.215463824384
       2: x0 = (  0.43,  0.43) -> converged at (  0.09, -0.71) with f = -1.031628453490
       3: x0 = (  1.59,  1.59) -> converged at (  1.70, -0.80) with f = -0.215463824384
       4: x0 = (  0.18,  0.18) -> converged at (  0.09, -0.71) with f = -1.031628453490
       5: x0 = (  0.81,  0.81) -> converged at ( -0.09,  0.71) with f = -1.031628453490
       6: x0 = ( -0.46, -0.46) -> converged at ( -0.09,  0.71) with f = -1.031628453487
       7: x0 = (  0.61,  0.61) -> converged at (  0.09, -0.71) with f = -1.031628453490
       8: x0 = (  0.76,  0.76) -> converged at ( -0.09,  0.71) with f = -1.031628453484
       9: x0 = (  0.87,  0.87) -> converged at ( -0.09,  0.71) with f = -1.031628453490
      10: x0 = (  0.76,  0.76) -> converged at ( -0.09,  0.71) with f = -1.031628453489
      11: x0 = (  1.23,  1.23) -> converged at (  1.70, -0.80) with f = -0.215463824384
      12: x0 = ( -0.87, -0.87) -> converged at (  0.09, -0.71) with f = -1.031628453486
      13: x0 = (  1.03,  1.03) -> converged at (  0.09, -0.71) with f = -1.031628453490
      14: x0 = ( -0.77, -0.77) -> converged at ( -0.09,  0.71) with f = -1.031628453490
      15: x0 = ( -0.25, -0.25) -> converged at (  0.09, -0.71) with f = -1.031628453490
      16: x0 = (  0.21,  0.21) -> converged at (  0.09, -0.71) with f = -1.031628453488
      17: x0 = ( -0.27, -0.27) -> converged at (  0.09, -0.71) with f = -1.031628453490
      18: x0 = (  0.31,  0.31) -> converged at ( -0.09,  0.71) with f = -1.031628453490
      19: x0 = (  1.57,  1.57) -> converged at (  1.61,  0.57) with f = 2.104250310311
      24: x0 = ( -0.14, -0.14) -> converged at (  0.09, -0.71) with f = -1.031628453490
      27: x0 = ( -0.13, -0.13) -> converged at (  0.09, -0.71) with f = -1.031628453490
     155: x0 = (  0.60,  0.60) -> converged at (  0.09, -0.71) with f = -1.031628453490

    best solution:
     x = (  0.09, -0.71) -> f = -1.031628453490

**Question B:** Create a 3D scatter plot of where the optimizer converges, and color the dots by the associated function values.

```python
# write your code here
```

**Answer:**

```python
# a. main
fig = plt.figure()
ax = fig.add_subplot(1,1,1,projection='3d')
cs = ax.scatter(xs[:,0],xs[:,1],fs,c=fs);

# b. add labels
ax.set_xlabel('$x_1$')
ax.set_ylabel('$x_2$')
ax.set_zlabel('$f$')

# c. invert xaxis
ax.invert_xaxis()

# d. colorbar
fig.colorbar(cs);
```

**Question C:** Plot the function values at the solutions as a function of the starting values.

```python
# write your code here
```

**Answer:**

```python
# a. main
fig = plt.figure()
ax = fig.add_subplot(1,1,1,projection='3d')
cs = ax.scatter(x0s[:,0],x0s[:,1],fs,c=fs);

# b. add labels
ax.set_xlabel('$x_1$')
ax.set_ylabel('$x_2$')
ax.set_zlabel('$f$')

# c. invert xaxis
ax.invert_xaxis()

# d. colorbar
fig.colorbar(cs);
```

```python
sm.init_printing(pretty_printing=False)
```

# Problem: Solve the consumer problem with income risk I

Define the following **variables** and **parameters**:

* $m_t$ is cash-on-hand in period $t$
* $c_t$ is consumption in period $t$
* $y_t$ is income in period $t$
* $\Delta \in (0,1)$ is income risk
* $r$ is the interest rate
* $\beta > 0$, $\rho > 1$, $\nu > 0 $, $\kappa > 0$, $\xi > 0$ are utility parameters

In the **second period** the household solves:

$$
\begin{aligned}
v_{2}(m_{2}) &= \max_{c_{2}}\frac{c_{2}^{1-\rho}}{1-\rho}+\nu\frac{(m_{2}-c_{2}+\kappa)^{1-\rho}}{1-\rho} \\
 \text{s.t.} \\
c_{2} & \in [0,m_{2}]
\end{aligned}
$$

In the **first period** the household solves:

$$
\begin{aligned}
v_{1}(m_{1}) & =
\max_{c_{1}}\frac{c_{1}^{1-\rho}}{1-\rho}+\beta\mathbb{E}_{1}\left[v_2(m_2)\right] \\
\text{s.t.} \\
m_2 &= (1+r)(m_{1}-c_{1})+y_{2} \\
y_{2} &= \begin{cases}
1-\Delta & \text{with prob. }0.5\\
1+\Delta & \text{with prob. }0.5
\end{cases}\\
c_{1} & \in [0,m_{1}]\\
\end{aligned}
$$

The **basic functions** are:

```python
def utility(c,rho):
    return c**(1-rho)/(1-rho)

def bequest(m,c,nu,kappa,rho):
    return nu*(m-c+kappa)**(1-rho)/(1-rho)

def v2(c2,m2,rho,nu,kappa):
    return utility(c2,rho) + bequest(m2,c2,nu,kappa,rho)

def v1(c1,m1,rho,beta,r,Delta,v2_interp):

    # a. v2 value, if low income
    m2_low = (1+r)*(m1-c1) + 1-Delta
    v2_low = v2_interp([m2_low])[0]

    # b. v2 value, if high income
    m2_high = (1+r)*(m1-c1) + 1+Delta
    v2_high = v2_interp([m2_high])[0]

    # c. expected v2 value
    v2 = 0.5*v2_low + 0.5*v2_high

    # d. total value
    return utility(c1,rho) + beta*v2
```

The **solution functions** are:

```python
def solve_period_2(rho,nu,kappa,Delta):

    # a. grids
    m2_vec = np.linspace(1e-8,5,500)
    v2_vec = np.empty(500)
    c2_vec = np.empty(500)

    # b. solve for each m2 in grid
    for i,m2 in enumerate(m2_vec):

        # i. objective
        obj = lambda c2: -v2(c2,m2,rho,nu,kappa)

        # ii. initial value (consume half)
        x0 = m2/2

        # iii. optimizer
        result = optimize.minimize_scalar(obj,x0,method='bounded',bounds=[1e-8,m2])

        # iv. save
        v2_vec[i] = -result.fun
        c2_vec[i] = result.x

    return m2_vec,v2_vec,c2_vec

def solve_period_1(rho,beta,r,Delta,v1,v2_interp):

    # a. grids
    m1_vec = np.linspace(1e-8,4,100)
    v1_vec = np.empty(100)
    c1_vec = np.empty(100)

    # b. solve for each m1 in grid
    for i,m1 in enumerate(m1_vec):

        # i. objective
        obj = lambda c1: -v1(c1,m1,rho,beta,r,Delta,v2_interp)

        # ii. initial guess (consume half)
        x0 = m1*1/2

        # iii. optimize
        result = optimize.minimize_scalar(obj,x0,method='bounded',bounds=[1e-8,m1])

        # iv. save
        v1_vec[i] = -result.fun
        c1_vec[i] = result.x

    return m1_vec,v1_vec,c1_vec
```

**Question A:** Find optimal consumption in the first period as funcition of cash-on-hand, and plot it.

```python
rho = 8
kappa = 0.5
nu = 0.1
r = 0.04
beta = 0.94
Delta = 0.5

# b. solve
# write your code here

# c. plot
# write your code here
```

**Answer:**

```python
# b. solve
def solve(rho,beta,r,Delta,nu,kappa,v1):

    # a. solve period 2
    m2_vec,v2_vec,c2_vec = solve_period_2(rho,nu,kappa,Delta)

    # b. construct interpolator
    v2_interp = interpolate.RegularGridInterpolator((m2_vec,), v2_vec,
                                                    bounds_error=False,fill_value=None)

    # b. solve period 1
    m1_vec,v1_vec,c1_vec = solve_period_1(rho,beta,r,Delta,v1,v2_interp)

    return m1_vec,c1_vec

m1_vec,c1_vec = solve(rho,beta,r,Delta,nu,kappa,v1)

# c. plot
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.plot(m1_vec,c1_vec)
ax.set_xlabel('$m_1$')
ax.set_ylabel('$c_1$')
ax.set_title('consumption function in period 1')
ax.set_xlim([0,4])
ax.set_ylim([0,2.5]);
```

**Question B:** Find optimal consumption in the first period as funcition of cash-on-hand, and plot it, assuming that

$$
y_{2} = \begin{cases}
1-\sqrt{\Delta} & \text{with prob. }0.1\\
1-\Delta & \text{with prob. }0.4\\
1+\Delta & \text{with prob. }0.4\\
1+\sqrt{\Delta} & \text{with prob. }0.1
\end{cases}
$$

which add some low probability tail events, but does not change mean income. Give an interpretation of the change in the consumption function.

```python
# write your code here
```

**Answer:**

```python
def v1_alt(c1,m1,rho,beta,r,Delta,v2_interp):

    # a. expected v2 value
    Ra = (1+r)*(m1-c1)
    v2 = 0
    y2s = [1-np.sqrt(Delta),1-Delta,1+Delta,1+np.sqrt(Delta)]
    probs = [0.1,0.4,0.4,0.1]
    for y2,prob in zip(y2s,probs):
        m2 = Ra + y2
        v2 += prob*v2_interp([m2])[0]

    # b. total value
    return utility(c1,rho) + beta*v2

m1_vec_alt,c1_vec_alt = solve(rho,beta,r,Delta,nu,kappa,v1_alt)

# plot
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.plot(m1_vec,c1_vec,label='original')
ax.plot(m1_vec_alt,c1_vec_alt,label='new')
ax.legend(loc='upper left')
ax.set_xlabel('$m_1$')
ax.set_ylabel('$c_1$')
ax.set_title('consumption function in periode 1')
ax.set_xlim([0,4])
ax.set_ylim([0,2.5]);
```

# Problem: Solve the consumer problem with income risk II

Define the following **variables** and **parameters**:

* $m_t$ is cash-on-hand in period $t$
* $c_t$ is non-durable consumption in period $t$
* $d_t$ is durable consumption in period $t$ (only adjustable in period 1)
* $y_t$ is income in period $t$
* $\Delta \in (0,1)$ is income risk
* $r$ is the interest rate
* $\beta > 0$, $\rho > 1$, $\alpha \in (0,1)$, $\nu > 0 $, $\kappa > 0$, $\xi > 0$ are utility parameters

In the **second period** the household solves:

$$
\begin{aligned}
v_{2}(m_{2},d_{2}) &= \max_{c_{2}}\frac{c_{2}^{1-\rho}}{1-\rho}+\alpha\frac{d_{2}^{1-\rho}}{1-\rho}+\nu\frac{(m_{2}+d_{2}-c_{2}+\kappa)^{1-\rho}}{1-\rho} \\
\text{s.t.} \\
c_{2} & \in [0,m_{2}]
\end{aligned}
$$

In the **first period** the household solves:

$$
\begin{aligned}
v_{1}(m_{1}) &= \max_{c_{1},d_{1}}\frac{c_{1}^{1-\rho}}{1-\rho}+\alpha\frac{d_{1}^{1-\rho}}{1-\rho}+\beta\mathbb{E}_{1}\left[v_2(m_2,d_2)\right]\\&\text{s.t.}&\\
m_2 &= (1+r)(m_{1}-c_{1}-d_{1})+y_{2} \\
y_{2} &= \begin{cases}
1-\Delta & \text{with prob. }0.5\\
1+\Delta & \text{with prob. }0.5
\end{cases}\\
c_{1}+d_{1} & \in [0,m_{1}]\\
\end{aligned}
$$

Choose **parameters**:

```python
rho = 2
alpha = 0.1
kappa = 0.5
nu = 0.1
r = 0.04
beta = 0.94
Delta = 0.5

# b. solve
# write your code here

# c. plot
# write your code here
```

The **basic functions** are:

```python
def utility(c,d,alpha,rho):
    return c**(1-rho)/(1-rho) + alpha*d**(1-rho)/(1-rho)

def bequest(m,c,d,nu,kappa,rho):
    return nu*(m+d-c+kappa)**(1-rho)/(1-rho)

def v2(c2,d2,m2,alpha,rho,nu,kappa):
    return utility(c2,d2,alpha,rho) + bequest(m2,c2,d2,nu,kappa,rho)

def v1(c1,d1,m1,alpha,rho,beta,r,Delta,v2_interp):

    # a. v2 value, if low income
    m2_low = (1+r)*(m1-c1-d1) + 1-Delta
    v2_low = v2_interp([m2_low,d1])[0]

    # b. v2 value, if high income
    m2_high = (1+r)*(m1-c1-d1) + 1+Delta
    v2_high = v2_interp([m2_high,d1])[0]

    # c. expected v2 value
    v2 = 0.5*v2_low + 0.5*v2_high

    # d. total value
    return utility(c1,d1,alpha,rho) + beta*v2
```

The **solution function for period 2** is:

```python
def solve_period_2(alpha,rho,nu,kappa,Delta):

    # a. grids
    m2_vec = np.linspace(1e-8,5,200)
    d2_vec = np.linspace(1e-8,5,200)
    v2_grid = np.empty((200,200))
    c2_grid = np.empty((200,200))

    # b. solve for each m2 in grid
    for i,m2 in enumerate(m2_vec):
        for j,d2 in enumerate(d2_vec):

            # i. objective
            obj = lambda c2: -v2(c2,d2,m2,alpha,rho,nu,kappa)

            # ii. initial value (consume half)
            x0 = m2/2

            # iii. optimizer
            result = optimize.minimize_scalar(obj,x0,method='bounded',bounds=[1e-8,m2])

            # iv. save
            v2_grid[i,j] = -result.fun
            c2_grid[i,j] = result.x

    return m2_vec,d2_vec,v2_grid,c2_grid
```

**Question A:** Solve for consumption in period 2 and plot the consumption function.

```python
# write your code here
```

**Answer:**

```python
# a. solve
m2_vec,d2_vec,v2_grid,c2_grid = solve_period_2(alpha,rho,nu,kappa,Delta)

# b. grids
m2_grid,d2_grid = np.meshgrid(m2_vec,d2_vec,indexing='ij')

# c. main
fig = plt.figure()
ax = fig.add_subplot(1,1,1,projection='3d')
cs = ax.plot_surface(m2_grid,d2_grid,c2_grid,cmap=cm.jet)

# d. add labels
ax.set_xlabel('$m_2$')
ax.set_ylabel('$d_2$')
ax.set_zlabel('$c_2$')

# e. invert xaxis
ax.invert_xaxis()

# f. add colorbar
fig.colorbar(cs);
```

**Question B:** Find optimal consumption and choices of durables in the first period as a function of cash-on-hand and plot it.

```python
# write your code here
```

**Answer:**

```python
# a. define solve function
def solve_period_1(alpha,rho,beta,r,Delta,v1,v2_interp):

    # a. grids
    m1_vec = np.linspace(1e-4,4,100)
    v1_vec = np.empty(100)
    c1_vec = np.empty(100)
    d1_vec = np.empty(100)

    # b. solve for each m1 in grid
    for i,m1 in enumerate(m1_vec):

        # i. objective
        obj = lambda x: -v1(x[0],x[1],m1,alpha,rho,beta,r,Delta,v2_interp)

        # ii. initial guess
        x0 = [m1*1/3,m1*1/3]

        # iii. bounds and constraitns
        bound = (1e-8,m1-1e-8)
        bounds = (bound, bound)
        ineq_con = {'type': 'ineq', 'fun': lambda x: m1-x[0]-x[1]}

        # iv. optimize
        result = optimize.minimize(obj,x0, method='SLSQP',
                                   bounds=bounds,
                                   constraints=[ineq_con])

        #result = optimize.minimize(obj,x0, method='Nelder-Mead')

        # v. save
        v1_vec[i] = -result.fun
        c1_vec[i] = result.x[0]
        d1_vec[i] = result.x[1]

    return m1_vec,v1_vec,c1_vec,d1_vec

# b. construct interpolator
v2_interp = interpolate.RegularGridInterpolator((m2_vec,d2_vec), v2_grid,
                                                bounds_error=False,fill_value=None)

# c. solve period 1
m1_vec,v1_vec,c1_vec,d1_vec = solve_period_1(alpha,rho,beta,r,Delta,v1,v2_interp)

# d. plot
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.plot(m1_vec,c1_vec,label='non-durable consumption')
ax.plot(m1_vec_alt,d1_vec,label='durable consumption')
ax.legend(loc='upper left')
ax.set_xlabel('$m_1$')
ax.set_xlim([0,4])
ax.set_ylim([0,2.5]);
```

```python

```

```python

```


# some simple univariate polynomials

* computing with real univariate polynomials ...
* it seems that the gcd function for floating point numbers does not give what we expect
* can you fix this?
    * hint 1: Euclid's algorithm is used for the gcd computations
    * hint 2: this problem has been studied
    * hint 3: the product of the two polynomials is not the exact product
    * hint 4: check out the code relating to domains

```python
# example how to use numpy arrays to define polynomial systems ...

from sympy import symbols

from numpy import arange, dot
from numpy.random import rand

x = symbols('x')
d = 3  # degree
n = 2  # size of system

# array of monomials xb, degree d
ie = arange(d+1)
xb = x**ie  # Stetter denotes this by b(x)

# random coefficients
C = rand(n,d+1)

# polynomial system
P = dot(C,xb)

print("random polynomial system: \n {}\n".format(P))

# convert to sympy polynomial (P[i] are just expressions)
print("polynomial P[0]: {}".format(P[0].as_poly()))
```

```python
# division with remainder
from sympy import div
q, r = div(P[1], P[0])
print(" quotient  q = {} \n remainder r = {}".format(q,r))

# check result
print(" should be zero: {}".format(q*P[0] + r - P[1]))
```

```python
# get some help
help(div)
```

```python
# compute the gcd
from sympy import gcd
q = gcd(P[0],P[1])
print(" gcd of P[0] and P[1] : {}".format(q))
```

```python
# a less trivial case of gcd ???? try a few different random polynomials ...
q = gcd(P[0]*P[1], P[0])
print(" gcd of P[0]*P[1] and P[0] : {}".format(q))
```

```python
# get some help!
help(gcd)
```

```python
# do the same computation with rationals
from sympy import Rational
from numpy import vectorize # (not necessarily fast)
q_vect = vectorize(Rational)

Cq = q_vect(C)
Q = dot(Cq,xb)

q = gcd(Q[0]*Q[1], Q[0])
print(" gcd of Q[0]*Q[1] and Q[0] : {}".format(q))

_, r = div(Q[0], q)
print(" remainder of division Q[0]/q : {}".format(r))

print(" difference Q-P : {}".format(Q-P))

print(" difference Q[0]*Q[1] - P[0]*P[1] : {}".format((Q[0]*Q[1]-P[0]*P[1]).as_poly()))
```


```python
from resources.workspace import *
```

    /Users/brajard/Documents/recherche/collaboration/bigdata/DAPPER

### The Gaussian (i.e. Normal) distribution

Consider the random variable with a Gaussian distribution with mean $\mu$ (`mu`) and variance $P$. We write its probability density function (**pdf**) as
$$ p(x) = N(x|\mu,P) = (2 \pi P)^{-1/2} e^{-(x-\mu)^2/2P} \, .  \qquad \qquad (1) $$

---
**Exc 2.2:** Code it up (complete the code below)! Hints:
* Note that `**` is the power operator in Python.
* As in Matlab, $e^x$ is available as `exp(x)`

```python
# Univariate (scalar), Gaussian pdf
def pdf_G_1(x,mu,P):
    # pdf_values = ### INSERT ANSWER HERE ###
    return sp.stats.norm.pdf(x,loc=mu,scale=sqrt(P))
```

```python
show_answer('pdf_G_1')
```

pdf_values = 1/sqrt(2*pi*P)*exp(-0.5*(x-mu)**2/P)
# Version using the scipy (sp) library:
# pdf_values = sp.stats.norm.pdf(x,loc=mu,scale=sqrt(P))

Let's plot the pdf.

```python
mu  = 0                 # mean     of distribution
P   = 25                # variance of distribution
P12 = sqrt(P)           # std. dev of distribution

# Plotting
N  = 201                # num of grid points
xx = linspace(-20,20,N) # grid
dx = xx[1]-xx[0]        # grid spacing
pp = pdf_G_1(xx,mu,P)   # pdf values
plt.subplot(211)        # allocate plot panel
plt.plot(xx,pp);        # plot
```

    <IPython.core.display.Javascript object>

This could for example be the pdf of a stochastic noise variable. It could also describe our uncertainty about a parameter (or state), which we model as randomness in the Bayesian paradigm.

**Exc 2.4:** Change `P` in the above code, and re-run the cell. Look at the figure.
 * How does the pdf curve change when you increase P?
 * Re-set `P=25` and re-run (this is a convienient value for examples)

**Exc 2.6:** Recall $p(x)$ from eqn (1). The following are helpful points to remember how it looks. Use pen, paper, and calculus. Hint: it's typically easier to analyse $\log p(x)$ rather than $p(x)$ itself.
 * Where is the location of the mode (maximum) of the distribution? I.e. where $\frac{d p(x)}{d x} = 0$.
 * Where is the inflection point? I.e. where $\frac{d^2 p(x)}{d x^2} = 0$.
 * What is the value of $\frac{d^2 \log p(x)}{d x^2}$ at the mode?

#### The multivariate (i.e. vector) case
Here's the pdf of the *multivariate* Gaussian:
\begin{align}
N(x|\mu,P) &= |2 \pi P|^{-\frac{1}{2}} e^{-\frac{1}{2}\|x-\mu\|^2_P} \, , \\\
\end{align}
where $|.|$ represents the determinant, and $\|.\|_W$ represents the norm with weighting: $\|x\|^2_W = x^T W^{-1} x$.

The following implements this pdf. Take a moment to digest the code; in particular, it should be noted that `pdf_G_m()` can accept multiple `x` vectors at once (assembled into a matrix), whence the `xx` naming convention.

```python
from numpy.linalg import det, inv

def weighted_norm22(xx,W):
    # Computes the norm of each row vector of xx, as weighted by W.
    return np.sum((xx @ inv(W)) * xx, axis=1)

def pdf_G_m(xx,mu,P):
    return 1/sqrt(det(2*pi*P))*exp(-0.5*weighted_norm22(xx-mu,P))
```

The following code plots the pdf as contour (equi-density) curves.
The plot appears in the above figure.

```python
def list_2_array(grid): return array([xi.ravel() for xi in grid]).T
def square_reshape(X):  return X.reshape(int(sqrt(len(X))),-1)

grid = np.meshgrid(xx,xx)
grid = list_2_array(grid)

pp = pdf_G_m(grid, 0, P*array([[1,0.7],[0.7,1]]))
pp = square_reshape(pp)

plt.subplot(212)
plt.contour(xx,xx,pp);
plt.axis('equal');
plt.show()
```

    /Users/brajard/anaconda/envs/nn/lib/python3.6/site-packages/matplotlib/cbook/deprecation.py:106: MatplotlibDeprecationWarning: Adding an axes using the same arguments as a previous axes currently reuses the earlier instance.  In a future version, a new instance will always be created and returned.  Meanwhile, this warning can be suppressed, and the future behavior ensured, by passing a unique label to each axes instance.
      warnings.warn(message, mplDeprecation, stacklevel=1)

**Exc 2.8:**
 * Set the correlation to 0. How do the contours look?
 * Set the correlations to 0.99. How do the contours look?

**Exc 2.9:** Go play the [correlation game](http://guessthecorrelation.com/)

### Bayes' rule
Bayes' rule is how we do inference. For continuous random variables, $x$ and $y$, it reads:

$$ p(x|y) = \frac{p(x) \, p(y|x)}{p(y)} \, , \qquad \qquad (2)$$

or, in words:

$$
\text{"posterior" (pdf of $x$ given $y$)}
\; = \;
\frac{\text{"prior" (pdf of $x$)}
\; \times \;
\text{"likelihood" (pdf of $y$ given $x$)}}
{\text{"normalization" (pdf of $y$)}}
$$.

**Exc 2.10:** Derive Bayes' rule from the definition of [conditional pdf's](https://en.wikipedia.org/wiki/Conditional_probability#Kolmogorov_definition).

```python
show_answer('BR deriv')
```

Wikipedia

Computers generally work with discrete, numerical representations of mathematical entities.
Numerically, pdfs may be represented by their `values` on a grid, such as `xx` from above. Bayes' rule (2) then consists of *(grid-)point-wise* multiplication, as shown below.

```python
def Bayes_rule(prior_values,lklhd_values,dx):
    pp = prior_values * lklhd_values   # pointwise multiplication
    posterior_values = pp/(sum(pp)*dx) # normalization
    return posterior_values
```

**Exc 2.12:** Why does `Bayes_rule()` not need the values of the denominator, $p(y)$, as input?

```python
show_answer('BR grid normalization')
```

Because it can compute $p(y)$ as
the factor needed to normalize to 1,
as required by the definition of pdfs.

That's what the #normalization line does.

Here's the proof that the normalization (which makes pp sum to 1) is equivalent to dividing by $p(y)$:
$$\texttt{sum(pp)*dx} \approx \int p(x) p(y|x) \, dx = \int p(x,y) \, dx = p(y) \, .$$

In fact, since normalization is so simple, we often don't bother to do it until it's strictly necessary. Therefore we often simplify Bayes' rule (2) as
$$ p(x|y) \propto p(x) \, p(y|x) \, .  \qquad \qquad (3) $$

---
The code below show's Bayes' rule in action. Again, remember that the only thing it's doing is multiplying the prior and likelihood at each gridpoint. Move the sliders with the arrow keys to animate it.

```python
%matplotlib inline

b = 0 # prior mean
B = 1 # prior variance

@interact(y=(-10,10,1),log_R=(-2,5,0.5))
def animate_Gaussian_Bayes(y=4.0,log_R=1):

    R = exp(log_R)

    prior     = lambda x: pdf_G_1(x,b,B)
    lklhd     = lambda x: pdf_G_1(y,x,R)

    post_vals = Bayes_rule(prior(xx),lklhd(xx),xx[1]-xx[0])

    plt.figure(figsize=(10,4))
    plt.plot(xx,prior(xx)     ,label='prior N(x|0,1)')
    plt.plot(xx,lklhd(xx)     ,label='likelihood N(y|x,R)')
    plt.plot(xx,post_vals     ,label='posterior - pointwise')

    ### Uncomment this block AFTER doing the exercise ###
    ### that defines Bayes_rule_Gaussian()            ###
    #mu, P     = Bayes_rule_Gaussian(b,B,y,R)
    #postr     = lambda x: pdf_G_1(x,mu,P)
    #plt.plot(xx,postr(xx),'--',label='posterior - parametric')

    plt.ylim(ymax=0.6)
    plt.legend()
    plt.show()
```

**Exc 2.14:** Answer the following by moving the sliders and seeing what happens.
 * What happens to the posterior when $R \rightarrow \infty$ ?
 * What happens to the posterior when $R \rightarrow 0$ ?
 * Where is the posterior when $R = B$ ? (try moving around $y$)
 * Does the posterior scale (width) depend on $y$?
 * Forgetting about its location and scale, what is the shape of the posterior? Does this depend on $R$ or $y$? Can you see why?
 * Can you see a shortcut to computing this posterior rather than having to do the pointwise multiplication?

**Exc 2.15*:** Implement a "uniform" (or "flat" or "box") distribution pdf and call it `pdf_U_1(x,mu,P)`. These formulae for its mean/variance will be useful. In the above animations, replace `pdf_G_1` with your new `pdf_U_1` (both for the prior and likelihood). Assure that everything is working correctly.
 - Why (in the figure) are the walls of the pdf (ever so slightly) inclined?
 - What happens when you move the prior and likelihood very far apart? Is the fault of the implementation, or the fundamental assumptions (uniform distribution)?
 - Re-do Exc 2.14, now with `pdf_U_1`.
 - Now test a Gaussian prior with a uniform likelihood.
 - Restore `pdf_G_1` (both the prior and likelihood) in the animation (for later use).

```python
show_answer('pdf_U_1')
```

def pdf_U_1(x,mu,P):
    # Univariate (scalar), Uniform pdf

    pdf_values = ones((x-mu).shape)

    a = mu - sqrt(3*P)
    b = mu + sqrt(3*P)

    pdf_values[x<a] = 0
    pdf_values[x>b] = 0

    height = 1/(b-a)
    pdf_values *= height

    return pdf_values

### Gaussian-Gaussian Bayes

The above animation shows Bayes' rule in 1 dimension. Previously, we saw how a Gaussian looks in 2 dimensions. Can you imagine how Bayes' rule looks in 2 dimensions? In higher dimensions, these things get difficult to imagine, let alone visualize.

Similarly, the size of the calculations required for Bayes' rule poses a difficulty. Indeed, the following exercise shows that (pointwise) multiplication for all grid points becomes a preposterious notion in high dimensions.

**Exc 2.16:**
 * (a) How many point-multiplications are needed on a grid with $N$ points in $m$ dimensions? (Imagine an $m$-dimensional cube where each side has a grid with $N$ points on it)
 * (b) Suppose we model 15 physical quanitites, on each grid point, on a discretized model of Earth. Assume the resolution is $1^\circ$ for latitude (110km), $1^\circ$ for longitude. How many variables are there in total? This is the dimensionality ($m$) of the problem.
 * (c) Suppose each variable is has a pdf represented with a grid using only $N=10$ points. How many multiplications are necessary to calculate Bayes rule (jointly) for all variables on our Earth model?

```python
show_answer('Dimensionality a')
show_answer('Dimensionality b')
show_answer('Dimensionality c')
```

$N^m$

$15 * 360 * 180 = 972'000 \approx 10^6$

$10^{10^6}$

In response to this computational difficulty, we try to be smart and do something more analytical ("pen-and-paper"): we only compute the parameters (mean and (co)variance) of the posterior pdf.

This is doable and quite simple in the Gaussian-Gaussian case.
With a prior $p(x) = N(x|b,B)$ and a likelihood $p(y|x) = N(y|x,R)$, the posterior will be given by
\begin{align}
p(x|y)
&= N(x|\mu,P) \qquad \qquad (4)
\, ,
\end{align}
where, in the univarite (1-dimensional) case:
\begin{align}
    P &= 1/(1/B + 1/R) \, , \qquad \qquad (5) \\\
  \mu &= P(b/B + y/R) \, .  \qquad \qquad (6)
\end{align}

#### Exc  2.18 'Gaussian Bayes':
Derive the above expressions for $P$ and $\mu$.
*Hint: you need eqns (1) and (3).*

```python
show_answer('BR Gauss')
```

We can ignore factors that do not depend on $x$.

\begin{align}
p(x|y)
&= \frac{p(x) \, p(y|x)}{p(y)} \\
&\propto p(x) \, p(y|x) \\
&=       N(x|b,B) \, N(y|x,R) \\
&\propto \exp \Big( \frac{-1}{2} \Big( (x-b)^2/B + (x-y)^2/R \Big) \Big) \\
&\propto \exp \Big( \frac{-1}{2} \Big( (1/B + 1/R)x^2 - 2(b/B + y/R)x \Big) \Big) \\
&\propto \exp \Big( \frac{-1}{2} \Big( x - \frac{b/B + y/R}{1/B + 1/R} \Big)^2 \cdot (1/B + 1/R) \Big) \, .
\end{align}

The last line can be identified as $N(x|\mu,P)$ as defined above.

**Exc 2.20:** Do some light algebra to show that eqns (5) and (6) can be rewritten as
\begin{align}
    P &= (1-K)B \, ,  \qquad \qquad (8) \\\
  \mu &= b + K (y-b)  \qquad \quad (9) \, ,
\end{align}
where $K = B/(B+R)$, which is called the "Kalman gain".

**Exc 2.22*:** Consider the formula for $K$ and its role in the previous couple of equations... Why do you think $K$ is called a "gain"?

```python
show_answer('KG 2')
```

Because it

drags the estimate from $b$ "towards" $y$.

is between 0 and 1.

weights the observation noise level (R) vs. the total noise level (B+R).

In the multivariate case (and with $H=I$), the same holds for its eigenvectors.

**Exc 2.24:** Implement a Gaussian-Gaussian Bayes' rule by completing the code below.

```python
def Bayes_rule_Gaussian(b,B,y,R):
    ### INSERT ANSWER HERE ###
    return mu,P
```

```python
show_answer('BR Gauss code')
```

P  = 1/(1/B+1/R)
mu = P*(b/B+y/R)
# Gain version:
#     KG = B/(B+R)
#     P  = (1-KG)*B
#     mu = b + KG*(y-b)

**Exc 2.26:** Then, go back to the animation above, and uncomment the block that makes use of `Bayes_rule_Gaussian()`. Make sure its curve coincides with that which uses pointwise multiplication (i.e. `Bayes_rule()`).

**Exc 2.28:** More questions related to the above animation:
 * Does the width (i.e. scale) for the posterior depend on the location $y$ of the likelihood?
 * Is the width (i.e. scale) for the posterior always smaller that that of prior and likelihood? What does this mean information-wise?
   * Do you think this is always the case, also for non-Gaussian distributions?
 * What if you're pretty sure about something, and you get a wildly different indication (observation). What is your posterior certainty?

**Exc 2.30*:** Why are we so fond of the Gaussian assumption?

```python
show_answer('why Gaussian')
```

Pragmatic: leads to least-squares problems, which lead to linear systems of equations.
   This was demonstrated by the simplicity of the parametric Gaussian-Gaussian Bayes' rule.

The central limit theorem (CLT) and all of its implications.

The intuitive condition "ML estimator = sample average" implies the sample is drawn from a Gaussian.

For more, see chapter 7 of: Probability theory: the logic of science (Edwin T. Jaynes), which is an excellent book for understanding probability and statistics.

### Next: [Univariate (scalar) Kalman filtering](T3 - Univariate Kalman filtering.ipynb)


# Solutions: Corollary 0.0.4 in $\mathbb R^2$

*These are **solutions** to the worksheet on corollary 0.0.4. Please **DO NOT LOOK AT IT** if you haven't given the worksheet a fair amount of thought.*

In this worksheet we will run through the proof of Corollary 0.0.4 from Vershynin. We will "pythonize" the proof step-by-step in the case of a polytope in $\mathbb R^2$ and visualize it. Please fill in the code wherever indicated. Here is the corollary (slightly generalized) for reference:

**Corollary 0.0.4 (Generalized)**: Let $P$ be a polytope in $\mathbb R^n$ with $N$ vertices. Then $P$ can be covered by at most $N^{\lceil (\text{diam}(T)/\epsilon)^2 \rceil}$ Euclidean balls of radii $\epsilon > 0$.

```python
# Some useful imports:
import numpy as np
import matplotlib.pyplot as plt
import itertools as it
import math
```

## The Proof of Corollary 0.0.4 in $\mathbb R^2$

Fix $\epsilon > 0$ and a polytope $P$ in $\mathbb R^n$ (in our case $n = 2$). Denote by $T$ the set of vertices of $P$.

```python
# Set epsilon to your favorite positive number
epsilon = 1.0

# Represent a polytope P in R2 by listing its points clockwise
# Represent the points as numpy arrays of length 2
P =  [np.array([0,0]),
      np.array([0,1]),
      np.array([1,1]),
      np.array([1.5,.5]),
      np.array([1,0])]

# Set N to the number of vertices of P (don't hard-code it in if you want to be able to change P easily later)
N = len(P)
```

Let us define the centers of the balls as follows. \
Let $k := \lceil (\text{diam}(T)/\epsilon)^2 \rceil$. Recall that the $\text{diam}(T) = \sup_{x, y \in T} \lvert x - y \rvert$.

```python
# Compute diam(T)
diamT = 0
for i in range(N):
    for j in range(i+1, N):
        d_xixj = np.sqrt(np.sum((P[i] - P[j])**2))
        diamT = d_xixj if (d_xixj > diamT) else diamT

# Compute k
k = math.ceil((diamT/epsilon)**2)
```

Consider the set
\begin{equation}
\mathcal N := \left\{ \frac{1}{k} \sum_{j=1}^k x_j : x_j \text{ are vertices of } P \right\}
\end{equation}

```python
# Construct \mathcal N

calN = []

# This gives an iterator over all combinations
# of k elements of P with replacement.
combinations = it.combinations_with_replacement(P, k)

# Compute the vector average and append it to calN
for comb in combinations:
    vec_sum = np.array([0,0])
    for vec in comb:
        vec_sum = vec_sum + vec
    calN.append(vec_sum / k)
```

We claim that the family of $\epsilon$-balls centered at $\mathcal N$ satisfy the conclusion of the corollary. To check this, note that the polytope $P$ is the convex hull of the set of its vertices. Thus we apply Theorem 0.0.2 to any point $x \in P = \text{conv}(T)$ and deduce that $x$ is within distance $\text{diam(T)} / \sqrt k \leq \epsilon$ from some point in $\mathcal N$. This shows that the $\epsilon$-balls centered at $\mathcal N$ indeed cover $P$.

```python
# We visualize the covering here
# Feel free to play around with the visualization!
scale = 10
figure, axes = plt.subplots(figsize=(scale, scale))
axes.axis('equal')
axes.set_xlim([-1, 1.5])
axes.set_ylim([-1,1.5])
plt.fill([p[0] for p in P], [p[1] for p in P], 'y', fill = True)
plt.plot([p[0] for p in calN], [p[1] for p in calN], 'or')
for p in calN:
    axes.add_artist(plt.Circle((p[0], p[1]), epsilon, fill = False))
```

We can bound the cardinality of $\mathcal N$ by noting that there are $N^k$ ways to choose $k$ out of $N$ vertices with repetition. Thus $|\mathcal N| \leq N^k = N^{\lceil (\text{diam}(T)/\epsilon)^2 \rceil}$. In fact we can be more clever by noticing that the order in which we choose the elements does not matter (this is addressed in exercise 0.0.6).

## Further Questions
At least in $\mathbb R^2$, Corollary 0.0.4 is rather wasteful. How can we come up with a more efficient covering of a polytope?
Is there a way to cleverly construct a subset of $\mathcal N$ that gets the job done?

Copyright (c) 2020 TRIPODS/GradStemForAll 2020 Team


# Stochastic Processes:
Data Analysis and Computer Simulation

# Distribution function and random number

# 4. Random walk

# 4.1. The model system (1D random walk)

# 4.2. As binomial distribution

- The total number of steps to the right: $n_+$
- The total number of steps to the left: $n_-$
- The total number of steps: $m=n_+ + n_-$
- The current location: $l=n_+ - n_-$

\begin{equation}
\therefore \ \ \ n_+=\frac{m+l}{2}, \ \ \ \ \ \ n_-=\frac{m-l}{2}\tag{E1}
\end{equation}

\begin{equation}
%\tilde
{P}(l,m)\rightarrow P_{\rm Binomial}(n_+;m)
%\frac{dn_+}{dl}
=P_{\rm Binomial}(n_-;m)
%\frac{dn_-}{dl}
\tag{E2}
\end{equation}

\begin{equation}
\hspace{20mm}
=
%\frac{1}{2}
\frac{m!}{n_+!(m-n_+)!}{{p}^{n_+}}{{(1-p)}^{m-n_+}}\tag{E3}
\end{equation}

# 4.3. As normal distribution (for $n_+,m\gg 1$)
\begin{equation}
P_{\rm Binomial}(n_+;m)
%\xrightarrow[n_+\to cont.]{n_+,m\gg 1}\
\simeq
\frac{1}{\sqrt{2\pi {{\sigma'}^{2}}}}\exp \left[ -\frac{{{(n_+-{{\mu' }_{1}})}^{2}}}{2{{\sigma' }^{2}}} \right]\tag{E4}
\end{equation}

\begin{equation}
{\rm with}\hspace{5mm}
\mu_1'=\langle n_+\rangle=mp,\ \ \
\sigma'^2=\langle n_+^2\rangle - \langle n_+\rangle^2=mp(1-p) \tag{E5, E6}
\end{equation}

Recall that $n_+=(m+l)~/~2$

\begin{equation}
P_{\rm Binomial}(n_+;m)
%\rightarrow\frac{1}{\sqrt{2\pi mp(1-p)}}
\simeq\exp \left[ -\frac{{{(l-{m(2p-1)})}^{2}}}{8mp(1-p)} \right]\tag{E7}
\end{equation}

\begin{equation}
\therefore \ \
%\tilde
{P}(l,m)= P_{\rm Binomial}(n_+;m)\frac{dn_+}{dl}
= P_{\rm Binomial}(n_+;m)\frac{1}{2}\tag{E8}
\end{equation}

\begin{equation}
\simeq\frac{1}{\sqrt{2\pi \sigma''^2}}
\exp \left[ -\frac{{{(l-{\mu''_1})}^{2}}}{2\sigma''^2} \right]\tag{E9}
\end{equation}

\begin{equation}
%\hspace{20mm}
{\rm with}\hspace{5mm}
\mu''_1=\langle l\rangle=m(2p-1),\ \ \
\sigma''^2=\langle l^2\rangle - \langle l\rangle^2=4mp(1-p) \tag{E10, E11}
\end{equation}

# 4.4. By computer simulation

```python
% matplotlib inline
import numpy as np # import numpy library as np
import math # use mathematical functions defined by the C standard
import matplotlib.pyplot as plt # import pyplot library as plt
plt.style.use('ggplot') # use "ggplot" style for graphs
```

```python
p = 0.5 # set p, propability to take a step to the right
M = 1000 # M = number of total steps
N = 100000 # N = number of independent random walkers
ave = M*(2*p-1) # average of the location L after M steps Eq.(E10)
std = np.sqrt(4*M*p*(1-p)) # standard deviation of L after M steps Eq.(E11)
print('p =',p,'M =',M)
L = np.zeros(N)
np.random.seed(0) # initialize the random number generator with seed=0
for i in range(N): # repeat independent random walks N times
    step=np.random.choice([-1,1],M) # generate random sampling from -1 or +1 M times and store them as step
    L[i]=np.sum(step) # calculate l after making M random steps and store it as L
nmin=np.int(ave-std*5)
nmax=np.int(ave+std*5)
nbin=np.int((nmax-nmin)/4)
plt.hist(L,range=[nmin,nmax],bins=nbin,normed=True) # plot normalized histogram of L
x = np.arange(nmin,nmax,0.01/std) # create array of x from nmin to nmax with increment 0.01/std
y = np.exp(-(x-ave)**2/(2*std**2))/np.sqrt(2*np.pi*std**2) # calculate the theoretical distribution Eq.(E9)
plt.plot(x,y,lw=2,color='b') # plot y vs. x with blue line
plt.xlabel(r'$l$',fontsize=16) # set x-label
plt.ylabel(r'$P(l,m)$',fontsize=16) # set y-label
plt.legend([r'Gauss',r'histogram'], fontsize=16) # set legends
plt.xlim(ave-250,ave+250) # set x-range
plt.show() # display plots
```

- You may repeat the same simulation by choosing different values of total steps, for example $M=100$, $1,000$, $10,000$, and $100,000$ to see how the distribution changes with the total number of steps.

# 4.5. Connection with the diffusion constant $D$

## $P(x,t)$ from random walk

- Define $a$ as the length of a single step and $t_s$ as the time between subsequent steps.
- Define $x=al$ as the position of the random walker and $t=t_s m$ as the duration of time needed to take $m$ steps.
- Here we consider a drift free case $p=0.5$, i.e., $\mu_1=\langle l\rangle=m(2p-1)=0$.

\begin{equation}
P(x,t)=P(l,m)\frac{dl}{dx}=P(l,m)\frac{1}{a}
\tag{E12}
\end{equation}

\begin{equation}
%P_{\rm Binomial}(n_+;m)\rightarrow
=\frac{1}{a\sqrt{8\pi mp(1-p)}}
\exp \left[ -\frac{{l^2}}{8mp(1-p)} \right]\tag{E13}
\end{equation}

\begin{equation}
%P_{\rm Binomial}(n_+;m)\rightarrow
=\frac{1}{\sqrt{8\pi a^2 p(1-p)t/t_s}}
\exp \left[ -\frac{{x^2}}{8a^2p(1-p)t/t_s} \right]\tag{E14}
\end{equation}

\begin{equation}
\hspace{-10mm}
{\rm with}\hspace{5mm}
\mu_1=\langle x\rangle=0,\ \
\sigma^2=\langle x^2\rangle - \langle x\rangle^2=4a^2p(1-p)t/t_s \tag{E15, E16}
\end{equation}

## $P(x,t)$ from the diffusion equation

- Consier the 1-D diffusion equation with diffusion constant $D$
\begin{equation}
\frac{\partial}{\partial t} P(x,t)=D\frac{\partial ^2}{\partial x^2} P(x,t)
\tag{E17}
\end{equation}
\begin{equation}
{\rm with}\hspace{5mm}P(x,t=0)=\delta(x)
\tag{E18}
\end{equation}
- The solution is given by
\begin{equation}
P(x,t)=\frac{1}{\sqrt{4\pi Dt}}
\exp \left[ -\frac{{{x^2}}}{4Dt} \right]\tag{E19}
\end{equation}
- By comparing Eqs.(E14) and (E19) we can relate the diffusion constant $D$ to the variance of the position of random walkers
\begin{equation}
D=\frac{2a^2p(1-p)}{t_s}=\frac{\sigma^2}{2t}\tag{E20}
\end{equation}
- In this case, $\sigma^2$ is also referred to as the mean-square displacement


```python
%matplotlib notebook
from sympy import *
import numpy as np
import matplotlib.pyplot as plt
init_printing()
```

```python
var("r h")
```

```python
import math
A = (2*math.pi*r*h)+(2*math.pi*(r**2))
A
```

```python
eql=Eq(math.pi*(r**2)*h,1000)
eql
```

```python
h_=solve(eql,h)
h_
```

```python
f=A.subs(h,h_[0])
f
```

```python
df=f.diff()
ddf=df.diff()
```

```python
pc=solve(df)
pc
```

```python
eql.subs(r,5.41926)
```

```python
solve(eql.subs(r,5.41926),h)
```

```python

```


# Section 4.5 $\quad$ Linear Independence

**$\underline{Recall}$** The set $W$ of all vectors of the form
$
  \left[\begin{array}{c} a \\ b \\ a+b \\ \end{array} \right]
$
is a subspace of $\mathbb{R}^3$.

### Example 1

Show that each of the following sets is a spanning set for $W$
\begin{equation*}
  S_1 = \left\{
  \left[\begin{array}{c} 1 \\ 0 \\ 1 \\ \end{array} \right],
  \left[\begin{array}{c} 0 \\ 1 \\ 1 \\ \end{array} \right]
  \right\}, ~~~~~
  S_2 = \left\{
  \left[\begin{array}{c} 1 \\ 0 \\ 1 \\ \end{array} \right],
  \left[\begin{array}{c} 0 \\ 1 \\ 1 \\ \end{array} \right],
  \left[\begin{array}{c} 3 \\ 2 \\ 5 \\ \end{array} \right]
  \right\}
\end{equation*}

## Definition of Linear Dependency

>The vectors  $\mathbf{v}_1, \mathbf{v}_2, \cdots, \mathbf{v}_k$ in a vector space $V$ are said to be **linearly dependent** if

**Remark**
- $\mathbf{v}_1, \mathbf{v}_2, \cdots, \mathbf{v}_k$ are linearly independent if,

- $\mathbf{v}_1, \mathbf{v}_2, \cdots, \mathbf{v}_k$ are linearly dependent if,

- If $S = \{\mathbf{v}_1, \mathbf{v}_2, \cdots, \mathbf{v}_k\}$,

### Example 2

Determine whether the vectors
\begin{equation*}
  \mathbf{v}_1 = \left[\begin{array}{c} 3 \\ 2 \\ 1 \\ \end{array} \right],~~~~
  \mathbf{v}_2 = \left[\begin{array}{c} 1 \\ 2 \\ 0 \\ \end{array} \right],~~~~
  \mathbf{v}_3 = \left[\begin{array}{c} -1 \\ 2 \\ -1 \\ \end{array} \right],~~~~
\end{equation*}
are linearly independent.

```python
from sympy import *

A = Matrix([[3, 1, -1], [2, 2, 2], [1, 0, -1]]);

A.rank()
```

    2

### Example 3

Are the vectors
\begin{equation*}
  \mathbf{v}_1 = \left[\begin{array}{cc} 2 & 1 \\ 0 & 1 \\ \end{array} \right],~~~~
  \mathbf{v}_2 = \left[\begin{array}{cc} 1 & 2 \\ 1 & 0 \\ \end{array} \right],~~~~
  \mathbf{v}_3 = \left[\begin{array}{cc} 0 & -3 \\ -2 & 1 \\ \end{array} \right],~~~~
\end{equation*}
in $M_{22}$ are linearly independent?

```python
from sympy import *

A = Matrix([[2, 1, 0, 1], [1, 2, 1, 0], [0, -3, -2, 1]]);

A.rank()
```

    2

### Example 4

Determine whether the vectors
\begin{equation*}
  \mathbf{v}_1 = \left[\begin{array}{c} -1 \\ 1 \\ 0 \\ 0 \\\end{array} \right],~~~~
  \mathbf{v}_2 = \left[\begin{array}{c} -2 \\ 0 \\ 1 \\ 1 \\\end{array} \right],
\end{equation*}
are linearly independent.

```python
from sympy import *

A = Matrix([[-1, 1, 0, 0], [-2, 0, 1, 1]]);

A.rank()
```

    2

>**Theorem** Let $S = \{\mathbf{v}_1, \mathbf{v}_2, \cdots, \mathbf{v}_n\}$ be a set of $n$ vectors in $\mathbb{R}^n$.

### Example 5

Is $S = \{[1~~ 2~~ 3],~~ [0 ~~1~~2],~~ [3~~0~ -1]\}$ a linearly independent set of vectors in $\mathbb{R}^3$?

```python
from sympy import *

A = Matrix([[1, 2, 3], [0, 1, 2], [3, 0, -1]]);

A.rank()
```

    3

>**Theorem** Let $S_1$, $S_2$ be finite subsets of a vector space. Let $S_1$ be a subset of $S_2$. Then
-
-


Computational Seismology

Finite Differences Method - Acoustic Waves in 3D

---

This notebook is part of the supplementary material
to [Computational Seismology: A Practical Introduction](https://global.oup.com/academic/product/computational-seismology-9780198717416?cc=de&lang=en&#),
Oxford University Press, 2016.

##### Authors:
* Heiner Igel ([@heinerigel](https://github.com/heinerigel))
* Lion Krischer ([@krischer](https://github.com/krischer))
* Taufiqurrahman ([@git-taufiqurrahman](https://github.com/git-taufiqurrahman))

---

This notebook covers the following aspects:

* implementation of the 3D acoustic wave equation
* understanding the input parameters for the simulation and the plots that are generated
* understanding the concepts of stability (Courant criterion)
* modifying source and receiver locations and observing the effects on the seismograms
* allowing you to explore the finite-difference method

---

### Getting started
Before you start, make a copy of the original notebook (e.g., orig.ipynb).
Now, run all of the code. Understand the input parameters for the simulation and the plots that are generated. We may modify source and receiver locations and observe the effects on the seismograms. Finally, we can relate the time extrapolation loop with the numerical algorithm we developed.

### Stability (Courant criterion)
Now, we introduce parameter epsilon $\epsilon$ and we want to calculate the Courant criterion.
Determine numerically the stability limit of the code as accurately as possible by increasing the time step.

The Courant criterion is defined as

$$
\epsilon = c \frac{ dt}{dx} \leq 1
$$

With this information we can calculate the maximum possible and stable time step.

### Analytical Solution

In the code below we present the analytical solution for the acoustic wave equation

$$
\partial_t^2 p(x,t) \ - \ c^2 \Delta p(x,t) \ = s(x,t)
$$

assuming constant velocity c and infinite space. Note that in 1D and 2D this equation is mathematically equivalent to the problem of SH wave propagation (i.e., shear waves polarised perpendicular to the plane through source and receiver). In 3D it is (only) descriptive of pressure (sound) waves.

Analytical solution for inhomogeneous partial differential equations (i.e., with non-zero source terms) are usually developed using the concept of Green's functions $G(x, t; x_0, t_0)$. Green's functions are the solutions to the specific partial differential equations for $\delta$-function as source terms evaluated at $(x, t)$ and activated at $(x_0, t_0)$. Thus, we seek solutions to

$$
\partial_t^2 G(x,t;x_0, t_0) \ - \ c^2 \Delta G(x,t;x_0, t_0) \ = \delta (x-x_0) \delta (t-t_0)
$$

where $\Delta$ is the Laplace operator. We recall the definition of the  $\delta-$function as a generalised function with

$$
\delta(x) = \left\{
\begin{array}{ll}
	\infty &x=0 \\
	0 &x\neq 0
\end{array}
\right.
$$

and

$$
\int_{-\infty}^{\infty}\delta(x)dx\ = \ 1 \ , \   \int_{-\infty}^{\infty}f(x)\delta(x)dx\ = \ f(0) \
$$

When comparing numerical with analytical solutions the functions that - in the limit - lead to the $\delta-$function will become very important. An example is the boxcar function

$$
\delta_{bc}(x) = \left\{
\begin{array}{ll}
	1/dx &|x|\leq dx/2 \\
	0 &\text{elsewhere}
\end{array}
\right.
$$

fulfilling these properties as $dx\rightarrow0$. These functions are used to properly scale the source terms to obtain correct absolute amplitudes.

To describe analytical solutions for the acoustic wave equation we also make use of the unit step function, also known as the Heaviside function, defined as

$$
H(x) = \left\{
\begin{array}{ll}
    0 &x<0 \\
	1 &x \geq 0
\end{array}
\right.
$$

The Heaviside function is the integral of the $\delta-$function (and vice-versa the $\delta$-function is defined as the derivative of the Heaviside function). in 3D the Green's function is proportional to a $\delta-$function scaled by the distance from the source (geometrical spreading). Beware of the practical significance of this result for sound wave propagation

$$
G = \frac{1}{4 \pi c^2 r} \delta(t-r/c)
$$

$$
r = \sqrt{x^2+y^2+z^2}
$$

### Numerical Solution (Finite Differences Method)

In 3D the constant-density acoustic wave equation is given by

$$
\ddot{p}(x,y,z,t) \ = \ c(x,y,z)^2 (\partial_x^2 p(x,y,z,t) + \partial_y^2 p(x,y,z,t) + \partial_z^2 p(x,y,z,t)) \ + s(x,y,z,t)
$$

In accordance with the above developments we discretise space-time using

$$
p(x,y,z,t) \ \rightarrow \ p^n_{i,j,k} \ = \ p(n dt, i dx, j dy, k dz) \ .
$$

Using the 3-point operator for the 2nd derivatives in time leads us to the extrapolation scheme

$$
 \frac{p_{i,j,k}^{n+1} - 2 p_{i,j,k}^n + p_{i,j,k}^{n-1}}{dt^2} \ = \ c^2 ( \partial_x^2 p + \partial_y^2 p + \partial_z^2 p) \ + s_{i,j,k}^n
$$

where on the r.h.s. the space and time dependencies are implicitly assumed and the partial derivatives are approximated by

\begin{equation}
\begin{split}
\partial_x^2 p \ &= \ \frac{p_{i+1,j,k}^{n} - 2 p_{i,j,k}^n + p_{i-1,j,k}^{n}}{dx^2} \\
\partial_y^2 p \ &= \ \frac{p_{i,j+1,k}^{n} - 2 p_{i,j,k}^n + p_{i,j-1,k}^{n}}{dy^2} \\
\partial_z^2 p \ &= \ \frac{p_{i,j,k+1}^{n} - 2 p_{i,j,k}^n + p_{i,j,k-1}^{n}}{dz^2}  \ .
\end{split}
\end{equation}
Note that for a regular 3D grid $dz=dy=dx$

### Analytical and Numerical Comparisons
The code below is given with a 3-point difference operator. Compare the results from the numerical simulation with the 3-point operator with the analytical solution.

### High-order operators
Extend the code to higher order by adding the option to use a 5-point difference operator. The 5-pt weights are:
$
[-1/12, 4/3, -5/2, 4/3, -1/12] / dx^2
$.
Compare simulations of the 3-point and 5-point operators.

---

```python
# Import Libraries (PLEASE RUN THIS CODE FIRST!)
# ----------------------------------------------
import numpy as np
import matplotlib
# Show Plot in The Notebook
matplotlib.use("nbagg")
import matplotlib.pyplot as plt

# Sub-plot Configuration
# ----------------------
from matplotlib import gridspec
from mpl_toolkits.axes_grid1 import make_axes_locatable

# Ignore Warning Messages
# -----------------------
import warnings
warnings.filterwarnings("ignore")
```

```python
# Parameter Configuration
# -----------------------

nx   = 100           # number of grid points in x-direction
ny   = nx            # number of grid points in y-direction
nz   = nx            # number of grid points in z-direction
# Note that for a regular 3D grid dz = dy = dx
dx   = 0.5           # grid point distance in x-direction
dy   = dx            # grid point distance in y-direction
dz   = dx            # grid point distance in z-direction
c0   = 250.          # wave velocity in medium (m/s)
isx  = 50            # source location in grid in x-direction
isy  = isx           # source location in grid in y-direction
isz  = isx           # source location in grid in z-direction
irx  = 57            # receiver location in grid in x-direction
iry  = irx           # receiver location in grid in y-direction
irz  = irx           # receiver location in grid in z-direction
nt   = 251           # maximum number of time steps
dt   = 0.0005        # time step

# CFL Stability Criterion
# -----------------------
eps  = c0 * dt / dx  # epsilon value

print('Stability criterion =', eps)
```

    Stability criterion = 0.25

```python
# Plot Source Time Function
# -------------------------

f0   = 100. # dominant frequency of the source (Hz)
t0   = 4. / f0 # source time shift

print('Source frequency =', f0, 'Hz')

# Source time function (Gaussian)
# -------------------------------
src  = np.zeros(nt + 1)
time = np.linspace(0 * dt, nt * dt, nt)
# 1st derivative of a Gaussian
src  = -2. * (time - t0) * (f0 ** 2) * (np.exp(-1.0 * (f0 ** 2) * (time - t0) ** 2))

# Plot Source Time Function
# -------------------------
plt.ion()
fig1  = plt.figure(figsize=(12, 6))
gs1   = gridspec.GridSpec(1, 2, width_ratios=[1, 1], hspace=0.3, wspace=0.3)

# Plot Source Time Function
# -------------------------
ax1  = plt.subplot(gs1[0])
ax1.plot(time, src) # plot source time function
ax1.set_title('Source Time Function')
ax1.set_xlim(time[0], time[-1])
ax1.set_xlabel('Time (s)')
ax1.set_ylabel('Amplitude')

# Plot Source Spectrum
# --------------------
ax2  = plt.subplot(gs1[1])
spec = np.fft.fft(src) # amplitude of the source in frequency domain
freq = np.fft.fftfreq(spec.size, d = dt / 4.) # change time domain to frequency domain of the source
ax2.plot(np.abs(freq), np.abs(spec)) # plot frequency and spectrum of the source
ax2.set_xlim(0, 500) # only display frequency from 0 to 500 Hz
ax2.set_title('Source Spectrum')
ax2.set_xlabel('Frequency (Hz)')
ax2.set_ylabel('Amplitude')

ax2.yaxis.tick_right()
ax2.yaxis.set_label_position("right")

plt.show()
```

    Source frequency = 100.0 Hz

    <IPython.core.display.Javascript object>

```python
# Plot 2D Slices & Seismogram (PLEASE RERUN THIS CODE AGAIN AFTER SIMULATION!)
# ---------------------------------------------------------------------------

# Initialize Empty Pressure
# -------------------------
p    = np.zeros((nx, ny, nz)) # p at time n (now)
pold = np.zeros((nx, ny, nz)) # p at time n-1 (past)
pnew = np.zeros((nx, ny, nz)) # p at time n+1 (present)
d2px = np.zeros((nx, ny, nz)) # 2nd space derivative of p in x-direction
d2py = np.zeros((nx, ny, nz)) # 2nd space derivative of p in y-direction
d2pz = np.zeros((nx, ny, nz)) # 2nd space derivative of p in z-direction

# Initialize Velocity Model (assume homogeneous model)
# ----------------------------------------------------
c    = np.zeros((nx, ny, nz))
c    = c + c0                 # initialize wave velocity in model

# Initialize Grid
# ---------------
x    = np.arange(nx)
x    = x * dx                 # coordinate in x-direction
y    = np.arange(ny)
y    = y * dy                 # coordinate in y-direction
z    = np.arange(nz)
z    = z * dz                 # coordinate in z-direction

# Initialize Empty Seismogram
# ---------------------------
seis = np.zeros(nt)

# Analytical Solution
# -------------------
G    = time * 0.0
r    = np.sqrt((x[isx] - x[irx]) ** 2 + (y[isy] - y[iry]) ** 2 + (z[isz] - z[irz]) ** 2)
t_ar = time[0] + r / c0
i_ar = int(t_ar / dt)
G[i_ar] = 1.0 / (4 * np.pi * c0 ** 2 * r) # Calculate Green's function
Gc   = np.convolve(G, src)
Gc   = Gc[0:nt]
lim  = Gc.max() # get limit value from maximum amplitude of analytical solution

# Plot Position Configuration
# ---------------------------
plt.ion()
fig2 = plt.figure(figsize=(12, 12))
gs2  = gridspec.GridSpec(2,2,width_ratios=[1,1],height_ratios=[1,1],hspace=0.15, wspace=0.3)

# Plot 3D Wave Propagation
# ------------------------
# Note: comma is needed to update the variable

# 2D Slice at nz
# --------------
ax3  = plt.subplot(gs2[0])
leg1,= ax3.plot(isx, isz, 'r*', markersize=11) # plot source location
leg2,= ax3.plot(irx, irz, 'k^', markersize=8) # plot receiver location
slc1,= plt.plot(np.array([0,nx]),np.array([irx,irx]), linewidth=1, linestyle='--') # plot 2D slice at nx
slc2,= plt.plot(np.array([irx,irx]),np.array([0,ny]), linewidth=1, linestyle='--') # plot 2D slice at ny
im3  = ax3.imshow(p[:,:,isx],vmin=-lim, vmax=+lim, interpolation='nearest', cmap=plt.cm.RdBu)
div3 = make_axes_locatable(ax3)
cax3 = div3.append_axes("right", size="5%", pad=0.05) # size & position of colorbar
fig2.colorbar(im3, cax=cax3) # plot colorbar
ax3.set_title('Slice at nz = %d' % isz)
ax3.set_xlim(0, nz)
ax3.set_ylim(0, nz)
ax3.set_xlabel('nx')
ax3.set_ylabel('ny')
ax3.legend((leg1, leg2), ('Source', 'Receiver'), loc='upper right', fontsize=10, numpoints=1)

# 2D Slice at nx
# --------------
ax4  = plt.subplot(gs2[1])
im4  = ax4.imshow(p[irx,:,:],vmin=-lim, vmax=+lim, interpolation="nearest", cmap=plt.cm.RdBu)
div4 = make_axes_locatable(ax4)
cax4 = div4.append_axes("right", size="5%", pad=0.05) # size & position of colorbar
fig2.colorbar(im4, cax=cax4) # plot colorbar
ax4.set_title('Slice at nx = %d' % irx)
ax4.set_xlim(0, nz)
ax4.set_ylim(0, ny)
ax4.set_xlabel('nz')
ax4.set_ylabel('ny')

# 2D Slice at ny
# --------------
ax5  = plt.subplot(gs2[2])
im5  = ax5.imshow(p[:,irx,:],vmin=-lim, vmax=+lim, interpolation="nearest", cmap=plt.cm.RdBu)
div5 = make_axes_locatable(ax5)
cax5 = div5.append_axes("right", size="5%", pad=0.05) # size & position of colorbar
fig2.colorbar(im5, cax=cax5) # plot colorbar
ax5.set_title('Slice at ny = %d' % iry)
ax5.set_xlim(0, nz)
ax5.set_ylim(0, ny)
ax5.set_xlabel('nx')
ax5.set_ylabel('nz')

# Plot Seismogram
# ---------------
# Note: comma is needed to update the variable
ax6  = plt.subplot(gs2[3])
up61,= ax6.plot(time, seis) # update seismogram each time step
up62,= ax6.plot([0], [0], 'r|', markersize=15) # update time step position
ax6.set_ylim(-lim, lim)
ax6.set_xlim(time[0], time[-1])
ax6.set_title('Seismogram - Time Step (nt) = 0')
ax6.set_xlabel('Time [s]')
ax6.set_ylabel('Amplitude')
leg3,= ax6.plot(0,0,'r--',markersize=1)
leg4,= ax6.plot(0,0,'b-',markersize=1)
ax6.legend((leg3, leg4), ('Analytical', 'FD'), loc='upper right', fontsize=10, numpoints=1)

ax6.yaxis.tick_right()
ax6.yaxis.set_label_position("right")

plt.plot(time,Gc,'r--')
plt.show()
```

    <IPython.core.display.Javascript object>

```python

```

#### Solutions:

```python
# 3D Wave Propagation (Finite Difference Solution)
# ------------------------------------------------

# Point Operator (choose 3 or 5 point operator)
# ---------------------------------------------
op   = 5
print(op, '- point operator')

# Calculate Partial Derivatives
# -----------------------------
for it in range(nt):
    if op == 3: # use 3 point operator FD scheme
        for i in range(1, nz - 1):
                d2pz[:, :, i] = (p[:, :, i + 1] - 2 * p[:, :, i] + p[:, :, i - 1]) / dz ** 2
        for j in range(1, ny - 1):
                d2py[:, j, :] = (p[:, j + 1, :] - 2 * p[:, j, :] + p[:, j - 1, :]) / dy ** 2
        for k in range(1, nx - 1):
                d2px[k, :, :] = (p[k + 1, :, :] - 2 * p[k, :, :] + p[k - 1, :, :]) / dx ** 2

    if op == 5: # use 5 point operator FD scheme
        for i in range(2, nz - 2):
                d2pz[:, :, i] = (-1./12 * p[:,:,i+2] + 4./3  * p[:,:,i+1] - 5./2 * p[:,:,i] \
                                + 4./3  * p[:,:,i-1] - 1./12 * p[:,:,i-2]) / dz ** 2
        for j in range(2, ny - 2):
                d2py[:, j, :] = (-1./12 * p[:,j+2,:] + 4./3  * p[:,j+1,:] - 5./2 * p[:,j,:] \
                                + 4./3  * p[:,j-1,:] - 1./12 * p[:,j-2,:]) / dy ** 2
        for k in range(2, nx - 2):
                d2px[k, :, :] = (-1./12 * p[k+2,:,:] + 4./3  * p[k+1,:,:] - 5./2 * p[k,:,:] \
                                + 4./3  * p[k-1,:,:] - 1./12 * p[k-2,:,:]) / dx ** 2

    # Time Extrapolation
    # ------------------
    pnew = 2 * p - pold + (c ** 2) * (dt ** 2) * (d2px + d2py + d2pz)

    # Add Source Term at isx, isy and isz
    # -----------------------------------
    pnew[isx, isy, isz] = pnew[isx, isy, isz] + src[it] / (dx * dy * dz) * (dt ** 2)

    # Remap Time Levels
    # -----------------
    pold, p = p, pnew

    # Output Seismogram
    # -----------------
    seis[it] = p[irx, iry, irz]

    # Update data for Wave Propagation Plot
    # -------------------------------------
    idisp = 5 # display frequency
    if (it % idisp) == 0:
        ax3.imshow(p[:, :, isx],vmin=-lim, vmax=+lim, interpolation="nearest", cmap=plt.cm.RdBu)
        ax4.imshow(p[irx, :, :],vmin=-lim, vmax=+lim, interpolation="nearest", cmap=plt.cm.RdBu)
        ax5.imshow(p[:, irx, :],vmin=-lim, vmax=+lim, interpolation="nearest", cmap=plt.cm.RdBu)
        ax6.set_title('Seismogram - Time Step (nt) = %d' % it)
        up61.set_ydata(seis)
        up62.set_data(time[it], seis[it])
        fig2.canvas.draw()
```

    5 - point operator

```python

```

notebooks/Computational Seismology/The Finite-Difference Method/fd_ac3d_homogeneous_solution.ipynb


# Code sample

```python
def primes(n):
    if n==2:
        return [2]
    elif n<2:
        return []

    s=range(3,n+1,2)
    mroot = n ** 0.5
    half=(n+1)//2-1
    i=0
    m=3

    while m <= mroot:
        if s[i]:
            j=(m*m-3)/2
            s[j]=0
        while j<half:
            s[j]=0
            j+=m
        i=i+1
        m=2*i+3
    x = [2]
    for y in s:
        if y:
            x.append(y)
    return x
```

```python
print(primes(100))
```

## Python version

```python
def primes1(n):
    if n==2:
        return [2]
    elif n<2:
        return []

    s=[i for i in range(3,n+1,2)]  # python 3 syntax
    mroot = n ** 0.5
    half=(n+1)//2-1
    i=0
    m=3

    while m <= mroot:
        if s[i]:
            j=(m*m-3)//2  # python 3 syntax
            s[j]=0
        while j<half:
            s[j]=0
            j+=m
        i=i+1
        m=2*i+3
    x = [2]
    for y in s:
        if y:
            x.append(y)
    return x
```

```python
print(primes1(100))
```

    [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97]

## Library function and dot operation

```python
from math import sqrt  # import the function

def primes2(n):
    if n==2:
        return [2]
    elif n<2:
        return []

    s=[i for i in range(3,n+1,2)]
    mroot = sqrt(n)  # add the library function
    half=(n+1)//2-1
    i=0
    m=3

    while m <= mroot:
        if s[i]:
            j=(m*m-3)//2
            s[j]=0
        while j<half:
            s[j]=0
            j+=m
        i=i+1
        m=2*i+3
    x = [2]
    for y in s:
        if y:
            x.append(y)
    return x
```

```python
print(primes2(100))
```

    [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97]

## List Comprehension

```python
from math import sqrt

def primes3(n):
    if n==2:
        return [2]
    elif n<2:
        return []

    s=[i for i in range(3,n+1,2)]  # list comprehension syntax
    mroot = sqrt(n)
    half=(n+1)//2-1
    i=0
    m=3

    while m <= mroot:
        if s[i]:
            j=(m*m-3)//2
            s[j]=0
        while j<half:
            s[j]=0
            j+=m
        i=i+1
        m=2*i+3
    return [2] + [x for x in s if x]  # list comprehension syntax
```

```python
print(primes3(100))
```

    [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97]

## Data Structure

```python
from numpy import arange
from math import sqrt

def primes4(n):
    if n==2:
        return [2]
    elif n<2:
        return []

    s=arange(3,n+1,2)  # numpy array
    mroot = sqrt(n)
    half=(n+1)//2-1
    i=0
    m=3

    while m <= mroot:
        if s[i]:
            j=(m*m-3)//2
            s[j]=0
        while j<half:
            s[j]=0
            j+=m
        i=i+1
        m=2*i+3
    return [2] + [x for x in s if x]
```

```python
print(primes4(100))
```

    [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97]

# Benchmarking

### Creating a benchmarking function

```python
from time import time

def benchmarking(function, n_repeat):
    ti = time()
    for _ in range (n_repeat):
        function

    return (time() - ti)
```

```python
print(benchmarking(primes1(100), 100000000))  # using python 3
```

    1.99668288230896

```python
print(benchmarking(primes2(100), 100000000))  # using library function
```

    1.9916484355926514

```python
print(benchmarking(primes3(100), 100000000))  # using list comprehension syntax
```

    1.9629108905792236

```python
print(benchmarking(primes4(100), 100000000))  # using numpy array
```

    1.9116380214691162

### Module: [timeit](https://docs.python.org/3/library/timeit.html)

Using IPython magic commands

```python
%timeit primes1(100)  # using python 3
```

    5.03 µs ± 57.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

```python
%timeit primes2(100)  # using library function
```

    4.94 µs ± 26.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

```python
%timeit primes3(100)  # using list comprehension syntax
```

    4.45 µs ± 56.9 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

```python
%timeit primes4(100)  # using numpy array
```

    9.46 µs ± 485 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

# CPU profiling

Using IPython magic commands

```python
%load_ext line_profiler
```

```python
%lprun -f primes1 primes1(100)  # using python 3
```

```python
%lprun -f primes4 primes4(100)  # using numpy array
```

## Improving the code?

From here it is an iterative process.

```python
from math import sqrt

def primes5(n):
    if n==2:
        return [2]
    elif n<2:
        return []

    s=list(range(3,n+1,2))  # list
    mroot = sqrt(n)
    half=(n+1)//2-1
    i=0
    m=3

    while m <= mroot:
        if s[i]:
            j=(m*m-3)//2
            s[j]=0
        while j<half:
            s[j]=0
            j+=m
        i=i+1
        m=2*i+3
    return [2] + [x for x in s if x]
```

```python
print(primes5(100))
```

    [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97]

```python
%timeit primes5(100)  # using library function
```

    4.05 µs ± 89.3 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

# Native-speed code

* [Numba](http://numba.pydata.org/)
* [Cython](https://cython.org/)
* [F2PY](https://numpy.org/doc/stable/f2py/)

## Numba

```python
from math import sqrt
from numba import jit

@jit
def primes6(n):
    if n==2:
        return [2]
    elif n<2:
        return [0]  # all returns need have the same tipe. Zero is not rigth!

    s=list(range(3,n+1,2))  # list
    mroot = sqrt(n)
    half=(n+1)//2-1
    i=0
    m=3

    while m <= mroot:
        if s[i]:
            j=(m*m-3)//2
            s[j]=0
        while j<half:
            s[j]=0
            j+=m
        i=i+1
        m=2*i+3
    return [2] + [x for x in s if x]
```

```python
print(primes6(100))
```

    [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97]

```python
%timeit primes6(100)  # using library function
```

    982 ns ± 20.6 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

# Parallel running

[Multiprocessing](https://docs.python.org/3/library/multiprocessing.html)

```python
from sympy import isprime

def prime(i):
    if (isprime(i)):
        return(i)
```

```python
from multiprocessing import Pool

def multi_primes(N):
    pool = Pool(processes=8)
    input_number = [i for i in range(N)]
    x = pool.map(prime, input_number)
    return [i for i in x if i!=None]
```

```python
print(multi_primes(100))
```

    [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97]

```python
benchmarking(multi_primes(5000000),1)
```

    2.1457672119140625e-06

```python
benchmarking(primes6(5000000),1)
```

    2.384185791015625e-06


# Computing the capacity of a cube with a re-entrant corner

### Background

The capacity $\text{cap}(\Omega)$ of an isolated conductor $\Omega\subset\mathbb{R}^3$ with boundary $\Gamma$ measures its ability to store charges. It is defined as the ratio of the total surface equilibrium charge relative to its surface potential value. To compute the capacity, we need to solve the following exterior Laplace problem for the equilibrium potential $u$ with unit surface value:
$$
\begin{align}
-\Delta u &= 0\quad\text{in }\Omega^\text{+},\\
u &= 1\quad\text{on }\Gamma,\\
|u(\mathbf{x})| &=\mathcal{O}\left(|\mathbf{x}|^{-1}\right)\quad\text{as }|\mathbf{x}|\rightarrow\infty.
\end{align}
$$
Here $\Omega^\text{+}$ is the domain exterior to $\Omega$.
The total surface charge of an isolated conductor is given by Gauss law as
$$
\text{cap}(\Omega)=-\epsilon_0\int_{\Gamma}\frac{\partial u}{\partial\nu}(\mathbf{x})\,\mathrm{d}\mathbf{x}.
$$
Here, $\nu(\mathbf{x})$ is the outward pointing normal direction for $\mathbf{x}\in\Gamma$, and $\epsilon_0$ is the electric constant with value $\epsilon_0\approx 8.854\times 10^{-12}\,{\rm F/m}$. For simplicity, in the following we will exclude this constant and use the expression $\text{cap}(\Omega)=-\int_{\Gamma}\partial u/\partial\nu\,d\mathbf{x}$.

Using Green's representation theorem and noting that the exterior Laplace double layer potential is zero for constant densities, we can represent the solution $u$ as
$$
u(x) = -\int_{\Gamma} g(\mathbf{x},\mathbf{y})\phi(\mathbf{y})\,\mathrm{d}\mathbf{y} \quad\text{for all }\mathbf{x}\in\Omega^\text{+},
$$
where $\phi:={\partial u}/{\partial\nu}$ is the normal derivative of the exterior solution $u$ and $g(\mathbf{x},\mathbf{y}):=\frac{1}{4\pi|\mathbf{x}-\mathbf{y}|}$ is the Green's function of the 3D Laplacian. By taking boundary traces, we arrive at the following boundary integral equation of the first kind.
$$
1 = -\int_{\Gamma} g(\mathbf{x},\mathbf{y})\phi(\mathbf{y})\,\mathrm{d}\mathbf{y} =: -\mathsf{V}\phi(\mathrm{x})\quad\text{for all }\mathbf{x}\in\Gamma.
$$
The capacity is now simply given by
$$
\text{cap}(\Omega) = -\int_\Gamma \phi(\mathbf{x}) \,\mathrm{d}\mathbf{x}.
$$

To improve the convergence we will solve the preconditioned equation
$$
\mathsf{V}\tilde{\mathsf{W}}\phi = -1,
$$
where $\tilde{\mathsf{W}}$ is a regularised hypersingular operator defined by the weak form
$$
\langle \tilde{\mathsf{W}}w, v\rangle = \langle \mathsf{W}w, v\rangle + \langle w, 1\rangle\langle v, 1\rangle.
$$

### Implementation

We start with the usual imports.

```python
import bempp.api
import numpy as np
```

The grid re-entrant cube is predefined in the shapes module. By default it refines towards the singular corner.

```python
grid = bempp.api.shapes.reentrant_cube(h=0.05, refinement_factor=1)
```

Next, we define the right-hand side and the operator. We discretise a pair of hypersingular and single layer operators. Using the corresponding Bempp function guarantees that only one single layer operator needs to be discretised. The parameter `dual` means that we discretise the single-layer on the dual grid using piecewise constant basis functions and the hypersingular operator on the original grid using piecewise linear continuous basis functions. The regularisation is done via a built-in rank one operator that is added to the hypersingular operator.

```python
def one_fun(x, n, domain_index, res):
    res[0] = -1

slp, hyp, base_slp = \
    bempp.api.operators.boundary.laplace.single_layer_and_hypersingular_pair(
        grid, spaces='dual', return_base_slp=True)

rank_one_op = bempp.api.RankOneBoundaryOperator(
    hyp.domain, hyp.range, hyp.dual_to_range)
hyp_regularized = hyp + rank_one_op

rhs_fun = bempp.api.GridFunction(slp.range, fun=one_fun)

lhs = slp * hyp_regularized
```

We use GMRES to solve the system. To improve convergence we use a strong form discretisation that automatically preconditions with the mass matrix.

```python
x, info, it_count = bempp.api.linalg.gmres(
    lhs, rhs_fun, use_strong_form=True,
    return_iteration_count=True)

sol_fun = hyp_regularized * x

print("Number of iterations: {0}".format(it_count))

print("The capacity is {0}.".format(-sol_fun.integrate()[0,0]))
```

    Number of iterations: 6
    The capacity is 8.1104817108.

We now want to compute local residual error estimates for the computed solution. To do this, we use the already calculates single layer operator on the barycentrically refined grid.

First, we compute the map from element indices of the original grid to descendent element indices on the barycentric refinement. ``bary_map[i,:]`` are the indices of the elements in the barycentric refinement that are associated with the element with index ``i`` on the original grid.

```python
bary_map = grid.barycentric_descendents_map()
```

For the error estimator we want to compute element diameters. This is handled by the following function.

```python
def diameter(element):
    """Compue the diameter of an element."""
    corners = element.geometry.corners
    d0 = np.linalg.norm(corners[:,0] - corners[:,1])
    d1 = np.linalg.norm(corners[:,0] - corners[:, 2])
    d2 = np.linalg.norm(corners[:,1] - corners[:, 2])
    return np.max([d0, d1, d2])
```

We now evaluate the solution on the barycentrically refined grid. The function space on the barycentric refinement is the space of piecewise linear, discontinuous functions. This was used automatically by Bempp to assemble the base single layer operator on the barycentric refinement.

We define an identity mapping the solution into this space.

```python
map_to_base_space = bempp.api.operators.boundary.sparse.identity(sol_fun.space, base_slp.domain, base_slp.domain)
base_sol_fun = map_to_base_space * sol_fun

base_slp_times_sol = base_slp * base_sol_fun
```

We now evaluate the squared error estimate $\eta_{T}^2 = \text{diam}(T)\|\nabla \mathsf{V}\phi\|_{\mathcal{L}^2(T)}^2$ on each element $T$ of the barycentric refinement, and sum of the contributions of the barycentric triangles associated with each triangle of the original grid. We can ignore the right-hand side function as it is constant and therefore its surface gradient is zero.

```python
bary_grid = grid.barycentric_grid()

local_errors_squared_bary = np.zeros(
    bary_grid.leaf_view.entity_count(0), dtype='float64')
local_errors = np.zeros(
    grid.leaf_view.entity_count(0), dtype='float64')
bary_index_set = bary_grid.leaf_view.index_set()

# Compute the error estimates on the barycentric refinement

for element in bary_grid.leaf_view.entity_iterator(0):
    index = bary_index_set.entity_index(element)
    gradient_norm = base_slp_times_sol.surface_grad_norm(element)
    local_error_squared = diameter(element) * gradient_norm**2
    local_errors_squared_bary[index] = local_error_squared

# Sum up the local contributions for each triangle to obtain an error estimate on the
# original grid.

for m in range(bary_map.shape[0]):
    for n in range(bary_map.shape[1]):
        local_errors[m] += local_errors_squared_bary[bary_map[m, n]]

local_errors = np.sqrt(local_errors)
total_error = np.linalg.norm(local_errors)

print("Total H^(-1/2) error estimate for normal derivative: {0}".format(
        np.linalg.norm(local_errors)))
print("Error estimate for capacity: {0}".format(total_error**2))
```

    Total H^(-1/2) error estimate for normal derivative: 0.0
    Error estimate for capacity: 0.0

We now want to plot the errors on each element. We first have to reorder the vector with elementwise errors to fit to the indices of a piecewise constant function space. The numbering of the two is not necessarily identical and depends on the underlying grid manager. We use the ``index_set`` to access the numberings of elements.

```python
const_space = bempp.api.function_space(grid, "DP", 0)

# Resort the error contributions
sorted_local_errors = np.zeros_like(local_errors)
index_set = grid.leaf_view.index_set()
for element in grid.leaf_view.entity_iterator(0):
    index = index_set.entity_index(element)
    global_dof_index = const_space.get_global_dofs(element, dof_weights=False)[0]
    sorted_local_errors[global_dof_index] = local_errors[index]

bempp.api.GridFunction(const_space, coefficients=sorted_local_errors).plot()
```


```latex
%%latex
We define the binomial distribution $f$ as
\begin{align}
\Theta &= 0.011 \\
f(k;n,\Theta) &= {n \choose k}\Theta^k(1-\Theta)^{n-k}\\
\end{align}
Let $X$ be the measured isotope fractions, and $Y$ the real isotope fractions (i.e. if the natural abundance was zero).
\begin{align}
X_0 &= f(0;N,\Theta) \cdot Y_0 \\
X_1 &= f(1;N,\Theta) \cdot Y_0 + f(0;N-1,\Theta) \cdot Y_1 \\
X_2 &= f(2;N,\Theta) \cdot Y_0 + f(1;N-1,\Theta) \cdot Y_1 + f(0;N-2,\Theta) \cdot Y_2 \\
X_i &= \sum_{j=0}^{i} f(i-j;N-j,\Theta) \cdot Y_j
\end{align}

So we can define a matrix $F$, such that:
$$\forall i \ge j ~~~ F_{i,j} = f(i-j;N-j,\Theta) = {N-j \choose i-j} \Theta^{i-j} (1-\Theta)^{N-i}$$
and the rest are zeros. Then it is easy to see that $X = FY$ and since $F$ is non-singular we can solve $Y = F^{-1}X$.
```

We define the binomial distribution $f$ as
\begin{align}
\Theta &= 0.011 \\
f(k;n,\Theta) &= {n \choose k}\Theta^k(1-\Theta)^{n-k}\\
\end{align}
Let $X$ be the measured isotope fractions, and $Y$ the real isotope fractions (i.e. if the natural abundance was zero).
\begin{align}
X_0 &= f(0;N,\Theta) \cdot Y_0 \\
X_1 &= f(1;N,\Theta) \cdot Y_0 + f(0;N-1,\Theta) \cdot Y_1 \\
X_2 &= f(2;N,\Theta) \cdot Y_0 + f(1;N-1,\Theta) \cdot Y_1 + f(0;N-2,\Theta) \cdot Y_2 \\
X_i &= \sum_{j=0}^{i} f(i-j;N-j,\Theta) \cdot Y_j
\end{align}

So we can define a matrix $F$, such that:
$$\forall i \ge j ~~~ F_{i,j} = f(i-j;N-j,\Theta) = {N-j \choose i-j} \Theta^{i-j} (1-\Theta)^{N-i}$$
and the rest are zeros. Then it is easy to see that $X = FY$ and since $F$ is non-singular we can solve $Y = F^{-1}X$.

```python
#######################################################
# Here you place the input data - counts for each mass
# starting with M+0 (of course)
#
counts = [900, 100, 5, 900, 5000]
#
#######################################################

import numpy as np
from scipy.misc import comb
theta = 0.011

N = len(counts)
X = np.matrix(counts, dtype=float).T
X = X / X.sum()

F = np.matrix(np.zeros((N, N)))
for i in xrange(N):
    for j in xrange(i+1):
        F[i,j] = comb(N-j, i-j) * theta**(i-j) * (1-theta)**(N-j)

Y = F.I * X

print "The corrected isotope relative abundances are:"
print '-'*50
print '  |  '.join(map(lambda d: ' M + %d' % d, range(N)))
print '  |  '.join(map(lambda s: '%5.2f%%' % (s*100), Y.flat))
print '-'*50

```

    The corrected isotope relative abundances are:
    --------------------------------------------------
     M + 0  |   M + 1  |   M + 2  |   M + 3  |   M + 4
    13.78%  |   0.76%  |   0.03%  |  13.32%  |  72.93%
    --------------------------------------------------


```python
import sympy as sy
from IPython.display import display

omega, zeta, md, kd, dd, ke, de = sy.symbols("omega, zeta, md, kd, dd, ke, de")

hoge = sy.Symbol("hoge")

def calc_omega():
    return sy.sqrt((kd + ke) / md)

def calc_zeta():
    return (dd + de) / (2*sy.sqrt(md*(kd+ke)))

sol = sy.solve(
    [omega - calc_omega(), zeta - calc_zeta(), dd - hoge],
    [md, kd, dd]
)
```

```python
sol
```

    [(-(de + hoge)/(2*omega*zeta), -(ke*zeta + omega*(de + hoge)/2)/zeta, hoge),
     ((de + hoge)/(2*omega*zeta), (-ke*zeta + omega*(de + hoge)/2)/zeta, hoge)]

```python
sol[0]
```

    (-(de + hoge)/(2*omega*zeta), -(ke*zeta + omega*(de + hoge)/2)/zeta, hoge)

```python
sol[1]
```

    ((de + hoge)/(2*omega*zeta), (-ke*zeta + omega*(de + hoge)/2)/zeta, hoge)


$\def\*#1{\mathbf{#1}}$
$\DeclareMathOperator*{\argmax}{arg\,max}$

# Data Classification

```python
import time

import numpy as np
import scipy as sp
import matplotlib.pyplot as plt
import pandas as pd
import sklearn.datasets as datasets
import sklearn.metrics as metrics
import sklearn.utils as utils

import scipy.sparse.linalg as linalg

#classification
from sklearn.neighbors import KNeighborsClassifier
from sklearn import tree

from scipy.spatial.distance import pdist, squareform
from sklearn.preprocessing import StandardScaler

import seaborn as sns

%matplotlib inline
%config InlineBackend.figure_format = 'svg'
```

## Introduction

#### Classifier

A classifier is a function $M$ that predicts the class label $\hat{y} \in \{c_1, c_2, \ldots, c_k\}$ for a given point $\*x \in \mathbb{R}^d$ :

$$\hat{y} = M(\*x)$$

#### Supervised Learning

1. Collect a **Training Data Set** composed of points along with their known classes
2. Train Classifier
3. Make Predictions

#### Collect Training Data

A training set of points with their known classes :

|Weight|Texture|Label |
|------|-------|------|
|140g  |Smooth |Apple |
|130g  |Smooth |Apple |
|150g  |Bumpy  |Orange|
|170g  |Bumpy  |Orange|

```python
features = [[140, 1], [130, 1], [150, 0], [170, 0]]

labels = [0, 0, 1, 1]
```

#### Train Classifier

```python
classifier = tree.DecisionTreeClassifier().fit(features, labels)
```

#### Make Predictions

```python
print(classifier.predict([[160, 0]]))
```

    [1]

## Decision Tree

### Iris dataset

```python
# http://scikit-learn.org/stable/auto_examples/tree/plot_iris.html

# Load data
iris = datasets.load_iris()

# Select the two first features, i.e., sepal length and width
features = [0, 1]
X = iris.data[:, features]
y = iris.target

# Considert two classes : iris-setora and the other two types of irises
classes = [0, 1]
class_names = [iris.target_names[0], 'others']
n_classes = 2
y[y > 1] = 1

print("iris-setora :", np.sum(y == 0))
print("others :", np.sum(y == 1))

# Shuffle
idx = np.arange(X.shape[0])
np.random.seed(13)
np.random.shuffle(idx)
X = X[idx]
y = y[idx]

# Standardize
mean = X.mean(axis=0)
std = X.std(axis=0)
X = (X - mean) / std

# Train
classifier = tree.DecisionTreeClassifier().fit(X, y)

# Plot the decision boundary
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
plot_step = 0.02
xx, yy = np.meshgrid(np.arange(x_min, x_max, plot_step),
                     np.arange(y_min, y_max, plot_step))

Z = classifier.predict(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)
cs = plt.contourf(xx, yy, Z, cmap=plt.cm.Paired)

plt.xlabel(iris.feature_names[features[0]])
plt.ylabel(iris.feature_names[features[1]])

# Plot the training points
# Parameters
plot_colors = "bry"
for i, color in zip(range(n_classes), plot_colors):
    idx = np.where(y == i)
    plt.scatter(X[idx, 0], X[idx, 1], c=color, label=class_names[i],
                    cmap=plt.cm.Paired)

plt.suptitle("Decision surface of a decision tree")
plt.legend()
plt.show()

```

    iris-setora : 50
    others : 100

```python
from sklearn.externals.six import StringIO

dot_file = open("iris_decision_tree.dot", 'w')
tree.export_graphviz(classifier, out_file=dot_file,
                     feature_names=iris.feature_names,
                     class_names=class_names,
                     filled=True, rounded=True,
                     special_characters=True)
dot_file.close()
```

```python
from IPython.display import Image
import os
os.system('dot -Tpng iris_decision_tree.dot -o iris_decision_tree.png')
Image('iris_decision_tree.png')
```

* Class : majority rule
* Samples : number of elements in the region
* Value : size of classes in the region

### Decision Trees

Let $\*D = \{\*x_i, y_j\}^ n_{i=1}$ denote the training set. This set consists of $n$ pairs composed of a $d$-dimensional point $\*x_i$ and its corresponding class label $y_i \in \{c_1, c_2, \ldots, c_k\}$. It is assumed that each feature is either numeric or categorial.

$$
D =
\left(
\begin{array}{c|c}
        X & \*y\\
        \hline
  \*x_1 & y_{1}\\
  \*x_2 & y_{2}\\
  \vdots & \vdots \\
  \*x_n & y_{n}
\end{array}
\right)
=
\left(
\begin{array}{cccc|c}
        X_1 & X_2 & \cdots & X_d & \*y\\
        \hline
  x_{1,1} & x_{1,2} & \cdots & x_{1,d} & y_{1}\\
  x_{2,1} & x_{2,2} & \cdots & x_{2,d} & y_{2}\\
  \vdots  & \vdots  & \ddots & \vdots & \vdots \\
  x_{n,1} & x_{n,2} & \cdots & x_{n,d} & y_{n}
\end{array}
\right)
$$

In a **decision tree classifier**, the **data space is recursively partitioned**. This partition is represented by a **binary tree**, where each **node** corresponds to a **region** of the data space. The **root** node corresponds to the data space, each pair of **child nodes** corresponds to a partitioning of the partent node's region into two half-spaces by an *axis-parallel hyperplane*, and each external node (**leaf**) corresponds to a class label equal to the majority call among the points in the corresponding region.

An **Axis-Parallel Hyperplane** $h(\*x)$ is defined as the set of all points $\*x$ that satisfy the the following equation,

$$
h(\*x) : x_j + b = 0,
$$

where the choice of $b$ leads to different hyperplanes along the dimension $X_j$, with $j = 1, 2, \ldots, d$.

```python
# Load data
iris = datasets.load_iris()

# Select the two first features, i.e., sepal length and width
features = [0, 1]
X = iris.data[:, features]
y = iris.target
# Considert two classes : iris-setora and the other two types of irises
classes = [0, 1]
class_names = [iris.target_names[0], 'others']
n_classes = 2
y[y > 1] = 1

plt.xlabel(iris.feature_names[features[0]])
plt.ylabel(iris.feature_names[features[1]])

# Plot the training points
plot_colors = "br"
for i, color in zip(range(n_classes), plot_colors):
    idx = np.where(y == i)
    plt.scatter(X[idx, 0], X[idx, 1], c=color, label=class_names[i],
                    cmap=plt.cm.Paired)

#plt.axvline(x=5.55, color='g', linestyle='-')
#plt.axhline(y=2.55, color='0', linestyle='-')
```

#### Numerical attributes

A hyperplane defines a **decision**, or *split point*, that splits the data space $\mathcal{R}$ into two half-spaces $\mathcal{R}_Y$ and $\mathcal{R}_N$. For a **numerical attribute**, this split point is defined as follows,

$$
X_j \leqslant v,
$$

where $v = -b$ is in the domain of attribute $X_j$. The half-space $\mathcal{R}_Y$ denotes the set of all possible points that **satisfy this decision**. Conversely, the half-space $\mathcal{R}_N$, denotes all the possible points that **do not satisfy this decision**.

This split of the data space $\mathcal{R}$, induces a binary partition of the corresponding data points $\*D$ defined as follows,

* $\*D_Y = \{\*x : \*x \in \*D, x_j \leqslant v\}$
* $\*D_N = \{\*x : \*x \in \*D, x_j > v\}$

```python
v = 5.55

R = np.zeros(len(y))
spaces = [0, 1]
R[X[:,0] <= v] = 1 #Dy

# Plot the training points
plot_colors = "rb"
for i, color in zip(range(n_classes), plot_colors):
    space_markers = ['o', 's']
    for j in spaces:
        idx = np.where((y == i) & (R == j))
        plt.scatter(X[idx, 0], X[idx, 1], c=color, marker=space_markers[j])

plt.axvline(x=5.55, color='g', linestyle='-')
```

    <matplotlib.lines.Line2D at 0x7f5add32ffd0>

#### Categorical attributes

For a **categorical attribute**,  the split points is defined as follows,

$$
X_j \in V,
$$

where $V \subset dom(X_j)$, a strict subset of the domain of $X_j$.

#### Region Evaluation

The **purity** of a region is defined as follows,

$$
purity(D_j) = \max_{i} \left\{\frac{n_{ji}}{n_j}\right\},
$$

where $n_j = |\*D_j|$ and $n_{ji}$ is the number of points in $D_j$ labeled with $c_i$

#### Algorithm

This algorithms takes as input three arguments, the training data set $\*D$, the leaf size $\nu$, and the leaf purity threshold $\pi$. The best split point is determined with respect to a given evaluation function, and the partioning is performed rescursively on each half-space. The partitioning process stops if either the number of data points is less than $\nu$ or the purity is greater than $\pi$.

#### Split Point Evaluation Measure

The **Gini index** is used to measure the purity of a split point. The Gini index of a given data set $\*D$ is defined as follows,

$$
G(\*D) = 1 - \sum_{i = 1}^k  P(c_i|\*D)^2
$$

where $P(c_i|\*D)$ is the probability of class $c_i$ in the data set $\*D$ and $k$ is the number of classes.

* *If the partition is pure* : the porbability of the majority class is 1 and the probability of others is 0, therefore the Gini index is 0.
* *Otherwise* : each class is equally represented (*i.e.*, $P(c_i|\*D) = \frac{1}{k}$, for all $i = 1,\ldots,k$), therefore the Gini index is equal to $\frac{k-1}{k}$.

For a given **split point**, the **weighted Gini index** is defined as follows

$$
G(\*D_Y, \*D_n) = \frac{n_Y}{n} G(\*D_Y) + \frac{n_N}{n} G(\*D_Y)
$$

where, $n_Y$ and $n_N$ denote the number of points in $\*D_Y$ and $\*D_N$ respectively.
#### Computational Complexity

$$
O(dn^2 \log n)
$$

## k-Nearest Neighbors

#### Generate data points forming to different cycles

```python
X, y = datasets.make_circles(noise=.1, factor=.5)
print("X.shape:", X.shape)
print("unique labels: ", np.unique(y))
```

    X.shape: (100, 2)
    unique labels:  [0 1]

#### Plot the data

```python
plt.prism()  # this sets a nice color map
plt.scatter(X[:, 0], X[:, 1], c=y)
```

    <matplotlib.collections.PathCollection at 0x7f5adc142d30>

#### Take the first 50 examples for training and the rest for testing.

```python
X_train = X[:50]
y_train = y[:50]
X_test = X[50:]
y_test = y[50:]
```

#### Try to classify the points (in the two classes) using a k-nn classifier:

```python
knn = KNeighborsClassifier(n_neighbors=5)    # we specify that this knn should always use 5 neighbors
```

```python
knn.fit(X_train, y_train)

y_pred_train = knn.predict(X_train)
print("KNN validation accuracy on training cycles: ", knn.score(X_train, y_train))

y_pred_test = knn.predict(X_test)
print("KNN validation accuracy on test cycles: ", knn.score(X_test, y_test))
```

    KNN validation accuracy on training cycles:  0.96
    KNN validation accuracy on test cycles:  0.84

#### Plot the result

```python
plt.prism() # gives us a nice color map

plt.xlim(-1.5, 1.5)
plt.ylim(-1.5, 1.5)

plt.scatter(X_test[:, 0], X_test[:, 1], c=y_pred_test, marker='^')
plt.scatter(X_train[:, 0], X_train[:, 1], c=y_train)

print("Accuracy of KNN test set:", knn.score(X_test, y_test))
```

    Accuracy of KNN test set: 0.84

## Probabilistic Classification

We consider a dataset $D = \{\*x_1, \ldots, \*x_n\}$, with $y_i \in \{c_1, c_2, \ldots, c_k\}$ for all $i = 1, 2, \ldots, n$. The **predicted class** for a new data point $\*x$ is obtained as follows :

$$
\hat{y} = \argmax_{c_i}\{P(c_i | \*x)\}
$$

**Bayes' formula**

Let $E_1, E_2, \ldots, E_k$ be a partition of $\Omega$ :

* $\Omega = E_1 \cup E_2 \cup \ldots \cup E_k$
* $E_i \cap E_j = \emptyset$, for all pair $i, j \in \{1, 2, \ldots, k\}$

Based on the probability axioms and elementary set theory, it can be showed that the **marginal probability** $P(E)$ is equal to :

$$
P(E) = P(E \cap \Omega) = P(E_1 \cap E) + P(E_2 \cap E) + \ldots + P(E_k \cap E)
$$

Based on conditional probability, this can be rewritten as follows :

$$
P(E) = P(E_1) P(E | E_1) + P(E_2) P(E | E_2) + \ldots + P(E_k) P(E | E_k)
$$

This leads to the **Bayes's formula** :

$$
P(E_i|E) = \frac{P(E_i \cap E)}{P(E)} = \frac{P(E_i)P(E|E_i)}{\sum_{l = 1}^k P(E_l)P(E|E_l)}
$$

### Bayes Classifier

Based on this formula, The **predicted class** for a new data point $\*x$ is obtained as follows :
$$
\begin{align}
\hat{y} &= \argmax_{c_i}\{P(c_i | \*x)\}\\
        &= \argmax_{c_i}\Bigg\{\frac{P(\*x | c_i) P(c_i)}{P(\*x)}\Bigg\}\\
        &= \argmax_{c_i}\{P(\*x | c_i) P(c_i)\}
\end{align}
$$

**Prior Probability Estimator** :

$$
\hat{P}(c_i) = \frac{|\*D_i|}{|\*D|} = \frac{n_i}{n}
$$

where $\*D_i = \{\*x_j \in \*D : y_j = c_i\}$, $|\*D| = n$, and $|\*D_i| = n_i$.

**Likelyhood estimator**, *i.e.* and estimator for $P(\*x | c_i)$ : see previous labs/lectures.

### Naive Bayes Classifier

In this case we assume that attributes are independent. This allows to write the likelyhood as follows :

$$
P(\*x|c_i) = P(x_1, x_2, \ldots, x_n | c_i) = \prod_{j = 1}^d P(x_i|c_i)
$$

For **numerical attributes**, we can, for example, assume a normal distribution for each clas $c_i$ :

$$
P(x_j | c_i) \propto f(x_j) = \frac{1}{\sqrt{2\sigma_{ij}^2\pi} } \; e^{ -\frac{(x_-\mu_{ij})^2}{2\sigma_{ij}^2} }
$$

where $\mu_{ij}$ and $\sigma_{ij}^2$ denote the mean and variance for attribute $X_j$ and class $c_i$.

```python
from sklearn import datasets

iris = datasets.load_iris()

from sklearn.naive_bayes import GaussianNB

gnb = GaussianNB()

y_pred = gnb.fit(iris.data, iris.target).predict(iris.data)

print("Number of mislabeled points out of a total {} points : {}".format(iris.data.shape[0],
                                                                         (iris.target != y_pred).sum()))
```

    Number of mislabeled points out of a total 150 points : 6

For **categorical attributes**,

$$
P(\*x|c_i) = \prod_{j = 1}^d P(x_i|c_i) \approx \prod_{j = 1}^d \hat{f}(\*v_j | c_i) = \prod_{j = 1}^d \frac{n_i(\*v_j)}{n_i}
$$

where $n_i(\*v_j)$ is the observed frequency of the value $\*v_j$ for the attribute $X_i$ in the class $c_i$.

```python
# naive_bayes.MultinomialNB([alpha, ...]) : Naive Bayes classifier for multinomial models
# naive_bayes.BernoulliNB([alpha, binarize, ...]) : Naive Bayes classifier for multivariate Bernoulli models.
```

## Classifier Evaluation

### Performance Measure

* $\*D$ : the testing set composed of $n$ points in a $d$ dimensional space
* $k$ : the number of classes
* $M$ : the classifier
* $y_i$ : the "true" class that corresponds to $\*x_i \in \*D$
* $\hat{y}_i = M(\*x_i)$ : the predicted class

#### Error Rate

The fraction of incorrect predictions :

$$
Error\ Rate = \frac{1}{n} \sum_{i = 1}^n I(y_i \neq \hat{y}_i)
$$

#### Accuracy

The fraction of correct predictions :
$$
Accuracy = \frac{1}{n} \sum_{i = 1}^n I(y_i = \hat{y}_i) = 1 - Error\ Rate
$$

### Evaluation

The data set $\*D$ is split into two disjoint sets :

* **Training set** : used to train $M$
* **Testing set** : used to evaluate the performance measure (denoted $\theta$) of $M$

#### $K$-fold Cross-Validation

The data set $\*D$ is divided into $K$ equal-sized parts, denoted $\*D_1, \*D_2, \ldots, \*D_K$, called folds.

For each $i = 1,2,\ldots,K$, a model $M_i$ is trained and the corresponding measure $\theta_i$ is evaluated with respect to :

* $\*D_i$ : the testing set
* $\*D \backslash \*D_i = \cup_{j \neq i} \*D_j$

This lead estimate the expected and variance values of the performance measure as follows,

* $\displaystyle \hat{\mu}_{\theta} = E[\theta] = \frac{1}{K} \sum_{i=1}^K \theta_i$
* $\displaystyle \hat{\sigma}_{\theta}^2 = \frac{1}{K} \sum_{i=1}^K (\theta_i - \hat{\mu}_{\theta})^2$

This validation is performed $n$ times, where the data set $\*D$ is partitioned randomly.

## References

* The Google Developers channel : Machine Learning Recipes
* Spring 2015 Boston University CS591 "Tools and Techniques for Data Mining and Applications" course