# Stencil Penalty based Contrained Immersed Boundary Method
---

### Author: Marin Lauber

```python
import numpy as np
import matplotlib.pyplot as plt
import NSsolver as ns
try:
    plt.style.use("jupyter")
except OSerror:
    print("Using default ploting style")
```

The stencil penalty based constrained immersed boundary method (SPcIB), see [Bale et al. (2020)](https://www.sciencedirect.com/science/article/pii/S0045793018308272), is based on the standard IB method with a singluar force

\begin{equation}
    \begin{split}
    &\frac{\partial \vec{u}}{\partial t} + (\vec{u}\cdot\nabla)\vec{u} = -\nabla p + \frac{1}{Re}\nabla^2\vec{u} + \int\vec{f}(\vec{\xi}(s, t))\delta(\vec{\xi} - \vec{x}) \text{ d}\vec{x},\\
    &\nabla\cdot\vec{u} = 0,\\
    &\vec{u}(\vec{\xi}(s, t)) = \int_{\vec{x}}\vec{u}(\vec{x})\delta(\vec{x} - \vec{\xi} ) \text{ d}\vec{x} = \vec{u}_B(\vec{\xi}(s, t)).
\end{split}
\end{equation}

but splits the forcing into two contributions

\begin{equation}
    \vec{f}(\vec{\xi}, t) = \vec{f}_{ct}(\vec{\xi}, t) + \vec{f}_{p}(\vec{\xi}, t)
\end{equation}

where $\vec{f}_p(\vec{\xi}, t)$ is a correction term that accounts for the bad approximation of the pressure gradient with central difference stencil. It is obtained from

\begin{equation}
    \vec{f}_p(\vec{\xi}, t) = \nabla p - \hat{\nabla}p.
\end{equation}

$\hat{\nabla}$ is a modified gradient operator that penalises stencils that contain a discontinuity. Standard WENO smoothness indicators are used.

## WENO smoothing
---

The modified discrete approximation is given by

\begin{equation}
    d\hat{\phi}\vert_i = \sum_{k=1}^{r}\omega_k d\phi_k,
\end{equation}

where $d\phi_k$ is the discrete approximation of the derivative of $\phi$ on stencil $k$ using $r$ grid points, i.e. with three points, we have the forward, central and backward differencinf schemes. The standard gradient approxiation on a stencil $k$, with a mesh spacing $h$ is given by

\begin{equation}
    d\phi_k = \frac{1}{h}\sum^{r-1}_{j=0}a_{kj}^{r}\phi(i-r+k+j+2).
\end{equation}

The coefficients $a_{kj}^{r}$ depend on the type and order of the discrete gradient operator. The weight $\omega_k$ are given by by a normalisation of the weights $\psi_k$

\begin{equation}
    \omega_k = \frac{\psi_k}{\sum_{k=1}^{r}\psi_k},
\end{equation}

which are given by

\begin{equation}
    \psi_k = \frac{\gamma_k}{(\epsilon + \beta_k)^m}.
\end{equation}

$\epsilon:=1e^{-6}$ prevents the weights from diverging, $m$ and must be chosen large enough. The choice of $\gamma_k$ is not important as it is normaized after. $\beta_k$ is the smoothness parameter and is given by

\begin{equation}
    \beta_k = \sum_{p=1}^{r-1}\left(\sum_{j=0}^{r-1}d_{kjp}^{r}\phi(i-r+k+j+2)\right)^2.
\end{equation}

$d_{r}^{kjp}$ are tabulated coefficients.

### Example on a 3 point stencil

---

For a three point stencil we have $r=3$, $k=\{1, 2, 3\}$, the smoothness parameters are

$$
    \beta_k = \sum_{p=1}^{r-1}\left(\sum_{j=0}^{2}d_{kjp}^{2}\phi(i+k+j-1)\right)^2.
$$

```python
def grad_c(f, dx, m=4, kappa=1, eps=1e-6, c=5):

    grad = np.zeros(len(f))

    for i in range(2, len(f)-2):

        # smoothness parameter
        b1 = 13./12*(f[i-2] - 2*f[i-1] + f[i])**2 + 0.25*(3*f[i-2]-4*f[i-1]+f[i])**2
        b2 = 13./12*(f[i-1] - 2*f[i] + f[i+1])**2 + 0.25*(f[i-1]-f[i+1])**2
        b3 = 13./12*(f[i] - 2*f[i+1] + f[i+2])**2 + 0.25*(3*f[i]-4*f[i+1]+f[i+2])**2

        #weights
        psi1 = c/(eps + b1)**m
        psi2 = c*kappa/(eps + b2)**m
        psi3 = c/(eps + b3)**m

        # normalized weights
        sum_i = 1./(psi1+psi2+psi3)
        w1 = psi1 * sum_i
        w2 = psi2 * sum_i
        w3 = psi3 * sum_i

        # candidate stencils bwd, ct, fwd
        df1 = (3*f[i]-4*f[i-1]+f[i-2]) / (2*dx)
        df2 = (f[i+1] - f[i-1]) / (2*dx)
        df3 = (-3*f[i]+4*f[i+1]-f[i+2]) / (2*dx)

        # weighed sum
        grad[i] = w1*df1 + w2*df2 + w3*df3

    # boundary point at threated with forward and backward stencils
    grad[:2] = (-3*f[ :2]+4*f[1:3]-f[2:4]) / (2*dx)
    grad[-2:]=  (3*f[-2:]-4*f[-3:-1:]+f[-4:-2]) / (2*dx)

    return grad
```

To test this WENO scheme, we use the following discontinous function

\begin{equation}
    \phi(x) = \begin{cases}
        \tanh(\alpha x) & \text{if } x \le 0.5,\\
        C + \xi + \tanh(\alpha(x-1))& \text{else }.
    \end{cases}
\end{equation}

The derivative is easily evaluated analitically

\begin{equation}
    \frac{d\phi(x)}{dx} = \begin{cases}
        \alpha (1-\tanh(\alpha x)^2) & \text{if } x \le 0.5,\\
        \alpha (1-\tanh(\alpha (x-1))^2)& \text{else }.
    \end{cases}
\end{equation}

```python
def phi(x, Xi=1., alpha=1.):
    C = np.tanh(0.5*alpha) - np.tanh(-0.5*alpha)
    res = np.zeros_like(x)
    for i in range(len(x)):
        if (x[i]<=0.5):
            res[i] = np.tanh(alpha*x[i])
        else:
            res[i] = C + Xi + np.tanh(alpha*(x[i] - 1))
    return res

def dphi(x, alpha=1.):
    res = np.zeros_like(x)
    for i in range(len(x)):
        if (x[i]<=0.5):
            res[i] = alpha*(1 - np.tanh(alpha*x[i])**2)
        else:
            res[i] = alpha*(1 - np.tanh(alpha*(x[i] - 1))**2)
    return res
```

```python
x, dx = np.linspace(0, 1, 1024, retstep=True)
fig, ax = plt.subplots(2, 2)
ax[0, 0].plot(x, phi(x, .1, 1))
ax[0, 1].plot(x, phi(x, 1, 1))
ax[1, 0].plot(x, phi(x, .1, 10))
ax[1, 1].plot(x, phi(x, 1, 10));
```

```python
print("L-infinity norm of the error: %.3e" % np.max(abs(grad_c(phi(x,.1, 1), dx) - dphi(x, 1))))
print("L-infinity norm of the error: %.3e" % np.max(abs(grad_c(phi(x,1, 1), dx) - dphi(x, 1))))
print("L-infinity norm of the error: %.3e" % np.max(abs(grad_c(phi(x,.1, 10), dx) - dphi(x, 10))))
print("L-infinity norm of the error: %.3e" % np.max(abs(grad_c(phi(x,1, 10), dx) - dphi(x, 10))))
```

    L-infinity norm of the error: 6.370e-07
    L-infinity norm of the error: 6.370e-07
    L-infinity norm of the error: 6.369e-04
    L-infinity norm of the error: 6.369e-04

Interpolation is achieved using a standard 3-points kernel

\begin{equation}
    d(r)=\begin{cases}
        \frac{3}{4} - r^2 & \text{if } |r|\le 0.5,\\
        \frac{1}{2}(\frac{9}{4} - 3|r|+r^2)& \text{if } |r|\le 1.5,\\
        0 & \text{else.}
       \end{cases}
\end{equation}

```python
def kernel(r):
    return np.where(abs(r)<=.5, .75-r**2, np.where(abs(r)<=1.5,.5*(2.25-3*abs(r)+r**2), 0))
```

```python
x, dx = np.linspace(-2, 2, 128, retstep=True)
X = 0
plt.plot(x, kernel(x-X), "-o");
```

```python
def Ic(x, X, dx, u):
    # interpolate Eulerian velocity on Lagrangian interface
    return np.sum(u*kernel((X - x)/dx))

def Sc(x, X, F, ds=1):
    # Regularize the Lagrangian Force to the Eulerian points
    return kernel((X - x)/dx)*ds*F
```

We can check that the interpolation and regularization function work properly with two trivial examples. First we check that interpolating a uniform Eulerian velocity field results in the correct Lagrangian velocity

```python
u0 = np.ones_like(x)
assert np.isclose(Ic(x, 0, dx, u0), 1.), "Interpolated velocity should be 1"
```

```python
def get_f(x, X, u, V, dx, dt, rho):
    # interpolate to Lagrangian point
    Ur = Ic(x, X, dx, u)
    # compute Lagrangian force density
    F = rho/dt*(V - Ur)
    return Sc(x, X, F) # extrapolate to Eulerian points
```

The we can check that the forcing of a Lagrangian point moving at the same velocity as a unform Eulerian field is zero

```python
assert np.allclose(get_f(x, X, u0, V, dx, 1, 1), 0.), "forcing should be zero"
```

## 1D Piston

The fractional step is used to solve the coupled system. The singular force is added i the predictor step, as in the IBM

$$
    u^* = u^n + \Delta t (r_{\Delta t}(u^n) + f_{ct})
$$

$$
    \nabla\cdot(\nabla p) =  \frac{\rho}{\Delta t}\nabla\cdot u^* + \nabla\cdot f_{p}
$$

$$
u^{n+1} = u^* + \frac{\Delta t}{\rho}\left(\nabla p - f_p\right)
$$

```python
def update_cIB(x, X, u0, V, dx, dt=1, rho=1):

    u_n = u0
    flag = False

    for i in range(1):
        # first step
        u_1 = u_n + dt*ns.r(u_n)
        u_2 = u_1 + dt*get_f(x, X, u_1, V, dx, dt, rho)

        f_p = np.zeros_like(u_2)

        for k in range(10):

            sigma = ns.div(u_2, dx) - ns.div(f_p, dx)

            if k==9: flag = True
            p = ns.solve_pressure(np.ones_like(sigma), sigma, dx, verbose=flag)

            f_p = ns.grad(p, dx) - to_face(grad_c(p, dx))

        u_n = u_2 - dt/rho*(ns.grad(p, dx) - f_p)

    return p, u_2, sigma, u_n

def to_face(f):
    cell_c = np.zeros_like(f)
    cell_c[1:] = 0.5*(f[:-1] + f[1:])
    cell_c[0] = 0.5*(f[-1]+f[0])
    return cell_c
```

```python
N = 24
x, dx = np.linspace(-1, 1, N, retstep=True)
xs = x + 0.5*dx # scalar cell-centred values
X = 0.
V = 1
u0 = np.zeros_like(x)
dt = 1.
d = (x-X)/dx

p, us, sigma, u_n =  update_cIB(x, X, u0, V, dx, 1, 1)
print("Interface at X: %.2f" % X)
print(r"L inf: %.3e" % np.max(np.abs(u_n -  V)))
```

    Jacobi solver:
    	res0: 2.301e-01
     	res: 2.297e-11
     	iter: 323
    Interface at X: 0.00
    L inf: 9.717e-01

```python
ns.draw_results(x, xs, X, u0, u_n, p, kernel(d))
```


# Modélisation du stockage thermique

Une illustration du stockage

## Equations

### Couche 1
\begin{equation}
m_{sable} Cp_{sable} \frac{dT_1}{dt}=-\frac{\lambda}{e}(T_1-T_2)-\frac{T_1-T_{ext}}{Rth_{ext}}
\end{equation}

### Couche 2
\begin{equation}
m_{sable} Cp_{sable} \frac{dT_2}{dt}=\frac{\lambda}{e}((T_1-T_2)-(T_2-T_3))-\frac{T_2-T_{ext}}{Rth_{ext}}+\dot Q_{pac}
\end{equation}

avec

\begin{equation}
\dot Q_{pac}= \dot m_{pac}Cp_{pac}(T_{pac\_out1}-T_{pac\_in1})=\frac{T_{pac\_out1}+T_{pac\_in1}}{2 Rth_{tube}}
\end{equation}

Ce qui donne :
\begin{equation}
T_{pac\_out1}=\frac{(\dot m_{pac}-\frac{1}{2 Rth_{tube}})}{(\dot m_{pac}-\frac{1}{2 Rth_{tube}})} *T_{pac\_in1}
\end{equation}

### Couche 3

\begin{equation}
m_{sable} Cp_{sable} \frac{dT_3}{dt}=\frac{\lambda}{e}((T_2-T_3)-(T_3-T_4))-\frac{T_3-T_{ext}}{Rth_{ext}}+\dot Q_{pac}
\end{equation}

### Couche 4 à 10
**Similaire aux équations des couches précédentes**

### Couche 11 (béton)
\begin{equation}
m_{beton} Cp_{beton} \frac{dT_{11}}{dt}=r_{10/11}(T_{10}-T_{11})-r_{11/12}(T_{11}-T_{12})
\end{equation}

### Couche 12 et 13
**Similaire à celle de la couche 11**

## Résolution numérique

Schéma de Euler explicite

### Couche 1 ( et pour les autres couches)
\begin{equation}
T1_{n+1}=T1_{n}+\frac{dt}{m_{sable}Cp_{sable}}(-\frac{\lambda}{e}(T1_{n}-T2_{n})-\frac{T_{n}-Text_{n}}{Rth_{ext}})
\end{equation}

```python

```


Here I will have some examples showing how to use Sampyl. This is for version 0.2.2. Let's import it and get started. Sampyl is a Python package used to sample from probability distributions using Markov Chain Monte Carlo (MCMC). This is most useful when sampling from the posterior distribution of a Bayesian model.

Every sampler provided by Sampyl works the same way. Define $ \log P(\theta) $ as a function, then pass it to the sampler class. The class returns a sampler object, which you can then use to sample from $P(\theta)$. For samplers which use the gradient, $\nabla_{\theta} \log P(\theta)$, Sampyl uses [autograd](https://github.com/HIPS/autograd) to automatically calculate the gradients. However, you can pass in your own $\nabla_{\theta} \log P(\theta)$ functions.

Starting out simple, let's sample from a normal distribution.

```python
%matplotlib inline

import matplotlib.pyplot as plt
import sampyl as smp
from sampyl import np

# Autgrad throws some warnings that are useful, but this is
# a demonstration, so I'll squelch them.
import warnings
warnings.filterwarnings('ignore')
```

A normal distribution with mean $\mu$ and variance $\sigma^2$ is defined as:

$$
P(x,\mu, \sigma) = \frac{1}{\sigma \sqrt{2 \pi}} \; \mathrm{Exp}\left( \frac{-(x - \mu)^2}{2\sigma^2} \right)
$$

For numerical stability, it is typically better to deal with log probabilities, $\log{P(\theta)}$. Then for the normal distribution with known mean and variance,

$$
\log{P(x \mid \mu, \sigma)} = -\log{\sigma} - \frac{(x - \mu)^2}{2\sigma^2}
$$

where we can drop constant terms since the MCMC samplers only require something proportional to $\log{P(\theta)}$. We can easily write this as a Python function.

```python
mu, sig = 3, 2
def logp(x):
    return  -np.log(sig) - (x - mu)**2/(2*sig**2)
```

First we'll use a Metropolis-Hastings sampler. Each sampler requires a $\log{P(\theta)}$ function and a starting state. We have included a function to calculate the *maximum a posteriori* (MAP) to find the peak of the distribution for use as the starting state. Then you call the sample method and a chain of samples is returned.

```python
start = smp.find_MAP(logp, {'x':1.})
metro = smp.Metropolis(logp, start)
chain = metro.sample(10000, burn=2000, thin=4)
```

    Progress: [##############################] 10000 of 10000 samples

We can retrieve the chain by accessing the attributes defined by the parameter name(s) of `logp`.

```python
plt.plot(chain.x)
```

```python
_ = plt.hist(chain.x, bins=30)
_ = plt.vlines(mu, 0, 250, linestyles='--')
```

Here we have sampled from a normal distribution with a mean of 3, indicated with the dashed vertical line.

There is also a No-U-Turn Sampler (NUTS), which avoids the random-walk nature of Metropolis samplers. NUTS uses the gradient of $\log{P(\theta)}$ to make intelligent state proposals. You'll notice here that we don't pass in any information about the gradient. Instead, it is calculated automatically with [autograd](https://github.com/HIPS/autograd).

```python
nuts = smp.NUTS(logp, start)
chain = nuts.sample(2100, burn=100)
```

    Progress: [##############################] 2100 of 2100 samples

```python
plt.plot(chain)
```

```python
_ = plt.hist(chain.x, bins=30)
_ = plt.vlines(mu, 0, 250, linestyles='--')
```

## Bayesian estimation of phone call rates

Let's try something a little more complicated. Let's say you run a business and you put an advertisement in the paper. Then, to judge the effectiveness of the ad, you want to compare the number of incoming phone calls per hour before and after the placement of the add. Then we can build a Bayesian model using a Poisson likelihood with exponential priors for $\lambda_1$ and $\lambda_2$.

\begin{align}
P(\lambda_1, \lambda_2 \mid D) &\propto P( D \mid \lambda_1, \lambda_2)\, P(\lambda_1)\, P(\lambda_2) \\
P( D \mid \lambda_1, \lambda_2) &\sim \mathrm{Poisson}(D\mid\lambda_1)\,\mathrm{Poisson}(D\mid\lambda_2) \\
P(\lambda_1) &\sim \mathrm{Exp}(1) \\
P(\lambda_2) &\sim \mathrm{Exp}(1)
\end{align}

This analysis method is known as Bayesian inference or Bayesian estimation. We want to know likely values for $\lambda_1$ and $\lambda_2$. This information is contained in the posterior distribution $P(\lambda_1, \lambda_2 \mid D)$. To infer values for $\lambda_1$ and $\lambda_2$, we can sample from the posterior using our MCMC samplers.

```python
# Fake data for the day before and after placing the ad.
# We'll make the calls increase by 2 an hour. Record data for each
# hour over two work days.
before = np.random.poisson(7, size=16)
after = np.random.poisson(9, size=16)

# Define the log-P function here
def logp(lam1, lam2):
    # Rates for Poisson must be > 0
    if lam1 <= 0 or lam2 <=0:
        return -np.inf
    else:
        # logps for likelihoods
        llh1 = np.sum(before*np.log(lam1)) - before.size*lam1
        llh2 = np.sum(after*np.log(lam2)) - after.size*lam2

        # Exponential priors for lams
        lam1_prior = -lam1
        lam2_prior = -lam2
        return llh1 + llh2 + lam1_prior + lam2_prior
```

```python
start = smp.find_MAP(logp, {'lam1':1., 'lam2':1.})
metro = smp.Metropolis(logp, start)
chain = metro.sample(10000, burn=2000, thin=4)
```

    Progress: [##############################] 10000 of 10000 samples

Sampling returns a numpy record array which you can use to access samples by name. Variable names are taken directly from the argument list of `logp`.

```python
print(metro.var_names)
```

    ('lam1', 'lam2')

```python
plt.plot(chain.lam1)
plt.plot(chain.lam2)
```

Now to see if there is a significant difference between the two days. We can find the difference $\delta =  \lambda_2 - \lambda_1$, then find the probability that $\delta > 0$.

```python
delta = chain.lam2 - chain.lam1
_ = plt.hist(delta, bins=30)
_ = plt.vlines(2, 0, 250, linestyle='--')
```

```python
p = np.mean(delta > 0)
effect = np.mean(delta)
CR = np.percentile(delta, (2.5, 97.5))
print("{:.3f} probability the rate of phone calls increased".format(p))
print("delta = {:.3f}, 95% CR = {{{:.3f} {:.3f}}}".format(effect, *CR))
```

    0.888 probability the rate of phone calls increased
    delta = 1.146, 95% CR = {-0.775 3.116}

There true difference in rates is two per hour, marked with the dashed line. Our posterior is showing an effect, but our best estimate is that the rate increased by only one call per hour. The 95% credible region is {-0.865 2.982} which idicates that there is a 95% probability that the true effect lies with the region, as it indeed does.

We can alo use NUTS to sample from the posterior.

```python
nuts = smp.NUTS(logp, start)
chain = nuts.sample(2100, burn=100)
_ = plt.plot(chain.lam1)
_ = plt.plot(chain.lam2)
```

```python
delta = chain.lam2 - chain.lam1
_ = plt.hist(delta, bins=30)
_ = plt.vlines(2, 0, 250, linestyle='--')
p = np.mean(delta > 0)
effect = np.mean(delta)
CR = np.percentile(delta, (2.5, 97.5))
print("{:.3f} probability the rate of phone calls increased".format(p))
print("delta = {:.3f}, 95% CR = {{{:.3f} {:.3f}}}".format(effect, *CR))
```

## Linear models too

When you build larger models, it would be cumbersome to have to include every parameter as an argument in the logp function. To avoid this, you can declare the size of variables when passing in the starting state.

For instance, with a linear model it would be great to pass the coefficients as one parameter. First, we'll make some fake data, then infer the coefficients.

```python
# Number of data points
N = 200
# True parameters
sigma = 1
true_B = np.array([2, 1, 4])

# Simulated features, including a constant
X = np.ones((N, len(true_B)))
X[:,1:] = np.random.rand(N, 2)*2

# Simulated outcomes with normally distributed noise
y = np.dot(X, true_B) + np.random.randn(N)*sigma

data = np.ones((N, len(true_B) + 1))
data[:, :-1] = X
data[:, -1] = y
```

```python
fig, axes = plt.subplots(figsize=(7,4),ncols=2)
for i, ax in enumerate(axes):
    ax.scatter(X[:,i+1], y)
axes[0].set_ylabel('y')
axes[0].set_xlabel('X1')
axes[1].set_xlabel('X2')
```

```python
fig.savefig('linear_model_data.png')
```

```python
# Here, b is a length 3 array of coefficients
def logp(b, sig):
    if sig <=0:
        return -np.inf
    y_hat = np.dot(X, b)
    likelihood = smp.normal(y, mu=y_hat, sig=sig)

    prior_sig = smp.exponential(sig)

    # Uniform priors on coefficients
    prior_b = smp.uniform(b, lower=-100, upper=100)

    return likelihood + prior_sig + prior_b
```

```python
start = smp.find_MAP(logp, {'b': np.ones(3), 'sig': 1.},
                     bounds={'b':(-5, 10), 'sig':(0.01, None)})
metro = smp.Metropolis(logp, start)
chain = metro.sample(20000, burn=5000, thin=4)
```

    Progress: [##############################] 20000 of 20000 samples

```python
_ = plt.plot(chain.b)
```

And using NUTS too.

```python
start = smp.find_MAP(logp, {'b': np.ones(3), 'sig': 1.})
nuts = smp.NUTS(logp, start)
chain = nuts.sample(2100, burn=100)
```

    Progress: [##############################] 2100 of 2100 samples

```python
import seaborn as sb
```

```python
sb.set_context('notebook', font_scale=1.25)
```

```python
fig, axes = plt.subplots(figsize=(8,5), nrows=2, ncols=2)
for i, (row, param) in enumerate(zip(axes, [chain.b, chain.sig])):
    row[0].plot(param)
    row[0].set_ylabel('Sample value')
    #row[0].set_xlabel('Sample')
    row[0].set_title(['b', 'sig'][i])
    row[1].set_title(['b', 'sig'][i])
    if len(param.shape) > 1:
        for each in param.T:
            row[1].hist(each, alpha=0.8, histtype='stepfilled')
        row[1].set_yticklabels('')
        row[1].vlines([2,1,4], 0, 600, linestyles='--', alpha=0.5)

    else:
        row[1].hist(param, alpha=0.8, histtype='stepfilled')
        row[1].set_yticklabels('')
        row[1].vlines(1, 0, 600, linestyles='--', alpha=0.5)
    #row[1].set_xlabel('Sample value')
fig.tight_layout(pad=0.1, h_pad=1.5, w_pad=1)
```

```python
fig.savefig('linear_model_posterior.png')
```

```python

_ = plt.plot(chain.b)
```

```python
for each in chain.b.T:
    plt.hist(each, alpha=0.8, orientation='horizontal')
plt.hlines([2,1,4], 0, 600, linestyles='--', alpha=0.5)
```

```python
_ = plt.plot(chain.sig)
```

## Using one logp function for both logp and gradient

You can also use one `logp` function that returns both the logp value and the gradient. To let the samplers know about this, set `grad_logp = True`. I'm also using one argument `theta` as the parameter which contains the five $\beta$ coefficients and $\sigma$.

```python
from autograd import grad
grads = [grad(logp, 0), grad(logp, 1)]
def single_logp(theta):
    b, sig = theta[:5], theta[-1]
    logp_val = logp(b, sig)
    grad_val = np.hstack([each(b, sig) for each in grads])
    return logp_val, grad_val
```

```python
single_logp(start.tovector())
```

    (-107.03408315518031,
     array([ -4.84281457e-05,  -1.09379841e-04,   6.21644571e-05,
              1.65228255e-04,  -1.43789309e-04,   1.78887423e-03]))

```python
nuts = smp.NUTS(single_logp, {'theta':start.tovector()}, grad_logp=True)
chain = nuts.sample(1000)
```

    Progress: [##############################] 1000 of 1000 samples

```python
_ = plt.plot(chain.theta[:, :5])
```

## Sampling in parallel

We can make use of our multicore CPUs by running chains in parallel. To do this, simply request the number of chains you want when you call `sample`: `nuts.sample(1000, n_chains=4)`. Each chain is given its own process and the OS decides how to run the processes. Typically this means that each process will run on its own core. So, if you have four cores and four chains, they will all run in parallel. But, if you have two cores, only two will run at a time.

```python
nuts = smp.NUTS(logp, start)
chains = nuts.sample(1000, n_chains=2)
```

    Progress: [##############################] 1000 of 1000 samples

```python
fig, axes = plt.subplots(figsize=(10,3), ncols=2)
for ax, chain in zip(axes, chains):
    _ = ax.plot(chain.b)
```

The future
----------

* Write a module that makes it easier to build logp functions from distributions
* Add various functions such as autocorrelation, HPD, etc.
* Plots!


## A large portion of this code is taken from Aurélien Géron's: Hands-On machine learning with SciKit-Learn, Keras and Tensorflow (2nd edition). Also code taken from his github repo is included here. I have put comments based on information in the book as well as information I found elsewhere.

## Chapter 11

## Vanishing/Exploding gradients problem

Vanishing gradients:
  - If the gradients at the upper layer are small, they're divided and assigned to each neuron in the lower layer.
    These values are then further divided and assigned to each neron in the layer below it.
    As the number of layers are large in a Deep Neural Network, the gradients at the bottom become zero.
    During forward propagation, this causes the weights to remain the same and training never converges.
Exploding gradients:
  - If the loss function has a steep gradient at the location where you're computing it,
    there will be a large change in the weight values. This in turn can cause you to overshoot the minima
    and land on the other side of the loss function. If the gradient at that point is also large,
    this can cause another large change in the weight values, and another overshoot.
  - Sometimes these overshoots will get larger and larger getting you more and more away from the minima.
    Sometimes these overshoots will get you into a region of the loss function where the gradients are smaller,
    thus temporarily keeping the changes to the weights low. But if you get back down to where the
    gradients are large, you may find yourself again overshooting. This will cause you to oscillate,
    and never get to the minima.
  - More generally, neural networks suffer from unstable gradients.
    Different layers may learn at widely different speeds.

These problems result from using the Sigmoid activation function,
and the Standard Normal initial weight distribution
(0 mean, 1 std. dev.). With this initialization,
the input variance for each layer increases at the output of that layer.
A large variance means there are many large values being multiplied by the weights and added to get the next layer's input.
This causes the top layer's sigmoid function to get input that saturates it's output.
With a saturated output, the gradient is close to zero.
Backpropagation has no gradient to propagate back,
any small gradient keeps getting diluted at each lower layer.
The Sigmoid function has a mean of 0.5, which exacerbates the problem.
The Hyperbolic tangent function behaves better (since it's mean is 0).

### Initialization of weights and biases

  - If weights are initialized to the same value, then each neuron is equivalent to any other neuron. It's like saying you only have one neuron per layer. Such a network is impossible to converge.
  - Biases can be initialized to the same value. Assymetry is acheived due to randomly initialized weights as they are used when calculating the next value of the bias for each neuron.

### Glorot and He Initialization

Solution is to use Glorot initialization:
$$fan_{avg} = \frac{fan_{in} + fan_{out}}{2}$$
Initialize weights with Normal distribution with 0 mean, variance $\sigma^2 = \frac{1}{fan_{avg}}$

Or, a uniform distribution between $\pm$r, with $r = \sqrt{\frac{3}{fan_{avg}}} = \sqrt{3{\sigma}^2}$

Glorot initialization can speed up training considerably.

| Initialization | Activation functions | $\sigma^2$ (Normal) |
|---|---|---|
|Glorot|None, tanh, logistic, softmax|$\frac{1}{fan_{avg}}$|
|He|ReLU and variants|$\frac{2}{fan_{in}}$|
|LeCun|SELU|$\frac{1}{fan_{in}}$|

By default Keras uses Glorot initialization with a uniform distribution.

```python
# To use He init with 𝑓𝑎𝑛_𝑖𝑛:

# keras.layers.Dense(10, activation='relu', kernel_initializer='he_normal')
# or kernel_initializer='he_uniform'

# To use He init with 𝑓𝑎𝑛_𝑎𝑣𝑔:
# he_avg_init = keras.initializers.VarianceScaling(scale=2, mode='fan_avg', distribution='uniform')
# keras.layers.Dense(10, activation='sigmoid', kernel_initializer=he_avg_init)

# With distribution="uniform", samples are drawn from a uniform distribution within [-limit, limit], with
#     limit = sqrt(3 * scale / n).
#     n = number of input weights
# So you can specify the limits of the uniform distribution by setting the scale value
```

### Nonsaturating Activation Functions

ReLU (Rectified Linear Unit):
  - $$\begin{align}RELU(z) & = 0 \;\;\; if \; z < 0 \\
                       & = z \;\;\; if \; z >= 0 \end{align}$$
  - ReLU function thus does not saturate for positive values.
  - It is also fast to compute, since we only have to look at
    z to decide the output.
  - If a layer's weighted sums are negative for all input instances,
    the output is 0 (and stays 0 forever). This is the dying ReLU problem.
  - Use Leaky ReLU instead

**The problems with ReLU occur since it's derivative is zero sometimes (when z < 0). All the solutions below have non-zero derivatives for any z value.**

Leaky ReLU:
  - $$Leaky \; ReLU_\alpha(z) = max(\alpha z, z)$$
  - $\alpha$ is the slope of the function for negative z
  - Setting $\alpha$ to 0.2 (a huge leak) seems to perform better than 0.01 (a small leak)
  - Default value of $\alpha$ = 0.3
  - **Preferred if you're worried about runtime latency**
  - **Useful if you need your net to be as fast as possible**

RReLU (Randomized Leaky ReLU)
  - uses random $\alpha$ for training,
    and a fixed average $\alpha$ for testing.
    It performed fairly well and seemed to act as a regularizer
  - **Preferred if you're overfitting**

PReLU (Parametric Leaky ReLU)
  - allows $\alpha$ to be learned during training.
    It becomes a parameter that backprop can modify.
  - **strongly outperformed ReLU on large datasets, but risks overfitting on smaller datasets**
  - **Preferred if you have a huge training set**

ELU (Exponential Linear Unit):
  - **Outperformed all ReLUs**
  - **Training time was reduced**
  - **Network performed better on test set**
  - $$\begin{align}ELU_\alpha(z) & = \alpha(e^z - 1) & if \; z < 0 \\
                                 & = z & if \; z >= 0 \end{align}$$
  - ELU increases linearly with slope 1 for z > 0, and
    becomes more and more negative with an asymptote at -$\alpha$ as z becomes more negative
  - If $\alpha\,=\,1$, the function is smooth everywhere, including at z = 0.
    This allows Gradient Descent to speed up, as it does not bounce left and right of 0.
  - **ELU is slower to compute. It's faster training rate helps at training time,
    but slow computation hurts during testing.**

SELU (Scaled ELU):
  - If:
    - Your network consists only of a stack on Dense layers
    - All hidden layers use the SELU activation function
    - Input features are all standardized (mean 0, std. dev. 1)
    - All hidden-layer weights are initialized with LeCun normalization (kernel_initializer = 'lecun_normal')
    - Network's architecture must be sequential. No Recurrent Networks, Skip connections (Wide and Deep nets).
    - Some researchers have mentioned SELU works well for Convolutional Networks as well (even though there are no Dense layers).
  - If there is no regularization (ex. dropout or L1), SELU ensures the model is self-normalized which solves the vanishing/exploding gradients problem
**In general, prefer SELU > ELU > Leaky ReLU (and it's variants) > tanh > logistic**

```python
# This is how you use the ReLU/SELU functions
#
# model = keras.models.Sequential([
#     [ ... ]
#     keras.layers.Dense(10, kernel_initializer = "he_normal"),
#     keras.layers.LeakyReLU(alpha = 0.2),   # For LeakyReLU, this layer should come after each layer you want to
#                                            # apply it to. Default alpha = 0.3.
#                                            # For PReLU, replace this LeakyReLU() with PReLU().
#                                            # No implementation of RReLU() yet - you write your own.
#     [ ... ]
# ])

# layer = keras.layers.Dense(10, activation='selu', kernel_initializer='lecun_normal')
```

### Saturating activation functions

Hyperbolic tan function (tanh):
  - Useful in the output layer if you need to output a number between -1 and 1
  - Rarely used in hidden layers, except for RNNs

Logistic function (Sigmoid):
  - Useful in the output layer if you want to output a probability
  - Rarely used in hidden layers (except for the coding layer of variational autoencoders)

Softmax function:
  - Useful in the output layer to output probabilities for mutually exclusive classes
  - Rarely used in hidden layers

### Batch Normalization (BN)

**Batch Normalization is one of the most-used layers in deep neural networks. So much so that it is omitted from diagrams and it is assumed that it as there after each layer.**

Why do we need it:
  - The Glorot/He initialization plus any nonsaturating activation function reduces the danger of vanishing/exploding gradients at the beginning of training. But they can come back during training. BN addresses this issue.

How:
  - Add BN layer before/after the activation function.

How does it work:
  - Evaluates mean and std dev of the input over the current mini-batch (hence it's called Batch Normalization).

  $$\displaystyle\begin{align}\mu_B & = \frac{1}{m_B}\sum_{i=1}^{m_B}x^{(i)} \\
                          {\sigma_B}^2 & = \frac{1}{m_B}\sum_{i=1}^{m_B}{(x^{(i)} - \mu_B)}^2\end{align}$$
  - 0-centers and normalizes each input, then scales and shifts it according to two new parameter vectors per layer.

  $$\begin{align}\widehat{x}^{(i)} & = \frac{x^{(i)}\;-\;\mu_B}{\sqrt{{\sigma_B}^2\;+\;\epsilon}} \\
                           z^{(i)} & = \gamma\;\otimes\;\widehat{x}^{(i)}\;+\;\beta\end{align}$$

  <paragraph>where $\;\otimes \;=\;$ elementwise$\;$ multiplication</paragraph>

  <paragraph>$\epsilon \;=\; $small number to avoid divide-by-zero error, called a smoothing term, typically $10^{-5}$</paragraph>
  - During backpropagation, each batch-normalized layer learns:
    - $\gamma$ : the output scale vector
    - $\beta$  : the output offset vector
    - $\mu$    : final input mean vector (learned by using a moving average of the layer's input mean)
    - $\sigma$ : final input std dev vector (learned by using a moving average of the layer's input std dev)
  - The batch mean/std.dev are used during training, and final mean/std.dev are used after training.
  - **Benefits:**
    - **Improved neural networks**
    - **Strongly reduced vanishing gradients problem**
    - **Networks were also much less sensitive to weight initialization**
    - **Could use much larger learning rates, speeding up learning**
    - **Acts as a regularizer, reducing the need for any other regulatization technique**
    - **Does not affect shallower networks as much, but has a tremendous impact on deep networks.**
    - **Although converges faster, training per epoch is rather slow. All in all, wall time will be shorter.**

```python
# Add a Batch Normalization layer before/after each hidden layer's activation function,
# and optionally after the first layer in your model (as shown here).
#
# model = Sequential([  # In this tiny example with just 2 hidden layers, it's unlikely that Batch Normalization
#                       # will have a very positive impact. For deeper networks, it can make a tremendous difference.
#   Flatten(input_shape=[28, 28]),
#   BatchNormalization(),
#   Dense(300, activation='elu', kernel_initializer='he_normal'),
#   BatchNormalization(),
#   Dense(100, activation='elu', kernel_initializer='he_normal'),
#   BatchNormalization(),
#   Dense(10, activation='softmax')
# ])
# model.summary()  # This will show you the Batch Normalization layers. The number of parameters
#                  # added for the BN layers is 4 x (number of input parameters).
#                  # mu and sigma are not trainable via backprop, and this is what Keras shows at the bottom.
# [(var.name, var.trainable) for var in model.layers[1].variables] # will list trainable/untrainable parameters.
#
# model.layers[1].updates  # Shows the operations Keras created to train the trainable params (gamma, beta)
#                          # at each iteration. Since we're using the Tensorflow backend, these are TF operations.

# Shows how to add BN layers before the activation function.
# You should try adding them before and after the activation function,
# and choosing the way that works best.
# model = Sequential([
#   Flatten(input_shape=[28, 28]),
#   BatchNormalization(),
#   Dense(300, kernel_initializer='he_normal', use_bias=False), # Removed activation function.
#                                                               # Since BN layer already has bias,
#                                                               # remove bias from the layer using use_bias=False.
#   BatchNormalization(),
#   Activation('elu'),
#   Dense(100, kernel_initializer='he_normal', use_bias=False),
#   BatchNormalization(),
#   Activation('elu'),
#   Dense(10, activation='softmax')
# ])
#
```

${\bf Momentum}$ hyperparameter for BN layer:

BN layer has a lot of hyperparameters you can tweak.
Most defaults will be fine.
Usually you would need to tweak the momentum.
Given a new vector of input means or std dev (call it $\widehat{v}\,$) computed over the current batch,
the BN layer updates the running average $\widehat{v}$ using:
$$\widehat{v} = \widehat{v}\;\times\;momentum\;+\;v\;\times\;(1 - momentum)$$
A good momentum is typically close to 1 (ex. 0.9, 0.9, 0.999). You want more 9's for larger datasets, and smaller mini-batches.

${\bf Axis}$ hyperparameter for BN layer:

- Determines which axis should be normalized.
- Defaults to -1, which means the last axis. So if you have batch shape [batch size, features], the normalization is across the batch size for each feature.
- If BN layer was before the Flatten layer, batch shape would be [batch size, height, width], so normalization would be across the batch size and height for each width. This means we would get 28 means and 28 std devs for each column. If you want to use the 784 pixels independently, you should set axis=[1, 2].

### Gradient Clipping

- Gradient Clipping is used in Recurrent Neural Networks (RNN), since Batch Normalization is tricky. Other than RNNs, BN layer works fine.
- Gradient Clipping clips the gradient during backprop so it does not exceed some threshold. ex:

optimizer = keras.optimizers.SGD(clipvalue=1.0)  # all gradients clipped to $\pm$1.0

model.compile(loss='mse', optimizer=optimizer)

- clipvalue can change the orientation of the gradient vector. ex. if original gradient = [0.9, 100.0], after clipping it will be [0.9, 1.0]
- use clipnorm instead of clipvalue to keep orientation of gradient vector. This clips gradient if it's L2 norm > threshold. clipnorm=1.0 will change [0.9, 100.0] to [0.00899, 0.9999], preserving it's orientation.

## Using pretrained layers

- Transfer learning:
  - Use the lower layers of a DNN trained to solve a similar problem as the first layers of your network. ex. To recognize facial expressions, you can use a DNN that is trained to recognize faces. The lower layers of this network will have learned where the mouth, eyes, etc, are on the face. You can then add your upper layers to recognize facial expressions.
  - Transfer learning will work best when the inputs have similar low-level features.
  - The more similar the tasks are, the more layers you will want to use. For very similar tasks, keep all hidden layers, and replace just the output layer.
- Initially, try locking the weights for the pre-trained layers, so when training the upper layers, those weights remain the same.
- If that does not work, try unfreezing one or two top hidden layers to let them updated weights during backprop. The more training data you have, the more layers you can unfreeze. Reduce the learning rate when you unfreeze layers.
- If you still have problems:
  - and you have little training data, try dropping the top hidden layers and freezing all remaining hidden layers.
  - and you have plenty of training data, try replacing the top hidden layers instead of dropping them, and even adding more hidden layers.

### Transfer learning with Keras

```python
# If you use a model's weights as-is, any weight changes to the new model will affect the original as well.
# So you have to clone the original.
# This is just an example. Transfer learning does not work well with small dense networks, probably because
# - small networks learn few patterns
# - dense networks learn very specific patterns which are unlikely to be useful in other tasks
# Transfer learning works best with deep convolutional neural networks, which tend to learn
# feature detectors that are much more general (especially in the lower layers).
# Deep Convolutional networks in Chapter 14
#
# model_A = keras.models.load_model('my_model_A.h5')
# model_A_clone = keras.models.clone_model(model_A) # model_A is the model that solves a similar problem.
# model_A_clone.set_weights(model_A.get_weights())  # This is done since clone_model() does not clone weights.
# model_B_on_A = Sequential(model_A_clone.layers[:-1]) # Include all layers except output
# model_B_on_A.add(Dense(1, activation='sigmoid'))     # Our own output layer

# for layer in model_B_on_A.layers[:-1]:               # Freeze borrowed layer weights. This may be needed for the
#     layer.trainable = False                          # first few epochs until the new output layer has learned
#                                                      # reasonable weights. This is done since it's weights are
#                                                      # random and will wreck the borrowed layer weights.

# model_B_on_A.compile(loss='binary_crossentropy',     # You must always compile your model after you
#                      optimizer='sgd',                # freeze/unfreeze layers.
#                      metrics='accuracy')

# history = model_B_on_A.fit(X_train_B, y_train_B, epochs=4,         # Train the model for a few epochs
#                            validation_data=(X_valid_B, y_valid_B))

# for layer in model_B_on_A.layers[:-1]:               # Now our new layer must have learned good weights, so
#     layer.trainable = False                          # unfreeze the lower layers.

# optimizer = keras.optimizer.SGD(lr=1e-4)             # After unfreezing the lower layers, it is a good idea to
#                                                      # lower the learning rate to avoid damaging the
#                                                      # lower-layer weights.

# model_B_on_A.compile(loss='binary_crossentropy',     # You must always compile your model after you
#                      optimizer=optimizer,            # freeze/unfreeze layers.
#                      metrics='accuracy')

# history = model_B_on_A.fit(X_train_B, y_train_B, epochs=16,
#                            validation_data=(X_valid_B, y_valid_B))
```

### Unsupervised pretraining

Suppose you have a complex task but not much labeled training data, and cannot find a model trained on a similar task. You should:
  - try to get more labeled training data. If you cannot,
  - try to perform unsupervised pretraining:
    - Requires plenty of unlabeled training data
    - Use it to pretrain an unsupervised model (ex. autoencoder/GAN (Generative Adversarial Network)). See Ch 17.
    - Use the lower layers of the unsupervised model, add your output layer on top, and fine-tune the final network using supervised learning (with the labeled training data).

### Pretraining on an auxiliary task

If you don't have much labeled training data, another approach is to:
  - train a first network on an auxiliary task for which you have labeled data. Then reuse the lower layers for your actual task.
  - ex. for a Natural Language Processing (NLP) task, you can:
    - download a corpus of millions of text documents,
    - randomly mask out some words and train a model to predict what those words are.
    - This will train your model to "understand" the language to some extent.
    - Then you can reuse it for your actual task and fine-tune it for your labeled data.

 Self-supervised learning:
   - you automatically generate the labels from the data itself.
   - train a model on the resulting labeled dataset using supervised learning.
   - Does not require human labeling whatsoever, so it is best classified as a form of unsupervised learning.

## Faster Optimizers

Most popular algorithms:
  - momentum optimization
  - Nesterov Accelerated Gradient
  - AdaGrad
  - RMSProp
  - Adam
  - Nadam

### Momentum Optimization

  - Gradient Descent takes small regular steps down the slope. The step size is also a function of the gradient value. If the gradient is tiny, the step size will be tiny also.

$$\theta \leftarrow \theta - \eta\nabla_{\theta}J(\theta)$$

  - Momentum optimization changes the step size as a function of time, quickly taking larger and larger steps.

$$\begin{align}m & \leftarrow\beta m - \eta \nabla_{\theta}J(\theta) \\
          \theta & \leftarrow \theta + m \end{align}$$
          <paragraph>where $\beta$ is the momentum and should be set between 0 (high friction) and 1 (no friction).
          If $\beta$ = 0.9, the velocity is 10 x $\eta\nabla_{theta}J(\theta)$, so
          momentum optimization goes 10 times faster than gradient descent.</paragraph>
  - Benefits:
    - escapes from plateaus faster than Gradient Descent
    - can roll past local optima
    - For deep networks that don't use Batch Normalization, the upper layers will end up with inputs of very different scales. Momentum optimization helps a lot to converge for layers across all input scales.
  - Issues:
    - The optimizer may overshoot in one direction, then overshoot in another thus oscillating for a while before stabilizing at the minimum. Good to have friction to get rid of oscillations and speed up coverage.
  - momentum value of 0.9 usually works well in practice.

### Nesterov Accelerated Gradient (NAG) AKA Nesterov Momentum Optimization

  - This variant of momentum optimization is almost always faster than vanilla momentum optimization. In fact it is significantly faster than regular momentum optimization.
  - It measures the gradient of the cost function slightly ahead in the direction of the momentum. The vanilla momentum optimization computes the gradient before the momentum update to $\theta$. NAG updates the momentum to $(\theta + \beta \; m)$ and then applies the gradient to it. This works because in general the momentum will point in the right direction.

  $$\begin{align}m & \leftarrow\beta m - \eta \nabla_{\theta}J(\theta + \beta \; m) \\
          \theta & \leftarrow \theta + m \end{align}$$

```python
# optimizer = kears.optimizers.SGD(lr=0.0001, momentum=0.9, nesterov=True)  # Nesterov Accelerated Gradient (NAG),
#                                                                          # AKA Nesterov Momentum Optimization
```

### AdaGrad

  - It would be nice if the algorithm could correct it's direction to point more towards the global optimum. AdaGrad achieves this by scaling down the gradient vector along the steepest dimensions.

$$\begin{align}s & \leftarrow s + \nabla_{\theta}J(\theta) \otimes \nabla_{\theta} J(\theta) \\
         \theta  & \leftarrow \theta - \eta \nabla_{\theta} J(\theta) \div \sqrt{s + \epsilon} \end{align}$$
         <paragraph>Accululate the square of the gradient into the variable s.
    Then scale down the gradient vector by a factor of $\sqrt{s + \epsilon}$.
    Let's use the $\div$ symbol for elementwise division.
    $\epsilon$ is a smoothing term to avoid div-by-zero. Typically set to 1e-10. This decays the learning rate faster for steeper dimensions than for dimensions with gentler slopes. This is called an adaptive learning rate. </paragraph>
  - Benefits:
    - It helps point the update toward the global optimum.
    - This algorithm requires much less tuning of the learning rate $\eta$
    - Performs well for simple quadratic problems - may be efficient for Linear Regression.
  - Issues:
    - Often stops too early. The learning rate gets scaled down so much that the algorithm stops before reaching global optimum. You should not use it for Deep NNs.

### RMSProp

  - Since AdaGrad runs the risk of slowing down a bit too fast and never converging, RMSProp accumulates the gradients from only the most recent iterations (as opposed to all the gradients since the beginning of training). It does so by using exponential decay in during gradient accumulation (first equation below):

$$\begin{align}s & \leftarrow \beta s + (1 - \beta) \nabla_{\theta}J(\theta) \otimes \nabla_{\theta} J(\theta) \\
         \theta  & \leftarrow \theta - \eta \nabla_{\theta} J(\theta) \odiv \sqrt{s + \epsilon)} \end{align}$$
         <paragraph>Accumulate the square of the latest gradients into the variable s.
    Then scale down the gradient vector by a factor of $\sqrt{s + \epsilon}$.
    The $\odiv$ is the symbol for elementwise division.
    The decay rate is $\beta$.
    $\epsilon$ is a smoothing term to avoid div-by-zero. Typically set to 1e-10. This decays the learning rate faster for steeper dimensions than for dimensions with gentler slopes. This is called an adaptive learning rate. </paragraph>

  - The decay rate $\beta$ is typically set to 0.9 (which typically works well).
  - Can lead to solutions that generalize poorly on some datasets. So if you're seeing this, try using plain Nesterov Accelerated Gradient instead.
  - RMSProp was the preferred optimization algorithm for many researchers until Adam optimization came around.

```python
# optimizer = keras.optimizers.RMSprop(lr=0.001, rho=0.9) # rho is the decay rate beta in the above text
```

### Adam, AdaMax and Nadam Optimization

  - Adam (Adaptive Moment Estimation) combines the ideas of momentum optimization and RMSProp.
    - It keeps track of exponentially decaying average of past gradients
    - Exponentially decaying average of past squared gradients

 $$\begin{align}m & \leftarrow \beta_1 m - (1 - \beta_1) \nabla_{\theta} J(\theta) \\
                s & \leftarrow \beta_2 s + (1 - \beta_2) \nabla_{\theta}J(\theta) \otimes \nabla_{\theta} J(\theta) \\
      \widehat{m} & \leftarrow \frac{m}{1 - \beta_1^T} \\
      \widehat{s} & \leftarrow \frac{s}{1 - \beta_2^T} \\
           \theta & \leftarrow \theta + \;\eta \; \widehat{m} \; \odiv \sqrt{\widehat{s} + \epsilon}
      \end{align}$$
      <paragraph>T represents the iteration number starting at 1.
    $\widehat{m}$ and $\widehat{s}$ are initialized to 0 at the beginning of training, so the third and fourth equations are needed to boost them.</paragraph>
    - Momentum decay $\beta_1$ is typically initialized to 0.9
    - Scaling decay $\beta_2$ is typically initialized to 0.999
    - Smoothing term $\epsilon$ defaults to None, which tells Keras to use keras.backend.epsilon which is 1e-7. If you want, you can change it using keras.backend.set_epsilon()
    - Learning rate $\eta$ is typically set to 0.001. Since Adam is adaptive, it requires less tuning of $\eta$
    - Can lead to solutions that generalize poorly on some datasets. So if you're seeing this, try using plain Nesterov Accelerated Gradient instead.
  - AdaMax
    - replaces the L2 norm of parameter updates of Adam with the $L_{\infty}$ norm (the max value of the time-decayed gradients).
    - Adam performs better, so you can try this algorithm if you're seeing problems with AdaMax
  - Nadam (Nesterov Adam)
    - This is Adam + Nesterov trick, so it will converge slightly faster than Adam.
    - Can lead to solutions that generalize poorly on some datasets. So if you're seeing this, try using plain Nesterov Accelerated Gradient instead.

```python
# optimizer = keras.optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999)
```

### Training Sparse Models

- If you need
  - blazing fast model at runtime, or
  - you need to take up less memory
- then
  - train the model as usual
  - set it's tiny weights to 0 (using model.set_weights()?)
  - This may typically not lead to a very sparse model, and
  - may degrade the model's performance.
- Better option
  - Apply strong L1 regularization during training (this zeroes out many weights)
- If this is not enough
  - Use TensorFlow Model Optimization Toolkit (TF-MOT) which has a pruning API that iteratively removes connections during training based on their magnitude.

```python
# Untested code - check to make sure it works.
# To set model weights to zero for weights less than threshold.

# def set_low_weights_to_zero(layer, threshold=0.01):
#     weights = layer.get_weights()
#     weights[weights < threshold] = 0.0
#     layer.set_weights(weights)

# layer = model.layers[1]
# set_low_weights_to_zero(layer, threshold=0.001)
```

### Optimizer Comparison

Optimizer Comparison

Class	Convergence Speed	Convergence quality

SGD	*	**

SGD(momentum=...)	**	***

SGD(momentum=..., nesterov=True)	**	***

Adagrad	***	* (stops too early)

RMSProp	***	** or ***

Adam	***	** or ***

Nadam	***	** or ***

AdaMax	***	** or ***

## Learning Rate Scheduling

  - To find a good rough LR:
    - You can train your model at different exponential learning rates (say 0.001, 0.01, 0.1, 1.0).
    Then pick an LR that gives fast convergence without blowing up
  - Then you can pick an LR that is slightly lower than the best LR you picked above, and train your model. If you're satistifed with the error, this is fine. If you're not satisfied:
    - Select a range around the best rough LR you found above, and zoom in by trying at various LRs around it.
    - If you're satistifed with the loss, you can stop. If not, continue to zoom into the LR that gives you the error threshold you're trying to achieve. You may not know the threshold - in which case you can zoom in to see which LR is best.
  - Learning Schedules:
    - Power scheduling

        $$\eta(t) = \frac{\eta_0}{1 + \left(\frac{t}{s}\right)^c}$$

      <paragraph>t = iteration number

      $\eta_0$ = initial learning rate

      c = power (typically 1)

      s = number of steps before learning rate drops. After s steps it is down to $\eta_0/2$, after another s steps it is down to $\eta_0/3$, etc.
</paragraph>
    - Exponential scheduling
        $$\eta(t) = \eta_0 \; {0.1}^{\frac{t}{s}}$$
      <paragraph>t = iteration number

      $\eta_0$ = initial learning rate

      c = power (typically 1)

      s = number of steps before learning rate drops. Slashes LR by a factor of 10 every s steps
</paragraph>
      - Works well, converges fast
      - Easy to tune
      - 1cycle is best
    - Piecewise constant scheduling
      - Use a constant rate of say $\eta_0$ = 0.1 for 5 epochs, then a smaller rate of say $\eta_0$ = 0.01 for 50 epochs, etc. Requires fiddling around with the right sequence and number of epochs.
    - Performance scheduling
      - Measure validation error every N steps (just like for early stopping), then reduce LR by a factor of $\lambda$ when the error stops dropping
      - Works well, converges fast
      - Easy to tune
    - 1cycle scheduling
      - Choose an $\eta_1$ based on the rough LR algorithm shown above
      - Choose $\eta_0$ to be 0.1 $\eta_1$
      - Increase LR from $\eta_0$ to $\eta_1$ linearly halfway through training (half of the epochs).
        If we're using momentum, start with a high momentum (say 0.95), then drop it linearly to say 0.85.
      - Decrease LR from $\eta_1$ to $\eta_0$ linearly through the rest of training.
        If we're using momentum, increase momentum from the 0.85 level to 0.95

```python
# For the scheduling functions, the final LR is saved with the model
# unless your schedule function uses the epoch argument.
# In this case, when you load your model and want to continue training,
# you would set the fit() method's initial_epoch argument so the epoch starts
# at the right value.

# power scheduling
# optimizer = keras.optimizer.SGD(lr=0.01, decay=1e-4)  # decay is inverse of s.
                                                        # Keras assumes c = 1.

# exponential scheduling
# def exponential_decay(lr0, s):       # When you create a LearningRateScheduler instance with the exponential_decay_fn,
#     def exponential_decay_fn(epoch): # the LearningRateScheduler will update the optimizer's LR at the beginning
#         return lr0 * 0.1**(epoch / s)# of each epoch. You can also write your own callback to update LR at each step.
#     return exponential_decay_fn      # This makes sense if there are many steps per epoch.
#                                      # Alternatively, use keras.optimizers.schedules.
#                                      # The exponential_decay_fn() could take the LR as the second argument to return
#                                      # an updated LR value.
# exponential_decay_fn = exponential_decay(lr0=0.01, s=20)

# lr_scheduler = keras.callbacks.LearningRateScheduler(exponential_decay_fn)
# history = model.fit(X_train_scaled, y_train, [...], callbacks=[lr_scheduler])

# # piecewise constant scheduling
# def piecewise_constant_fn(epoch):
#     if epoch < 5:
#         return 0.01
#     elif epoch < 15:
#         return 0.005
#     else:
#         return 0.001

# # performance scheduling
# lr_scheduler = keras.callback.ReduceLROnPlateau(factor=0.5, patience=5) # Multiply LR by 0.5 if validation
#                                                                         # error is the same for 5 consecutifve epochs
#                                                                         # (given by the patience variable).
```

```python
# To update the LR at each step (rather than at each epoch),
# use the tf.keras.optimizers.schedules.
# This is specific to tf.keras - it is not available in the Keras API.
# The example below does the same thing as the exponential_decay_fn() above.
#
# s = 20 * len(X_train) // 32 # number of steps in 20 epochs (batch size = 32)
# learning_rate = keras.optimizers.schedules.ExponentialDecay(0.01, s, 0.1)
# optimizer = keras.optimizers.SGD(learning_rate)
```

```python
# 1cycle scheduling.
# This is how you would scan through the LR range to find the LR value that gives the minimum loss.

# K = keras.backend

# class ExponentialLearningRate(keras.callbacks.Callback):
#     def __init__(self, factor):
#         self.factor = factor
#         self.rates = []
#         self.losses = []
#     def on_batch_end(self, batch, logs):
#         self.rates.append(K.get_value(self.model.optimizer.lr))
#         self.losses.append(logs["loss"])
#         K.set_value(self.model.optimizer.lr, self.model.optimizer.lr * self.factor)

# def find_learning_rate(model, X, y, epochs=1, batch_size=32, min_rate=10**-5, max_rate=10):
#     init_weights = model.get_weights()
#     iterations = len(X) // batch_size * epochs
#     factor = np.exp(np.log(max_rate / min_rate) / iterations)
#     init_lr = K.get_value(model.optimizer.lr)
#     K.set_value(model.optimizer.lr, min_rate)
#     exp_lr = ExponentialLearningRate(factor)
#     history = model.fit(X, y, epochs=epochs, batch_size=batch_size,
#                         callbacks=[exp_lr])
#     K.set_value(model.optimizer.lr, init_lr)
#     model.set_weights(init_weights)
#     return exp_lr.rates, exp_lr.losses

# def plot_lr_vs_loss(rates, losses):
#     plt.plot(rates, losses)
#     plt.gca().set_xscale('log')
#     plt.hlines(min(losses), min(rates), max(rates))
#     plt.axis([min(rates), max(rates), min(losses), (losses[0] + min(losses)) / 2])
#     plt.xlabel("Learning rate")
#     plt.ylabel("Loss")
```

```python
# tf.random.set_seed(42)
# np.random.seed(42)

# model = keras.models.Sequential([
#     keras.layers.Flatten(input_shape=[28, 28]),
#     keras.layers.Dense(300, activation="selu", kernel_initializer="lecun_normal"),
#     keras.layers.Dense(100, activation="selu", kernel_initializer="lecun_normal"),
#     keras.layers.Dense(10, activation="softmax")
# ])
# model.compile(loss="sparse_categorical_crossentropy",
#               optimizer=keras.optimizers.SGD(lr=1e-3),
#               metrics=["accuracy"])
```

```python
# batch_size = 128
# rates, losses = find_learning_rate(model, X_train_scaled, y_train, epochs=1, batch_size=batch_size)
# plot_lr_vs_loss(rates, losses)
```

```python
# This shows how you can ramp up on the LR rate and ramp down on it at the halfway point.
#

# class OneCycleScheduler(keras.callbacks.Callback):
#     def __init__(self, iterations, max_rate, start_rate=None,
#                  last_iterations=None, last_rate=None):
#         self.iterations = iterations
#         self.max_rate = max_rate
#         self.start_rate = start_rate or max_rate / 10
#         self.last_iterations = last_iterations or iterations // 10 + 1
#         self.half_iteration = (iterations - self.last_iterations) // 2
#         self.last_rate = last_rate or self.start_rate / 1000
#         self.iteration = 0
#     def _interpolate(self, iter1, iter2, rate1, rate2):
#         return ((rate2 - rate1) * (iter2 - self.iteration)
#                 / (iter2 - iter1) + rate1)
#     def on_batch_begin(self, batch, logs):
#         if self.iteration < self.half_iteration:
#             rate = self._interpolate(0, self.half_iteration, self.start_rate, self.max_rate)
#         elif self.iteration < 2 * self.half_iteration:
#             rate = self._interpolate(self.half_iteration, 2 * self.half_iteration,
#                                      self.max_rate, self.start_rate)
#         else:
#             rate = self._interpolate(2 * self.half_iteration, self.iterations,
#                                      self.start_rate, self.last_rate)
#             rate = max(rate, self.last_rate)
#         self.iteration += 1
#         K.set_value(self.model.optimizer.lr, rate)
```

```python
# n_epochs = 25
# onecycle = OneCycleScheduler(len(X_train) // batch_size * n_epochs, max_rate=0.05)
# history = model.fit(X_train_scaled, y_train, epochs=n_epochs, batch_size=batch_size,
#                     validation_data=(X_valid_scaled, y_valid),
#                     callbacks=[onecycle])
```

## Avoiding Overfitting through Regularization

  - Deep Neural Networks have millions of parameters. So it is easy to overfit when using them. We need to regularize.
  - Techniques for regularization:
    - Early Stopping
    - L1 and L2 regularization
    - Dropout regularization
    - Max-norm regularization

### L1 and L2 Regularization

```python
# L1 regularization may create a sparse model (many weights are 0)
# L2 regularization constraints the connection weights

# layer = Dense(100, activation='elu',
#               kernel_initializer='he_normal',
#               kernal_regularizer=keras.regularizers.l2(0.01)) # To use L1 regularization, use l1(0.01).
#                                                               # To use both L1 and L2 regularization, use:
#                                                               # keras.regularizers.l1_l2(l1=0.01, l2=0.01)

# # Typically, you use the same regularizer for all layers, and the same init strategy for all layers.
# # Instead of repeating code, use functools.partial():

# from functools import partial

# RegularizedDense = partial(keras.layers.Dense,
#                            activation='elu',                   # Put all your common layer init here
#                            kernel_initializer='he_normal',
#                            kernel_regularizer=keras.regularizers.l2(0.01))
# model = Sequential([
#     Flatten(input_shape=[28, 28]),
#     RegularizedDense(300),
#     RegularizedDense(100),
#     RegularizedDense(10, activation='softmax', kernel_initializer='glorot_uniform') # Setting other arguments
# ])
```

### Dropout Regularization

  - Even a state of the art network gets a 1-2% accuracy boost by adding dropout.
    If you're already at 95% accuracy, 2% boost means error went down by 40%. This is huge.
  - Dropout algorithm allows each neuron a probability p of getting dropped out at each training step.
    If dropped, the neuron will be entirely ignored during this training step.
    It may become active in the next training step.
  - p = dropout rate, typically 0.1 - 0.5 (closer to 0.2-0.3 in RNNs, closer to 0.4-0.5 in CNNs)
  - After training, neurons don't get dropped anymore
  - Input neurons could be dropped
  - Output neurons are never dropped
  - A neuron has to not rely on any single input neuron, since it can be dropped out.
    Neurons thus become less sensitive to small changes in the input - they generalize better.
  - Another way to think about it:
    - At each training step, the network is different
    - If there are N possible neurons that can be dropped, there are $2^N$ possible networks
    - If you've run 10000 training steps, you've tried 10000 different networks (each with just one training instance).
    - The resulting network can be seen as an average ensemble of these networks
  - If p = 0.5 during training, a neuron is connected to 2 x as many neurons during testing. This is because during testing we don't drop neurons. This is why we have to compensate by multiplying each neuron's weight by 0.5. More generally, we need to multiply each neuron's output weight by the keep probability (1 - p) after training. Instead we can divide each neuron's output by the keep probability during training. The Dropout layer handles this for us.
  - In practice you drop neurons from only the top 1-3 layers (ignoring the output layer)
  - Slows convergence during training, by approximately a factor of 2.
  - It is only turned on during training, so no impact on inference speed.
  - Results in a much better model - worth the time and effort
  - For SELU activation function, use alpha dropout (it preserves mean and std dev of it's inputs)

```python
# model = Sequential([
#     Flatten(input_shape=[28, 28]),
#     Dropout(rate=0.2),
#     Dense(300, activation='elu', kernel_initializer='he_normal'),
#     Dropout(rate=0.2),
#     Dense(100, activation='elu', kernel_initializer='he_normal'),
#     Dropout(rate=0.2),
#     Dense(10, activation='softmax')
# ])

# Since neurons are missing due to dropout during training, and are present during validation,
# the training and validation losses cannot be compared.
# The model could be overfitting the training set, but have the same validation losses.
# To compare them, evaluate the training loss without dropout (i.e. after training).
# If full dropout is too strong, you can try dropout for only one layer (the one before the output).
```

### Monte Carlo (MC) Dropout

  - Can boost the performance of any trained model without retraining or modifying it
  - Provides a much better measure of the model's uncertainity
  - Simple to implement
  - MC Dropout is active during training (so it slows down training) and inference (slows down inference slightly).
    Usually during MC Dropout, you want to run inference 10 or more times to get better predictions.
    Due to this, predictions are slowed down by a factor of 10 or more.

```python
# y_probas = np.stack([model.predict(X_test_scaled, training=True)# With 10000 instances in the test set and 10 classes,
#                      for sample in range(100)])                 # model.predict() returns matrix of shape [10000, 10].
# y_proba = y_probas.mean(axis=0)                                 # Stacking 100 such matrices results in an array of
#                                                                 # shape [100, 10000, 10]. Averaging over axis=0, gives
#                                                                 # shape [10000, 10] - a single prediction.
# # Averaging over a multiple predictions with dropout on (we have training=True)
# # gives us a Monte Carlo estimate
# # that is more reliable than a single prediction with dropout off.
# # You can compare the y_probas values against the first instance prediction:
# np.round(model.predict(X_test_scaled[:1]), 2) # versus
# np.round(y_probas[:, :1], 2)

# To see what the first instance average prediction using MC dropout is:
# np.round(y_proba[:1], 2)

# Also check out the std dev of the probability estimates.
# If there is a large variance in the probability estimates,
# you may treat these predictions warily (based on if they
# were life threatening, so cricital in some other way.)
#
# y_std = y_probas.std(axis=0)
# np.round(y_std[:1], 2)

# The higher the number of samples you use (100 in this example),
# the more accurate the predictions and their uncertainity estimates.
# But inference time will also increase.

# If you have other layers in your model that behave in a special way (ex. BatchNormalization),
# then you should not force training mode as we did in the model.predict() function above.
# Instead, replace Dropout layers with the MCDropout class shown below.
# This will ensure that only the dropout layers force training=True.

# class MCDropout(Dropout):
#     def call(self, inputs):
#         return super().call(inputs, training=True)

# If your model was already trained using Dropout,
# - create a new model, identical to the original, except:
#   it replaces the Dropout layers with MCDropout.
# - copy the existing model's weights to your new model
# - run the prediction on the model and average the output as
#   shown earlier in this cell.
```

### Max-Norm Regularization

  - Each neuron's weights are constrained such that $\;||w||_2 \le r$
  - Implemented by computing the weight norm after each training step and rescaling $w \leftarrow w \frac{r}{||w||_2}$
  - Reducing r increases regularization.
  - Max-Norm Regularization can also alleviate the unstable gradients problems (if you're not using Batch Normalization)

```python
# Dense(100, activation='elu', kernel_initializer='he_normal',
#       kernal_constraint=keras.constraints.max_norm(1.))       # The value given to max_norm() is the r value above.
#                                                               # The axis value for max_norm() defaults to 0.
#                                                               # Dense layer weights shape=[num_inputs, num_neurons].
#                                                               # So axis=0 means max-norm constraint applies to each
#                                                               # neuron's weight vector.
#                                                               # For CNNs, set max_norm()'s axis appropriately -
#                                                               # Usually axis=[0, 1, 2].

# You can define your own function as the kernel_constraint.
# You can also constraint the bias terms using bias_constraint.
```

## Summary and Practical Guidelines

The top table below works in most cases. Please do not consider these defaults as hard rules though !!

Default DNN Configuration

    Hyperparameter	Default value

Kernel initializer	He initialization

Activation function	ELU

Normalization	None if shallow; Batch Norm if deep

Regularization	Early stopping (+L2 reg. if needed)

Optimizer	Momentum optimization (or RMSProp or Nadam)

Learning rate schedule	1cycle

DNN Configuration for a self-normalizing net

    Hyperparameter	Default value

Kernel initializer	LeCun initialization

Activation function	SELU

Normalization	None (self-normalization)

Regularization	Alpha dropout if needed

Optimizer	Momentum optimization (or RMSProp or Nadam)

Learning rate schedule	1cycle

  - Normalize the input features
  - If you find a pre-trained net that solves a similar problem, re-use it
  - Use unsupervised pre-training if you have a lot of unlabeled data
  - Use pre-training on an auxiliary task if you have a lot of labeled data for a similar task
  - Exceptions to the above guidelines:
    - Use L1 regularization to get a sparse model (optionally zero out the tiny weights after training)
      For an even sparser model, use TensorFlow Model Optimization Toolkit (TF-MOT).
      This will break self-normalization, so you should use the default configuration.
    - For a low-latency model (to perform lightning-fast predictions):
      - use fewer layers
      - fold the BN layers into the previous layers
      - use a faster activation function (ex. leaky ReLU or ReLU)
      - use a sparse model
      - reduce the float precision from 32-bits to 16-bits or 8-bits
      - use TensorFlow Model Optimization Toolkit (TF-MOT)
    - If you're building a risk-sensitive app, or latency is not important:
      - use MC dropout to boost performance and get more reliable probability estimates along with uncertainity estimates

```python

```


Starting from input parameters `sigma0`, `sigmadot0`, `sigmaddot0`, `psi2`, `psi1`, and `psi0`, representing the corresponding values of the Bondi data at time $u=0$, we want to integrate the initial-value problem to find all the Bondi data at all times.  We start with
\begin{equation}
  \sigma = \texttt{sigma0} + u\,\texttt{sigmadot0} + u^2\,\texttt{sigmaddot0}\,/\,2,
\end{equation}
impose an initial condition on $\psi_2$ given by
\begin{equation}
  \Im[ψ₂] = -\Im[ð²σ̄ + σ ∂σ̄/∂u]
\end{equation}
so that the mass parameter is real, and then solve for the remaining parameters according to
\begin{align}
    ψ̇₀ &= -ðψ₁ + 3 σ ψ₂ \\
    ψ̇₁ &= -ðψ₂ + 2 σ ψ₃ \\
    ψ̇₂ &= -ðψ₃ + 1 σ ψ₄ \\
    ψ₃ &= ð ∂σ̄/∂u \\
    ψ₄ &= - ∂²σ̄/∂u²
\end{align}

Sympy can do all the heavy lifting for us, including writing the actual code.  There's a small difficulty with getting $\eth$ to act like a linear operator, which is why we have the Ð function below.

```python
import sympy
from sympy import init_printing
init_printing()

u = sympy.symbols('u', real=True)
ð = sympy.symbols('ð', cls=sympy.Function)

dot = lambda f: sympy.Derivative(f, u, evaluate=True)
ddot = lambda f: sympy.Derivative(f, u, u, evaluate=True)
integral = lambda f: sympy.Integral(f, u).doit()
```

```python
def Ð(expr, expand=True):
    """Apply ð and expand by linearity over numbers and u**n

    This is inspired by https://stackoverflow.com/a/56924562/1194883,
    which makes clever use of sympy's `expr.replace(filter, func)`
    functionality to expand sums inside the argument of ð, and then
    I make dumb use of the basic replace functionality (with
    wildcards) to expand scalar multiplication, assuming the scalars
    are just literal numbers and powers of `u`.

    """
    import sympy
    c = sympy.Wild('c', properties=(lambda x: x.is_Number,))
    p = sympy.Wild('p')
    w = sympy.Wild('w')
    expr = ð(expr)
    return expr.expand().replace(
        lambda x: x.func == ð,
        lambda x: sympy.Add(*[x.func(a) for a in sympy.Add.make_args(x.args[0])])
    ).replace(ð(c * u**p * w), c * u**p * ð(w))

```

```python
def split_polynomial(expr, string):
    import sympy
    from sympy.polys.polyfuncs import horner
    from sympy.printing.pycode import pycode
    from sympy.codegen.ast import Assignment
    expr = sympy.Poly(expr, u)
    values = [sympy.factor(c) for c in reversed(expr.all_coeffs())]
    n_coeffs = len(values)
    symbols = sympy.symbols(string+f"_:{n_coeffs}")
    symbol = sympy.symbols(string)
    value = horner(sympy.Poly(reversed(symbols), u))
    print('\n'.join(pycode(Assignment(s, v), allow_unknown_functions=True) for s, v in zip(symbols, values)))
    print(pycode(Assignment(symbol, value)))
    return value
```

```python
print('ð = lambda x: x.eth')
print('conjugate = lambda x: x.bar')
σ = split_polynomial(sympy.symbols('sigma0') + u * sympy.symbols('sigmadot0') + u**2 * sympy.symbols('sigmaddot0') / 2, 'σ')
print('# ψ₄ = -∂²σ̄/∂u²')
ψ4 = split_polynomial(-ddot(σ.conjugate()), 'ψ4')
print('# ψ₃ = ð ∂σ̄/∂u')
ψ3 = split_polynomial(Ð(dot(σ.conjugate())), 'ψ3')
print('# ψ₂ = ∫ (-ðψ₃ + σψ₄) du')
ψ̇2 = sympy.collect(sympy.expand(-Ð(ψ3) + σ*ψ4), u)
ψ2 = split_polynomial(integral(ψ̇2) + sympy.symbols('psi2'), 'ψ2')
print('# ψ₁ = ∫ (-ðψ₂ + 2σψ₃) du')
ψ̇1 = sympy.collect(sympy.expand(-Ð(ψ2) + 2*σ*ψ3), u)
ψ1 = split_polynomial(integral(ψ̇1) + sympy.symbols('psi1'), 'ψ1')
print('# ψ₀ = ∫ (-ðψ₁ + 3σψ₂) du')
ψ̇0 = sympy.collect(sympy.expand(-Ð(ψ1) + 3*σ*ψ2), u)
ψ0 = split_polynomial(integral(ψ̇0) + sympy.symbols('psi0'), 'ψ0')
```

    ð = lambda x: x.eth
    conjugate = lambda x: x.bar
    σ_0 = sigma0
    σ_1 = sigmadot0
    σ_2 = (1/2)*sigmaddot0
    σ = u*(u*σ_2 + σ_1) + σ_0
    # ψ₄ = -∂²σ̄/∂u²
    ψ4_0 = -2*conjugate(σ_2)
    ψ4 = ψ4_0
    # ψ₃ = ð ∂σ̄/∂u
    ψ3_0 = ð(conjugate(σ_1))
    ψ3_1 = 2*ð(conjugate(σ_2))
    ψ3 = u*ψ3_1 + ψ3_0
    # ψ₂ = ∫ (-ðψ₃ + σψ₄) du
    ψ2_0 = psi2
    ψ2_1 = σ_0*ψ4_0 - ð(ψ3_0)
    ψ2_2 = -1/2*(-σ_1*ψ4_0 + ð(ψ3_1))
    ψ2_3 = (1/3)*σ_2*ψ4_0
    ψ2 = u*(u*(u*ψ2_3 + ψ2_2) + ψ2_1) + ψ2_0
    # ψ₁ = ∫ (-ðψ₂ + 2σψ₃) du
    ψ1_0 = psi1
    ψ1_1 = 2*σ_0*ψ3_0 - ð(ψ2_0)
    ψ1_2 = -1/2*(-2*σ_0*ψ3_1 - 2*σ_1*ψ3_0 + ð(ψ2_1))
    ψ1_3 = -1/3*(-2*σ_1*ψ3_1 - 2*σ_2*ψ3_0 + ð(ψ2_2))
    ψ1_4 = -1/4*(-2*σ_2*ψ3_1 + ð(ψ2_3))
    ψ1 = u*(u*(u*(u*ψ1_4 + ψ1_3) + ψ1_2) + ψ1_1) + ψ1_0
    # ψ₀ = ∫ (-ðψ₁ + 3σψ₂) du
    ψ0_0 = psi0
    ψ0_1 = 3*σ_0*ψ2_0 - ð(ψ1_0)
    ψ0_2 = -1/2*(-3*σ_0*ψ2_1 - 3*σ_1*ψ2_0 + ð(ψ1_1))
    ψ0_3 = -1/3*(-3*σ_0*ψ2_2 - 3*σ_1*ψ2_1 - 3*σ_2*ψ2_0 + ð(ψ1_2))
    ψ0_4 = -1/4*(-3*σ_0*ψ2_3 - 3*σ_1*ψ2_2 - 3*σ_2*ψ2_1 + ð(ψ1_3))
    ψ0_5 = -1/5*(-3*σ_1*ψ2_3 - 3*σ_2*ψ2_2 + ð(ψ1_4))
    ψ0_6 = (1/2)*σ_2*ψ2_3
    ψ0 = u*(u*(u*(u*(u*(u*ψ0_6 + ψ0_5) + ψ0_4) + ψ0_3) + ψ0_2) + ψ0_1) + ψ0_0

```python

```


```python
import numpy as np
from sympy import symbols, solve
import matplotlib.pyplot as plt
from astropy.table import Table
plt.style.use('seaborn-darkgrid')
```

```python
Pillar_Load = lambda C,H,W,Wc : C*H*(W+Wc)/W
PillarSt_Mark = lambda wh : 6.205*(0.64+(0.54*wh))
PillarSt_CSIRO = lambda wh : 6*(0.64+0.36*(0.69+0.44*wh)*wh)

```

```python
def Pillar_Strength(wh, formula =[], name =[]):
    S=[]
    i=0
    while (i<len(formula)):
        s= formula[i](wh)
        S.append(s)
        print ("Pillar Strength for", name[i],":",s)
        i=i+1
    return S;

def FoS(P,wh,formula=[],name=[]):
    S = Pillar_Strength(wh, formula, name)
    FoS=[]
    i=0
    while (i<len(formula)):
        f = S[i]/P
        FoS.append(f)
        print ("FoS for", name[i],":",f)
        i=i+1
    return FoS,S,name;

def wh(P,FoS,formula=[],name=[]):
    S= P*FoS
    w =symbols('w')
    wh=[]
    i=0
    while (i<len(formula)):
        sn=solve (formula[i](w)-S,w)
        def largest(a,n):
            max=a[0]
            for i in range(1,n):
                if a[i]>max:
                    max = a[i]
            return max;
        n=len(sn)
        sol = largest(sn,n)
        wh.append(sol)
        print ("w/h ratio for", name[i],":",sol)
        i=i+1
    return FoS,S,name;
```

```python

def FoS_Range(P,FoS_max, FoS_min, interval,formula,name):
    n = (FoS_max-FoS_min)/interval
    a =FoS_min
    F=[FoS_min]
    i=0
    while(i<n):
        a= a+interval
        F.append(a)
        i=i+1
    S=[]
    i=0
    while(i<n+1):
        b=P*F[i]
        S.append(b)
        i=i+1
    wh =[]
    w =symbols('w')
    i=0
    while(i<(n+1)):
        sni=solve (formula[0](w)-S[i],w)
        def largest(a,ni):
            max=a[0]
            for y in range(0,ni):
                if a[y]>max:
                    max = a[y]
            return max;
        ni=len(sni)
        sol = largest(sni,ni)
        wh.append(sol)
        i=i+1
    wh=np.asarray(wh)
    F=np.asarray(F)
    wh.astype(int)
    F.astype(int)
    plt.style.use('seaborn-darkgrid')
    #plt.figure(figsize=())
    plt.plot(F,wh,'r^')
    plt.xlabel("FoS")
    plt.ylabel(name)
    t = Table([F,wh],names=('FoS',name))
    print(t)
    return t,F,wh;

```

```python
def FoS_wh_Range(P,FoS_max, FoS_min, interval,wh_max,wh_min,formula,name):
    n = (FoS_max-FoS_min)/interval
    a =FoS_min
    F=[FoS_min]
    i=0
    while(i<n):
        a= a+interval
        F.append(a)
        i=i+1
    S=[]
    i=0
    while(i<n+1):
        b=P*F[i]
        S.append(b)
        i=i+1
    wh =[]
    w =symbols('w')
    i=0
    while(i<(n+1)):
        sni=solve (formula[0](w)-S[i],w)
        def largest(a,ni):
            max=a[0]
            for y in range(0,ni):
                if a[y]>max:
                    max = a[y]
            return max;
        ni=len(sni)
        sol = largest(sni,ni)
        wh.append(sol)
        i=i+1
    wh=np.asarray(wh)
    F=np.asarray(F)
    wh.astype(int)
    F.astype(int)
    w=np.delete(wh,np.argwhere(wh>=wh_max))
    F1=np.delete(F,np.argwhere(wh>=wh_max))
    w=np.delete(w,np.argwhere(wh<=wh_min))
    F=np.delete(F1,np.argwhere(wh<=wh_min))
    plt.style.use('seaborn-darkgrid')
    #plt.figure(figsize=())
    plt.plot(F,w,'r^')
    plt.xlabel("FoS")
    plt.ylabel(name)
    t = Table([F,w],names=('FoS',name))
    print(t)
    return t,F,w;
```

```python
Pillar_Strength(wh=2,formula=[PillarSt_Mark,PillarSt_CSIRO], name =["USA","AUS"])
```

    Pillar Strength for USA : 10.672600000000001
    Pillar Strength for AUS : 10.622399999999999

    [10.672600000000001, 10.622399999999999]

```python
FoS(P=9,wh=2,formula=[PillarSt_Mark,PillarSt_CSIRO], name =["USA","AUS"])
```

    Pillar Strength for USA : 10.672600000000001
    Pillar Strength for AUS : 10.622399999999999
    FoS for USA : 1.1858444444444445
    FoS for AUS : 1.1802666666666666

    ([1.1858444444444445, 1.1802666666666666],
     [10.672600000000001, 10.622399999999999],
     ['USA', 'AUS'])

```python
wh(P=9,FoS=1.2,formula=[PillarSt_Mark,PillarSt_CSIRO], name =["USA","AUS"])
```

    w/h ratio for USA : 2.03802190587042
    w/h ratio for AUS : 2.03336022215367

    (1.2, 10.799999999999999, ['USA', 'AUS'])

```python
FoS_Range(P=9,FoS_max=3,FoS_min= 0.5, interval= 0.1,formula=[PillarSt_Mark],name= "USA w/h ratio")
```

```python
a,b,c=FoS_wh_Range(P=9,FoS_max=3, FoS_min=0.5, interval=0.1,wh_max=6,wh_min=0.5,formula=[PillarSt_Mark],name= "USA w/h ratio")
```

```python
d,e,f=FoS_wh_Range(P=9,FoS_max=3, FoS_min=0.5, interval=0.1,wh_max=3,wh_min=0.5,formula=[PillarSt_CSIRO],name= "AUS w/h ratio")
```

```python
plt.plot(b,c,'r^',label='$Mark$')
plt.plot(e,f,'b^',label='$CSIRO$')
plt.xlabel('FoS')
plt.ylabel('w/h Ratio')
plt.legend()
plt.show()
```


# Knapsack Cover Inequalities

Consider the 0-1 knapsack Set

$K := \Big\{\{0, 1\}^{n} : \sum_{j = 1}^n w_j x_j \le b \Big\}, w_j > 0, \forall j \in N = \{1, 2, ..., n\}, b > 0$

**Definition (Cover)** A set $C \subseteq N$ is a *cover* if $\sum_{j \in C} w_j > b$. A cover is minimal if $\sum_{j \in C \backslash \{j\}} w_j \le b, \forall j \in C$.

**Prop** If $C \subseteq N$ is a cover for $K$, then $\sum_{j \in C} \le \vert C \vert - 1$ is valid for $K$.

The alternate reformulation of the Knapsack set is in terms of cover inequalities. In this tutorial, we will see how to generate a minimal cover, how to lift it to make it facet defining, and solve the resulting Knapsack problem.

```python
# First we import some packages
import numpy as np
from gurobipy import *
import time
```

### Generating problem instance

```python
def generateData(n): # n is the number of items
    np.random.seed(30) # Setting a seed value
    a = np.random.randint(1000, size=(n)) # Generating value of items
    w = np.random.randint(1000, size=(n))# Generating weight of items
    b = np.sum(w)*0.40 # Total weight of Knapsack
    return a, w, b
```

### Simply solving the Knapsack problem using Gurobi

```python
def solveKnapsackUsingGurobi(n, a, w, b, verbose = 1): # no. of items, value of items, weight of items, and Knapsack weight
    m = Model() # Preparing the model
    x = {i : m.addVar(lb=0, vtype=GRB.BINARY) for i in range(n)} # whether the item should be included in the Knapsack
    m.addConstr(sum([w[j] * x[j] for j in range(n)]) <= b) # Total weight should be less than the Knapsack capacity
    m.setObjective(sum([a[j] * x[j] for j in range(n)]), sense = GRB.MAXIMIZE) # Maximize the total value of the Knapsack
    m.Params.OutputFlag = 0; m.optimize()
    if m.status == 2:
        if verbose == 1:
            print('*********************** Gurobi ***************************')
            print('Items selected: ', {k for k in x if x[k].x > 0.4})
            print('Total value: ', m.objVal)
            print('**********************************************************')
        return {k for k in x if x[k].x > 0.4}, m.objVal
    else:
        print('Infeasible model')
        return {}, -float("inf")
```

### Let's start preparing the initial model

```python
def preparePartialModel(n, a, w, b):
    m = Model()
    x = {i : m.addVar(lb=0.0, ub = 1.0, vtype=GRB.CONTINUOUS, name = str(i)) for i in range(n)}
    m.addConstr(sum([w[j] * x[j] for j in range(n)]) <= b)
    c, zeta = generateCover(n, a, w, b, {k:0 for k in range(n)})

    m.addConstr(sum([x[j] for j in c]) <= len(c)-1)
    m.setObjective(sum([a[j] * x[j] for j in range(n)]), sense = GRB.MAXIMIZE)
    m.Params.OutputFlag = 0; m.Params.IntegralityFocus = 1
    return m

```

## Generating a cover/Seperation problem
To use cover inequalities in a cutting plane scheme, one is faced with the separation problem, that is, given a vector $\bar{x} \in \left[0, 1\right]^n$, find a cover inequality for $K$ that is violated by $\bar{x}$, or show that none exists. For this purpose, we solve the following problem:

$\zeta = \min\{\sum_{j \in C} (1-\bar{x}_j): C \text{ is a cover for } K\}$

- If $\zeta \ge 1$, then $\bar{x}$ satisfies all the cover inequalities for $K$.
- If $\zeta < 1$, then an optimal over to the above problem yields the violated minimal cover.

One can also write the above prblem as the following binary program. Let $z_j = 1$, if $j$ is in the violated cover $C$.

$$
	\begin{align}
	& \zeta = \underset{z}{\text{minimize}}
	& & \sum_{j=1}^n (1-\bar{x}_j) z_j \\
	& \text{subject to}
	& & \sum_{j= 1}^n w_j z_j \ge b + 1\\
	& & & z \in \{0,1\}^n
	\end{align}
$$

```python
def generateCover(n, a, w, b, xbar): # no. of items, value of items, weight of items, Knapsack weight, and xbar
     m1 = Model()
     z = {i:m1.addVar(lb=0.0, vtype=GRB.BINARY) for i in range(n)}
     m1.addConstr(sum([w[j] * z[j] for j in range(n)]) >= b + 1)
     m1.setObjective(sum([(1-xbar[j]) * z[j] for j in range(n)]), sense = GRB.MINIMIZE)
     m1.Params.OutputFlag = 0; m1.optimize()
     return [j for j in range(n) if z[j].x > 0.4], m1.objVal # return the cover and the value of zeta
```

## Lifting

This is esentially a procedure to convert a minimal cover inequality $\sum_{j \in C} x_j \le \vert C \vert - 1$ into a facet defining inequality for $K$.

### Sequential Lifting
- Choose an ordering $j_1, j_2, ..., j_l$ of the indices in $N \backslash C$. Set t = 1.
- The valid inequality $\sum_{i = 1}^{t-1} \alpha_{j_i} x_{j_i} + \sum_{j \in C} x_j \le \vert C \vert - 1$  has been obtained so far.
- To calculate the largest value of $\alpha_{j_t}$ for which the inequality
$\alpha_{j_t} x_t  + \sum_{i = 1}^{t-1} \alpha_{j_i} x_{j_i} + \sum_{j \in C} x_j \le \vert C \vert -1$ is valid, solve the following Knapsack problem:

$$
	\begin{align}
	& \xi_t = \underset{x}{\text{maximize}}
	& & \sum_{i = 1}^{t-1} \alpha_{j_i} x_{j_i} + \sum_{j \in C} x_j \\
	& \text{subject to}
	& & \sum_{i = 1}^{t-1} \alpha_{j_i} x_{j_i} + \sum_{j \in C} w_j x_j \le b - w_{j_t}\\
	& & & x \in \{0,1\}^{\vert C \vert + t - 1}
	\end{align}
$$
- Set  $\alpha_{j_t} = \vert C \vert - 1 - \xi_t$.
- Stop if $t = r$.

We remark that different orderings of $N \backslash C$ may produce different lifted inequalities. Furthermore, not all possible liftings can be derived from the above procedure.

```python
def sequentialLifting(c, n, a, w, b): # Cover set, no. of items, value of items, weight of items, and Knapsack weight.
    alpha = {k:0.0 for k in range(n) if k not in c} # coefficients of x_j, j not in cover
    m2 = Model()
    x_dash = {i:m2.addVar(lb=0.0, vtype=GRB.BINARY) for i in c}
    toBechecked = list(alpha.keys()); checked = [] # choosing the ordering of indices one-by-one and finding its coeffcient
    if len(toBechecked) != 0:
        i = toBechecked.pop(0);
        x_dash[i] = m2.addVar(lb=0.0, vtype=GRB.BINARY) # including a variable.
        m2.update()
        tmpc = sum([x_dash[j] * w[j] for j in c]) + sum([alpha[j]*x_dash[j] for j in checked]) # constraint of already in the Knapsack
        tmpo = sum([x_dash[j]  for j in c]) + sum([alpha[j]*x_dash[j] for j in checked])
        m2.setObjective(tmpo, sense = GRB.MAXIMIZE)
        m2.addConstr(tmpc <= b - w[i])
        m2.Params.OutputFlag = 0; m2.optimize()
        if len(c) - 1 - m2.objVal >= 1: # Checking if the current item can be included in the cover.
            alpha[i] = len(c) - 1 - m2.objVal
            checked.append(i)
    return alpha # return coefficients of x_j in the lifted inequality.

```

### Balas' lifiting of minimal cover inequalities
Let $C$ be a minimal cover for $K$, and let
$\sum_{j \in C} x_j + \sum_{j \in N \backslash C} w_j x_j \le \vert C \vert - 1$ be a lifting of the cover inequality associated with $C$. Up to permuting the indices, assume that C = \{1, ..., t\} and $a_1 \ge a_2 \ge ... \ge a_t$.

Let $\mu_0 =  0$, and $\mu_h = \sum_{l = 0}^h w_j,$ for $h = 1, ..., t$	. Let $\lambda = \mu_t - b$ (note that $\lambda > 0$).

If above inequality defines a facet of $conv(K)$, then the following hold for every $j \in N \backslash C$.
- If, for some $h$, $\mu_h ≤ w_j ≤ \mu_{h+1} − \lambda$, then $\alpha_j = h$.
- If, for some $h$, $\mu_{h+1} − \lambda < w_j < \mu_{h+1}$, then $h ≤ \alpha_j ≤ h + 1$ (i.e., select $\alpha_j = h+1$).

```python
def BalasLifting(c, n, a, w, b): # Cover set, no. of items, value of items, weight of items, and Knapsack weight.
    orderCoverWeightsList = sorted(c, key=lambda item: w[item], reverse =True)
    alpha = {k:0.0 for k in range(n) if k not in c} # coefficients of x_j, j not in cover
    mu = {0:0.0}
    ind = 0; cum = 0
    for k in orderCoverWeightsList:
        if ind == 0:
            mu[ind+1] = w[k]
            ind += 1; cum += w[k]
        else:
            mu[ind+1] = w[k] + cum
            ind += 1; cum += w[k]
    lamb = mu[len(mu)-1] - b

    for j in alpha:
        h = [h for h in mu if h < len(mu)-1 and mu[h] <= w[j] <= mu[h+1] - lamb]
        if len(h) != 0:
            alpha[j] = h[0]
        else:
            h = [h for h in mu if h < len(mu)-1 and mu[h] -lamb <= w[j] <= mu[h]]
            alpha[j] = h[-1]
    return alpha # return coefficients of x_j in the lifted inequality.
```

### Solving Knapsack problem using cover inequalities
Putting above peices together.

```python
def solveKnapsackUsingCoverInequalities(n, a, w, b, lifting = 'sequential', verbose = 1):
    m = preparePartialModel(n, a, w, b)
    m.optimize()
    if m.status == 2:
        x = {i:m.getVarByName(str(i)).x for i in range(n)}
        c, zeta = generateCover(n, a, w, b, x)
    else:
        print('Infeasible model')
    alpha = []

    while zeta < 1:
        m.addConstr(sum([m.getVarByName(str(i)) for i in c]) + sum([m.getVarByName(str(i)) * alpha[i] for i in alpha]) <= len(c) - 1)
        m.optimize()
        x = {i:m.getVarByName(str(i)).x for i in range(n)}
        c, zeta = generateCover(n, a, w, b, x)
        #print('Cover', sum([w[k] for k in c])-min([w[k] for k in c]) <= b)
        if lifting == 'sequential':
            alpha = sequentialLifting(c, n, a, w, b)
        elif lifting == 'Balas':
            alpha = BalasLifting(c, n, a, w, b)
        elif lifting == 'None':
            alpha = []
        else:
            print('lifting is unknown.. terminating')
            return 'NA'
        zeta = round(zeta, 4)
        #print(c, zeta, alpha)

    if verbose == 1:
        print('******************* Cover Inequality *********************')
        print('Items selected: ', {i:m.getVarByName(str(i)).x for i in range(n) if m.getVarByName(str(i)).x > 0.4} )
        print('Total value: ', sum([a[k] for k in range(n) if round(m.getVarByName(str(k)).x) > 0.4]))
        print('**********************************************************')
    return {i:m.getVarByName(str(i)).x for i in range(n) if m.getVarByName(str(i)).x > 0.4}, sum([a[k] for k in range(n) if round(m.getVarByName(str(k)).x) > 0.4]), len(m.getConstrs()) - 1

```

## Solve the models and print results

```python
import pandas as pd
n = 1000 # No. of items
results = []; results.append(['Total items', n])
a, w, b = generateData(n)
start = time.time()
xg, objg = solveKnapsackUsingGurobi(n, a, w, b, verbose = 0)
#print('Gurobi model took', time.time()-start); start = time.time()
timeF = time.time()-start
results.append(['No. of items selected', len(xg)])
results.append(['Value of items selected', objg])
results.append(['No. of covers generated', float("NAN")])
results.append(['Computational time', timeF])

for lift in ['None', 'sequential', 'Balas']:
    start = time.time()
    xc, objc, nc = solveKnapsackUsingCoverInequalities(n, a, w, b, lifting = lift, verbose = 0)
    timeF = time.time()-start
    #print('CG model took', time.time()- start)
    results[0].append(n)
    results[1].append(len(xc))
    results[2].append(objc)
    results[3].append(nc)
    results[4].append(timeF)

# Create the pandas DataFrame
pd.options.display.float_format = "{:,.2f}".format
df = pd.DataFrame(results, columns = ['Particulars', 'IP with Gurobi', 'Knapsack cover with no lifting', 'Knapsack cover with sequential lifting', 'Knapsack cover with Balas lifting'])
display(df)
```

    Academic license - for non-commercial use only - expires 2021-09-17
    Using license file C:\Users\Pramesh Kumar\gurobi.lic

      Particulars
      IP with Gurobi
      Knapsack cover with no lifting
      Knapsack cover with sequential lifting
      Knapsack cover with Balas lifting

      0
      Total items
      1,000.00
      1,000.00
      1,000.00
      1,000.00

      1
      No. of items selected
      561.00
      560.00
      560.00
      560.00

      2
      Value of items selected
      374,128.00
      374,076.00
      374,076.00
      374,076.00

      3
      No. of covers generated
      nan
      6.00
      6.00
      11.00

      4
      Computational time
      0.12
      0.42
      0.66
      5.07

Knapsack Cover Inequalities/Knapsack Cover Inequalities notebook.ipynb


```python
from Exercise import Exercise, MarkdownBlock
from config import URL, TOKEN
import json

import numpy as np
import sympy as sp
import matplotlib.pyplot as plt

import matplotlib as mpl
mpl.rcParams['figure.dpi'] = 100

Exercise.URL = URL
Exercise.TOKEN = TOKEN
```

```python
mu, sigma = 0, 1.5 # mean and standard deviation
s0 = np.random.normal(mu, sigma, 10)
mu, sigma = 3, 2 # mean and standard deviation
s1 = np.random.normal(mu, sigma, 10)
```

```python
plt.ylim(-0.1, 1.1)
plt.scatter(s0, y=[0 for i in range(len(s))], marker="+")
plt.scatter(s1, y=[0 for i in range(len(s))], marker="x")

from pylab import *
# Create some test data
dx = 0.01
X  = np.arange(-3, 9, dx)
# Y  = exp(-X ** 2)
def normal_dist(x , mean , sd):
    prob_density = (np.pi*sd) * np.exp(-0.5*((x-mean)/sd)**2)
    return prob_density
Y = normal_dist(X, 3, 2)

# Normalize the data to a proper PDF
Y /= (dx * Y).sum()

# Compute the CDF
CY = np.cumsum(Y * dx)

# Plot both
plot(X, Y)
plot(X, CY, 'r--')

# plt.ylim(0.1, 1)
```

```python
import numpy as np
import statsmodels.api as sm # recommended import according to the docs
import matplotlib.pyplot as plt

ecdf = sm.distributions.ECDF(s0)

x = np.linspace(min(s0), max(s0))
y = ecdf(x)
plt.step(x, y)
plt.show()
```

```python

```