# Logistic Regression

Most of the machine learning concepts in my recent posts have had convenient closed-form solutions for optimal weights. Most problems in machine learning don't!
For example, neural networks usually have to use stochastic gradient descent to optimize weights.

In this post, I walk through logistic regression (which can also be thought of as a one-neuron neural network!) Optimal weights in logistic regression requires an iterative optimizer.

See also:
 - [MLPR notes](http://www.inf.ed.ac.uk/teaching/courses/mlpr/2017/notes/w3c_logistic_regression.html)
 - Section 17.4 of Barber's Bayesian Reasoning and Machine Learning
 - Section 4.3.2 of Bishop's Pattern Recognition and Machine Learning

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import minimize
from scipy.optimize import check_grad
```

```python
# helper functions you can skip over :D
def hide_ticks(plot):
    plot.axes.get_xaxis().set_visible(False)
    plot.axes.get_yaxis().set_visible(False)

SAVE = True
def maybe_save_plot(filename):
    if SAVE:
        plt.tight_layout()
        plt.savefig('images/' + filename, bbox_inches="tight")
```

## Problem and data generation

Logistic regression can be used for classification. For example, if I have a bunch of observed data points that belong to class A and class B, I can pick a new point and ask which class the model thinks the point belongs to.

For this demo, I'll generate two groups of points that correspond to the two classes.

```python
N = 100
D = 2

def get_observations(N):
    assert N % 2 == 0, 'sorry, N should be even for the following code to work'

    center_A = [-1, -1]
    center_B = [1, 1]

    data_A = np.random.randn(N//2, D) + center_A
    data_B = np.random.randn(N//2, D) + center_B

    shuffle_indices = np.arange(N)
    np.random.shuffle(shuffle_indices)

    # stack the data and labels, and then shuffle it according to our shuffle indices
    data = np.vstack((data_A, data_B))[shuffle_indices]
    labels = np.hstack((np.ones(N//2), np.zeros(N//2)))[shuffle_indices]

    return data, labels

fuzzy_data, fuzzy_labels = get_observations(N)
```

```python
def plot_labeled_data(ax, data, labels):
    '''
    ax: matplotlib axis
    data: (N, D) 2D data points
    labels: (N,) label of 0 or 1
    '''
    ax.plot(*data[labels == 0].T, '.')
    ax.plot(*data[labels == 1].T, '.')

plt.figure(figsize=(6, 6))
plot_labeled_data(plt, fuzzy_data, fuzzy_labels)
plt.axis('square')
maybe_save_plot('2018-05-22-fuzzy')
plt.show()
```

Note: We could also learn this without iterative optimization using [Bayes classifiers](https://github.com/jessstringham/notebooks/blob/master/2018-05-04-ipywidgets-for-learning-logistic-sigmoid-and-bayes-classifiers.ipynb). But these require quadratically more parameters than logistic regression (see Bishop for more info).

## Logistic sigmoid

Logistic regression learns the weights of a logistic sigmoid function. The logistic sigmoid is given by:

$$\sigma(\textbf{w}^{\top}\textbf{x} + b) = \frac{1}{1 + \exp(-\textbf{w}^{\top}\textbf{x} + b)}$$

where $\textbf w$ and $b$ are the weights I want to learn and $\textbf{x}$ is the input data.
$\sigma$ ranges from 0 to 1. In the case of logistic regression, its value is treated as the probability that a point $\textbf{x}$ falls into one of the two classes.

I'll define a `sigmoid_with_bias` function I can use. (Heads up, I combine the bias with the rest of the weights.)

```python
def sigmoid_with_bias(bias_and_weights, x):
    '''
    bias_and_weights: (D + 1,) bias and array of weights in one array
    x: (N, D) data
    '''
    # use w[0] as the bias term, and the rest as weights
    bias = bias_and_weights[0]
    weights = bias_and_weights[1:]
    return 1/(1 + np.exp(-(weights.T @ x + bias)))
```

```python
xs = np.linspace(-5, 5, 100)
plt.plot(xs, sigmoid_with_bias(np.array([0, 1]), xs[None, :]))
plt.title('Logistic sigmoid')
maybe_save_plot('2018-05-22-sigmoid')
plt.show()
```

*(Oh hey, my [ipywidgets post](https://github.com/jessstringham/notebooks/blob/master/2018-05-04-ipywidgets-for-learning-logistic-sigmoid-and-bayes-classifiers.ipynb) has an interactive example of how the weights and bias influence the shape of the sigmoid.)*

### Sigmoid with two input variables

My data has two input variables, so I'd like a function that takes in two inputs and returns the probability that the point belongs to class A. The `sigmoid_with_bias` defined above can do this.

To demonstrate, I can plot sigmoid with two input dimensions (and a 3rd output dimension) using a contour plot.

Below I plot the contours for a $\sigma$ using $\textbf w = \begin{bmatrix}3/2 \\ -2\end{bmatrix}$ and $b = 2$. The graph shows a region where $\sigma$ is close to 1, another region where $\sigma$ is close to 0, and a sloping boundary in between. If I were to view the boundary from the side, I'd get something like the sigmoid shape shown above.

If a data point is in the area on this slope, I'd say things like "I'm 80% sure that this point is in class B". This is useful for where the data from the two classes overlap, such as the example data.

```python
MIN_GRID_VAL = -5
MAX_GRID_VAL = 5

xgrid = np.linspace(MIN_GRID_VAL, MAX_GRID_VAL, 100)
ygrid = xgrid

def plot_sigmoid(ax, w, data, labels):
    '''
    ax: matplotlib axis
    w: (D,) numpy array of weights for the sigmoid, where first value is bias term
    data: (N, D - 1) data points
    labels: (N,) labels 1s and 0s
    '''
    X, Y = np.meshgrid(xgrid, ygrid)

    # I'm not sure if there's a better way to do this. I'm flattening the grid down, computing
    # the results, and reshaping it back into the form wanted by `contour`
    flattened_x_y = np.dstack((X, Y)).reshape(-1, 2).T
    sigmoid_vals = sigmoid_with_bias(w, flattened_x_y).reshape(xgrid.shape[0], ygrid.shape[0])

    CS = ax.contour(X, Y, sigmoid_vals, colors='k', levels=np.arange(0, 1, 0.125))
    ax.clabel(CS, inline=1, fontsize=13)

    return sigmoid_vals
```

```python
w = np.hstack([2, 3/2, -2])
data = fuzzy_data
labels = fuzzy_labels

plt.figure(figsize=(6, 6))

sigmoid_vals = plot_sigmoid(plt, w, data, labels)

plt.imshow(
    sigmoid_vals,
    interpolation='bilinear',
    origin='lower',
    extent=(MIN_GRID_VAL, MAX_GRID_VAL, MIN_GRID_VAL, MAX_GRID_VAL)
)

plt.axis('square')
maybe_save_plot('2018-05-22-2d-sigmoid')
plt.show()
```

If I drop this sigmoid on the Fuzzy Data, it looks like this $\textbf w$ and $b$ make a terrible boundary.

```python
plt.clf()
plt.figure(figsize=(6, 6))
plot_sigmoid(plt, w, fuzzy_data, fuzzy_labels)
plot_labeled_data(plt, fuzzy_data, fuzzy_labels)
plt.axis('square')
maybe_save_plot('2018-05-22-bad-fit')
plt.show()
```

## Optimizers

Later I'll need to use an optimizer to find the best weights, so here I'll show how `minimize` works. I'll try to minimize the function $x^2 - 2x$.
I also need to provide the gradient of $f$. Because that can be tricky to get right, I'll run `check_grad`, a method in `scipy` that numerically checks the gradient. I'll also need an initial weight for the minimizer to start at.

When I run this, the minimum is in `example_optimizer_result.x`, and is $x = 1$.

```python
# Check gradients at an arbitrary spot!
# `check_grad` returns the difference, which is close to 0.
# Heads up, I'm not sure how close to 0 to expect this to be, so I chose an arbitrary value that worked.
# Look into this more before using it for reals.
def assert_grad_is_close(f, grad_f, x0):
    assert np.isclose(check_grad(f, grad_f, x0=x0), 0, atol=1e-2), 'grads are diff: {}'.format(check_grad(f, grad_f, x0=x0))

def f(x):
    return x**2 - 2 * x

def grad_f(x):
    return 2 * x - 2

assert_grad_is_close(f, grad_f, np.array([3]))

example_optimizer_result = minimize(f, x0=3, jac=grad_f, method='BFGS')
```

```python
plt.plot(xgrid, f(xgrid), label='loss function')
plt.plot(example_optimizer_result.x, f(example_optimizer_result.x), 'Xk', label='minimum')
plt.title('loss over values of x')
plt.xlabel('x')
plt.ylabel('loss')
plt.legend()
maybe_save_plot('2018-05-22-optimizer')
plt.show()
```

## Maximum Likelihood of Logistic Regression

One way to learn a better value of $\textbf w$ and $b$ given the labeled observations is by using maxmimum likelihood.  I'll use the equations from Chapter 17 of [Barber](http://www0.cs.ucl.ac.uk/staff/d.barber/brml/).

(Heads up, I moved the bias $b$ into weights $\textbf w$ by adding a column of 1's to $\textbf x$.)

To find the $\textbf w$ using maximum likelihood, I find $\textbf w^*$ so that

$$\textbf w^* = \mathrm{argmax}_{\textbf w} \log P(\mathcal{D} \mid \textbf w)$$

where $\mathcal{D}$ is the observed data. Using $\textbf x_n$ as an input and $y_n$ as a label from an observation, the log likelihood is given by the sum over all observations of the probabilities, or

\begin{align}
\log P(\mathcal{D} \mid \textbf w) &= \sum_n \log P(\textbf x_n, y_n \mid \textbf w) . \\
\end{align}

For this logistic regression set-up, $\log P(\textbf x_n, y_n \mid \textbf w)$ becomes

$$\log P(\textbf x_n, y_n \mid \textbf w) = y_n\log\sigma(\textbf w^\top \textbf x_n) + (1 - y_n)\log(1 - \sigma(\textbf w^\top \textbf x_n)).$$

The goal is to find the value of $\textbf w$ that maximizes $\log P(\mathcal{D} \mid \textbf w)$.
There is no closed-form solution, so I'll use an iterative optimizer. As above, iterative optimizers sometimes require the gradient with respect to the weights, which for this logistic regression set-up is given by

$$\nabla_{\textbf w} \log P(\mathcal D \mid \textbf w) = \sum_n(y_n - \sigma(\textbf w^\top \textbf x_n))\textbf  x_n. $$

While there isn't a closed-form solution, it turns out that $\log P(\mathcal{D} \mid \textbf w)$ does have a single maximum, so I don't need to worry about local maxima. This makes logistic regression different than other models where multiple runs gives different results.

Finally, I'll add a regularizer to keep the weights reasonable. The equations with regularization are given by

$$\textbf w^* = \mathrm{argmax}_{\textbf w} \log P(\mathcal{D} \mid \textbf w) + \lambda \textbf w^{\top} \textbf w$$

$$\nabla_{\textbf w} \left(\log P(\mathcal{D} \mid \textbf w) + \lambda \textbf w^{\top} \textbf w\right) = \sum_n(y_n - \sigma(\textbf w^\top \textbf x_n))\textbf  x_n + 2 \lambda \mathbf{w}. $$

I return to why regularization is important below.

### Coding up equations

Now I translate the equations for $\log P(\mathcal{D} \mid \textbf w)$ and $\nabla_{\textbf w} \log P(\mathcal D \mid \textbf w)$ and the regularization terms into code.

In general:
 - $\log P(\mathcal{D} \mid \textbf w)$ should return a scalar
 - $\nabla_{\textbf w} \log P(\mathcal D \mid \textbf w)$ should return a vector the same size as $\textbf w$.

There are a few catches:
 - The optimizer I'm using, `minimize` , is a minimizer, so I'll actually minimize the _negative_ log likelihood.
 - $\log$ overflows when the sigmoid start returning values that round to 0. There might be better ways to solve this, but I avoid this by adding a tiny offset.
 - Because of how I'm dealing with the bias term, I have to explicitly make an $\textbf x$ with a column of ones here called `data_with_bias`.
 - I'm returning the loss function as well so I can plot it. Eh.

```python
def fit_logistic_regression(data, labels, regularization_constant, initial_w):
    epsilon = 1e-9

    data_with_bias = np.hstack((
        np.ones((data.shape[0], 1)),
        data,
    ))

    def loss(w):
        return -np.sum(
            labels * np.log(sigmoid_with_bias(w, data.T) + epsilon)
            + (1 - labels) * np.log(1 - sigmoid_with_bias(w, data.T) + epsilon)
        ) + regularization_constant * (w.T @ w)

    def grad_loss(w):
        return -(labels - sigmoid_with_bias(w, data.T)) @ data_with_bias + 2 * regularization_constant * w

    assert_grad_is_close(loss, grad_loss, w)

    opt_result = minimize(loss, x0=w, method='BFGS', jac=grad_loss)

    assert opt_result.success, opt_result

    return opt_result, loss
```

## Optimizing for logistic regression

Now I optimize those functions and plot the results.

```python
log_reg_optimizer_result, loss_func = fit_logistic_regression(
    fuzzy_data,
    fuzzy_labels,
    regularization_constant=0.1,
    initial_w=w,
)

print(log_reg_optimizer_result)
```

          fun: 22.596598925923256
     hess_inv: array([[ 0.16123441, -0.00642625,  0.06831963],
           [-0.00642625,  0.15614122, -0.00025405],
           [ 0.06831963, -0.00025405,  0.22012547]])
          jac: array([  3.01092133e-07,  -2.53026262e-07,  -6.82767868e-07])
      message: 'Optimization terminated successfully.'
         nfev: 16
          nit: 13
         njev: 16
       status: 0
      success: True
            x: array([-0.29020881, -1.35188296, -1.78343628])

          fun: 15.919840119847468
     hess_inv: array([[ 0.17848323, -0.05018526,  0.00905405],
           [-0.05018526,  0.26530471,  0.17622941],
           [ 0.00905405,  0.17622941,  0.44592668]])
          jac: array([ -2.61269672e-06,   4.82700074e-06,  -2.26616311e-06])
      message: 'Optimization terminated successfully.'
         nfev: 19
          nit: 16
         njev: 19
       status: 0
      success: True
            x: array([ 0.1293201 , -2.03361611, -2.73868849])

### Plotting the loss function

To visualize what the minimizer found, I'll plot the loss function, or the negative log likelihood with regularization, and the optimal $w$ found. While the real loss surface has three input dimensions, one for the bias and two for the weights, for visualization, I'll just vary one of the weights.

```python
w0, w1, w2 = log_reg_optimizer_result.x
loss_xgrid = np.linspace(w1 - 2, w1 + 2, 100)

plt.figure(figsize=(8, 6))

# Plot the loss function
loss_over_w1 = np.hstack((
    loss_func(np.array([ w0, i, w2]))
    for i in loss_xgrid
))

plt.plot(loss_xgrid, loss_over_w1, label='loss function')
plt.plot(w1, loss_func(log_reg_optimizer_result.x), 'Xk', label='minimum')
plt.title('loss over values of w1')
plt.xlabel('w1')
plt.ylabel('loss')
plt.legend()
plt.show()
```

### Logistic sigmoid on the data with fitted weights

Plotting the logistic sigmoid with the fitted weights looks much better!

```python
plt.figure(figsize=(8, 6))
plot_sigmoid(plt, log_reg_optimizer_result.x, fuzzy_data, fuzzy_labels)
plot_labeled_data(plt, fuzzy_data, fuzzy_labels)
plt.axis('square')
maybe_save_plot('2018-05-22-fitted-weights')
plt.show()
```

## Regularization

Regularization is important in logistic regression. One problem with logistic regression is that if data is linearly separable, the boundary becomes super steep.

I'll generate the data points, run logistic regression without regularization on it, and plot the boundary.

```python
def get_lin_sep_observations(N):
    assert N % 2 == 0, 'sorry, N should be even for the following code to work'

    center_A = [-4, -4]
    center_B = [0, 0]

    data_A = 4 * np.random.rand(N//2, D) + center_A
    data_B = 4 * np.random.rand(N//2, D) + center_B

    shuffle_indices = np.arange(N)
    np.random.shuffle(shuffle_indices)

    # stack the data and labels, and then shuffle it according to our shuffle indices
    data = np.vstack((data_A, data_B))[shuffle_indices]
    labels = np.hstack((np.ones(N//2), np.zeros(N//2)))[shuffle_indices]

    return data, labels

lin_sep_data, lin_sep_labels = get_lin_sep_observations(N)
```

```python
plt.figure(figsize=(6, 6))

plot_labeled_data(plt, lin_sep_data, lin_sep_labels)
plt.axis('square')
maybe_save_plot('2018-05-22-lin-sep')
plt.show()
```

```python
no_reg_optimizer_result, _ = fit_logistic_regression(
    lin_sep_data,
    lin_sep_labels,
    regularization_constant=0,  # setting the regularization constant as 0 is the same as no regularization
    initial_w=w,
)

reg_optimizer_result, _ = fit_logistic_regression(
    lin_sep_data,
    lin_sep_labels,
    regularization_constant=0.1,
    initial_w=w,
)

very_reg_optimizer_result, _ = fit_logistic_regression(
    lin_sep_data,
    lin_sep_labels,
    regularization_constant=10,
    initial_w=w,
)
```

The main thing I want to show is that with no regularization constant, the boundary will become as steep as it can until the optimizer gives up. I also show the effect of a higher regularization constant.

```python
fig, axs = plt.subplots(1, 3, figsize=(18, 6), sharex=True, sharey=True)

axs[0].axis([-5, 5, -5, 5])

plot_sigmoid(axs[0], no_reg_optimizer_result.x, lin_sep_data, lin_sep_labels)
plot_labeled_data(axs[0], lin_sep_data, lin_sep_labels)
axs[0].set_title('No regularization')

plot_sigmoid(axs[1], reg_optimizer_result.x, lin_sep_data, lin_sep_labels)
plot_labeled_data(axs[1], lin_sep_data, lin_sep_labels)
axs[1].set_title('With regularization, lambda = 0.1')

plot_sigmoid(axs[2], very_reg_optimizer_result.x, lin_sep_data, lin_sep_labels)
plot_labeled_data(axs[2], lin_sep_data, lin_sep_labels)
axs[2].set_title('With regularization, lambda = 10')

maybe_save_plot('2018-05-22-reg-vs-no')
plt.show()
```

```python

```


# The Implicit Backward Time Centered Space (BTCS) Difference Equation for the Heat Equation
## The Heat Equation
The Heat Equation is the first order in time ($t$) and second order in space ($x$) Partial Differential Equation:
$$  \frac{\partial u}{\partial t} = \frac{\partial^2 u}{\partial x^2},$$
The equation describes heat transfer on a domain
$$ \Omega = \{ t \geq, 0\leq x \leq 1\}. $$
with an initial condition at time $t=0$ for all $x$ and boundary condition on the left ($x=0$) and right side ($x=1$).

## Backward Time Centered Space (BTCS) Difference method
This note book will illustrate the Backward Time Centered Space (BTCS) Difference method for the Heat Equation with the __initial conditions__
$$ u(x,0)=2x, \ \ 0 \leq x \leq \frac{1}{2}, $$
$$ u(x,0)=2(1-x), \ \ \frac{1}{2}  \leq x \leq 1, $$
and __boundary condition__
$$ u(0,t)=0,  u(1,t)=0. $$

```python
# LIBRARY
# vector manipulation
import numpy as np
# math functions
import math

# THIS IS FOR PLOTTING
%matplotlib inline
import matplotlib.pyplot as plt # side-stepping mpl backend
import warnings
warnings.filterwarnings("ignore")
from IPython.display import HTML

HTML('''
<form action="javascript:code_toggle()"><input type="submit" value="Click here to toggle on/off the raw code."></form>''')

```

<form action="javascript:code_toggle()"><input type="submit" value="Click here to toggle on/off the raw code."></form>

## Discete Grid
The region $\Omega$ is discretised into a uniform mesh $\Omega_h$. In the space $x$ direction into $N$ steps giving a stepsize of
$$ h=\frac{1-0}{N},$$
resulting in
$$x[i]=0+ih,$$
and into $N_t$ steps in the time $t$ direction giving a stepsize of
$$ k=\frac{1-0}{N}$$
resulting in
$$t[i]=0+ik.$$
The Figure below shows the discrete grid points for $N=10$ and $N_t=15$, the red dots are the unknown values, the green dots are the known boundary conditions and the blue dots are the known initial conditions of the Heat Equation.

```python
N=10
Nt=200
h=1/N
k=1/Nt
r=k/(h*h)
time_steps=15
time=np.arange(0,(time_steps+.5)*k,k)
x=np.arange(0,1.0001,h)
X, Y = np.meshgrid(x, time)
fig = plt.figure()
plt.plot(X,Y,'ro');
plt.plot(x,0*x,'bo',label='Initial Condition');
plt.plot(np.ones(time_steps+1),time,'go',label='Boundary Condition');
plt.plot(x,0*x,'bo');
plt.plot(0*time,time,'go');
plt.xlim((-0.02,1.02))
plt.xlabel('x')
plt.ylabel('time (ms)')
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
plt.title(r'Discrete Grid $\Omega_h,$ h= %s, k=%s'%(h,k),fontsize=24,y=1.08)
plt.show();
```

## Discrete Initial and Boundary Conditions

The discrete initial conditions are
$$ w[i,0]=2x[i], \ \ 0 \leq x[i] \leq \frac{1}{2} $$
$$ w[i,0]=2(1-x[i]), \ \ \frac{1}{2}  \leq x[i] \leq 1 $$
and the discete boundary conditions are
$$ w[0,j]=0,  w[10,j]=0, $$
where $w[i,j]$ is the numerical approximation of $U(x[i],t[j])$.

The Figure below plots the inital and boundary conditions for $t[0]=0.$

```python
w=np.zeros((N+1,time_steps+1))
b=np.zeros(N-1)
# Initial Condition
for i in range (1,N):
    w[i,0]=2*x[i]
    if x[i]>0.5:
        w[i,0]=2*(1-x[i])

# Boundary Condition
for k in range (0,time_steps):
    w[0,k]=0
    w[N,k]=0

fig = plt.figure(figsize=(8,4))
plt.plot(x,w[:,0],'o:',label='Initial Condition')
plt.plot(x[[0,N]],w[[0,N],0],'go',label='Boundary Condition t[0]=0')
#plt.plot(x[N],w[N,0],'go')
plt.xlim([-0.1,1.1])
plt.ylim([-0.1,1.1])
plt.title('Intitial and Boundary Condition',fontsize=24)
plt.xlabel('x')
plt.ylabel('w')
plt.legend(loc='best')
plt.show()
```

## The Implicit Backward Time Centered Space (BTCS) Difference Equation
The implicit Backward Time Centered Space (BTCS) difference equation of the Heat Equation is
\begin{equation*}
\frac{w_{ij+1}-w_{ij}}{k}=\frac{w_{i+1j+1}-2w_{ij+1}+w_{i-1j+1}}{h^2}
\end{equation*}
Rearranging the equation we get
\begin{equation}
-rw_{i-1j+1}+(1+2r)w_{ij+1}-rw_{i+1j+1}=w_{ij}
\end{equation}
for $i=1,...9$ where $r=\frac{k}{h^2}$.

This gives the formula for the unknown term $w_{ij+1}$ at the $(ij+1)$ mesh points
in terms of $x[i]$ along the jth time row.

Hence we can calculate the unknown pivotal values of $w$ along the first row of $j=1$ in terms of the known boundary conditions.

This can be written in matrix form
$$ A\mathbf{w}_{j+1}=\mathbf{w}_{j} +\mathbf{b}_{j+1} $$
for which $A$ is a $9\times9$ matrix:
$$
\left(\begin{array}{cccc cccc}
1+2r&-r& 0&0&0 &0&0&0\\
-r&1+2r&-r&0&0&0 &0&0&0\\
0&-r&1+2r &-r&0&0& 0&0&0\\
0&0&-r&1+2r &-r&0&0& 0&0\\
0&0&0&-r&1+2r &-r&0&0& 0\\
0&0&0&0&-r&1+2r &-r&0&0\\
0&0&0&0&0&-r&1+2r &-r&0\\
0&0&0&0&0&0&-r&1+2r&-r\\
0&0&0&0&0&0&0&-r&1+2r\\
\end{array}\right)
\left(\begin{array}{c}
w_{1j+1}\\
w_{2j+1}\\
w_{3j+1}\\
w_{4j+1}\\
w_{5j+1}\\
w_{6j+1}\\
w_{7j+1}\\
w_{8j+1}\\
w_{9j+1}\\
\end{array}\right)=
\left(\begin{array}{c}
w_{1j}\\
w_{2j}\\
w_{3j}\\
w_{4j}\\
w_{5j}\\
w_{6j}\\
w_{7j}\\
w_{8j}\\
w_{9j}\\
\end{array}\right)+
\left(\begin{array}{c}
rw_{0j+1}\\
0\\
0\\
0\\
0\\
0\\
0\\
0\\
rw_{10j+1}\\
\end{array}\right).
$$
It is assumed that the boundary values $w_{0j}$ and $w_{10j}$ are known for $j=1,2,...$, and $w_{i0}$ for $i=0,...,10$ is the initial condition.

The Figure below shows the matrix $A$ and its inverse $A^{-1}$ in color plot form for $r$.

```python
A=np.zeros((N-1,N-1))
for i in range (0,N-1):
    A[i,i]=1+2*r

for i in range (0,N-2):
    A[i+1,i]=-r
    A[i,i+1]=-r

Ainv=np.linalg.inv(A)
fig = plt.figure(figsize=(12,4));
plt.subplot(121)
plt.imshow(A,interpolation='none');
plt.xticks(np.arange(N-1), np.arange(1,N-0.9,1));
plt.yticks(np.arange(N-1), np.arange(1,N-0.9,1));
clb=plt.colorbar();
clb.set_label('Matrix elements values');
clb.set_clim((-1,1));
plt.title('Matrix A r=%s'%(np.round(r,3)),fontsize=24)

plt.subplot(122)
plt.imshow(Ainv,interpolation='none');
plt.xticks(np.arange(N-1), np.arange(1,N-0.9,1));
plt.yticks(np.arange(N-1), np.arange(1,N-0.9,1));
clb=plt.colorbar();
clb.set_label('Matrix elements values');
clb.set_clim((-1,1));
plt.title(r'Matrix $A^{-1}$ r=%s'%(np.round(r,3)),fontsize=24)

fig.tight_layout()
plt.show();
```

## Results
To numerically approximate the solution at $t[1]$ the matrix equation becomes
$$ \mathbf{w}_{1}=A\mathbf{w}_{0} +\mathbf{b}_{0} $$
where all the right hand side is known.
To approximate solution at time $t[2]$ we use the matrix equation
$$ \mathbf{w}_{2}=A\mathbf{w}_{1} +\mathbf{b}_{1}. $$
Each set of numerical solutions $w[i,j]$ for all $i$ at the previous time step is used to approximate the solution $w[i,j+1]$.
The left and right plot below show the numerical approximation $w[i,j]$ of the Heat Equation using the BTCS method for $x[i]$ for $i=0,...,10$ and time steps $t[j]$ for $j=1,...,15$.

```python
fig = plt.figure(figsize=(12,6))

plt.subplot(121)
for j in range (1,time_steps+1):
    w[1:(N),j]=np.dot(Ainv,w[1:(N),j-1])
    plt.plot(x,w[:,j],'o:',label='t[%s]=%s'%(j,time[j]))
plt.xlabel('x')
plt.ylabel('w')
#plt.legend(loc='bottom', bbox_to_anchor=(0.5, -0.1))
plt.legend(bbox_to_anchor=(-.4, 1), loc=2, borderaxespad=0.)

plt.subplot(122)
plt.imshow(w.transpose())
plt.xticks(np.arange(len(x)), x)
plt.yticks(np.arange(len(time)), time)
plt.xlabel('x')
plt.ylabel('time')
clb=plt.colorbar()
clb.set_label('Temperature (w)')
plt.suptitle('Numerical Solution of the  Heat Equation r=%s'%(np.round(r,3)),fontsize=24,y=1.08)
fig.tight_layout()
plt.show()
```

```python

```

Chapter 08 - Heat Equations/.ipynb_checkpoints/03_Heat Equation-Crank_Nicolson-checkpoint.ipynb


## Лекция 6

## Методы спуска (Descent methods)
## Пять способов получить градиентный спуск

## На прошлом занятии...
1. Условия оптимальности
2. Введение в теорию двойственности

## Что такое методы спуска?

Последовательность $x_k$ генерируется по правилу

$$
x_{k+1} = x_k + \alpha_k h_k
$$

так что

$$
f(x_{k+1}) < f(x_k)
$$

Направление $h_k$ называется *направлением убывания*.

**Замечание**: существуют методы,

которые не требуют
монотонного убывания функции

от итерации к итерации.

```python
def DescentMethod(f, x0, epsilon, **kwargs):

    x = x0

    while StopCriterion(x, f, **kwargs) > epsilon:

        h = ComputeDescentDirection(x, f, **kwargs)

        alpha = SelectStepSize(x, h, f, **kwargs)

        x = x + alpha * h

    return x

```

## Способ 1: направление убывания
Рассмотрим линейную аппроксимацию дифференцируемой функции $f$ вдоль некоторого направления убывания $h, \|h\|_2 = 1$:

$$
f(x + \alpha h) = f(x) + \alpha \langle f'(x), h \rangle + o(\alpha)
$$

Из условия убывания

$$
f(x) + \alpha \langle f'(x), h \rangle + o(\alpha) < f(x)
$$

и переходя к пределу при $\alpha \rightarrow 0$:

$$
\langle f'(x), h \rangle \leq 0
$$

Также из неравенства Коши-Буняковского-Шварца

$$
\langle f'(x), h \rangle \geq -\| f'(x) \|_2 \| h \|_2 = -\| f'(x) \|_2
$$

Таким образом, направление антиградиента
$$
h = -\dfrac{f'(x)}{\|f'(x)\|_2}
$$
даёт направление **наискорейшего локального** убывания функции$~f$.

В итоге метод имеет вид
$$
x_{k+1} = x_k - \alpha f'(x_k)
$$

## Способ 2: схема Эйлера решения ОДУ

Рассмотрим обыкновенное диференциальное уравнение вида:

$$
\frac{dx}{dt} = -f'(x(t))
$$

и дискретизуем его на равномерной сетке с шагом $\alpha$:
$$
\frac{x_{k+1} - x_k}{\alpha} = -f'(x_k),
$$
где $x_k \equiv x(t_k)$ и $\alpha = t_{k+1} - t_k$ - шаг сетки.

Отсюда получаем выражение для $x_{k+1}$
$$
x_{k+1} = x_k - \alpha f'(x_k),
$$
которое в точности совпадает с выражением

для градиентного спуска.

Такая схема называется явной или прямой схемой Эйлера.

**Вопрос:** какая схема называется неявной или обратной?

## Способ 3: манипуляции с необходимым условием

\begin{align*}
& f'(x) = 0\\
& -\alpha f'(x) = 0\\
& x - \alpha f'(x) = x\\
& x_k - \alpha f'(x_k) = x_{k+1}
\end{align*}
Это **HE** доказательство корректности, только формальное

получение выражения!

## Способ 4: минимизация квадратичной оценки сверху
#### (А. В. Гасников Метод универсального градиентного спуска https://arxiv.org/abs/1711.00394)

Глобальная оценка сверху на функцию $f$ в точке $x_k$:
$$
f(y) \leq f(x_k) + \langle f'(x_k), y - x_k \rangle + \frac{L}{2} \|y - x_k \|_2^2 = g(y),
$$
где $\lambda_{\max}(f''(x)) \leq L$ для всех допустимых $x$.

Справа — квадратичная форма, точка минимума которой имеет

аналитическое выражение:

\begin{align*}
& g'(y^*) = 0 \\
& f'(x_k) + L (y^* - x_k) = 0 \\
& y^* = x_k - \frac{1}{L}f'(x_k) = x_{k+1}
\end{align*}

Этот способ позволяет оценить значение шага как $\frac{1}{L}$. Однако часто константа $L$ неизвестна.

## Способ 5: концепция доверительных областей

Минимизация линейной аппроксимации в шаре радиуса $r$ с центром в точке $x_k$
\begin{align}
& \min \; f(x_k) + \langle f'(x_k), x - x_k \rangle\\
\text{s.t. } & \| x - x_k \|^2_2 \leq r^2
\end{align}

Используя метод множителей Лагранжа и условия ККТ, получим
$$
x_{k+1} = x_k - \frac{1}{2\lambda} f'(x_k),
$$
где $\lambda$ — множитель Лагранжа, и
$$
\lambda = \frac{\| f'(x_k)\|_2}{2r}
$$

**Вопрос:** как выбрать $r$?

**Вопрос:** какое соотношение между константой Липшица $L$ градиента $f'(x_k)$ из способа 4 и $\lambda$?

## Итого: метод градиентного спуска — дёшево и сердито
```python
def GradientDescentMethod(f, x0, epsilon, **kwargs):

    x = x0

    while StopCriterion(x, f, **kwargs) > epsilon:

        h = ComputeGradient(x, f, **kwargs)

        alpha = SelectStepSize(x, h, f, **kwargs)

        x = x - alpha * h

    return x

```

## Как выбрать шаг $\alpha_k$? (J. Nocedal, S. Wright Numerical Optimization, $\S$ 3.1.)

Список подходов:
- Постоянный шаг
$$
\alpha_k = \overline{\alpha}
$$
- Априорно заданная последовательность, например
$$
\alpha_k = \dfrac{\overline{\alpha}}{\sqrt{k+1}}
$$
- Наискорейший спуск
$$
\alpha_k = \arg\min_{\alpha \geq 0} f(x_k - \alpha f'(x_k))
$$
- Требование **достаточного** убывания, требование **существенного** убывания и условие кривизны: для некоторых $\beta_1, \beta_2$, таких что $0 < \beta_1 < \beta_2 < 1$ найти $x_{k+1}$ такую что

    - Достаточное убывание: $f(x_{k+1}) \leq f(x_k) - \beta_1 \alpha_k \langle f'(x_k), h_k \rangle$ или
    $ f(x_k) - f(x_{k+1}) \geq \beta_1 \alpha_k \langle f'(x_k), h_k \rangle
    $
    - Существенное убывание: $f(x_{k+1}) \geq f(x_k) - \beta_2 \alpha_k \langle f'(x_k), h_k \rangle$ или
    $
    f(x_k) - f(x_{k+1}) \leq \beta_2 \alpha_k \langle f'(x_k), h_k \rangle
    $
    - Условие кривизны: $\langle f'(x_{k+1}), h_k \rangle \geq \beta_2 \langle f'(x_k), h_k \rangle$

Обычно коэффициенты выбирают так: $\beta_1 \in (0, 0.3)$, а $\beta_2 \in (0.9, 1)$

### Анализ и мотивация подходов к выбору шага $\alpha_k$
- Постоянный шаг: самое простое и неэффективное решение
- Априорно заданная последовательность: немногим лучше постоянного шага
- Наискорейший спуск: самое лучшее решение, но применимо только если вспомогательная задача решается аналитически или ооооооочень быстро.

То есть почти всегда неприменимо :)
- Требование достаточного убывания, требование существенного убывания и условие кривизны:
    - требование достаточного убывания гарантирует, что функция в точке $x_{k+1}$ не превосходит линейной аппроксимации с коэффициентом наклона $\beta_1$
    - требование существенного убывания гарантирует, что функция в точке $x_{k+1}$ убывает не меньше, чем линейная аппроксимация c коэффициентом наклона $\beta_2$
    - условие кривизны гарантирует, что угол наклона касательной в точке $x_{k+1}$ не меньше, чем угол наклона касательной в точке $x_k$,

умноженный на $\beta_2$

Требование существенного убывания и условие кривизны

обеспечивают убывание функции по выбранному направлению $h_k$.

Обычно выбирают одно из них.

#### Альтернативные названия
- Требование достаточного убывания $\equiv$ правило Армихо
- Требование достаточного убывания + условие кривизны $\equiv$ правило Вольфа
- Требование достаточного убывания + требование существенного убывания $\equiv$ правило Гольдштейна

## Зачем нужно условие существенного убывания?

```python
%matplotlib notebook
import matplotlib.pyplot as plt
import ipywidgets as ipywidg
import numpy as np

def gd(f, gradf, x0, step, eps=1e-6, maxiter=10):
    x = x0
    x_hist = [x]
    n_iter = 0
    while abs(gradf(x)) > eps and n_iter < maxiter:
        x = x - step * gradf(x)
        x_hist.append(x)
        n_iter += 1
    return x, x_hist

f = lambda x: x**2
gradf = lambda x: 2 * x

fig = plt.figure()
ax = fig.add_subplot(1, 1, 1)
plt.rc("text", usetex=True)

def update(x0, step):
    _, x_hist = gd(f, gradf, x0, step)
    x = np.linspace(-np.max(x_hist) - 1, np.max(x_hist) + 1)
    ax.clear()
    ax.plot(x, f(x), color="r", label="$f(x) = x^2$")
    y_hist = np.array([f(x) for x in x_hist])
    x_hist = np.array(x_hist)
    plt.quiver(x_hist[:-1], y_hist[:-1], x_hist[1:]-x_hist[:-1], y_hist[1:]-y_hist[:-1],
               scale_units='xy', angles='xy', scale=1, width=0.005, color="green", label="Descent path")
    ax.legend()
    fig.canvas.draw()

step_slider = ipywidg.FloatSlider(value=0.8, min=0, max=1.2, step=0.1, description="Step")
x0_slider = ipywidg.FloatSlider(value=1.5, min=-4, max=4, step=0.1, description="Initial point")
_ = ipywidg.interact(update, x0=x0_slider, step=step_slider)
```

    <IPython.core.display.Javascript object>

Failed to display Jupyter Widget of type interactive.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

```python
def plot_alpha(f, x, h, alphas):
    df = np.zeros_like(alphas)
    for i, alpha in enumerate(alphas):
        df[i] = f(x + alpha * h)
    plt.plot(alphas, df)
    plt.xlabel(r"$\alpha$", fontsize=18)
    plt.ylabel(r"$f(x + \alpha h)$", fontsize=18)
```

```python
plot_alpha(lambda x: x**2, 0.5, -1, np.linspace(1e-3, 1.01, 10))
```

    <IPython.core.display.Javascript object>

### Backtracking

```python
def SelectStepSize(x, f, h, rho, alpha0, beta1, beta2):

    # 0 < rho < 1

    # alpha0 - initial guess of step size

    # beta1 and beta2 - constants from conditions

    alpha = alpha0

    # Check violating sufficient decrease and curvature conditions

    while (f(x - alpha * h) >= f(x) +

           beta1 * alpha grad_f(x_k).dot(h)) and

          (grad_f(x - alpha * h).dot(h) <= beta2 *

           grad_f(x_k).dot(h)):

        alpha *= rho

    return alpha
```

## Теоремы сходимости (Б.Т. Поляк Введение в оптимизацию, гл. 1, $\S$ 4;  гл. 3, $\S$ 1; Ю.Е. Нестеров Введение в выпуклую оптимизацию, $\S$ 2.2)
От общего к частному:

**Теорема 1.**
Пусть

- $f(x)$ дифференцируема на $\mathbb{R}^n$,
- градиент $f(x)$ удовлетворяет условию Липшица с константой $L$
- $f(x)$ ограничена снизу
- $\alpha = const$ и $0 < \alpha < \frac{2}{L}$

Тогда для градиентного метода выполнено:

$$
\lim\limits_{k \to \infty} f'(x_k) = 0,
$$

а функция монотонно убывает $f(x_{k+1}) < f(x_k)$.

** Теорема 2.**
Пусть
- $f(x)$ дифференцируема на $\mathbb{R}^n$,
- градиент $f(x)$ непрерывен
- множество $\{ x: f(x) \leq f(x_0) \}$ ограничено
- $\alpha_k = \arg\min\limits_{\alpha \geq 0} f(x_k - \alpha f'(x_k))$

Тогда
$$
f'(x_k) \to 0, \; k \to \infty \qquad x_{k_i} \to x^*
$$

**Теорема 3.** Пусть
- $f(x)$ дифференцируема на $\mathbb{R}^n$
- $f(x)$ выпукла
- $f'(x)$ удовлетворяет условию Липшица с константой $L$
- $\alpha = \dfrac{1}{L}$

Тогда
$$
f(x_k) - f^* \leq \dfrac{2L \| x_0 - x^*\|^2_2}{k+4}
$$

**Теорема 4.**
Пусть

- $f(x)$ дифференцируема на $\mathbb{R}^n$,
- градиент $f(x)$ удовлетворяет условию Липшица с константой $L$
- $f(x)$ является сильно выпуклой с константой $l$
- $\alpha = const$ и $0 < \alpha < \frac{2}{L}$

Тогда градиентный метод сходится к единственной точке глобального минимума $x^*$ с линейной скоростью:

$$
\| x_k - x^* \|_2 \leq cq^k, \qquad 0 \leq q < 1
$$

**Теорема 5.**
Пусть

- $f(x)$ дифференцируема на $\mathbb{R}^n$,
- градиент $f(x)$ удовлетворяет условию Липшица с константой $L$
- $f(x)$ является сильно выпуклой с константой $l$
- $\alpha = \dfrac{2}{l + L}$

Тогда для градиентного метода выполнено:
$$
\| x_k - x^* \|^2_2 \leq \left( \dfrac{M - 1}{M + 1} \right)^k \|x_0 - x^*\|^2_2 \qquad f(x_k) - f^* \leq \dfrac{L}{2} \left( \dfrac{M - 1}{M + 1} \right)^{2k} \| x_0 - x^*\|^2_2,
$$
где $M = \frac{L}{l}$

**Теорема 6.**
Пусть
- $f(x)$ дважды дифференцируема и $l\mathbf{I} \preceq f''(x) \preceq L\mathbf{I}$ для всех $x$
- $\alpha = const$ и $0 < \alpha < \frac{2}{L}$

Тогда
$$
\| x_k - x^*\|_2 \leq \|x_0 - x^*\|_2 q^k, \qquad q = \max(|1 - \alpha l|, |1 - \alpha L|) < 1
$$
и минимальное $q^* = \dfrac{L - l}{L + l}$ при $\alpha^* = \dfrac{2}{L + l}$

### От чего зависит $q^*$ и как это использовать?
Из Теорем 5 и 6 имеем
$$
q^* = \dfrac{L - l}{L + l} = \dfrac{L/l - 1}{L/l + 1} = \dfrac{M - 1}{M + 1},
$$
где $M$ - оценка числа обусловленности $f''(x)$.

**Вопрос**: что такое число обусловленности матрицы?

- При $M \gg 1$, $q^* \to 1 \Rightarrow$ оооочень **медленная** сходимости градиентного метода. Например при $M = 100$: $q^* \approx 0.98 $
- При $M \simeq 1$, $q^* \to 0 \Rightarrow$ **ускорение** сходимости градиентного метода. Например при $M = 4$: $q^* = 0.6 $

**Вопрос**: какая геометрия у этого требования?

**Мораль**: необходимо сделать оценку $M$ как можно ближе к 1!

О том, как это сделать, Вам будет предложено подумать

в домашнем задании :)

## Выбор начального приближения

Обратите внимание, что ни в одной из теорем приведённых выше не было условий на начальное приближение $x_0$!

- Для невыпуклой функции градиентный спуск сойдётся к некоторой стационарной точке, которая может быть как локальным минимумом, так и седловой точкой
- Для выпуклых функций стационарная точка будет точкой минимума
- Способы выбора начального приближения:
    - случайная точка
    - безградиентные методы случайного поиска, например [эволюционный алгоритм](https://docs.scipy.org/doc/scipy-0.18.1/reference/generated/scipy.optimize.differential_evolution.html), алгоритм муравьиной колонии (ant colony) и т.д.
    - обычно эти методы могут дать лишь незначительное уменьшение функции, однако сильно отойти от локальных минимумов

## Вычислительный аспект и эксперименты
1. Для каждого шага метода нужно хранить только текущую точку и вектор градиента: $O(n)$ памяти
2. Поиск $\alpha_k$:
    - дан априори
    - ищется из аналитического решения задачи наискорейшего спуска
    - заканчивается за конечное число шагов
3. Для каждого шага метода нужно вычислять линейную комбинацию векторов: $O(n)$ вычислений + высокопроизводительные реализации

### Pеализация градиентного спуска

```python
import numpy as np

def GradientDescent(f, gradf, x0, epsilon, num_iter, line_search,
                    disp=False, callback=None, **kwargs):
    x = x0.copy()
    iteration = 0
    opt_arg = {"f": f, "grad_f": gradf}
    for key in kwargs:
        opt_arg[key] = kwargs[key]
    while True:
        gradient = gradf(x)
        alpha = line_search(x, -gradient, **opt_arg)
        x = x - alpha * gradient
        if callback is not None:
            callback(x)
        iteration += 1
        if disp:
            print("Current function val =", f(x))
            print("Current gradient norm = ", np.linalg.norm(gradf(x)))
        if np.linalg.norm(gradf(x)) < epsilon:
            break
        if iteration >= num_iter:
            break
    res = {"x": x, "num_iter": iteration, "tol": np.linalg.norm(gradf(x))}
    return res
```

### Выбор шага

```python
def constant_step(x, gradient, **kwargs):
    return 1e-3

# For quadratic function x'Ax
def exact_linesearch(x, gradient, **kwargs):
    return max(0, -x.dot(A.dot(gradient)) / gradient.dot(A.dot(gradient)))

def backtracking(x, descent_dir, **kwargs):
    f = kwargs["f"]
    grad_f = kwargs["grad_f"]
    if kwargs["method"] == "Armijo":
        beta1 = kwargs["beta1"]
        rho = kwargs["rho"]
        alpha = 1
        while f(x + alpha * descent_dir) >= f(x) + beta1 * alpha * grad_f(x).dot(descent_dir) or \
                 np.isnan(f(x + alpha * descent_dir)):
            alpha *= rho
            if alpha < 1e-16:
                break
    return alpha
```

### Зависимость от обусловленности матрицы $f''(x)$
Рассмотрим задачу
$$
\min f(x),
$$
где
$$
f(x) = x^{\top}Ax, \; A =
\begin{bmatrix}
1 & 0\\
0 & \gamma
\end{bmatrix}
$$

$$
f'(x) = 2Ax
$$

```python
def my_f(x, A):
    return x.dot(A.dot(x))

def my_gradf(x, A):
    return 2 * A.dot(x)
```

```python
plt.rc("text", usetex=True)

gammas = [0.1, 0.5, 1, 2, 3, 4, 5, 10, 20, 50, 100, 1000, 5000, 10000]
num_iter_converg = []
for g in gammas:
    A = np.array([[1, 0],
                  [0, g]], dtype=np.float64)
    f = lambda x: my_f(x, A)
    gradf = lambda x: my_gradf(x, A)
    x0 = np.random.rand(A.shape[0])
#     x0 = np.sort(x0)
#     x0 = x0[::-1]
#     x0 = np.array([g, 1], dtype=np.float64)
#     print x0[1] / x0[0]
    res = GradientDescent(f, gradf, x0, 1e-7, 10000, exact_linesearch)
    num_iter_converg.append(res["num_iter"])

plt.figure(figsize=(8, 6))
plt.loglog(gammas, num_iter_converg)
plt.xticks(fontsize = 20)
plt.yticks(fontsize = 20)
plt.xlabel(r"$\gamma$", fontsize=20)
plt.ylabel(r"Number of iterations with $\varepsilon = 10^{-7}$", fontsize=20)
```

    <IPython.core.display.Javascript object>

    Text(0,0.5,'Number of iterations with $\\varepsilon = 10^{-7}$')

- При неудачном начальном приближении сходимость для плохо обусловенной задачи очень медленная
- При случайном начальном приближении сходимость может быть гораздо быстрее теоретических оценок
- **Open problem:** получить оценки сходимости, использующие случайное начальное приближение, а не худшее из возможных

## Эксперимент на многомерной задаче
Пусть $A \in \mathbb{R}^{m \times n}$. Рассмотрим систему линейных неравенств: $Ax \leq 1$ при условии $|x_i| \leq 1$ для всех $i$.

**Определение.** Аналитическим центром системы неравенств $Ax \leq 1$ при условии $|x_i| \leq 1$ является решение задачи
$$
f(x) = - \sum_{i=1}^m \log(1 - a_i^{\top}x) - \sum_{i = 1}^n \log (1 - x^2_i) \to \min_x
$$
$$
f'(x) - ?
$$

```python
import numpy as np
n = 100
m = 200
x0 = np.zeros(n)
A = np.random.rand(n, m)
```

### Точное решение с помощью CVXPy

```python
import cvxpy as cvx
print(cvx.installed_solvers())
x = cvx.Variable(n, 1)

obj = cvx.Minimize(cvx.sum_entries(-cvx.log(1 - A.T * x)) -
                   cvx.sum_entries(cvx.log(1 - cvx.square(x))))
prob = cvx.Problem(obj)
prob.solve(solver="SCS", verbose=True)
x = x.value
print("Optimal value =", prob.value)
```

    ['ECOS', 'ECOS_BB', 'SCS', 'LS']
    ----------------------------------------------------------------------------
    	SCS v1.2.6 - Splitting Conic Solver
    	(c) Brendan O'Donoghue, Stanford University, 2012-2016
    ----------------------------------------------------------------------------
    Lin-sys: sparse-indirect, nnz in A = 20700, CG tol ~ 1/iter^(2.00)
    eps = 1.00e-03, alpha = 1.50, max_iters = 2500, normalize = 1, scale = 1.00
    Variables n = 500, constraints m = 1200
    Cones:	soc vars: 300, soc blks: 100
    	exp vars: 900, dual exp vars: 0
    Setup time: 2.36e-03s
    ----------------------------------------------------------------------------
     Iter | pri res | dua res | rel gap | pri obj | dua obj | kap/tau | time (s)
    ----------------------------------------------------------------------------
         0| 1.32e+02  2.71e+03  7.93e-01 -3.35e+04 -3.86e+03  0.00e+00  5.32e-03
        80| 5.19e-05  6.60e-04  3.64e-05 -6.50e+02 -6.50e+02  1.44e-13  2.14e-01
    ----------------------------------------------------------------------------
    Status: Solved
    Timing: Solve time: 2.14e-01s
    	Lin-sys: avg # CG iterations: 2.95, avg solve time: 2.52e-04s
    	Cones: avg projection time: 2.36e-03s
    ----------------------------------------------------------------------------
    Error metrics:
    dist(s, K) = 1.8168e-05, dist(y, K*) = 2.2204e-16, s'y/|s||y| = 8.7910e-11
    |Ax + s - b|_2 / (1 + |b|_2) = 5.1898e-05
    |A'y + c|_2 / (1 + |c|_2) = 6.6006e-04
    |c'x + b'y| / (1 + |c'x| + |b'y|) = 3.6366e-05
    ----------------------------------------------------------------------------
    c'x = -650.1131, -b'y = -650.0658
    ============================================================================
    Optimal value = -650.1131344566362

### Решение с помощью градиентного спуска

```python
f = lambda x: -np.sum(np.log(1 - A.T.dot(x))) - np.sum(np.log(1 - x*x))
grad_f = lambda x: np.sum(A.dot(np.diagflat(1 / (1 - A.T.dot(x)))), \
                          axis=1) + 2 * x / (1 - np.power(x, 2))
def my_callback(x, array):
    array.append(x)
```

```python
x_conv = []
callback = lambda x: my_callback(x, x_conv)

res = GradientDescent(f, grad_f, x0, 1e-4, 100, backtracking, callback=callback,
                        method="Armijo", beta1=0.1, rho=0.7)
print("Gradient norm = {} and optimal value = {}".format(
    np.linalg.norm(grad_f(res["x"])), f(res["x"])))
grad_conv = [np.linalg.norm(grad_f(x)) for x in x_conv]
plt.figure(figsize=(8,6))
plt.semilogy(grad_conv, label=r"$\| f'(x_k) \|_2$")
plt.semilogy([np.linalg.norm(x - np.array(x_k)) for x_k in x_conv], label=r"$\|x_k - x^*\|_2$")
plt.semilogy([np.linalg.norm(prob.value - f(np.array(x_k))) for x_k in x_conv], label=r"$\|f(x_k) - f^*\|_2$")
plt.semilogy([np.linalg.norm(np.array(x_conv[i]) - np.array(x_conv[i+1])) for i in range(len(x_conv) - 1)], label=r"$\|x_k - x_{k+1}\|_2$")
plt.semilogy([np.linalg.norm(f(np.array(x_conv[i])) - f(np.array(x_conv[i+1]))) for i in range(len(x_conv) - 1)], label=r"$\|f(x_k) - f(x_{k+1})\|_2$")
plt.xlabel(r"Number of iteration, $k$", fontsize=20)
plt.ylabel(r"Convergence rate", fontsize=20)
plt.xticks(fontsize = 20)
plt.yticks(fontsize = 20)
plt.legend(loc="best", fontsize=20)
```

    Gradient norm = 4.499503227551099e-05 and optimal value = -650.1470673985286

    /Users/alex/anaconda3/envs/cvxpy/lib/python3.6/site-packages/ipykernel_launcher.py:1: RuntimeWarning: invalid value encountered in log
      """Entry point for launching an IPython kernel.

    <IPython.core.display.Javascript object>

    <matplotlib.legend.Legend at 0x181bfa6278>

### Зависимость скорости сходимости от параметров линейного поиска

Для простоты используем только условие достаточного убывания

#### Зависимость от $\rho$

От величины коэффициента $\rho$ зависит насколько агрессивно будет уменьшаться величина шага в процессе поиска.

```python
rhos = [0.4 + i * 0.05 for i in range(8)]
hist_x = []
callback = lambda x: my_callback(x, hist_x)
plt.figure(figsize=(10, 6))
for rho in rhos:
    GradientDescent(f, grad_f, x0, 1e-4, 100, backtracking, callback=callback,
                    method="Armijo", beta1=0.1, rho=rho)
    grad_norm = [np.linalg.norm(grad_f(x)) for x in hist_x]
    plt.semilogy(range(1, len(grad_norm) + 1), grad_norm, label=r"$\rho$ = " + str(rho))
    hist_x = []
plt.xlabel(r"Number of iteration, $k$", fontsize=18)
plt.ylabel(r"Norm of gradient, $\| f'(x_k) \|_2$", fontsize=20)
plt.xticks(fontsize = 18)
plt.yticks(fontsize = 20)
_ = plt.legend(loc="best", fontsize=18)
```

    <IPython.core.display.Javascript object>

    /Users/alex/anaconda3/envs/cvxpy/lib/python3.6/site-packages/ipykernel_launcher.py:1: RuntimeWarning: invalid value encountered in log
      """Entry point for launching an IPython kernel.

#### Зависимость от $\beta_1$

От коэффициента $\beta_1$ зависит угол наклона линейной аппроксимации, с которой сравнивается значение в новой точке $x_k - \alpha_k f'(x_k)$.

```python
betas = [0.05 + i * 0.05 for i in range(7)]
hist_x = []
callback = lambda x: my_callback(x, hist_x)
plt.figure(figsize=(8, 6))
for beta in betas:
    GradientDescent(f, grad_f, x0, 1e-4, 100, backtracking, callback=callback,
                    method="Armijo", beta1=beta, rho=0.6)
    grad_norm = [np.linalg.norm(grad_f(x)) for x in hist_x]
    plt.semilogy(range(1, len(grad_norm) + 1), grad_norm, label=r"$\beta_1$ = " + str(np.round(beta, 3)))
    hist_x = []
plt.xlabel(r"Number of iteration, $k$", fontsize=20)
plt.ylabel(r"Norm of gradient, $\| f'(x_k) \|_2$", fontsize=20)
plt.xticks(fontsize = 20)
plt.yticks(fontsize = 20)
plt.legend(loc="best", fontsize=18)
```

    <IPython.core.display.Javascript object>

    /Users/alex/anaconda3/envs/cvxpy/lib/python3.6/site-packages/ipykernel_launcher.py:1: RuntimeWarning: invalid value encountered in log
      """Entry point for launching an IPython kernel.

    <matplotlib.legend.Legend at 0x181c87ecf8>

## Pro & Contra

Pro
- легко реализовать
- сходимость как минимум к стационарной точке
- параметры при выборе шага влияют на сходимость не столь сильно
- имеет многочисленные вариации

Contra
- линейная сходимость
- очень сильно зависит от числа обусловленности $f''(x)$, выбор начального приближения может помочь
- не является оптимальным для выпуклых функций с липшицевым градиентом и сильновыпуклых функций (см. [ускорение Нестерова](https://blogs.princeton.edu/imabandit/2013/04/01/acceleratedgradientdescent/))

Модификациям градиентного спуска будет посвящено отдельное занятие.

## Резюме
1. Методы спуска
2. Направление убывания
3. Метод градиентного спуска
4. Правила выбора шага
5. Теоремы сходимости
6. Эксперименты


Chapter 20 of A guided tour of mathematical methods for the physical sciences deals with Potential Methods. These methods deal with (vector) fields that let themselves be described by (scalar) potentials. For example, the gravitational field due to the Earth's mass can be described by a potential $V$, so that the gravitional field is $$\mathbf{g} = -\nabla V.$$

In Chapter 6, we learned that the gravitational field at position $\mathbf{r}$ in a medium with density $\rho$ is
\begin{equation}
\nabla\cdot\mathbf{g}(\mathbf{r}) = -4\pi G\rho(\mathbf{r}).
\label{eq:g}
\end{equation}

Combining these equations, we find that the potential $V$ satisifies Laplace's equation:
$$ \nabla^2 V(\mathbf{r}) = 4\pi G\rho(\mathbf{r}),$$
the topic of Chapter 20. Let us illustrate the solutions for the gravitational acceleration with an application to gravity on/in/over the Earth.

The gravitational acceleration -- or the magnitude of the gravitational field -- due to a point mass $M$ is
$$g = \frac{G M}{r^2}:$$

```python
def g(m,r):
    return G*m/r**2
```

It turns out (see Chapters 6 and 8) that this is also the right expression outside ($r \ge R$) any finite spherically symmetric mass $M$. So let's explore gravity and Earth, by first defining some constants in SI units:

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib notebook # comment out in colab
import pandas as pd
from scipy import interpolate

rho = 5510 # The average density of the Earth in kg/m^3
R = 6371*10**3 # The approximate average radius the Earth in m
G = 6.67*10**(-11) # The Gravitational constant in m^3 s^(-2) kg^-1
```

For a mass M of radius $R$, the gravitational acceleration for $r \ge R$ is

```python
rs_out = np.linspace(R,2*R)
M = 4*np.pi*R**3*rho/3
g_out = g(M,rs_out)
plt.figure()
plt.plot(rs_out,g_out)
plt.xlabel('Distance to the centre (km)')
plt.ylabel('Gravitational acceleration (m/s$^2$)')
plt.show()
```

    <IPython.core.display.Javascript object>

Now let us consider the gravitational acceleration as we decend into a homogeneous sphere with (total) mass M. From the Theorem of Gauss in Chapter 8, we learned that only mass from the centre to the observer contributes to the acceleration.
We can calculate the mass below each position in the Earth and determine its contribution to the gravitational acceleration:

```python
rs = np.linspace(1,R,num=200) # skipping r=0, to avoid a warning by python about dividing by zero to get g.
m_in = (4/3)*np.pi*rho*rs**3
g_in = g(m_in,rs)
```

For this case, we also derived an analytic expression in equation 8.7 of Chapter 8: $$ g = \frac{GMr}{R^3},$$
where $r\le R$:

```python
g_in2 = G*M*rs/R**3
```

Just to be sure, let us compare the answers in a figure:

```python
plt.figure()
plt.plot(rs/1000,g_in,label='numerical')
plt.plot(rs/1000,g_in2,'r--',label='analytical')
plt.xlabel('Distance to the centre (km)')
plt.ylabel('Gravitational acceleration (m/s$^2$)')
plt.legend()
plt.show()
```

    <IPython.core.display.Javascript object>

Gravity is zero at the centre and increases linearly for a sphere with a constant density.
### Homework
The same way we derived the analytic expression for the gravitational acceleration inside a sphere with constant density, show that the analytic solution
for a linear density profile is:
$$g(r) = 4\pi G \left(\frac{ar}{3}+\frac{br^2}{4}\right)$$

Now we pick things up one notch. Most rocks you can find outside have a density of less than 3000 kg per cubic metre. However, it was Cavendish who figured out that the mass of the Earth is on the order of $6\times 10^{24}$ kg. (This means the average density of the Earth is around 5510 kg per cubic metre!) To better honour the Earth's density profile, let's assume it is linear, with density of 2500 kg per cubic metre at the surface, then increasing linearly to match the total mass of the Earth:
$$ \rho(r) = ar+b,$$
where we have two boundary conditions:

$\rho(R) = 2500$ and $\int_0^R\rho(r)dr= M$, then:

```python
rhoR= 2500
a = (12/R)*(rhoR/3-M/(4*np.pi*R**3))
b= rhoR-a*R
lin_rhos = b+ a*rs # is an array of equally spaced values that are on this linear density profile
```

```python
g_lin = 4*np.pi*G*((b*rs)/3+(a*rs**2)/4)
```

Or compute the mass under each point $r$ and determine the gravitational acceleration due to this mass. We do this by summing the contributions of shells with a constant density:

```python
def gravity(ii,rhos):
    M=0
    dr = rs[1]-rs[0]
    # find mass inside rs[ii]:
    for i in range(0,ii+1):
        M += dr*4*np.pi*rhos[i]*rs[i]**2
    return g(M,rs[ii])
```

Compute the gravitational acceleration for each mass under a radius $r$:

```python
g_lin2 = []
for i in range(len(rs)):
    g_lin2.append(gravity(i,lin_rhos))
```

And let's compare the two ways of computing the gravitational acceleration for a linear density distribution in a sphere:

```python
plt.figure()
plt.plot(rs/1000,g_in,label='average')
plt.plot(rs/1000,g_lin,label='lin')
plt.plot(rs/1000,g_lin2,label='lin2')
plt.xlabel('Distance to the centre (km)')
plt.ylabel('Gravitational acceleration (m/s$^2$)')
plt.legend()
plt.show()
```

    <IPython.core.display.Javascript object>

Now the gravitational acceleration is still zero at the centre of the Earth, but increases faster than for an Earth of constant density. Both models "predict" the gravitational acceleration we roughly find everywhere at the surface of the Earth. It is just that the model with the linear increase in density with depth honours what Cavendish learned about the Earth's mass/density. One more step: from seismic wavespeed measurements, we can say even more about the density in the Earth. How, is one of the topics of our stage 3 Solid Earth Geophysics course, but here's an estimate of the best radial model of density of the Earth:

```python
url="http://ds.iris.edu/files/products/emc/data/PREM/PREM_ANISOTROPIC.csv"
c=pd.read_csv(url, usecols=[0,1], names=['radius', 'density'], header=None)
prem_rhos = c['density']
prem_rs = c['radius']
plt.figure()
plt.plot(prem_rs/1000,prem_rhos,'o',label='PREM')
plt.plot(rs/1000,lin_rhos,'-.k',label='linear')
mean_rhos = 5510*np.ones(len(rs))
plt.plot(rs/1000,mean_rhos,'--g',label='average')
plt.ylabel('Density (kg/m$^3$)')
plt.xlabel('Distance to the centre (km)')
plt.legend()
plt.show()
```

    <IPython.core.display.Javascript object>

Note that generally density increases with depth, but there is a large jump from the rocky mantle to the iron core.

To determine the gravitational acceleration everywhere in an Earth with such an irregular density distribution, we are going to numerically integrate (sum) shells of constant density again. First, we'll create an earth that honours this model, but with regularly spaced shells, and we'll plot our three density models:

```python
f = interpolate.interp1d(prem_rs, prem_rhos)
rhos = f(rs)
```

Estimating $g(r)$ and plotting the results:

```python
g_prem = []
for i in range(0,len(rs)):
    g_prem.append(gravity(i,rhos))
plt.figure()
plt.plot(rs/1000,g_prem,'r',label='PREM')
plt.plot(rs/1000,g_lin,'-.k',label='linear')
plt.plot(rs/1000,g_in,'--g',label='average')
plt.legend()
plt.xlabel('Distance to the centre (km)')
plt.ylabel('Gravitational acceleration (m/s$^2$)')
#plt.savefig('g_inside.pdf')
plt.show()
```

    <IPython.core.display.Javascript object>

We can now clearly see the imprint on the gravitational acceleration of the jump in density between the mantle and the core, in particular.

### Variations on spherical symmetry in the Earth
However, the Earth is heterogeneous and NOT (quite) spherically symmetric. Below is a result from the GRACE mission (figure from wikipedia), showing that variations in $g$ from this best fitting ellipsoid:

[source](https://earthobservatory.nasa.gov/features/GRACE/page3.php).

The topography and colours look dramatic, but the scale means the Earth's density varies on the order 100 mGal or 1 mm/s$^2$. On an absolute value of almost 10 m/$s^2$, variations on our spherical models are (only) on the order of 1 part in 10000. Still, these variations allow Geophysicists to explain features in the subsurface related to tectonic plates, resources, and other geologic features.

```python

```


# Introduction to Convolutional Neural Networks

Convolutional neural networks or more commonly called by their abreviation "CNNs", is one of the most used architecure types of Neural Networks. These networks were inspired by the field of image processing, they use as building block the convolution operation instead of simple matrix multiplications. The use of convolutions have some advantages, such as:

* Reduced memory requirements: with convolutions the weights are filter or kernels that pass through the whole image, for example a 3x3 kernel has in total 9 weights and can be used in any image whose dimensions are equal or bigger than the ones from the kernel, if we used a linear layer the number of weights increase linearly with the number of pixels in the image.

* Convolutions are equivariant: this means that with convolutions we are able to identify features of an image regardless of their orientation. With a trained CNN for recognizing MNIST digits we can rotate the numbers or crop them and still be able to identify the number, this would be a hard task for linear layers.

## What is a Convolution extacly?

A convolution is the process of adding each element of the image to its local neighbors, weighted by the kernel.

Convolutions also have certain parameters that control how they behave and the shape of their output:

* **Stride**:  specifies how much we move the convolution filter at each step.

Convolution with stride of 1: \\
 \\
Convolution with stride of 2: \\
 \\

* **Padding**: these are extra values added arround the input image in order to achieve a desirable size of the reuslting operation. Normally the padding values are zeroes but others can be used like the values at the opposite edge.\\

 \\

* **Dilation**: The dilations is the ammount of gap between the kernle weights, it can be seen as expanding the kernel to a bigger one and filling the gaps with zeroes. The default value of dilation is 1. \\

\\

The dimension of the output of a convolution can be calculated as:

\begin{equation}
D_{o u t}=\left\lfloor\frac{D_{i n}+2 \times \text { padding }-\text { dilation } \times(\text { kernel_size }-1)-1}{\text { stride }}+1 \right\rfloor
\end{equation}

## Convolutional Neural Network for MNIST digits classification

Now we will try to solve the MNIST digit classification problem but using convolutions, we hope that by doing this we can increase accuracy.

```
import torch
import torch.nn as nn
import torch.nn.functional as F
import torchvision.datasets as datasets
import torchvision
import numpy as np
import matplotlib.pyplot as plt
```

If the dataset download doesn't work you can download the dataset from [here](http://yann.lecun.com/exdb/mnist/). Then save the .gz fikles under ./MNIST/raw/

```
!git clone https://github.com/ferngonzalezp/MNIST
```

```
transform = torchvision.transforms.ToTensor
train_dataset = datasets.MNIST(root='./',download=True, train=True, transform=transform())
test_dataset = datasets.MNIST(root='./', train=False, transform=transform())
```

```
example = train_dataset[np.random.randint(0,len(train_dataset))]
plt.imshow(example[0][0], cmap='gray')
plt.title(example[1])
```

We will build a simple convolutional neural network for this task. The idea behind this CNN is to have some feature extraction layers using convolutions and then a fully connected output layer to compute the prediction of the labels.

 \\

To reduce the dimensionality of the image we use a pooling operation.  This enables us to reduce the number of parameters, which both shortens the training time and combats overfitting. Pooling layers downsample each feature map independently, reducing the height and width, keeping the depth intact.

The most common type of pooling is max pooling which just takes the max value in the pooling window. Contrary to the convolution operation, pooling has no parameters. It slides a window over its input, and simply takes the max value in the window. Similar to a convolution, we specify the window size and stride.

Here is the result of max pooling using a 2x2 window and stride 2. Each color denotes a different window. Since both the window size and stride are 2, the windows are not overlapping. \\

 \\

The activation we use between convolutions is the rectified Linear unit (ReLU) and softmax in the output. We build our CNN pytorch module as follows: \\

```
class cnn(nn.Module):
  def __init__(self):
    super(cnn,self).__init__()
    self.feature_extraction = nn.Sequential(
        nn.Conv2d(1,16,kernel_size=3,stride=1,padding=1),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=2, stride=2),
        nn.Conv2d(16,32,kernel_size=3,stride=1,padding=1),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=2, stride=2),
    )
    self.fc = nn.Sequential(
        nn.Linear(32*7*7,10),
        nn.Softmax(dim=1)
    )
  def forward(self,x):
    x = self.feature_extraction(x)
    x = torch.flatten(x,1)
    return self.fc(x)
```

Now the remaining part of the code remains the same just as in classification with MLP.
____

```
#Use GPU if available
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
```

```
model = cnn().to(device)
```

```
print(model)
pytorch_total_params = sum(p.numel() for p in model.parameters())
print("number of parameters in model: %d"%(pytorch_total_params))
```

```
#Hyperparameters
batch_size = 100
n_epochs = 10
learn_rate = 2e-1
```

```
#DataLoaders
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=batch_size)
```

```
#Loss function and Optimizer
criterion = nn.BCELoss()
optimizer = torch.optim.SGD(model.parameters(),lr=learn_rate)
```

```
#Training Loop
train_loss = []
val_loss = []
for epoch in range(n_epochs):  # loop over the dataset multiple times
    epoch_loss = 0.0
    running_loss = 0.0
    for i, data in enumerate(train_loader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data
        inputs = inputs.to(device)
        labels = F.one_hot(labels,10).type_as(inputs)
        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        epoch_loss += loss.item()
        # print statistics
        running_loss += loss.item()

        if i % 50 == 49:    # print every 50 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 50))
            running_loss = 0.0
    train_loss.append(epoch_loss/(i+1))
    #Evaluation of the trained model
    correct = 0
    total = 0
    epoch_loss = 0.0
    print("validating...")
    with torch.no_grad():
        for i, data in enumerate(test_loader, 0):
            inputs, labels = data
            inputs = inputs.to(device)
            outputs = model(inputs)
            predicted = torch.argmax(outputs,dim=1)
            loss = criterion(outputs, F.one_hot(labels,10).type_as(inputs))
            labels =  labels.type_as(inputs)
            total += labels.shape[0]
            correct += (predicted == labels).sum().item()
            epoch_loss += loss.item()
    print('Accuracy of the network on the test images: %d %%' % (
        100 * correct / total))
    val_loss.append(epoch_loss/(i+1))

print('Finished Training')
```

```
plt.plot(train_loss, label='training')
plt.plot(val_loss, label='validation')
plt.legend()
plt.title('Loss')
plt.xlabel('Epochs')
plt.ylabel('Loss')
```

We can observe thet we achieve a higher accuracy with a CNN and with less training, even after the first epoch we got the same accuracy of the treined MLP.

```
#Evaluation of the trained model
correct = 0
total = 0
with torch.no_grad():
    for i, data in enumerate(test_loader, 0):
        inputs, labels = data
        inputs = inputs.to(device)
        labels =  labels.type_as(inputs)
        outputs = model(inputs)
        predicted = torch.argmax(outputs,dim=1)
        total += labels.shape[0]
        correct += (predicted == labels).sum().item()

print('Accuracy of the network on the test images: %d %%' % (
    100 * correct / total))
```

    Accuracy of the network on the test images: 98 %

Let's plot some image to see how the model performs.

```
from math import ceil
n_samples = 6
id = np.random.randint(0,len(test_dataset),n_samples)
rows = ceil(n_samples/3)
plt.figure(figsize=(rows*5,10))
for i in range(len(id)):
  images = test_dataset[id[i]][0].to(device)
  pred_labels = torch.argmax(model(images.unsqueeze(0)),dim=1)[0]
  plt.subplot(rows,3,i+1)
  plt.imshow(images[0].cpu(), cmap='gray')
  plt.title('prediction: %d'%(pred_labels))
```

# Tricks that help with performance, convergence and generalization

Using convolutional layers helps a lot in the performance of commputer vision, however for real wolrs applications and more difficult tasks it may not be enough. For that reason there exists a bag of tricks that drive CNNs towards greater accuracy, generalization and capacity for handling more complicated tasks. Here we will talk about some of the most used that are present nowadays in most of deep learning models.

## Batch Normalization

Batch normalization is used to fix the variance of the input of an activation function. This helps in reducuing the internal covariance shift in a neural networks as well as more practical benefits such as reducing the probability of having vanishing or exploding gradients and reducing generalization error. What BatchNorm does is first it calculates the mean and variance of the current mini-batch:

\begin{equation}
\mu_{B}=\frac{1}{m} \sum_{i=1}^{m} x_{i} \text { , and } \sigma_{B}^{2}=\frac{1}{m} \sum_{i=1}^{m}\left(x_{i}-\mu_{B}\right)^{2}
\end{equation}

Then normalizes the input of the layer as:

\begin{equation}
\hat{x}_{i}^{(k)}=\frac{x_{i}^{(k)}-\mu_{B}^{(k)}}{\sqrt{\sigma_{B}^{(k)^{2}}+\epsilon}}
\end{equation}

Where $k \in[1, d] \text { is the dimension number and } i \in[1, m] \text{ is the ith layer of the network}; \mu_{B}^{(k)} \text { and } \sigma_{B}^{(k)} \text{ are the per dimension mean and variance}$

$\epsilon$ is added in the denominator for numerical stability and is an arbitrarily small constant. The resulting normalized activation $\hat{x}_{i}^{(k)}$ have zero mean and unit variance if $\epsilon$ is not taken into account. To restore the representation power of the network, a transformation step then follows as:

\begin{equation}
y_{i}^{(k)}=\gamma^{(k)} \hat{x}_{i}^{(k)}+\beta^{(k)}
\end{equation}

Where $\gamma$ and $\beta$ are learnable parameters. Implementing BatchNorm is very straightforward in pytorch. In [pytorch](https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html#torch.nn.BatchNorm2d) we have the module **BatchNorm1d, BatchNorm2d, BatchNorm3d** depending in the dimensionality of our input, we need also to specify the number of features.  In a module we can use batchNorm like this:

```
self.feature_extraction = nn.Sequential(
        nn.Conv2d(1,16,kernel_size=3,stride=1,padding=1),
        nn.BatchNorm2d(16),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=2, stride=2),
        nn.Conv2d(16,32,kernel_size=3,stride=1,padding=1),
        nn.BatchNorm2d(32),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=2, stride=2),
    )
```

## Momentum and other optimizers

Stochastic gradient descent is the basis of a family of gradient-based algorithms used for deep learning. As we have seen, SGD is not perfetc and there are ways to boost its performance. One of the most fundamental techniques used to better the convergence of SGD is momentum.  The most intuitive way to understand momentum is making an anlogy with the physical world, imagine we have a sphere that goes with a initial speed, if the sphere encounters an uphill it may get stucked because its momentum is not enough for conquering gravity. We can overcome this problem if we add a bigger slope before hill so we can reach our goal at the end.

In gradient descent, momentum adapts the learning rate in order to accelerate convergence.

In stochastic gradient descent with momentum, information of the previous updating step is used to update the current one:

\begin{equation}
\begin{array}{l}
v_{t+1}=\mu * v_{t}+g_{t+1} \\
p_{t+1}=p_{t}-\operatorname{lr} * v_{t+1}
\end{array}
\end{equation}

Where $p, g, v \text{ and } \mu$ denote the parameters, gradient, velocity and momentum respectively. In this case momentum is an hyperparameter where its value we can set, if it is 0 then we have normal SGD.

As said earlier there are many algorithms that can be used for opimizing a Neural Network, In the pytorch [documentation](https://pytorch.org/docs/stable/optim.html#algorithms) you can find more information about them and see what is available for you to solve your problem.

## Data augmentations

Generally the data we have may not be enough to achieve generalization, for example we may train a model to recognize images but it may fail is the image is corrupted with noise or is rotated, on the contrary of human perception where we still may recognize the content of an image even if it is disorted.

One way to overcome this and achieve greater performance is by performing transformations to the model during training. We may do these transofrmations randomly during the training loop, and by doing this we virtually increase the samples the model see during training, this is why it is called data augmentation.

```
transforms = torchvision.transforms.Compose([
            torchvision.transforms.RandomAffine(60),
            torchvision.transforms.ToTensor(),
]
)
```

```
train_dataset = datasets.MNIST(root='./',download=True, train=True, transform=transforms)
```

```
example = train_dataset[np.random.randint(0,len(train_dataset))]
plt.imshow(example[0][0], cmap='gray')
plt.title(example[1])
```

# Build a classificator for the Fashion MNIST dataset

Now we will build a classificator for the fashion MNIST dataset. In the words of the author:

"Fashion-MNIST is a dataset of Zalando's article images—consisting of a training set of 60,000 examples and a test set of 10,000 examples. Each example is a 28x28 grayscale image, associated with a label from 10 classes. We intend Fashion-MNIST to serve as a direct drop-in replacement for the original MNIST dataset for benchmarking machine learning algorithms. It shares the same image size and structure of training and testing splits."

to download the dataset you can clone github repo with the command:
```
!git clone git@github.com:zalandoresearch/fashion-mnist.git
```

Or you can use the function in [torchvision](https://pytorch.org/vision/stable/datasets.html#fashion-mnist).

This practice will be free, and use of the documentation, working in groups and asking the instructor is encouraged. At the end you should build and test your model which needs to be reproductible in another computer. This problem have the following requirements:

1.   You should build two models: One an MLP and the other a CNN.
2.   You should use both models on MNIST and Fashion-MNIST.
3.   In the MLP model, keeping all the parameters the same, study the effect of increasing the number of layers of the network, use 2, 4 and 8 hidden layers. Make a graph of the accuray respect to the number of layers for both MNIST and Fashion-MNIST.
4.   Your can use batchNorm, dropout or any other techique that helps
 the number of layers and parameters of convolutions is free too.
5.   Use both SGD and SGD with momentum optimizer in the MLP an CNN, make a graph comparing the training loss curve with both optimizers for both datasets.
6.   Feel free to work in groups, search the web and ask questions but reporting remains individual.
7.   The metric to be used is accuracy but you can use as loss function binary cross entropy as before or any other.
8.   You must include the code of everything you did which should give all of the results.

At the end you should write a report containing the following:

*   The training and validations curves of both models for the two datasets.
*   The graph comparing the effect of number of layers in the MLP for the two datastes.
*   The graph comparing the effect of optimizer descibred in (5).
*   A table of accuracy containing rows for the dataset and colummns of model.
*   A table summarizing the hyperparameters used for both models.

At the end we will make a leaderboard to see who could design a model that achieves the highest accuracy for Fashion-MNIST.

```
import torch
import torch.nn as nn
import torch.nn.functional as F
import torchvision.datasets as datasets
import torchvision
import numpy as np
import matplotlib.pyplot as plt
```

```
#MLP for classification

def layers(depth,width):
  layers = []
  layers.append(nn.Linear(28**2,width))
  layers.append(nn.ReLU())
  for i in range(depth-1):
    layers.append(nn.Linear(width,width))
    layers.append(nn.ReLU())
  layers.append(nn.Linear(width,10))
  layers.append(nn.Softmax(dim=1))
  return layers

class mlp(nn.Module):
  def __init__(self, depth, width):
    super(mlp,self).__init__()
    self.model = nn.Sequential(*layers(depth,width))
  def forward(self,x):
    x = torch.flatten(x,1)
    return self.model(x)
```

```
#Build your CNN model

```

```
#Hyperparameters
batch_size = 64
n_epochs = 20
learn_rate = 1e-1
depth = 4            # This controls the number of layers of the MLP
dataset = 'MNIST'    # MNIST or FMNIST
opt = 'sgd_momentum' # sgd or sgd_momentum
momentum = 0.9
model_type = 'mlp' #choose betwen mlp or cnn
```

```
#Use GPU if available
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
```

```
#Dataset
if dataset == 'MNIST':
  transform = torchvision.transforms.ToTensor
  train_dataset = datasets.MNIST(root='./',download=True, train=True, transform=transform())
  test_dataset = datasets.MNIST(root='./', train=False, transform=transform())
if dataset == 'FMNIST':
  transform = torchvision.transforms.ToTensor
  train_dataset = datasets.FashionMNIST(root='./',download=True, train=True, transform=transform())
  test_dataset = datasets.FashionMNIST(root='./', train=False, transform=transform())
```

```
if model_type == 'mlp':
  model = mlp(depth=depth,width=32).to(device)
if model_type == 'cnn':
  model = cnn().to(device)
print(model)
```

```
#DataLoaders
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=batch_size)
```

```
example = train_dataset[np.random.randint(0,len(train_dataset))]
plt.imshow(example[0][0], cmap='gray')
plt.title(example[1])
```

```
#Loss function and Optimizer
criterion = nn.BCELoss()
if opt == 'sgd':
  optimizer = torch.optim.SGD(model.parameters(),lr=learn_rate)
if opt == 'sgd_momentum':
  optimizer = torch.optim.SGD(model.parameters(),lr=learn_rate, momentum=momentum)
```

```
#Training Loop
train_loss = []
val_loss = []
metric = []
for epoch in range(n_epochs):  # loop over the dataset multiple times
    epoch_loss = 0.0
    running_loss = 0.0
    for i, data in enumerate(train_loader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data
        inputs = inputs.to(device)
        labels = F.one_hot(labels,10).type_as(inputs)
        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        epoch_loss += loss.item()
        # print statistics
        running_loss += loss.item()

        if i % 50 == 49:    # print every 50 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 50))
            running_loss = 0.0
    train_loss.append(epoch_loss/(i+1))
    #Evaluation of the trained model
    correct = 0
    total = 0
    epoch_loss = 0.0
    print("validating...")
    with torch.no_grad():
        for i, data in enumerate(test_loader, 0):
            inputs, labels = data
            inputs = inputs.to(device)
            outputs = model(inputs)
            predicted = torch.argmax(outputs,dim=1)
            loss = criterion(outputs, F.one_hot(labels,10).type_as(inputs))
            labels =  labels.type_as(inputs)
            total += labels.shape[0]
            correct += (predicted == labels).sum().item()
            epoch_loss += loss.item()
    accuracy = 100 * correct / total
    print('Accuracy of the network on the test images: %d %%' % (accuracy))
    metric.append(accuracy)
    val_loss.append(epoch_loss/(i+1))
if model_type=='mlp':
  torch.save(metric,model_type + '_metric_'+ dataset +'_'+ opt + '_'+ str(depth)+'_layers.py')
  torch.save(train_loss,model_type + '_loss_'+ dataset +'_'+ opt + '_' + str(depth)+'_layers.py')
if model_type=='cnn':
  torch.save(metric,model_type + '_metric_'+ dataset +'_'+ opt +'.py')
  torch.save(train_loss,model_type + '_loss_'+ dataset +'_'+ opt +'.py')
print('Finished Training')
```

```
plt.plot(train_loss, label='training')
plt.plot(val_loss, label='validation')
plt.legend()
plt.title('Loss_'+model_type+'_'+dataset)
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.savefig('Loss_'+model_type+'.png')
```

```
metric1 = torch.load('/content/mlp_metric_MNIST_sgd_2_layers.py')
metric2 = torch.load('/content/mlp_metric_MNIST_sgd_4_layers.py')
metric3 = torch.load('/content/mlp_metric_MNIST_sgd_6_layers.py')

plt.plot(metric1, label='2 layers')
plt.plot(metric2, label='4_layers')
plt.plot(metric3, label='6_layers')
plt.legend()
plt.title('Accuracy MNIST')

```

```
metric1 = torch.load('/content/mlp_metric_FMNIST_sgd_2_layers.py')
metric2 = torch.load('/content/mlp_metric_FMNIST_sgd_4_layers.py')
metric3 = torch.load('/content/mlp_metric_FMNIST_sgd_6_layers.py')

plt.plot(metric1, label='2 layers')
plt.plot(metric2, label='4_layers')
plt.plot(metric3, label='6_layers')
plt.legend()
plt.title('Accuracy Fashion MNIST')

```

```
metric1 = torch.load('/content/mlp_metric_MNIST_sgd_4_layers.py')
metric2 = torch.load('/content/mlp_metric_MNIST_sgd_momentum_4_layers.py')
loss1 = torch.load('/content/mlp_loss_MNIST_sgd_4_layers.py')
loss2 = torch.load('/content/mlp_loss_MNIST_sgd_momentum_4_layers.py')
fig = plt.figure(figsize=(10,5))
plt.subplot(1,2,1)
plt.plot(metric1, label='sgd')
plt.plot(metric2, label='sgd_momentum')
plt.legend()
plt.title('Accuracy MNIST '+model_type)

plt.subplot(1,2,2)
plt.plot(loss1, label='sgd')
plt.plot(loss2, label='sgd_momentum')
plt.legend()
plt.title('training loss MNIST '+model_type)
```

```
metric1 = torch.load('/content/mlp_metric_FMNIST_sgd_4_layers.py')
metric2 = torch.load('/content/mlp_metric_FMNIST_sgd_momentum_4_layers.py')
loss1 = torch.load('/content/mlp_loss_FMNIST_sgd_4_layers.py')
loss2 = torch.load('/content/mlp_loss_FMNIST_sgd_momentum_4_layers.py')
fig = plt.figure(figsize=(10,5))
plt.subplot(1,2,1)
plt.plot(metric1, label='sgd')
plt.plot(metric2, label='sgd_momentum')
plt.legend()
plt.title('Accuracy Fashion MNIST '+model_type)

plt.subplot(1,2,2)
plt.plot(loss1, label='sgd')
plt.plot(loss2, label='sgd_momentum')
plt.legend()
plt.title('training loss Fashion MNIST '+model_type)

```

```
metric1 = torch.load('/content/cnn_metric_MNIST_sgd.py')
metric2 = torch.load('/content/cnn_metric_MNIST_sgd_momentum.py')
loss1 = torch.load('/content/cnn_loss_MNIST_sgd.py')
loss2 = torch.load('/content/cnn_loss_MNIST_sgd_momentum.py')
fig = plt.figure(figsize=(10,5))
plt.subplot(1,2,1)
plt.plot(metric1, label='sgd')
plt.plot(metric2, label='sgd_momentum')
plt.legend()
plt.title('Accuracy MNIST '+model_type)

plt.subplot(1,2,2)
plt.plot(loss1, label='sgd')
plt.plot(loss2, label='sgd_momentum')
plt.legend()
plt.title('training loss MNIST '+model_type)
```

```
metric1 = torch.load('/content/cnn_metric_FMNIST_sgd.py')
metric2 = torch.load('/content/cnn_metric_FMNIST_sgd_momentum.py')
loss1 = torch.load('/content/cnn_loss_FMNIST_sgd.py')
loss2 = torch.load('/content/cnn_loss_FMNIST_sgd_momentum.py')
fig = plt.figure(figsize=(10,5))
plt.subplot(1,2,1)
plt.plot(metric1, label='sgd')
plt.plot(metric2, label='sgd_momentum')
plt.legend()
plt.title('Accuracy Fashion MNIST '+model_type)

plt.subplot(1,2,2)
plt.plot(loss1, label='sgd')
plt.plot(loss2, label='sgd_momentum')
plt.legend()
plt.title('training loss Fashion MNIST '+model_type)
```


Neural network for importance sampling supplemented variational inference.

## Motivation

As the visualization of shallow neural network on MNIST dataset shows, the fitting of the PDF of the posterior via variational inference needs further finization. This calls for adding more (trainable) degree of freedom to the inferencer. At the same time, however, we shall keep both the PDF and the sampler of the inference-distribution explicit. Thus, the approach of adding these degree of freedom onto the parameterization of the inference-distribution would be too hard to establish. Instead, we decide to add them onto the weights of the samples sampled from the inference-distribution, thus treat this approach as an importance-sampling.

(Furthermore, the weight is a map from the parameter-space to $\mathbb{R}$, which thus avoids the redundancy of degree of freedom as the approach of neural network sampler (which maps from some low-dimensional hidden space to the high-dimensional parameter-space).)

## Theory

### Loss-function

Let $\theta \in \mathbb{R}^{N_d}$ the parameters, and $p(\theta)$ the PDF of the posterior over the parameter-space. Let $q(\theta; z)$ the inference distribution, with parameter $z$ to be trained. Let
\begin{equation}
  \hat{q}(\theta; z, w) := \omega (\theta; w) q(\theta, z),
\end{equation}
where $\omega (\theta; w)$, mapping to $\mathbb{R}$, is the weights of the importance sampling, with parameter $w$ to be trained.

The loss-function is
\begin{align}
  \mathcal{L} (z, w)
    & := \textrm{KL}(\hat{q} \| p) \\
    & = \int \hat{q} (\theta)
          \left[ \ln \hat{q} (\theta; z, w) - \ln p (\theta) \right]
        d \theta \\
    & = \int q(\theta) \omega (\theta; w)
          \left[ \ln \omega (\theta; w) + \ln q (\theta; z)
                 - \ln p (\theta) \right]
        d \theta \\
    & = \mathbb{E}_{\theta_s \sim q(z)}
          \left\{ \omega (\theta_s; w)
            \left[ \ln \omega (\theta_s; w) + \ln q (\theta_s; z)
                   - \ln p (\theta_s) \right] \right\},
\end{align}
where $s = 1, \ldots, N_s$. Further, for $\theta \in \{\theta_1, \ldots, \theta_{N_s}\} \sim q(z)$, let
\begin{equation}
  \omega(\theta; w) := \textrm{softmax} \left( \zeta (\theta; w) \right) * N_s,
\end{equation}
the normalization of $\hat{q}(\theta; z, w)$ can always be ensured:
\begin{align}
  1 & = \int \hat{q}(\theta; z, w) d\theta \\
    & = \mathbb{E}_{\theta_s \sim q(z)} \left[ \omega (\theta; w) \right] \\
    & = N_s * \mathbb{E}_{\theta_s \sim q(z)}
                \left[ \textrm{softmax} \left( \zeta (\theta; w) \right) \right] \\
    & = N_s * \frac{1}{N_s}
        \sum_s \textrm{softmax}_s \left( \zeta (\theta; w) \right) \\
    & = N_s \frac{1}{N_s} \times 1.
\end{align}

This methods extends the representability of pure variational inference by adding a weight-function. This weight-function can be implemented by a neural network by the utility of its universality.

We do not prefer a large variance of the weights, thus a penalty on this variance in the loss is called for. Thus
\begin{align}
  \mathcal{L}(z, w)
    & \to \mathcal{L}(z, w; \sigma) \\
    & = \mathbb{E}_{\theta_s \sim q(z)}
          \left\{ \omega (\theta_s; w)
            \left[ \ln \omega (\theta_s; w) + \ln q (\theta_s; z)
                   - \ln p (\theta_s) \right] \right\} \\
     & + \mathbb{E}_{\theta_s \sim q(z)}
           \left\{ \left(
             \frac{ \omega(\theta_s; w) - 1 }{ \sigma }
             \right)^2 \right\},
\end{align}
where $\sigma$ is a hyper-parameter.

The machine will automatically minimizes this loss by tuning the trainable parameters $(z, w)$, and then find a "harmonic" way of combining the $\omega$ and $q$, so as to represent the target $p(\theta)$.

### Measurement of Performance

The performance of the inference can be measured explicitly. For each sample $\theta$, define error:
\begin{equation}
  \mathcal{E} (\theta; z, w)
    := \ln p(\theta) - \ln q(\theta, z) - \ln \omega(\theta, w).
\end{equation}
If the inference works, we would expect that the $\{\mathcal{E}(\theta_s) \mid s = 1, \ldots, N_s\}$ distributes around the zero-point. With this, we have, $\forall f(\cdot)$,
\begin{equation}
  \mathbb{E}_{\theta \sim p(\theta)} \left[ f(\theta) \right]
    = \mathbb{E}_{\theta \sim q(\theta)} \left[
        \omega(\theta) \exp\{\mathcal{E}(\theta)\} f(\theta) \right].
\end{equation}
The smaller $\{|\mathcal{E}(\theta)| \mid \theta \sim q \}$, the better the performance is.

Thus, this measurement of performance can also help us in fine-tuning the hyper-parameters, like the $\sigma$, or those in the optimization process.

## Implementation

```python
import matplotlib.pyplot as plt
import numpy as np
import tensorflow as tf
import tensorflow.contrib.distributions as tfd
from collections import namedtuple
from tensorflow.contrib.layers import xavier_initializer
np.random.seed(1)
tf.set_random_seed(1)

def softplus(x, limit=30):
  if x > limit:
    return x
  else:
    return np.log(1.0 + np.exp(x))

LossInfo = namedtuple('LossInfo',
    ('samples, sample_logits, sample_weights, log_sample_weights,'
     'log_ps, log_qs, kl_divergence, penalty, loss, errors, gvs'))

def make_loss(log_p, inference_dist, logits, logits_scale=1e+2,
              penalty_scale=1e+0, n_samples=10, use_logits=True):
  """
  Args:
    log_p: Callable from tensor of the shape `[n_dims]` to scalar.
    inference_dist: An instance of `tfd.Distribution`.
    logits: Callable from tensor of the shape `[n_samples, n_dims]`
      to tensor of the shape `[n_samples]`.
      * It would be benefit if the output tensor is bounded, such as
        by employing the activation `lambda x: scale * tf.nn.tanh(x)`
        with some postive float `scale`.
    logits_scale: Positive float.
    n_samples: Positive integer.
  Returns:
    An instance of `LossInfo`.
  """
  with tf.name_scope('Samples'):
    # shape `[n_samples, n_dims]`
    samples = inference_dist.sample(n_samples)
  with tf.name_scope('Logits'):
    # shape: `[n_samples]`
    if use_logits:
      sample_logits = logits(samples)
    else:
      sample_logits = tf.zeros([n_samples])
  with tf.name_scope('KLDivergence'):
    # shape: `[n_samples]`
    sample_weights = tf.nn.softmax(sample_logits) * n_samples
    # shape: `[n_samples]`
    log_sample_weights = tf.log(sample_weights)
    # The batch-supplement may not ensured in `log_p`,
    # so we employ `tf.map_fn` for vectorization
    # shape: `[n_samples]`
    log_ps = tf.map_fn(log_p, samples)
    # Notice `tfd.Distribution.log_prob()` is batch-supplemented,
    # shape: `[n_samples]`
    log_qs = inference_dist.log_prob(samples)
    # shape: `[]`
    kl_divergence = tf.reduce_mean(
      sample_weights * (log_sample_weights + log_qs - log_ps),
      axis=0)
  with tf.name_scope('Penalty'):
    # shape: `[]`
    _, sample_logits_std = tf.nn.moments(sample_logits/logits_scale,
                                         axes=[0])
    # shape: `[]`
    penalty = penalty_scale * tf.nn.relu(sample_logits_std - 1.0)
  with tf.name_scope('Loss'):
    # shape: `[]`
    loss = kl_divergence + penalty
  with tf.name_scope('LogEvidence'):
    # shape: `[]`
    log_evidence = (tf.reduce_logsumexp(log_ps - log_qs, axis=0)
                    - tf.log(tf.to_float(n_samples)))
  with tf.name_scope('Errors'):
    # shape: `[n_samples]`
    errors = log_ps - log_qs - log_sample_weights - log_evidence
  return LossInfo(samples, sample_logits, sample_weights, log_sample_weights,
                  log_ps, log_qs, kl_divergence, penalty, loss, errors, None)
```

## Experiments

```python
def get_inference_dist(n_dims):
  with tf.name_scope('InferenceDistribution'):
    loc = tf.get_variable('loc', [n_dims], 'float32')
    scale = tf.get_variable('scale', [n_dims], 'float32')
    inference_dist = tfd.Independent(
        tfd.NormalWithSoftplusScale(loc, scale))
  return inference_dist

def get_logits(n_hiddens_list, scale):
  def logits(samples):
    """The `logits` argument of `make_loss()`."""
    with tf.name_scope('LogitsNeuralNetwork'):
      hiddens = samples
      for n_hiddens in n_hiddens_list:
        hiddens = tf.layers.dense(
            hiddens, n_hiddens, activation=tf.nn.sigmoid,
            kernel_initializer=xavier_initializer())
        # Dropping out demages the representability of the ANN.
        #hiddens = tf.layers.dropout(hiddens)  # thus shall not use.
      # shape: `[n_samples, 1]`
      #outputs = tf.layers.dense(hiddens, 1)
      outputs = scale * tf.layers.dense(
          hiddens, 1, activation=tf.tanh,
          kernel_initializer=xavier_initializer())
      # shape: `[n_samples]`
      return tf.squeeze(outputs, axis=1)
  return logits

def test(log_p, n_dims, n_hiddens_list=None, use_logits=True,
         logits_scale=1e+2, penalty_scale=1e+0, lr=1e-3, n_iters=10**4,
         max_trails=2, error_threshold=5, scale=1e+1):
  if n_hiddens_list is None:
    n_hiddens_list = []
  inference_dist = get_inference_dist(n_dims)
  logits = get_logits(n_hiddens_list, scale)
  loss_info = make_loss(log_p, inference_dist, logits,
                        logits_scale, penalty_scale,
                        use_logits=use_logits)
  optimizer = tf.train.RMSPropOptimizer(learning_rate=lr)
  gvs = optimizer.compute_gradients(
      loss_info.loss, tf.trainable_variables())
  #gvs = [(tf.clip_by_value(g, -5., 5.), v) for g, v in gvs]  # clip.
  train_op = optimizer.apply_gradients(gvs)
  sess = tf.Session()
  sess.run(tf.global_variables_initializer())

  print('Loss (before):', sess.run(loss_info.loss))
  print('Penalty (before):', sess.run(loss_info.penalty))

  loss_vals = []
  for step in range(1, n_iters+1):
    sess.run(train_op)

    # The value of loss can sometimes temporally be `NaN`, and in
    # the next `sess.run()` becomes non-`NaN` (strange!). So, we
    # employ the following strategy:
    loss_val = sess.run(loss_info.loss)
    n_trials = 0
    while np.isnan(loss_val) and n_trials < max_trails:
      loss_val = sess.run(loss_info.loss)
      n_trials += 1
    if n_trials == max_trails:
      print(sess.run([loss_info.log_sample_weights, loss_info.log_ps,
                      loss_info.log_qs, loss_info.kl_divergence,
                      loss_info.penalty]))
      print('Always `NaN`, finally stopped at step {}.'.format(step))
      sess.close()
      return
    loss_vals.append(loss_val)

  print('Loss (after):', sess.run(loss_info.loss))
  print('Penalty (after):', sess.run(loss_info.penalty))
  visualize(loss_vals, loss_info, sess, error_threshold)
  sess.close()

def visualize(loss_vals, loss_info, sess,
              error_threshold):
  samples = []
  log_sample_weights = []
  logits = []
  errors = []
  for i in range(100):
    result = sess.run([loss_info.samples, loss_info.log_sample_weights,
                       loss_info.sample_logits, loss_info.errors])
    sample_vals, log_sample_weight_vals, logit_vals, error_vals = result
    samples += [_ for _ in sample_vals]
    log_sample_weights += [_ for _ in log_sample_weight_vals]
    logits += [_ for _ in logit_vals]
    errors += [_ for _ in error_vals]
  samples = np.array(samples)
  log_sample_weights = np.array(log_sample_weights)
  logits = np.array(logits)

  #err_mean = np.mean(errors)
  #err_std = np.std(errors)
  #errors = np.array([_ for _ in errors if abs(_ - err_mean) < 3*err_std])
  errors = np.array(errors)

  # Visualize
  fig = plt.figure()

  ax1 = fig.add_subplot(222)
  ax1.plot(loss_vals, label='loss')
  ax1.legend()

  ax2 = fig.add_subplot(221)
  ax3 = fig.add_subplot(223)
  ax4 = fig.add_subplot(224)
  count = 0
  for l, e, lw, s in zip(
      logits, errors, log_sample_weights, samples[:,0]):
    if abs(e) < error_threshold:
      count += 1
      ax2.scatter(e, l, c='blue', alpha=0.2)
      ax3.scatter(e, lw, c='blue', alpha=0.2)
      ax4.scatter(s, lw, c='blue', alpha=0.2)
  ax2.set_xlabel('$\mathcal{E}$')
  ax2.set_ylabel('logits')
  ax3.set_ylabel('$\ln \omega$')
  ax4.set_xlabel('$\\theta^{\\alpha=0}$')
  print('{} important samples'.format(count))
  print('Importance ratio:', count / len(log_sample_weights))

  plt.show()
```

### High-dimensional Gaussian distribution

The target is the PDF of distribution
$\mathcal{N} \left(0, \textrm{softplus}(10)\right)$
on $\mathbb{R}^{N_d}$.

```python
def get_log_p(n_dims):
  target_dist = tfd.Independent(
      tfd.NormalWithSoftplusScale(loc=tf.zeros(n_dims),
                                  scale=10*tf.ones(n_dims)))
  return target_dist.log_prob
```

```python
tf.reset_default_graph()

n_dims = 10000
log_p = get_log_p(n_dims)
kwargs = {
  'use_logits': True,
  'lr': 1e-3,
  'n_hiddens_list': [10, 10, 10],
  'n_iters': 20000,
  'error_threshold': 5,
  'scale': 10,
}
print('PARAMETERS:', kwargs, '\n')
test(log_p, n_dims, **kwargs)
```

* Conclusion: so far so good.

### More Complex Distribution

```python
def get_log_p(n_dims):
  dist = tfd.Independent(tfd.Gamma(1.2*tf.ones([n_dims]),
                                   1.0*tf.ones([n_dims])))
  def log_p(x):
    return dist.log_prob(tf.exp(x))
  return log_p
```

```python
%%time

tf.reset_default_graph()

n_dims = 10
log_p = get_log_p(n_dims)
kwargs = {
  'use_logits': False,  # base-line!
  'lr': 1e-4,
  'n_iters': 30000,
  'error_threshold': 2,
}
print('PARAMETERS:', kwargs, '\n')
test(log_p, n_dims, **kwargs)
```

```python
%%time

tf.reset_default_graph()

n_dims = 10
log_p = get_log_p(n_dims)
kwargs = {
  'use_logits': True,
  'lr': 1e-4,
  'n_hiddens_list': [10],
  'n_iters': 30000,
  'error_threshold': 2,
  'scale': 10,
}
print('PARAMETERS:', kwargs, '\n')
test(log_p, n_dims, **kwargs)
```

```python
%%time

tf.reset_default_graph()

n_dims = 10
log_p = get_log_p(n_dims)
kwargs = {
  'use_logits': True,
  'lr': 1e-4,
  'n_hiddens_list': [10, 10],
  'n_iters': 30000,
  'error_threshold': 2,
  'scale': 10,
}
print('PARAMETERS:', kwargs, '\n')
test(log_p, n_dims, **kwargs)
```

```python
%%time

tf.reset_default_graph()

n_dims = 10
log_p = get_log_p(n_dims)
kwargs = {
  'use_logits': True,
  'lr': 1e-4,
  'n_hiddens_list': [100],
  'n_iters': 30000,
  'error_threshold': 2,
  'scale': 10,
}
print('PARAMETERS:', kwargs, '\n')
test(log_p, n_dims, **kwargs)
```

```python
%%time

tf.reset_default_graph()

n_dims = 10
log_p = get_log_p(n_dims)
kwargs = {
  'use_logits': True,
  'lr': 1e-5,
  'n_hiddens_list': [100],
  'n_iters': 30000,
  'error_threshold': 2,
  'scale': 10,
}
print('PARAMETERS:', kwargs, '\n')
test(log_p, n_dims, **kwargs)
```

* Conclusion: XXX