# Small NODEC example
A small example to compare NODEC with optimal control on a toy dynamical system.

```python
# To edit source files and automatically refresh code
%load_ext autoreload
%autoreload 2

# Load custom modules path
import sys
sys.path.append('../../')

# Custom modules path
import nnc.controllers.baselines.ct_lti.dynamics as dynamics
import nnc.controllers.baselines.ct_lti.optimal_controllers as oc
from nnc.controllers.neural_network.nnc_controllers import\
     NeuralNetworkController, NNCDynamics

# Computation and plot helpers for this example
from small_example_helpers import EluTimeControl, evaluate_trajectory, todf
from small_example_helpers import compare_trajectories, grand_animation

# progress bar
from tqdm.cli import tqdm

# Other libraries for computing
import torch
import numpy as np
import pandas as pd

# ODE solvers with gradient flow
from torchdiffeq import odeint

# plots
import plotly
plotly.offline.init_notebook_mode()
```

## Continuous Time Time-Invariant Dynamics
First we define our dynamics as:
\begin{equation}\label{eq:ld}
\dfrac{dx}{dt} = \langle A, x^{\intercal} \rangle + \langle B, u^{\intercal}\rangle
\end{equation}
where:
\begin{align}
x &: \text{a state vector over $N$ nodes.}\\
A &: \text{an $N\times N$ interaction matrix, indicating nodal interactions.}\\
u &: \text{a control signal vector of $M\leq N$ independent control signals.}\\
B &: \text{an $N\times M$ driver matrix, indicating control signal effects on nodes.}\\
\end{align}

In this example we parametrize the system as:

### System Parametrization

\begin{align}
A &=
\begin{pmatrix}
    0 & 1 \\
    1 & 0 \\
\end{pmatrix}\\
B &=
\begin{pmatrix}
    1 \\
    0 \\
\end{pmatrix}
\end{align}

Essentially, we evaluate control over a 2-node undirected graph with control signal applied over one node (single driver node).

```python
# Basic setup for calculations, graph, number of nodes, etc.
dtype = torch.float32
device = 'cpu'
training_session = True

# interaction matrix
A = torch.tensor([
    [0., 1.],
    [1., 0.]
])

# driver matrix
B = torch.tensor([
    [1.],
    [0.]
])

# interaction matrix dimensions denote how many nodes we have in the network
n_nodes = A.shape[-1]
# column dimension implies the number of driver nodes.
n_drivers = B.shape[-1]

# implementing the dynamics
linear_dynamics = dynamics.ContinuousTimeInvariantDynamics(A, B, dtype, device)

```

The exact implementation of the dynamics:

```python
dynamics.ContinuousTimeInvariantDynamics??
```

### Experimental Setting
We aim to control the system from initial state $x(0)=(0.5, 0.5)$ to $x^* = (1,-1)$ within time $T=1$.

```python
# total time for control
T = 1
# we evaluate two points in time, first point is matched to initial
# state and  second one is matched to the terminal state
t = torch.linspace(0, T, 2)

# initial state is set as follows, but can be chosen arbirtarily:
x0 = torch.tensor([[
    0.5, 0.5
]])

# same applies for target state:
x_target = torch.tensor([[
    1, -1
]])
```

### Control  Baseline
Here we use the minimum energy optimal control as baseline based on the following work:
- Yan, G., Ren, J., Lai, Y. C., Lai, C. H., & Li, B. (2012). Controlling complex networks: How much energy is needed?. Physical review letters, 108(21), 218703.

For CT-LTI systems that satisfy controllability assumption, this baseline achieves the maximum theoretical performance, i.e. it cannot be surpassed by any other.

```python
# baseline definition
oc_baseline = oc.ControllabiltyGrammianController(
    alpha = A,
    beta = B,
    t_infs = T,
    x0s = x0, # a batch with one sample
    x_infs = x_target, # a batch with one sample
    simpson_evals = 100,
    progress_bar=None,
    use_inverse=False,
)
```

Below, we create on line 2 a `lambda` expression function to make the dynamics compatible with required method signatue, and then we evolve the system with `odeint` method from `torchdiffeq` package (line 3).
The result of `odeint` is a tensor of shape `[timesteps, state_size]`.
The reached state corresponds to the last index of the result `[-1]`.
As we print on line 4, optimal control reached the target state after control $x(T)\approx x^*$.

```python
timesteps = torch.linspace(0, T, 2)
oc_dynamics = lambda t,x: linear_dynamics(t, x, oc_baseline(t, x))
xT_oc = odeint(oc_dynamics, x0.unsqueeze(0), t=timesteps)[-1]
print(str(xT_oc.flatten().detach().cpu().numpy()))
```

### Neural Network Control
We define a custom neural network for learning contro.
In this example we pick a fully connected architecture with 2 layers of n+4 neurons
and exponential linear unit (ELU) activation.
The content of the class can be found below:

```python
EluTimeControl??
```

We define a training routine for the neural networks.
- Note the `x0.detach()` and `t.detach()` on lines 7,8 respectively to avoid unexpected gradient flows.
- It is important is to notice on the next cell, in line 11 that we include no energy regularization term in the loss.
- We then do the backpropagation with autograd on line 13 and then let the optimizer (Adam) to update the network parameters.
- We prefer to train with variable step method `dopri5` on line 8, but we evaluate with constant step size on line 16, to limit performance advantages due to step size selection.

```python
def train(nnc_dyn, epochs, lr, t, n_timesteps=40): #simple training
    optimizer = torch.optim.Adam(nnc_dyn.parameters(), lr=lr)
    trajectories = [] # keep trajectories for plots
    for i in tqdm(range(epochs)):
        optimizer.zero_grad() # do not accumulate gradients over epochs

        x = x0.detach()
        x_nnc = odeint(nnc_dyn, x, t.detach(), method='dopri5')
        x_T = x_nnc[-1, :] # reached state at T

        loss = ((x_target - x_T)**2).sum() # !No energy regularization

        loss.backward()
        optimizer.step()

        trajectory = evaluate_trajectory(linear_dynamics,
                                         nnc_dyn.nnc,
                                         x0,
                                         T,
                                         n_timesteps,
                                         method='rk4',
                                         options=dict(step_size = T/n_timesteps)
                                        )
        trajectories.append(trajectory)
    return torch.stack(trajectories).squeeze(-2)
```

## Experimental Results
We can now apply training, collect trajctories and save the model.
Notice that we decrease learning rate on lines 11 and 14.
NODEC is very sensitive to learning rate and often requires several lr-adapaptions to converge to desirable performance.
Such, adaptions can be automated as discussed in the paper.

```python
n_timesteps = 40 #relevant for plotting, ode solver is variable step
linear_dynamics = dynamics.ContinuousTimeInvariantDynamics(A, B, dtype, device)

if training_session:
    torch.manual_seed(0)
    neural_net = EluTimeControl(n_nodes, n_drivers)
    nnc_model = NeuralNetworkController(neural_net)
    nnc_dyn = NNCDynamics(linear_dynamics, nnc_model)
    # time to train now:
    t = torch.linspace(0, T, n_timesteps)
    t1 = train(nnc_dyn, 200, 0.1, t) # , 200 epochs, learning rate 0.1
    df1 = todf(t1, lr=0.1)
    t2 = train(nnc_dyn, 300, 0.01, t) # , 300 epochs, learning rate 0.01
    df2 = todf(t2, lr=0.01)
    torch.save(neural_net, 'trained_elu_net.torch')
    alldf = pd.concat([df1, df2], ignore_index=True)

    alldf.to_csv('all_trajectories.csv')
else:
    neural_net = torch.load('trained_elu_net.torch')
    alldf = pd.read_csv('all_trajectories.csv', index_col=0)
    nnc_model = NeuralNetworkController(neural_net)
    nnc_dyn = NNCDynamics(linear_dynamics, nnc_model)
```

Now we can check the reached for NODEC, and as we observe it is close to the target.
More epochs on lower learnig rates will further improve this result.

```python
t = torch.linspace(0, T, 2)
x = x0.detach()
ld_controlled_lambda = lambda t, x_in: linear_dynamics(t, u=neural_net(t, x_in), x=x_in)
x_all_nn = odeint(ld_controlled_lambda, x0, t, method='rk4',
                  options = dict(step_size=T/n_timesteps))
x_T = x_all_nn[-1, :]
print(str(x_T.flatten().detach().cpu().numpy()))
```

Now we compare NODEC to optimal control (OC) in terms of: (i) controlled trajectories (left) and (ii) total energy (right). As we see both methods are extremely close.

```python
fig_comparison, _, _ = compare_trajectories(linear_dynamics,
                     oc_baseline,
                     nnc_model,
                     x0,
                     x_target,
                     T,
                     x1_min=-3,
                     x1_max=3,
                     x2_min=-1.5,
                     x2_max=1.5,
                     n_points=200,
                    )
fig_comparison
```

Finally we animate the learning effort of NODEC through the training process. We observe how it slowly converges to the OC trajectory, both in terms of state values and energy.

```python
animation =  grand_animation(alldf,
                             linear_dynamics,
                             oc_baseline,
                             nnc_model,
                             x0,
                             x_target,
                             T,
                            frame_duration=50) # 50-100 should be ok.
animation
```

## Conclusion
This notebook gave us a quick yet insightful demonstration of NODEC and its performance compared to optimal control.
Results conclude that NODEC can achieve the control goal. Implicit energy regularization is also observed.
Larger scale demonstrations on complex dynamics are found in the paper results.


Introduction to NumPy & SciPy - the core numerical scientific packages

This tutorial is prepared by ACM Student Chapter of King Abdullah University of Science and Technology (KAUST).

Material has been adapted from the following:

 - David Ketcheson's [Introduction to NumPy and Matplotlib](http://nbviewer.ipython.org/github/ketch/AMCS252/blob/master/2_Introduction_to_Numpy%20and_Matplotlib.ipynb)

- Software Carpentry's [Numerical analysis with NumPy](http://nbviewer.ipython.org/github/geocarpentry/2014-01-30-mit/blob/gh-pages/lessons/vignettes/Numerical%20analysis%20with%20NumPy.ipynb)

- J.R. Johansson's [Numpy - multidimensional data arrays](http://nbviewer.ipython.org/github/jrjohansson/scientific-python-lectures/blob/master/Lecture-2-Numpy.ipynb) and [SciPy - Library of scientific algorithms for Python](http://nbviewer.ipython.org/github/jrjohansson/scientific-python-lectures/blob/master/Lecture-3-Scipy.ipynb#SciPy---Library-of-scientific-algorithms-for-Python)

- Official [Tentative NumPy tutorial](http://www.scipy.org/Tentative_NumPy_Tutorial)

**Prerequisites:** Introduction to Python

## NumPy

Python includes a package for numerical computation called numpy, which is an essential tool for scientific computing.
It is based on multidimensional arrays (vectors and matrices) and provides a large library of functions and operators that work efficiently on these objects.
In this way, any numerical algorithm is expressed as operations on arrays and in many cases runs almost as quickly as the equivalent C code.

Like MATLAB, `NumPy` relies on BLAS and LAPACK for efficient linear algebra computations.

`NumPy` is used in:

Numerical Analysis, Linear algebra, Solution of differential equations, Statistical analysis and many more...

### Why to use NumPy?

Consider the following list and suppose we want to perform some simple algebraic operations:

```
price = [5.99, 10.25, 2.0, 40.99, 5.60, 63.49]
```

```
price*2
```

    [5.99, 10.25, 2.0, 40.99, 5.6, 63.49, 5.99, 10.25, 2.0, 40.99, 5.6, 63.49]

```
price+0.5
```

Clearly, if we wanted to double the value of each element and add half, this cannot be done easily. Instead, we would have to use a loop:

```
for i in range(len(price)):
    price[i] = price[i]*2
    price[i] = price[i]+0.5
print price
```

    [12.48, 21.0, 4.5, 82.48, 11.7, 127.48]

But, there should be a more convinient way...

### Getting started

We start, by importing the NumPy module. Import statements like this are the typical way of getting access to functions in Python. The most general way to import a module is the following:

```
from numpy import *
```

The above imports all functions included in the package and can call them simply by their name.

```
sin(3*pi/2)*exp(2)
```

    -7.3890560989306504

Instead we can import the numpy module and tell Python that we want to refer to numpy by the short abbreviation *np*. This is recommended since we keep track of which module the functions belong to.

```
import numpy as np
```

### Arrays

NumPy's main object is the homogeneous multidimensional array.
It is a table of elements - all of the same type - indexed by a tuple of positive integers.

NumPy's array class is called *ndarray* and the package includes basic utility functions for creating and manipulate arrays in a similar fashion to MATLAB.
For example we can create a one-dimensional array by:

```
price = np.array([5.99, 10.25, 2.0, 40.99, 5.60, 63.49])
print type(price)
price = price*2+0.5
print price
```

    <type 'numpy.ndarray'>
    [  12.48   21.      4.5    82.48   11.7   127.48]

There are other ways to create one-dimensional arrays:

```
x = np.linspace(-1,1,5)
print x
```

    [-1.  -0.5  0.   0.5  1. ]

```
y = np.arange(0,1,0.2)
print y
```

    [ 0.   0.2  0.4  0.6  0.8]

---
### Exercise 7

Change the inputs of the *linspace* and *arange* function until you understand exactly what they do. Using both ways, create a vector of starting from -1.5 to 4.5 inclusive with a step of 0.5.

```
# type your solution here
```

##### Solution

```
#%load solutions/1D_array.py
```

We can always find out what a function does by typing its name, followed by a question mark:

```
np.arange?
```

Resize the help window (to get it out of your way) by dragging the divider. Double click on the divider to close it.

### Multidimensional arrays

We can create multidimensional arrays, like the following:

```
A = np.array([[1,2.4,-13],[4.1,5,0],[7.2,8,9]])
print "A = ", A
print type(A)
print A.shape
print A.dtype
```

    A =  [[  1.    2.4 -13. ]
     [  4.1   5.    0. ]
     [  7.2   8.    9. ]]
    <type 'numpy.ndarray'>
    (3, 3)
    float64

To see all different attributes of the array object type its name followed by a fullstop and press *tap*. Note that some attributes are basically functions and require parentheses (with or without arguments), e.g. *conjugate* and others don't, e.g. *size*:

```
A = np.array([[1,2+4j,3j],[4j,5,6-5j],[7j,8,9+3j]]) # dtype=complex
print A.conjugate()
print A.size
```

    [[ 1.-0.j  2.-4.j  0.-3.j]
     [ 0.-4.j  5.-0.j  6.+5.j]
     [ 0.-7.j  8.-0.j  9.-3.j]]
    9

We can also define the shape of the array:

```
np.random.uniform(0,1,size=(5,5)) # 5 x 5 matrix with elements from a uniform distribution
```

    array([[ 0.18894904,  0.4016313 ,  0.30167486,  0.04452608,  0.29880108],
           [ 0.85862765,  0.18898635,  0.46421297,  0.6303044 ,  0.56185042],
           [ 0.78056999,  0.82705839,  0.63247392,  0.07986216,  0.64999473],
           [ 0.07852346,  0.27204531,  0.85521149,  0.94891731,  0.7810846 ],
           [ 0.47172365,  0.57284027,  0.57618836,  0.30150177,  0.54945194]])

#### Special arrays

The function *zeros* creates an array full of zeros, the function *ones* creates an array full of ones and *eye* an identity array. Function *diag* is used for a diagonal array with specified diagonal elements. A matrix with random entries is generated by the *random* function.

```
print np.zeros((3,4))
print np.ones((2,3))
print np.eye(3)
```

    [[ 0.  0.  0.  0.]
     [ 0.  0.  0.  0.]
     [ 0.  0.  0.  0.]]
    [[ 1.  1.  1.]
     [ 1.  1.  1.]]
    [[ 1.  0.  0.]
     [ 0.  1.  0.]
     [ 0.  0.  1.]]

```
print np.diag(x,0)
print np.diag([2,3,4])
print np.random.random((2,2))
```

    [[-1.   0.   0.   0.   0. ]
     [ 0.  -0.5  0.   0.   0. ]
     [ 0.   0.   0.   0.   0. ]
     [ 0.   0.   0.   0.5  0. ]
     [ 0.   0.   0.   0.   1. ]]
    [[2 0 0]
     [0 3 0]
     [0 0 4]]
    [[ 0.89073408  0.92386659]
     [ 0.43499535  0.1673178 ]]

Notice that there are two pairs of parentheses used when calling the *zeros*, *ones* and *random* command, because we are passing as argument (e.g. the Python tuple *(3,4)*), which is the shape of the resulting array.

### Reshape

Manipulation of arrays can be done in many ways:

```
A = np.arange(1,10)
print "A = ", A
A = A.reshape((3,3))
print A
```

    A =  [1 2 3 4 5 6 7 8 9]
    [[1 2 3]
     [4 5 6]
     [7 8 9]]

```
B = arange(15).reshape((3,5))
print "B = ", B
print B.ravel() # flatten the array
```

    B =  [[ 0  1  2  3  4]
     [ 5  6  7  8  9]
     [10 11 12 13 14]]
    [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14]

### Indexing

We can slice numpy arrays just as other programming languages, but the indexing is a little different. See the examples below:

```
print A
print A[0,0]
print A[1:3,0:2]
print A[1:3,[0,1]]
```

    [[1 2 3]
     [4 5 6]
     [7 8 9]]
    1
    [[4 5]
     [7 8]]
    [[4 5]
     [7 8]]

Note that the fundamental difference from MATLAB is that the indexing starts from zero and is exclusive of the second index. Hence, in the above example 1:3 refers to the second and third row and 0:2 to the first and second column.

We can index in some clever ways too:

```
print A
```

    [[1 2 3]
     [4 5 6]
     [7 8 9]]

```
print A[0,:] # row selection
print A[:,0] # column selection
```

    [1 2 3]
    [1 4 7]

```
print A[:,1:] # all rows, from 2nd column until the end
print A[-1,:] # last row, all columns
```

    [[2 3]
     [5 6]
     [8 9]]
    [7 8 9]

We can change array's elements:

```
A[:,2] = [.3,.4,9.12]
print A
print A.dtype
```

    [[1 2 0]
     [4 5 0]
     [7 8 9]]
    int64

Notice that the elements of A remain integers. That is because A has type *int*. The following does the desired change:

```
A = A.astype(float)
A[:,2] = [.3,.4,9.12]
print A
```

    [[ 1.    2.    0.3 ]
     [ 4.    5.    0.4 ]
     [ 7.    8.    9.12]]

There is an easy way to reverse a vector:

```
a = arange(10)
print a
print a[: :-1] # reverse a
```

    [0 1 2 3 4 5 6 7 8 9]
    [9 8 7 6 5 4 3 2 1 0]

We can also index an array using boolean arrays:

```
b = a>5
print b
a[b] = 0  # All elements of 'a' higher than 5 become 0
print a
```

    [False False False False False False  True  True  True  True]
    [0 1 2 3 4 5 0 0 0 0]

### Array and matrix multiplication

```
print a**2
print 10*sin(np.pi/a[1:5]) # avoiding division with zero!
print A*A
```

    [ 0  1  4  9 16 25  0  0  0  0]
    [  1.22464680e-15   1.00000000e+01   8.66025404e+00   7.07106781e+00]
    [[  1.       4.       0.09  ]
     [ 16.      25.       0.16  ]
     [ 49.      64.      83.1744]]

By default, NumPy uses component-wise multiplication. For a matrix-matrix (or matrix-vector) multiplication, we use the *dot* function:

```
np.dot(A,A)
```

    array([[  11.1   ,   14.4   ,    3.836 ],
           [  26.8   ,   36.2   ,    6.848 ],
           [ 102.84  ,  126.96  ,   88.4744]])

### Other basic operations

```
A = array(A,dtype='int')
print A
```

    [[1 2 0]
     [4 5 0]
     [7 8 9]]

```
print A.sum() # sum of all elements
print A.min()
print A.max()
```

    36
    0
    9

```
print A.sum(axis=0)     # sum of each column
print A.min(axis=1)     # min of each row
print A.cumsum(axis=1)  # cumulative sum along each row
```

    [12 15  9]
    [0 0 7]
    [[ 1  3  3]
     [ 4  9  9]
     [ 7 15 24]]

In Python *axis* refers to the dimensions of arrays; *axis=0* refers to the colums and *axis=1* to the rows.

### Stacking and splitting

The *vstack* and *hstack* commands can be used to stack arrays and *hsplit* to split.

```
A = floor(10*random.random((2,2)))
print A
B = floor(10*random.random((2,2)))
print B
```

    [[ 4.  0.]
     [ 6.  7.]]
    [[ 1.  2.]
     [ 5.  3.]]

```
print np.vstack((A,B))
print np.hstack((B,A))
```

    [[ 4.  0.]
     [ 6.  7.]
     [ 1.  2.]
     [ 5.  3.]]
    [[ 1.  2.  4.  0.]
     [ 5.  3.  6.  7.]]

```
A = floor(10*random.random((2,12)))
print A
```

    [[ 6.  7.  5.  2.  3.  6.  2.  4.  7.  5.  2.  9.]
     [ 8.  4.  4.  5.  7.  0.  5.  3.  3.  3.  6.  3.]]

```
print np.hsplit(A,3)       # Split A into 3 arrays
print np.hsplit(A,(3,4))   # Split A after the third and the fourth column
```

    [array([[ 6.,  7.,  5.,  2.],
           [ 8.,  4.,  4.,  5.]]), array([[ 3.,  6.,  2.,  4.],
           [ 7.,  0.,  5.,  3.]]), array([[ 7.,  5.,  2.,  9.],
           [ 3.,  3.,  6.,  3.]])]
    [array([[ 6.,  7.,  5.],
           [ 8.,  4.,  4.]]), array([[ 2.],
           [ 5.]]), array([[ 3.,  6.,  2.,  4.,  7.,  5.,  2.,  9.],
           [ 7.,  0.,  5.,  3.,  3.,  3.,  6.,  3.]])]

---
### Exercise 8

1. Create a vector with values ranging from 10 to 99
2. Find indices of non-zero elements from [1,2,0,0,4,0]. Hint: use np.nonzero
3. Declare a 8x8 matrix and fill it with a checkerboard pattern of zeros and ones, without a loop.
4. Create a 5x3 matrix whose $(i,j)$ entry is equal to $i\times j^2$ without using a loop. Hint: use np.fromfunction.
5. Create a 4x4 array with numbers from a normal distribution with mean = 3 and standard deviation = 1.5. Hint: use np.random.normal. Sum the values of each row and store the result in a new array.

```
# type your solution here
```

##### Solution

```
#%load solutions/array_operations.py
```

Now let's work with some real data. Ensure that the stockholm_temp.txt in the same directory with this notebook.
This file contains the [daily average temperatures](http://bolin.su.se/data/stockholm/homogenized_daily_mean_temperatures.php)
according to observations of Stockholm for the years 1756 - 2012.
The following code checks if the file exists, opens it, prints the first lines and stores the data in an array.

```
import os
filename = 'data/stockholm_temp.txt'
if os.path.exists(filename):
    # look at the first 3 lines
    with open(filename,'r') as f:
        print '\n'.join(f.readlines()[:3])
    # alternative look at the first head lines
    !head 'data/stockholm_temp.txt'

    data = np.loadtxt(filename) # alternative we can use getfromtxt command

else:
    print 'Weather data does not exist, please download "stockholm_temp.txt" from Github.'
```

    1.756000000000000000e+03 1.000000000000000000e+00 1.000000000000000000e+00 -8.699999999999999289e+00

    1.756000000000000000e+03 1.000000000000000000e+00 2.000000000000000000e+00 -9.199999999999999289e+00

    1.756000000000000000e+03 1.000000000000000000e+00 3.000000000000000000e+00 -8.599999999999999645e+00

    1.756000000000000000e+03 1.000000000000000000e+00 1.000000000000000000e+00 -8.699999999999999289e+00
    1.756000000000000000e+03 1.000000000000000000e+00 2.000000000000000000e+00 -9.199999999999999289e+00
    1.756000000000000000e+03 1.000000000000000000e+00 3.000000000000000000e+00 -8.599999999999999645e+00
    1.756000000000000000e+03 1.000000000000000000e+00 4.000000000000000000e+00 -7.700000000000000178e+00
    1.756000000000000000e+03 1.000000000000000000e+00 5.000000000000000000e+00 -7.200000000000000178e+00
    1.756000000000000000e+03 1.000000000000000000e+00 6.000000000000000000e+00 -1.600000000000000089e+00
    1.756000000000000000e+03 1.000000000000000000e+00 7.000000000000000000e+00 6.999999999999999556e-01
    1.756000000000000000e+03 1.000000000000000000e+00 8.000000000000000000e+00 1.300000000000000044e+00
    1.756000000000000000e+03 1.000000000000000000e+00 9.000000000000000000e+00 2.399999999999999911e+00
    1.756000000000000000e+03 1.000000000000000000e+00 1.000000000000000000e+01 8.000000000000000444e-01

Since the data are in the form
*Year
- Month
- Day
- Temperature*
we would like to express them in a better way. We define a `dtype`, that is the type for each column: int, int, int, float:

```
dt = np.dtype([('Year', 'int16'), ('Month', 'int8'), ('Day', 'int8'), ('Temp', 'float64')])
data = np.loadtxt(filename,dtype=dt)
data[:10] # first 10 entries of our data
```

    array([(1756, 1, 1, -8.7), (1756, 1, 2, -9.2), (1756, 1, 3, -8.6),
           (1756, 1, 4, -7.7), (1756, 1, 5, -7.2), (1756, 1, 6, -1.6),
           (1756, 1, 7, 0.7), (1756, 1, 8, 1.3), (1756, 1, 9, 2.4),
           (1756, 1, 10, 0.8)],
          dtype=[('Year', '<i2'), ('Month', 'i1'), ('Day', 'i1'), ('Temp', '<f8')])

We can now look for the example only on the temperatures:

```
data['Temp']
```

    array([-8.7, -9.2, -8.6, ...,  0.2,  2.6,  2.2])

or find all leap years:

```
years = np.unique(data['Year']) # making an array with each year appearing only once
leap_years = np.zeros((1,),dtype='int16') # initializing array for leap years
for i in range(len(years)):
    if np.fmod(years[i],4)==0 and ((np.fmod(years[i],100)==0 and np.fmod(years[i],400)==0) or np.fmod(years[i],100)!=0):
        leap_years = np.append(leap_years,years[i]) # adding a leap year
leap_years = leap_years[1:] # removing dummy first value
print leap_years
```

    [1756 1760 1764 1768 1772 1776 1780 1784 1788 1792 1796 1804 1808 1812 1816
     1820 1824 1828 1832 1836 1840 1844 1848 1852 1856 1860 1864 1868 1872 1876
     1880 1884 1888 1892 1896 1904 1908 1912 1916 1920 1924 1928 1932 1936 1940
     1944 1948 1952 1956 1960 1964 1968 1972 1976 1980 1984 1988 1992 1996 2000
     2004 2008 2012]

---
### Exercise 9

Make an array with the temperatures of the 5th day of every month.

```
# type your solution here
```

##### Solution

```
#%load solutions/temperatures.py
```

### Linear algebra

Handling *ndarrays* is really powerful. Of the many things one can do with NumPy's arrays is linear algebra:

```
x = np.random.random(10)
print x
np.linalg.norm(x,np.inf) # maximum norm of a vector
```

    [ 0.05834554  0.05282837  0.39608145  0.37972821  0.59102374  0.38928025
      0.00254778  0.4471012   0.22718756  0.32243128]

    0.59102373985998502

We can use this module to solve a linear system of equations $Ax=b$:

```
A = np.array([[0,2],[8,0]])
b = np.array([1,2])
print A
print b
```

    [[0 2]
     [8 0]]
    [1 2]

```
x = np.linalg.solve(A,b)
print x
```

    [ 0.25  0.5 ]

We can also solve eigenvalue problems:

```
lamda,V = np.linalg.eig(A)
print lamda
print V
```

    [ 4. -4.]
    [[ 0.4472136  -0.4472136 ]
     [ 0.89442719  0.89442719]]

Notice we have put two variables on the left of the equals sign, to assign the outputs of *eig()* to two different variables. Here *lamda* are the eigenvalues and the columns of *V* are the eigenvectors.

---
### Exercise 10

Create a diagonal matrix using the *lamda* variable and then multiply from the left with V and from the right with the inverse of V (use *np.linalg.inv()*). What do you find?

```
# type your solution here
```

##### Solution

```
#%load solutions/linear_algebra.py
```

## SciPy

`NumPy` is complemented by `SciPy`; a library that adds more MATLAB-like functionality.

`SciPy` is a collection of mathematical algorithms and convenience functions that provides high-level commands and classes for Nowdays, everything from parallel programming to web and data-base subroutines and classes have been made available to Python All of this power is also accessible through the the mathematical libraries in SciPy `NumPy` and `SciPy`.

`SciPy` is used in:

Numerical Analysis, Optimization, Linear algebra, Image and signal processing, Statistical analysis, Data processing, Data transformation and query and many more...

### Getting started

As with NumPy, we can access all packages in SciPy:

```
from scipy import *
```

### Special functions

SciPy provides implementations of a large set of special functions, usually used in mathematics, physics and engineering.
Available functions include airy, elliptic, bessel, gamma, beta, hypergeometric, parabolic cylinder, mathieu, spheroidal wave, struve, and kelvin.

For a list of functions see SciPy's documention at http://docs.scipy.org/doc/scipy/reference/special.html#module-scipy.special.
or try

```
from scipy import special
#help(special)
```

For example we can look at the Bessel functions.

Bessel functions are a family of solutions to Bessel’s differential equation with real or complex order alpha:
\begin{align}
    x^2 \frac{d^2 y}{d x^2} + x \frac{d y}{d x} + (x^2 −\alpha^2)y = 0
\end{align}

```
from scipy.special import jn, jn_zeros

x = 0.0

n = 0    # order
# Bessel function of first kind, zero order
print "J_%d(%f) = %f" % (n, x, jn(n, x))

n = 1    # order
# Bessel function of first kind, first order
print "J_%d(%f) = %f" % (n, x, jn(n, x))
```

    J_0(0.000000) = 1.000000
    J_1(0.000000) = 0.000000

The next two lines allow plotting inside the ipython notebook. More during the next sessions...

```
# The next two
%pylab inline --no-import-all
import matplotlib.pyplot as plt
```

    Populating the interactive namespace from numpy and matplotlib

We can now plot the Bessel functions of first kind for different values of order:

```
x = linspace(0, 12, 100)

fig, ax = plt.subplots()
ax.plot(x,zeros(len(x)),'--k')
for n in range(4):
    ax.plot(x, jn(n, x), label=r"$J_%d(x)$" % n)
ax.legend();
```

and check for their zeros...

```
# zeros of Bessel functions
n = 0 # order
m = 4 # number of roots to compute
jn_zeros(n, m)
```

    array([  2.40482556,   5.52007811,   8.65372791,  11.79153444])

### Integration

The numerical evaluation of the definite integral of a function is called numerical quadrature, or simply quadature.
SciPy provides a series of functions for different kind of quadrature, for example the quad, dblquad and tplquad for single, double and triple integrals, respectively.

```
from scipy.integrate import quad
```

```
xl = 0  # the lower limit of x
xu = 10 # the upper limit of x

val, abserr = quad(lambda x: jn(0,x), xl, xu)

print "integral value =", val, ", absolute error =", abserr
```

    integral value = 1.06701130396 , absolute error = 7.43478946065e-14

We can pass extra arguments to integrand function, by using the `args` keyword argument.
For example we can use `args=(3,)` for the third order Bessel function.

---
### Exercise 11

Create a function that returns the Bessel function of first kinf for different orders and then find the integral.

```
# type your solution here
```

##### Solution

```
#%load solutions/bessel_integral.py
```

### Interpolation

The `interp1d` function accepts given arrays describing points X and Y data.
It returns a function that can be evaluated everywhere in the domain for an arbitrary value of $x \in X$ and the image it is corresponding interpolated value. Options for interpoland kinds are linear, quadratic and cubic spline interpolation.

```
from scipy.interpolate import interp1d
```

```
x = np.linspace(0, 20, 20)
y = lambda x: jn(5,x) # sampled on given values x
f1 = interp1d(x, y(x)) # linear interpolation
f2 = interp1d(x, y(x), kind='quadratic')
f3 = interp1d(x, y(x), kind='cubic')
```

```
# new points where we want to evaluate the interpoland
xnew = np.linspace(0, 20, 100)

fig, ax = plt.subplots(figsize=(10,4))
plt.plot(x,y(x),'o',xnew,f1(xnew),'-', xnew, f2(xnew),'--',xnew, f3(xnew),'--',xnew,y(xnew),'k')
plt.legend(['data', 'linear', 'quadratic','cubic','true'], loc='best')
plt.show()
```

## Sparse matrices

Sparse matrices are often useful in numerical analysis dealing with large systems.
Whenever the problem involves vectors or matrices with mostly zero values then one should take advantage of the sparsity.
SciPy has an extensive library for sparse matrices, with basic linear algebra operations (such as equation solving, eigenvalue calculations, etc).

To create a sparse matrix we have to choose the format that will be stored in:

```
from scipy.sparse import *
```

```
# dense matrix
M = array([[10,0,0,0], [0,.3,0,0], [0,2,1.5,0], [-3,0,0,100]])
print M
```

    [[  10.     0.     0.     0. ]
     [   0.     0.3    0.     0. ]
     [   0.     2.     1.5    0. ]
     [  -3.     0.     0.   100. ]]

```
# convert from dense to sparse
M_sparse = csr_matrix(M); # or csc_matrix(M)
print M_sparse
```

      (0, 0)	10.0
      (1, 1)	0.3
      (2, 1)	2.0
      (2, 2)	1.5
      (3, 0)	-3.0
      (3, 3)	100.0

We can go back to the dense form:

```
print M_sparse
print M_sparse.todense()
```

      (0, 0)	10.0
      (1, 1)	0.3
      (2, 1)	2.0
      (2, 2)	1.5
      (3, 0)	-3.0
      (3, 3)	100.0
    [[  10.     0.     0.     0. ]
     [   0.     0.3    0.     0. ]
     [   0.     2.     1.5    0. ]
     [  -3.     0.     0.   100. ]]

The easiest way to create a sparse matrix is using the `lil_matrix` command. It creates an empty matrix and then we allogating non-zero entries by using matrix indexing. This way we avoid creating large dense matrices.

```
A = lil_matrix((4,4)) # empty 4x4 sparse matrix using a linked list format
A[0,0] = 1
A[1,1] = 3
A[2,2] = A[2,1] = 1
A[3,3] = A[3,0] = 1
A
```

    <4x4 sparse matrix of type '<type 'numpy.float64'>'
    	with 6 stored elements in LInked List format>

We can also change the format based on the sparsity:

```
A = csr_matrix(A); A # if the rows are sparse
```

    <4x4 sparse matrix of type '<type 'numpy.float64'>'
    	with 6 stored elements in Compressed Sparse Row format>

```
A = csc_matrix(A); A # if the columns are sparse
```

    <4x4 sparse matrix of type '<type 'numpy.float64'>'
    	with 6 stored elements in Compressed Sparse Column format>

---
### Exercise 12

Construct a $10000 \times 10000$ `lil_matrix` $A$ and add values to it by setting all rows and columns $2000$ to $3000$ into random values. Also set random diagonal entries (you can do this without a loop!). Then use the `spy` function from `matplotlib.pyplot` to "spy" the matrix.

Using a vector $b$ with $10000$ random values solve the system $A x = b$.
To do so use the direcet solver `linsolve` from `scipy.sparse.linalg.dsolve`.
Change to different sparse formats (CSR and CSC) and measure the time needed to solve the system. You can use the `time` package.

```
# type your solution here
```

##### Solution

```
#%load solutions/sparse_matrices.py
```

## References:

#### NumPy:

- [NumPy tutorial](http://www.scipy.org/Tentative_NumPy_Tutorial)
- [NumPy tutorial video](http://showmedo.com/videotutorials/series?name=i9KuJuNcG)
- [NumPy: lock 'n load](http://mentat.za.net/numpy/intro/intro.html)
- [NumPy for Matlab users](http://www.scipy.org/NumPy_for_Matlab_Users)

#### SciPy
- [SciPy getting started](http://www.scipy.org/getting-started.html)
- [SciPy tutorial](https://docs.scipy.org/doc/scipy-0.14.0/reference/tutorial/index.html)

#### More general

- [General Python tutorial](http://docs.python.org/tutorial/)
- [Dive Into Python](http://diveintopython.org/)
- [Hans Petter Langtangen book](http://link.springer.com/book/10.1007/978-3-642-02475-7/page/1)

### Other packages
Besides NumPy and Scipy, there are many other useful Python packages for scientific computing. Here is a short list:

- [matplotlib](http://www.matplotlib.org/) - plotting and visualization
- [sympy](http://sympy.org/) - symbolic computation
- [mpi4py](http://mpi4py.scipy.org/) - parallel computing
- [petsc4py](http://code.google.com/p/petsc4py/), [pytrilinos](http://trilinos.sandia.gov/packages/pytrilinos/) - Python bindings for the "big 2" parallel scientific libraries
- [pyCUDA](http://mathema.tician.de/software/pycuda), [pyOpenCL](http://mathema.tician.de/software/pyopencl) - GPGPU computing
- [FENiCS](http://fenicsproject.org/), [FiPy](http://www.ctcms.nist.gov/fipy/), [PyClaw](http://numerics.kaust.edu.sa/pyclaw/) - solve complicated PDEs with very sophisticated numerical methods
- [networkX](http://networkx.github.com/), [pygraphviz](http://networkx.lanl.gov/pygraphviz/) - graphs

*Copyright 2015, Maruan Al-Shedivat, Yiannis Hadjimichael, ACM Student Chapter.*


# Lecture 24 - Adaptive Learning Rates

# Backpropagation in a Nutshell

Suppose you have the data $\{x_i\}_{i=1}^N \in \mathbb{R}^D$ with labels $\{d_i\}_{i=1}^N \in \mathbb{R}^K$. You want to find a mapper that learns the input samples $x_i$ and maps it to a label response $d_i$, i.e. *classification*.

Consider the following objective function:

$$J(w) = \frac{1}{2} \sum_{j=1}^p e_p^2 = \frac{1}{2} \sum_{j=1}^p \left(d_p-y_p\right)^2$$

where $p$ is the dimensionality of your desired values.

1. **Forward Pass:**
For any neuron $i$ receiving input signals from neurons $j$:

$$y_i = \phi\left(\sum_{j=1}^M w_{ij}x_j\right)$$

where $\phi(\bullet)$ is a pre-defined activation function. (The weights and biases of the network have been initialized to some random value.)

2. **Backward Pass:** Compute the gradient by defining local errors at every layer and neuron

$$\delta_i = \phi'(v_i)\sum_j \delta_j w_{ij}$$

$$\Delta w_{ij} = - \eta \delta_j y_i$$

and update the parameters:

$$w_{ij}^{(n+1)} = w_{ij}^{(n)} + \Delta w_{ij}$$

where $\eta$ is the learning rate.

# Adaptive Linear Systems

In regression, we found an analytic solution for the *optimal* weights of our regressor.

For some data $\{x_i\}_{i=1}^N$ and labels $\{d_i\}_{i=1}^N$, we found that the optimal weights are:

$$w^* = (X^TX)^{-1} X^Td$$

using the mean squared error function $J = \frac{1}{2N} \sum_{i=1}^N (d_i - wx_i)^2$

Note that if we were operating in the feature space of $X$, $\phi(X)$, the optimal weights are:

$$w^* = (\phi(X)^T\phi(X))^{-1} \phi(X)^Td$$

Assuming, the data $X$ is demeaned, $R = X^TX$ is called the *covariance* of the input data and $P = X^Td$ is called the *cross-covariance* of the input data with the desired signal.

Such analytic solution only exists if the model is a *linear function on the parameters*, i.e., $y = Xw$ or $y = \phi(X) w$, which is not the case for an MLP output!

* We can find a solution by performing a *search* of the performance surface (governed by the error function $J$).

# The Error Surface for a Linear Neuron

* The mean squared error function $J = \frac{1}{2N} \sum_{i=1}^N e_i^2$ is a *convex* function and therefore we can apply convex optimization techniques to search for the *minima* of this function.
    * The most common method to optimize the least squares error is the *steepest descent*.

* The error surface lies in a space with a horizontal axis for each weight and one vertical axis for the error.
    * For a linear neuron with a squared error, it is a quadratic bowl.
    * Vertical cross-sections are parabolas.
    * Horizontal cross-sections are ellipses.

* For multi-layer, non-linear nets the error surface is much more complicated.
    * But locally, a piece of a quadratic bowl is usually a very good apparoximation.

## Convergence Speed of Full-Batch Learning

* Going "downhill" reduces the error, but the direction of steepest descent does not point at the minimum unless the ellipse is a circle.
    * The gradient is big in the direction in which only want to travel a small distance.
    * The gradient is small in the direction in which we want to travel a large distance

* Even for non-linear multi-layer nets, the error surface is locally quadratic, so the same speed issues apply.

## Learning Rate

$$w^{(t+1)} = w^{(t)} - \eta \nabla J(w^{(t)})$$

* If the learning rate is big, the weights slosh to and from the ravine.
    * If the learning rate is too big, this oscillation diverges.

* What we would like to achieve:
    * Move quickly in direction with small but consistent gradients.
    * Move slowly in directions with big but inconsistent gradients.

## Stochastic Gradient Descent - Online Learning

* If the dataset is redundant, the gradient on the first half is almost identical to the gradient on the second half.
    * So instead of computing the ful gradient, update the weights using the gradient on the first half and then get a gradient for the new weights on the second half.
    * The extreme version of this approach updates weights after each point. It's called **online learning**.

* Mini-batches are usually better than online.
    * Less computation is used for updating the weights.
    * Computing the gradient for many cases simultaneously uses matrix-matrix multiplications which are very efficient, especially with GPUs.

* Mini-batches need to be balanced for classes.

## Full-Batch vs Mini-Batch Learning

* If we use the full gradient computed from all the training cases, **batch learning**, there are many clever ways to speed up (e.g. non-linear conjugate gradient).
    * The optimization community has studied the general problem of optimizing smooth non-linear functions for many years.
   * Multilayer neural nets are not typical of the problems they study so their methods may need a lot of modifications.

* For large neural networks with very large and highly redundant training sets, it is nearly always best to use **mini-batch learning**.
    * The mini-batches may need to be quite big when adapting fancy methods.
    * Big mini-batches are more computationaly efficient.

# Mini-Batch Gradient Descent

* We start by guessing a learning rate.
    * If the error keeps getting worse or oscillates wildly, reduce the learning rate.
    * If the error is falling fairly consistently but slowly, increase the learning rate.

* We can write a simple routine to automate this way of adjusting the learning rate.

* Towards the end of mini-batch learning it nearly always helps to turn down the learning rate.
    * This removes fluctuations in the final weights caused by the variations between mini-batches.

* Turn down the learning rate when the error stops decreasing.
    * Use the error on a separate validation set.

# A Bag of Tricks for Mini-Batch Gradient Descent

**Initializing the Weights**

* If two hidden units have exactly the same bias and exactly the same incoming and outgoing weights, they will always get exactly the same gradient.
    * So they can never learn to be different features.
    * We break symmetry by initializing the weights to have small random values.

* If a hidden unit has a big *fan-in*, small changes on many of its incoming weights can cause the learning to overshoot.
    * We generally want smaller incoming weights when the fan-in is big, so initialize the weights to be proportional to $\sqrt{\text{fan-in}}$.

**Shifting the Input (Demeaning)**
* When using the steepest descent, shifting the input values makes a big difference!
    * It usually helps to transform each component of the input vector so that it has zero mean over the whole training set.
* The hyperbolic tangent produces hidden activations that are roughly zero mean.
    * In this repect it's better than the logistic

**Scalling the Input (Unit Variance)**
* When using steepest descent, scaling then input values makes a big difference.
    * It usually helps to transform each component of the input vector over the whole training set.

**Decorrelate the Input components**
* For a linear neuron, we get a big win by decorrelating each component of the input from the other input components.

* There are several different ways to decorrelate inputs. A reasonable method is to use *Principal Component Analysis (PCA)*.
    * Drop the principal components with the smallest eigenvalues.
        * This achieves some dimensionality reduction.
    * Divide the remaining principal components by the square roots of their eigenvalues. For a linear neuron, this converts an axis aligned elliptical error surface into a circular one.

* For a circular error surface, the gradient points straight towards the minimum.

**Common Problems**

**Plateau**
* If we start with a very big learning rate, the weights of each hidden unit will all bcome very big and positive or very big and negative.
    * The error derivatives for the hidden units will all become tiny and the error will not decrease.
    * This is usualy a *plateau*, but can be misunderstood as a local minimum.

**Turning Down Learning Rate**
* Turning down the learning rate reduces the random fluctuations in the error due to the different gradients on different mini-batches.
    * So we get a quick win.
    * But then we get slower learning.
* So, don't turn down the learning rate too soon!

# Optimization for Training Networks

Here are a few ways to speed up mini-batch learning:

**Momentum**
* Instead of using the gradient to change *position* of the weight *particle*, use it to change the *velocity*.

**Adaptive Learning Rate**
* Use separate adaptive learning rates for each parameter.
* Slowly adjust the rate using the consistency of the gradient for that parameter.

**RMSProp (Root Mean Square Propagation)**
* Use separate adaptive learning rates for each parameter.
* Divide the learning rate for a weight by a running average of the magnitudes of recent gradients for that weight.
    * This is the mini-batch version of just using the sign of the gradient (method called **RProp** designed for full-batch learning).

**ADAM (Adaptive Moment Estimation)**
* Use separate adaptive learning rates for each parameter.
* Adaptation of RMSProp, running averages of both gradient and second moments are used

# The Momentum Learning

* The momentum learning can be applied to full-batch learning or mini-batch learning.
    * Probably the commonest recipe to learn deep neural nets is to use stochastic gradient descent with mini-batches combined with momentum.

* Imagine a ball on the error surface. The location of the ball in the horizontal plane represents the weight vector.
    * The ball starts off by following the gradient, but once it has velocity, it no longer does steepest descent.
    * Its momentum makes it keep going in the previous direction.
    * We need to introduce *viscosity* so that the velocity dies off when are getting closer to the solution.

* It damps oscillations in directions of high curvature by combining gradients with opposite signs.

* It builds up speed in directions with a gentle but consistent gradient.

\begin{align}
\Delta w_{ji}^{(t)} &= \alpha \Delta w_{ji}^{(t-1)} + \eta  \nabla J(w_{ji}^{(t)})\\
&= \alpha \Delta w_{ji}^{(t-1)} + \eta \delta_j^{(t)} y_i^{(t)}
\end{align}

The effect of the gradient is to increment the previous velocity. The velocity also decays by $\alpha$ which is slightly less than 1 (generally, $\alpha=0.9$).

* At the beginning of learning there may be very large gradients.
    * So it pays off to use a small momentum (e.g. $\alpha=0.5$).
    * Once the large gradients have disappeared and the weights are stuck in a ravine the momentum can be smoothly raised to its final value (e.g. $\alpha=0.9$ or even $0.99$).

* This allows us to learn at a rate that would have caused divergent oscillations without momentum (case of increased learning rate only).

## Nesterov's Accelerated Gradient Descent

* The standard momentum *first* computes the gradient at the current location and *then* takes a big jump in the direction of the updated accumulated gradient.

* Ilya Sutskever (2012 unpublished) suggested a new form of momentum that often works better.
    * Inspired by the Nesterov method for optimizing convex functions.
* *First* make a big jump in the direction of the previous accumulated gradient.

* *Then* measure the gradient where you end up and make a correction.

$$\Delta w_{ji}^{(t)} = \alpha \Delta w_{ji}^{(t-1)} + \eta \nabla J(w_{ji}^{(t-1)} + \eta \Delta w_{ji}^{(t-1)})$$

# (General) Adaptive Learning Rate

* In multilayer neural networks, the appropriate learning rates can vary widely between weights:
    * The magnitude of the gradient are often very different for different layers, especially if the initial weights are small.
    * The fan-in of a unit determines the size of the "overshoot" effects caused by simultaneously changing many of the incoming weights of a unit to correct the same error.

* So use a global learning rate multiplied by an appropriate local gain that is determined empirically for each weight.

* Start with a local gain of 1 for every weight.

* Increase the local gain if the gradient for that weight does not change sign.

* Use small additive increases and multiplicative decreases.
    * This ensures that big gains decay rapidly when oscillations start.
    * If the gradient is totally random the gain will hover around 1 when we increase by plus $\delta$ half the time and decrease by times $1-\delta$ half the time.

$$\Delta w_{ji} = -\eta g_{ji} \nabla J(w_{ji})$$

\begin{align}
\text{If } &\left(\nabla J(w_{ji}^{(t)}) \times \nabla J(w_{ji}^{(t-1)})\right) \geq 0 \\
&\text{then } g_{ji}(t) = g_{ji}(t-1) + \delta \\
&\text{else } g_{ji}(t) = g_{ji}(t-1) \times \delta \\
\end{align}

* Need to limit the gains to lie in some reasonable range, e.g. $[0.1,10]$ or $[0.01,100]$.

* Use full batch learning or very large mini-batches.
    * This ensures that changes in the sign of the gradient are not mainly due to the sampling error of a mini-batch.

* Adaptive learning rates can be combined with momentum (Jacobs, 1989).

# RProp

* RProp stands for *Resilient BackPropagation*.

* The magnitude of the gradient can be very different for different weights and can change during learning.
    * This make it hard to choose a single glocal learning rate.

* For full-batch learning, we can deal with this variation by only using the sign of the gradient.
    * The weight updates are all of the same magnitude.
    * This escapes from plateaus with tiny gradients quickly.

$$\Delta w_{ji} = -\eta g_{ji}\text{sign}\left( \nabla J(w_{ji})\right)$$

\begin{align}
\text{If } &\left(\nabla J(w_{ji}^{(t)}) \times \nabla J(w_{ji}^{(t-1)})\right) \geq 0 \\
& \text{then } g_{ji}(t) = g_{ji}(t-1)
\times \delta_1 \\
& \text{else } g_{ji}(t) = g_{ji}(t-1) \times \delta_2
\end{align}

* RProp combines the idea of only using the sign of the gradient with the idea of adapting the learning rate separately for each weight.
    * Increase the learning rate for a weight multiplicatively if the signs of its last two gradients agree.
    * Otherwise decrease the step size multiplicatively.

* Use full batch learning or very big mini-batches.
   * This ensures that changes in the sign of the gradient are not mainly due to the sampling error of a mini-batch.

# RMSProp

* RProp is equivalent to using the gradient but also dividing by the size of the gradient.
    * The problem with mini-batch RProp is that we divide by a different number for each mini-batch.

* RMSProp keeps a moving average of the squared gradient for each weight.

$$\Delta w_{ji}^{(t)} = \gamma \Delta w_{ji}^{(t-1)} + (1-\gamma) \left(\nabla J(w_{ji}^{(t)})\right)^2$$

$$w_{ji}^{(t+1)} = w_{ji}^{(t)} - \frac{\eta}{\sqrt{\Delta w_{ji}^{(t)}}} \nabla J(w_{ji}^{(t)}) $$

# ADAM

* ADAM is a combination of RMSProp and momentum.

* ADAM keeps both moving average of the gradient and the squared gradient.

* Adam includes bias corrections to the estimates of both the ﬁrst-order moments (the momentum term) and the (uncentered) second-order moments to account for their initializationat the origin.
    * Thus, unlike in ADAM, the RMSProp second-order moment estimate may have high biasearly in training.

$$m_w^{(t+1)} = \beta_1 m_w^{(t)} + (1-\beta_1)\nabla J(w^{(t)})$$

$$v_w^{(t+1)} = \beta_2 v_w^{(t)} + (1-\beta_2)\left(\nabla J(w^{(t)})\right)^2$$

$$\hat{m}_w = \frac{m_w^{(t+1)}}{1 - \beta_1^{t+1}}\text{ (bias correction)}$$

$$\hat{v}_w = \frac{v_w^{(t+1)}}{1 - \beta_2^{t+1}}\text{ (bias correction)}$$

$$w^{(t+1)} = w^{(t)} - \eta \frac{\hat{m}_w}{\sqrt{\hat{v}_m} + \epsilon}$$

* Often used $\beta_1 = 0.9$, $\beta_2 = 0.999$ and $\epsilon = 10^{-8}$.

# Summary of Learning Methods for Neural Networks

* For small datasets (e.g. 10,000 samples) or bigger datasets without much redundacy, use a full-batch method.
    * AdaGrad, RProp, ...

* For big, redundant datasets use mini-batches.
    * Try gradient descent with momentum.
    * Try RMSProp.
    * Try ADAM.

*Why there is no simple recipe?*

* There are lots of different network architectures
* Tasks differ a lot
    * Some require very accurate weights, some don't.
    * Some have many very rare cases (e.g. words).

# Recommended Reading

Chapter 8 "Optimization for Training Deep Models" from Deep Learning by Ian Goodfellow
* http://www.deeplearningbook.org/

```python

```


```python
import sympy as sy
from IPython.display import display
sy.init_printing()

A = sy.Array([
    [
        [1, 2, 3], [4, 5, 6], [7, 8, 9]
    ],
    [
        [10, 11, 12], [13, 14, 15], [16, 17, 18]
    ],
    [
        [19, 20, 21], [22, 23, 24], [25, 26, 27]
    ]
])

A
```

$\displaystyle \left[\begin{matrix}\left[\begin{matrix}1 & 2 & 3\\4 & 5 & 6\\7 & 8 & 9\end{matrix}\right] & \left[\begin{matrix}10 & 11 & 12\\13 & 14 & 15\\16 & 17 & 18\end{matrix}\right] & \left[\begin{matrix}19 & 20 & 21\\22 & 23 & 24\\25 & 26 & 27\end{matrix}\right]\end{matrix}\right]$

```python

```

```python
A[0, :, :]
```

$\displaystyle \left[\begin{matrix}1 & 2 & 3\\4 & 5 & 6\\7 & 8 & 9\end{matrix}\right]$

```python
type(sy.Matrix(A[0, :, :]))
```

    sympy.matrices.dense.MutableDenseMatrix

```python
B = sy.Matrix([[1, 2, 3]]).T

sy.Matrix(A[0, :, :]) * B
```

$\displaystyle \left[\begin{matrix}14\\32\\50\end{matrix}\right]$

```python
b = sy.randMatrix(3,3)
c = sy.randMatrix(3,3)
b
```

$\displaystyle \left[\begin{matrix}83 & 14 & 18\\29 & 28 & 89\\77 & 90 & 92\end{matrix}\right]$

```python
c
```

$\displaystyle \left[\begin{matrix}30 & 3 & 72\\88 & 31 & 13\\53 & 73 & 1\end{matrix}\right]$

```python
d = sy.tensor.Array([b,c], (2, 3, 3))
d
```

$\displaystyle \left[\begin{matrix}\left[\begin{matrix}83 & 14 & 18\\29 & 28 & 89\\77 & 90 & 92\end{matrix}\right] & \left[\begin{matrix}30 & 3 & 72\\88 & 31 & 13\\53 & 73 & 1\end{matrix}\right]\end{matrix}\right]$

```python
d = sy.permutedims(d, (1,0,2))
d
```

$\displaystyle \left[\begin{matrix}\left[\begin{matrix}83 & 14 & 18\\30 & 3 & 72\end{matrix}\right] & \left[\begin{matrix}29 & 28 & 89\\88 & 31 & 13\end{matrix}\right] & \left[\begin{matrix}77 & 90 & 92\\53 & 73 & 1\end{matrix}\right]\end{matrix}\right]$

```python
B = sy.tensor.Array(b, (1, 3, 3))
B = B.reshape(3, 3, 1)
B
```

$\displaystyle \left[\begin{matrix}\left[\begin{matrix}83\\14\\18\end{matrix}\right] & \left[\begin{matrix}29\\28\\89\end{matrix}\right] & \left[\begin{matrix}77\\90\\92\end{matrix}\right]\end{matrix}\right]$

```python
C = sy.tensor.Array(c, (1, 3, 3))
C = C.reshape(3, 3, 1)
C
```

$\displaystyle \left[\begin{matrix}\left[\begin{matrix}30\\3\\72\end{matrix}\right] & \left[\begin{matrix}88\\31\\13\end{matrix}\right] & \left[\begin{matrix}53\\73\\1\end{matrix}\right]\end{matrix}\right]$

```python
D = sy.tensor.Array([B, C])
D
```

$\displaystyle \left[\begin{matrix}\left[\begin{matrix}83\\14\\18\end{matrix}\right] & \left[\begin{matrix}29\\28\\89\end{matrix}\right] & \left[\begin{matrix}77\\90\\92\end{matrix}\right]\\\left[\begin{matrix}30\\3\\72\end{matrix}\right] & \left[\begin{matrix}88\\31\\13\end{matrix}\right] & \left[\begin{matrix}53\\73\\1\end{matrix}\right]\end{matrix}\right]$

```python
D.shape
```

```python
D = sy.tensorcontraction(D, (2, 3, 3))
```


# Taylor Expanding the Non-linearities

```python
import sympy as sp
```

```python
# Define the symbols
gamma_max = sp.Symbol('\gamma_{max}')
nu_max = sp.Symbol(r'\nu_{max}')
Kd = sp.Symbol('K_D')
phi_R = sp.Symbol('\phi_R')
phi_O = sp.Symbol('\phi_O')
lam = sp.Symbol('\lambda')
phi_P = 1 - phi_R

# Define the self replciator steady state equations
c_AA = (nu_max * phi_P / lam) - 1
lam_eq = gamma_max * phi_R * (c_AA / (c_AA + Kd))
phiR_soln = sp.solve(lam_eq - lam, phi_R)[0]
```

```python
def taylor(function, x0, n, x):
    i = 0
    p = 0
    while i <= n:
        p = p + (function.diff(x, i).subs(x, x0))/(sp.factorial(i))*(x - x0)**i
        i += 1
    return p
phiR_expand = taylor(phiR_soln, x=lam, x0=0, n=2)

```

```python
phiR_expand
```

$\displaystyle - \frac{\lambda^{2} \left(- 4 K_{D} \gamma_{max} \nu_{max} + \gamma_{max}^{2} + 2 \gamma_{max} \nu_{max} + \nu_{max}^{2} - \frac{\left(\gamma_{max}^{2} \nu_{max} + \gamma_{max} \nu_{max}^{2}\right)^{2}}{\gamma_{max}^{2} \nu_{max}^{2}}\right)}{4 \gamma_{max} \nu_{max} \sqrt{\gamma_{max}^{2} \nu_{max}^{2}}} + \frac{\lambda \left(- \gamma_{max} + \nu_{max} - \frac{- \gamma_{max}^{2} \nu_{max} - \gamma_{max} \nu_{max}^{2}}{\sqrt{\gamma_{max}^{2} \nu_{max}^{2}}}\right)}{2 \gamma_{max} \nu_{max}} + \frac{\gamma_{max} \nu_{max} - \sqrt{\gamma_{max}^{2} \nu_{max}^{2}}}{2 \gamma_{max} \nu_{max}}$

```python

```


# Timber Harvesting Model - Cubic Spline Approximation

**Randall Romero Aguilar, PhD**

This demo is based on the original Matlab demo accompanying the  Computational Economics and Finance 2001 textbook by Mario Miranda and Paul Fackler.

Original (Matlab) CompEcon file: **demdp01.m**

Running this file requires the Python version of CompEcon. This can be installed with pip by running

    !pip install compecon --upgrade

Last updated: 2021-Oct-01

## About

Profit maximizing owner of a commercial tree stand must decide when to clearcut the stand.

```python
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
from compecon import DPmodel, BasisSpline, demo, NLP
```

### Model Parameters
Assuming that the unit price of biomass is $p=1$, the cost to clearcut-replant is $\kappa=0.2$, the stand carrying capacity $s_{\max}=0.5$, biomass growth factor is $\gamma=10\%$ per period, and the annual discount factor $\delta=0.9$.

```python
price = 1.0
kappa = 0.2
smax  = 0.5
gamma = 0.1
delta = 0.9
```

### State Space
The state variable is the stand biomass, $s\in [0,s_{\max}]$.

Here, we represent it with a cubic spline basis, with $n=200$ nodes.

```python
n = 200
basis = BasisSpline(n, 0, smax, labels=['biomass'])
```

### Action Space
The action variable is $j\in\{0:\text{'keep'},\quad 1:\text{'clear cut'}\}$.

```python
options = ['keep', 'clear-cut']
```

### Reward function
If the farmer clears the stand, the profit is the value of selling the biomass $ps$ minus the cost of clearing and replanting $\kappa$, otherwise the profit is zero.

```python
def reward(s, x, i , j):
    return (price * s - kappa) * j
```

### State Transition Function
If the farmer clears the stand, it begins next period with $\gamma s_{\max}$ units. If he keeps the stand, then it grows to $s + \gamma (s_{\max} - s)$.

```python
def transition(s, x, i, j, in_, e):
    if j:
        return np.full_like(s, gamma * smax)
    else:
        return s + gamma * (smax - s)
```

### Model Structure
The value of the stand, given that it contains $s$ units of biomass at the beginning of the period, satisfies the Bellman equation

\begin{equation} V(s) = \max\left\{(ps-\kappa) + \delta V(\gamma s_{\max}),\quad \delta V[s+\gamma(s_{\max}-s)]  \right\}   \end{equation}

To solve and simulate this model, use the CompEcon class ```DPmodel```.

```python
model = DPmodel(basis, reward, transition,
                discount=delta,
                j=options)

S = model.solve()
```

    Solving infinite-horizon model collocation equation by Newton's method
    iter change       time
    ------------------------------
       0       3.1e-01    0.0160
       1       1.1e-01    0.0209
       2       1.1e-02    0.0259
       3       4.6e-03    0.0309
       4       1.6e-16    0.0389
    Elapsed Time =    0.04 Seconds

The ```solve``` method retuns a pandas ```DataFrame```, which can easily be used to make plots. To see the first 10 entries of `S`, we use the `head` method

```python
S.head()
```

      biomass
      value
      resid
      j*
      value[keep]
      value[clear-cut]

      biomass

      0.000000
      0.000000
      0.067287
      -1.387779e-17
      keep
      0.067287
      -0.132713

      0.000250
      0.000250
      0.067320
      -7.491474e-09
      keep
      0.067320
      -0.132462

      0.000500
      0.000500
      0.067353
      -5.953236e-09
      keep
      0.067353
      -0.132212

      0.000750
      0.000750
      0.067387
      -1.332984e-09
      keep
      0.067387
      -0.131962

      0.001001
      0.001001
      0.067420
      7.309628e-10
      keep
      0.067420
      -0.131712

## Analysis

To find the biomass level at which it is indifferent to keep or to clear cut, we interpolate as follows:

```python
scrit = NLP(lambda s: model.Value_j(s).dot([1,-1])).broyden(0.0)[0]
vcrit = model.Value(scrit)
print(f'When the biomass is {scrit:.5f} its value is {vcrit:.5f} regardless of the action taken by the manager')
```

    When the biomass is 0.30669 its value is 0.17402 regardless of the action taken by the manager

### Plot the Value Function and Optimal Action

```python
fig1 = plt.figure(figsize=[8,5])
gs = gridspec.GridSpec(2, 1,height_ratios=[3.5, 1])

S[['value[keep]', 'value[clear-cut]']].plot(ax= plt.subplot(gs[0]))
plt.title('Action-Contingent Value Functions')
plt.xlabel('')
plt.ylabel('Value of Stand')
plt.xticks([])

ymin,ymax = plt.ylim()
plt.vlines(scrit,ymin, vcrit,linestyle=':')
demo.annotate(scrit, ymin,'$s^*$',ms=0)

S['j*'].cat.codes.plot(ax=plt.subplot(gs[1]))
plt.title('Optimal Action')
plt.ylabel('Action')
plt.ylim([-0.25,1.25])
plt.yticks([0,1],options)
plt.legend([])
```

### Plot Residuals

```python
S['resid2'] = 100*S.resid / S.value

fig2 = demo.figure('Bellman Equation Residual','','Percent Residual')
plt.plot(S['resid2'])
plt.hlines(0,0,smax,'k')
```

###  Simulation

The path followed by the biomass is computed by the ```simulate()``` method. Here we simulate 32 periods starting with a biomass level $s_0 = 0$.

```python
H = model.simulate(32, 0.0)

fig3 = demo.figure('Timber harvesting simulation','Period','Biomass')
plt.plot(H['biomass'])

```

```python
#demo.savefig([fig1, fig2, fig3])
```

_build/jupyter_execute/notebooks/dp/01 Timber Harvesting Model - Cubic Spline Approximation.ipynb


```python
from sympy import *
init_printing(use_unicode=True)
```

```python
Matrix([[1,-1],[3,4],[0,2]])
Matrix([1,2,3])
```

$$\left[\begin{matrix}1 & -1\\3 & 4\\0 & 2\end{matrix}\right]$$

$$\left[\begin{matrix}1\\2\\3\end{matrix}\right]$$

```python
M = Matrix([[1,2,3],[3,2,1]])
N = Matrix([0,1,1])

M
M.shape
```