# Homework 1
*This notebook includes both coding and written questions. Please hand in this notebook file with all the outputs and your answers to the written questions.*

This assignment covers linear filters, convolution and correlation.

```python
# Setup
from __future__ import print_function
import numpy as np
import matplotlib.pyplot as plt
from time import time
from skimage import io

%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

# for auto-reloading extenrnal modules
%load_ext autoreload
%autoreload 2
```

    The autoreload extension is already loaded. To reload it, use:
      %reload_ext autoreload

## Part 1: Convolutions
### 1.1 Commutative Property (10 points)
Recall that the convolution of an image $f:\mathbb{R}^2\rightarrow \mathbb{R}$ and a kernel $h:\mathbb{R}^2\rightarrow\mathbb{R}$ is defined as follows:
$$(f*h)[m,n]=\sum_{i=-\infty}^\infty\sum_{j=-\infty}^\infty f[i,j]\cdot h[m-i,n-j]$$

Or equivalently,
\begin{align}
(f*h)[m,n] &= \sum_{i=-\infty}^\infty\sum_{j=-\infty}^\infty h[i,j]\cdot f[m-i,n-j]\\
&= (h*f)[m,n]
\end{align}

Show that this is true (i.e. prove that the convolution operator is commutative: $f*h = h*f$).

**Your Answer:** *Write your solution in this markdown cell. Please write your equations in [LaTex equations](http://jupyter-notebook.readthedocs.io/en/latest/examples/Notebook/Typesetting%20Equations.html).*

### 1.2 Shift Invariance (10 points)
Let $f$ be a function $\mathbb{R}^2\rightarrow\mathbb{R}$. Consider a system $f\xrightarrow{s}g$, where $g=(f*h)$ with some kernel $h:\mathbb{R}^2\rightarrow\mathbb{R}$. Also consider functions $f'(m,n) = f(m-m_0, n-n_0)$ and $g'(m,n) = g(m-m_0, n-n_0)$.

Show that $S$ defined by any kernel $h$ is a Shift Invariant system by showing that $g' = (f'*h)$.

**Your Answer:** *Write your solution in this markdown cell. Please write your equations in [LaTex equations](http://jupyter-notebook.readthedocs.io/en/latest/examples/Notebook/Typesetting%20Equations.html).*

### 1.3 Implementation (30 points)

In this section, you will implement two versions of convolution:
- `conv_nested`
- `conv_fast`

First, run the code cell below to load the image to work with.

```python
# Open image as grayscale
img = io.imread('dog.jpg', as_grey=True)

# Show image
plt.imshow(img)
plt.axis('off')
plt.title("Isn't he cute?")
plt.show()
```

Now, implement the function **`conv_nested`** in **`filters.py`**. This is a naive implementation of convolution which uses 4 nested for-loops. It takes an image $f$ and a kernel $h$ as inputs and outputs the convolved image $(f*h)$ that has the same shape as the input image. This implementation should take a few seconds to run.

*- Hint: It may be easier to implement $(h*f)$*

We'll first test your `conv_nested` function on a simple input.

```python
from filters import conv_nested

# Simple convolution kernel.
kernel = np.array(
[
    [1,0,1],
    [0,0,0],
    [1,0,0]
])

# Create a test image: a white square in the middle
test_img = np.zeros((9, 9))
test_img[3:6, 3:6] = 1

# Run your conv_nested function on the test image
test_output = conv_nested(test_img, kernel)

# Build the expected output
expected_output = np.zeros((9, 9))
expected_output[2:7, 2:7] = 1
expected_output[5:, 5:] = 0
expected_output[4, 2:5] = 2
expected_output[2:5, 4] = 2
expected_output[4, 4] = 3

# Plot the test image
plt.subplot(1,3,1)
plt.imshow(test_img)
plt.title('Test image')
plt.axis('off')

# Plot your convolved image
plt.subplot(1,3,2)
plt.imshow(test_output)
plt.title('Convolution')
plt.axis('off')

# Plot the exepected output
plt.subplot(1,3,3)
plt.imshow(expected_output)
plt.title('Exepected output')
plt.axis('off')
plt.show()

# Test if the output matches expected output
assert np.max(test_output - expected_output) < 1e-10, "Your solution is not correct."
```

Now let's test your `conv_nested` function on a real image.

```python
from filters import conv_nested

# Simple convolution kernel.
# Feel free to change the kernel to see different outputs.
kernel = np.array(
[
    [1,0,-1],
    [2,0,-2],
    [1,0,-1]
])

out = conv_nested(img, kernel)

# Plot original image
plt.subplot(2,2,1)
plt.imshow(img)
plt.title('Original')
plt.axis('off')

# Plot your convolved image
plt.subplot(2,2,3)
plt.imshow(out)
plt.title('Convolution')
plt.axis('off')

# Plot what you should get
solution_img = io.imread('convoluted_dog.jpg', as_grey=True)
plt.subplot(2,2,4)
plt.imshow(solution_img)
plt.title('What you should get')
plt.axis('off')

plt.show()
```

Let us implement a more efficient version of convolution using array operations in numpy. As shown in the lecture, a convolution can be considered as a sliding window that computes sum of the pixel values weighted by the flipped kernel. The faster version will i) zero-pad an image, ii) flip the kernel horizontally and vertically, and iii) compute weighted sum of the neighborhood at each pixel.

First, implement the function **`zero_pad`** in **`filters.py`**.

```python
from filters import zero_pad

pad_width = 20 # width of the padding on the left and right
pad_height = 40 # height of the padding on the top and bottom

padded_img = zero_pad(img, pad_height, pad_width)

# Plot your padded dog
plt.subplot(1,2,1)
plt.imshow(padded_img)
plt.title('Padded dog')
plt.axis('off')

# Plot what you should get
solution_img = io.imread('padded_dog.jpg', as_grey=True)
plt.subplot(1,2,2)
plt.imshow(solution_img)
plt.title('What you should get')
plt.axis('off')

plt.show()
```

Next, complete the function **`conv_fast`** in **`filters.py`** using `zero_pad`. Run the code below to compare the outputs by the two implementations. `conv_fast` should run significantly faster than `conv_nested`.
Depending on your implementation and computer, `conv_nested` should take a few seconds and `conv_fast` should be around 5 times faster.

```python
from filters import conv_fast

t0 = time()
out_fast = conv_fast(img, kernel)
t1 = time()
out_nested = conv_nested(img, kernel)
t2 = time()

# Compare the running time of the two implementations
print("conv_nested: took %f seconds." % (t2 - t1))
print("conv_fast: took %f seconds." % (t1 - t0))

# Plot conv_nested output
plt.subplot(1,2,1)
plt.imshow(out_nested)
plt.title('conv_nested')
plt.axis('off')

# Plot conv_fast output
plt.subplot(1,2,2)
plt.imshow(out_fast)
plt.title('conv_fast')
plt.axis('off')

# Make sure that the two outputs are the same
if not (np.max(out_fast - out_nested) < 1e-10):
    print("Different outputs! Check your implementation.")
```

### Note for 2D image convolution
1. Flip the kernel by row, then by column
2. slide the kernel over the image
3. if it is required the output shape = input shape, add zero padding

For any kernel size k, p_row = p_column =
\begin{align}
p & = (k - 1)/2
\end{align}

### Extra Credit 1 (10 points)
Devise a faster version of convolution and implement **`conv_faster`** in **`filters.py`**. You will earn extra credit only if the `conv_faster` runs faster (by a fair margin) than `conv_fast` **and** outputs the same result.

```python
from filters import conv_faster

t0 = time()
out_fast = conv_fast(img, kernel)
t1 = time()
out_faster = conv_faster(img, kernel)
t2 = time()

# Compare the running time of the two implementations
print("conv_fast: took %f seconds." % (t1 - t0))
print("conv_faster: took %f seconds." % (t2 - t1))

# Plot conv_nested output
plt.subplot(1,2,1)
plt.imshow(out_fast)
plt.title('conv_fast')
plt.axis('off')

# Plot conv_fast output
plt.subplot(1,2,2)
plt.imshow(out_faster)
plt.title('conv_faster')
plt.axis('off')

# Make sure that the two outputs are the same
if not (np.max(out_fast - out_faster) < 1e-10):
    print("Different outputs! Check your implementation.")
```

---
## Part 2: Cross-correlation

Cross-correlation of two 2D signals $f$ and $g$ is defined as follows:
$$(f\star{g})[m,n]=\sum_{i=-\infty}^\infty\sum_{j=-\infty}^\infty f[i,j]\cdot g[i-m,j-n]$$

### 2.1 Template Matching with Cross-correlation (12 points)
Suppose that you are a clerk at a grocery store. One of your responsibilites is to check the shelves periodically and stock them up whenever there are sold-out items. You got tired of this laborious task and decided to build a computer vision system that keeps track of the items on the shelf.

Luckily, you have learned in CS131 that cross-correlation can be used for template matching: a template $g$ is multiplied with regions of a larger image $f$ to measure how similar each region is to the template.

The template of a product (`template.jpg`) and the image of shelf (`shelf.jpg`) is provided. We will use cross-correlation to find the product in the shelf.

Implement **`cross_correlation`** function in **`filters.py`** and run the code below.

*- Hint: you may use the `conv_fast` function you implemented in the previous question.*

```python
from filters import cross_correlation

# Load template and image in grayscale
img = io.imread('shelf.jpg')
img_grey = io.imread('shelf.jpg', as_grey=True)
temp = io.imread('template.jpg')
temp_grey = io.imread('template.jpg', as_grey=True)

# Perform cross-correlation between the image and the template
out = cross_correlation(img_grey, temp_grey)

# Find the location with maximum similarity
y,x = (np.unravel_index(out.argmax(), out.shape))

# Display product template
plt.figure(figsize=(25,20))
plt.subplot(3, 1, 1)
plt.imshow(temp)
plt.title('Template')
plt.axis('off')

# Display cross-correlation output
plt.subplot(3, 1, 2)
plt.imshow(out)
plt.title('Cross-correlation (white means more correlated)')
plt.axis('off')

# Display image
plt.subplot(3, 1, 3)
plt.imshow(img)
plt.title('Result (blue marker on the detected location)')
plt.axis('off')

# Draw marker at detected location
plt.plot(x, y, 'bx', ms=40, mew=10)
plt.show()

```

#### Interpretation
How does the output of cross-correlation filter look? Was it able to detect the product correctly? Explain what problems there might be with using a raw template as a filter.

**Your Answer:** *Write your solution in this markdown cell.*

---
### 2.2 Zero-mean cross-correlation (6 points)
A solution to this problem is to subtract the mean value of the template so that it has zero mean.

Implement **`zero_mean_cross_correlation`** function in **`filters.py`** and run the code below.

```python
from filters import zero_mean_cross_correlation

# Perform cross-correlation between the image and the template
out = zero_mean_cross_correlation(img_grey, temp_grey)

# Find the location with maximum similarity
y,x = (np.unravel_index(out.argmax(), out.shape))

# Display product template
plt.figure(figsize=(30,20))
plt.subplot(3, 1, 1)
plt.imshow(temp)
plt.title('Template')
plt.axis('off')

# Display cross-correlation output
plt.subplot(3, 1, 2)
plt.imshow(out)
plt.title('Cross-correlation (white means more correlated)')
plt.axis('off')

# Display image
plt.subplot(3, 1, 3)
plt.imshow(img)
plt.title('Result (blue marker on the detected location)')
plt.axis('off')

# Draw marker at detcted location
plt.plot(x, y, 'bx', ms=40, mew=10)
plt.show()

```

You can also determine whether the product is present with appropriate scaling and thresholding.

```python
def check_product_on_shelf(shelf, product):
    out = zero_mean_cross_correlation(shelf, product)

    # Scale output by the size of the template
    out = out / float(product.shape[0]*product.shape[1])

    # Threshold output (this is arbitrary, you would need to tune the threshold for a real application)
    out = out > 0.025

    if np.sum(out) > 0:
        print('The product is on the shelf')
    else:
        print('The product is not on the shelf')

# Load image of the shelf without the product
img2 = io.imread('shelf_soldout.jpg')
img2_grey = io.imread('shelf_soldout.jpg', as_grey=True)

plt.imshow(img)
plt.axis('off')
plt.show()
check_product_on_shelf(img_grey, temp_grey)

plt.imshow(img2)
plt.axis('off')
plt.show()
check_product_on_shelf(img2_grey, temp_grey)
```

---
### 2.3 Normalized Cross-correlation (12 points)
One day the light near the shelf goes out and the product tracker starts to malfunction. The `zero_mean_cross_correlation` is not robust to change in lighting condition. The code below demonstrates this.

```python
from filters import normalized_cross_correlation

# Load image
img = io.imread('shelf_dark.jpg')
img_grey = io.imread('shelf_dark.jpg', as_grey=True)

# Perform cross-correlation between the image and the template
out = zero_mean_cross_correlation(img_grey, temp_grey)

# Find the location with maximum similarity
y,x = (np.unravel_index(out.argmax(), out.shape))

# Display image
plt.imshow(img)
plt.title('Result (red marker on the detected location)')
plt.axis('off')

# Draw marker at detcted location
plt.plot(x, y, 'rx', ms=25, mew=5)
plt.show()
```

A solution is to normalize the pixels of the image and template at every step before comparing them. This is called **normalized cross-correlation**.

The mathematical definition for normalized cross-correlation of $f$ and template $g$ is:
$$(f\star{g})[m,n]=\sum_{i,j} \frac{f[i,j]-\overline{f_{m,n}}}{\sigma_{f_{m,n}}} \cdot \frac{g[i-m,j-n]-\overline{g}}{\sigma_g}$$

where:
- $f_{m,n}$ is the patch image at position $(m,n)$
- $\overline{f_{m,n}}$ is the mean of the patch image $f_{m,n}$
- $\sigma_{f_{m,n}}$ is the standard deviation of the patch image $f_{m,n}$
- $\overline{g}$ is the mean of the template $g$
- $\sigma_g$ is the standard deviation of the template $g$

Implement **`normalized_cross_correlation`** function in **`filters.py`** and run the code below.

```python
from filters import normalized_cross_correlation

# Perform normalized cross-correlation between the image and the template
out = normalized_cross_correlation(img_grey, temp_grey)

# Find the location with maximum similarity
y,x = (np.unravel_index(out.argmax(), out.shape))

# Display image
plt.imshow(img)
plt.title('Result (red marker on the detected location)')
plt.axis('off')

# Draw marker at detcted location
plt.plot(x, y, 'rx', ms=25, mew=5)
plt.show()
```

## Part 3: Separable Filters

### 3.1 Theory (10 points)
Consider an $M_1\times{N_1}$ image $I$ and an $M_2\times{N_2}$ filter $F$. A filter $F$ is **separable** if it can be written as a product of two 1D filters: $F=F_1F_2$.

For example,
$$F=
\begin{bmatrix}
1 & -1 \\
1 & -1
\end{bmatrix}
$$
can be written as a matrix product of
$$F_1=
\begin{bmatrix}
1  \\
1
\end{bmatrix},
F_2=
\begin{bmatrix}
1 & -1
\end{bmatrix}
$$
Therefore $F$ is a separable filter.

Prove that for any separable filter $F=F_1F_2$,
$$I*F=(I*F_1)*F_2$$

**Your Answer:** *Write your solution in this markdown cell. Please write your equations in [LaTex equations](http://jupyter-notebook.readthedocs.io/en/latest/examples/Notebook/Typesetting%20Equations.html).*

### 3.2 Complexity comparison (10 points)
Consider an $M_1\times{N_1}$ image $I$ and an $M_2\times{N_2}$ filter $F$ that is separable (i.e. $F=F_1F_2$).

(i) How many multiplication operations do you need to do a direct 2D convolution (i.e. $I*F$)?

(ii) How many multiplication operations do you need to do 1D convolutions on rows and columns (i.e. $(I*F_1)*F_2$)?

(iii) Use Big-O notation to argue which one is more efficient in general: direct 2D convolution or two successive 1D convolutions?

**Your Answer:** *Write your solution in this markdown cell. Please write your equations in [LaTex equations](http://jupyter-notebook.readthedocs.io/en/latest/examples/Notebook/Typesetting%20Equations.html).*

Now, we will empirically compare the running time of a separable 2D convolution and its equivalent two 1D convolutions. The Gaussian kernel, widely used for blurring images, is one example of a separable filter. Run the code below to see its effect.

```python
# Load image
img = io.imread('dog.jpg', as_grey=True)

# 5x5 Gaussian blur
kernel = np.array(
[
    [1,4,6,4,1],
    [4,16,24,16,4],
    [6,24,36,24,6],
    [4,16,24,16,4],
    [1,4,6,4,1]
])

t0 = time()
out = conv_nested(img, kernel)
t1 = time()
t_normal = t1 - t0

# Plot original image
plt.subplot(1,2,1)
plt.imshow(img)
plt.title('Original')
plt.axis('off')

# Plot convolved image
plt.subplot(1,2,2)
plt.imshow(out)
plt.title('Blurred')
plt.axis('off')

plt.show()
```

In the below code cell, define the two 1D arrays (`k1` and `k2`) whose product is equal to the Gaussian kernel.

```python
# The kernel can be written as outer product of two 1D filters
k1 = None  # shape (5, 1)
k2 = None  # shape (1, 5)

### YOUR CODE HERE
pass
### END YOUR CODE

# Check if kernel is product of k1 and k2
if not  np.all(k1 * k2 == kernel):
    print('k1 * k2 is not equal to kernel')

assert k1.shape == (5, 1), "k1 should have shape (5, 1)"
assert k2.shape == (1, 5), "k2 should have shape (1, 5)"
```

We now apply the two versions of convolution to the same image, and compare their running time. Note that the outputs of the two convolutions must be the same.

```python
# Perform two convolutions using k1 and k2
t0 = time()
out_separable = conv_nested(img, k1)
out_separable = conv_nested(out_separable, k2)
t1 = time()
t_separable = t1 - t0

# Plot normal convolution image
plt.subplot(1,2,1)
plt.imshow(out)
plt.title('Normal convolution')
plt.axis('off')

# Plot separable convolution image
plt.subplot(1,2,2)
plt.imshow(out_separable)
plt.title('Separable convolution')
plt.axis('off')

plt.show()

print("Normal convolution: took %f seconds." % (t_normal))
print("Separable convolution: took %f seconds." % (t_separable))
```

```python
# Check if the two outputs are equal
assert np.max(out_separable - out) < 1e-10
```


# Poiseuille flow

$$
\renewcommand{\DdQq}[2]{{\mathrm D}_{#1}{\mathrm Q}_{#2}}
\renewcommand{\drondt}{\partial_t}
\renewcommand{\drondx}{\partial_x}
\renewcommand{\drondy}{\partial_y}
\renewcommand{\drondtt}{\partial_{tt}}
\renewcommand{\drondxx}{\partial_{xx}}
\renewcommand{\drondyy}{\partial_{yy}}
\renewcommand{\dx}{\Delta x}
\renewcommand{\dt}{\Delta t}
\renewcommand{\grandO}{{\mathcal O}}
\renewcommand{\density}[2]{\,f_{#1}^{#2}}
\renewcommand{\fk}[1]{\density{#1}{\vphantom{\star}}}
\renewcommand{\fks}[1]{\density{#1}{\star}}
\renewcommand{\moment}[2]{\,m_{#1}^{#2}}
\renewcommand{\mk}[1]{\moment{#1}{\vphantom{\star}}}
\renewcommand{\mke}[1]{\moment{#1}{e}}
\renewcommand{\mks}[1]{\moment{#1}{\star}}
$$

In this tutorial, we consider the classical $\DdQq{2}{9}$ to simulate a Poiseuille flow modeling by the Navier-Stokes equations.

```python

from six.moves import range
%matplotlib inline
```

## The $\DdQq{2}{9}$ **for Navier-Stokes**

The $\DdQq{2}{9}$ is defined by:

* a space step $\dx$ and a time step $\dt$ related to the scheme velocity $\lambda$ by the relation $\lambda=\dx/\dt$,

* nine velocities $\{(0,0), (\pm1,0), (0,\pm1), (\pm1, \pm1)\}$, identified in pylbm by the
numbers $0$ to $8$,

* nine polynomials used to build the moments

$$ \{1, \lambda X, \lambda Y, 3E-4, (9E^2-21E+8)/2, 3XE-5X, 3YE-5Y,X^2-Y^2, XY\},$$

where $E = X^2+Y^2$.

* three conserved moments $\rho$, $q_x$, and $q_y$,

* nine relaxation parameters (three are $0$ corresponding to conserved moments): $\{0,0,0,s_\mu,s_\mu,s_\eta,s_\eta,s_\eta,s_\eta\}$, where $s_\mu$ and $s_\eta$ are in $(0,2)$,

* equilibrium value of the non conserved moments
$$
\begin{aligned}\mke{3} &= -2\rho + 3(q_x^2+q_y^2)/(\rho_0\lambda^2), \\ \mke{4} &= \rho-3(q_x^2+q_y^2)/(\rho_0\lambda^2), \\ \mke{5} &= -q_x/\lambda, \\ \mke{6} &= -q_y/\lambda, \\ \mke{7} &= (q_x^2-q_y^2)/(\rho_0\lambda^2), \\ \mke{8} &= q_xq_y/(\rho_0\lambda^2),\end{aligned}
$$

where $\rho_0$ is a given scalar.

This scheme is consistant at second order with the following equations (taken $\rho_0=1$)
$$
\begin{gathered} \drondt\rho + \drondx q_x + \drondy q_y = 0,\\ \drondt q_x + \drondx (q_x^2+p) + \drondy (q_xq_y) = \mu \drondx (\drondx q_x + \drondy q_y ) + \eta (\drondxx+\drondyy)q_x, \\ \drondt q_y + \drondx (q_xq_y) + \drondy (q_y^2+p) = \mu \drondy (\drondx q_x + \drondy q_y ) + \eta (\drondxx+\drondyy)q_y,\end{gathered}
$$
with $p=\rho\lambda^2/3$.

## Build the simulation with pylbm
In the following, we build the dictionary of the simulation step by step.

### The geometry

The simulation is done on a rectangle of length $L$ and width $W$. We can use $L=W=1$.

We propose a dictionary that build the geometry of the domain. The labels of the bounds can be specified to different values for the moment.

```python
import numpy as np
import matplotlib.pyplot as plt

import pylbm

L, W = 1., 1.
dico_geom = {'box':{'x':[0,L], 'y':[-.5*W,.5*W], 'label':list(range(4))}}
geom = pylbm.Geometry(dico_geom)
print(geom)
geom.visualize(viewlabel=True)
```

### The stencil

The stencil of the $\DdQq{2}{9}$ is composed by the nine following velocities in 2D:
$$
\begin{gathered}v_0=(0,0),\\ v_1=(1,0), \quad v_2=(0,1), \quad v_3=(-1,0), \quad v_4=(0,-1), \\ v_5=(1,1), \quad v_6=(-1,1), \quad v_7=(-1,-1), \quad v_8=(1,-1).\end{gathered}
$$

```python
dico_sten = {
    'dim':2,
    'schemes':[{'velocities':list(range(9))}],
}
sten = pylbm.Stencil(dico_sten)
print(sten)
sten.visualize()
```

### The domain

In order to build the domain of the simulation, the dictionary should contain the space step $\dx$ and the stencils of the velocities (one for each scheme).

```python
dico_dom = {
    'space_step':.1,
    'box':{'x':[0,L], 'y':[-.5*W,.5*W], 'label':list(range(4))},
    'schemes':[{'velocities':list(range(9))}],
}
dom = pylbm.Domain(dico_dom)
print(dom)
dom.visualize(view_distance=True)
```

### The scheme

In pylbm, a simulation can be performed by using several coupled schemes. In this example, a single scheme is used and defined through a list of one single dictionary. This dictionary should contain:

* 'velocities': a list of the velocities
* 'conserved_moments': a list of the conserved moments as sympy variables
* 'polynomials': a list of the polynomials that define the moments
* 'equilibrium': a list of the equilibrium value of all the moments
* 'relaxation_parameters': a list of the relaxation parameters ($0$ for the conserved moments)
* 'init': a dictionary to initialize the conserved moments

(see the documentation for more details)

In order to fix the bulk ($\mu$) and the shear ($\eta$) viscosities, we impose
$$ s_\eta = \frac{2}{1+\eta d}, \qquad s_\mu = \frac{2}{1+\mu d}, \qquad d = \frac{6}{\lambda\rho_0\dx}.$$

The scheme velocity could be taken to $1$ and the inital value of $\rho$ to $\rho_0=1$, $q_x$ and $q_y$ to $0$.

In order to accelerate the simulation, we can use another generator. The default generator is Numpy (pure python). We can use for instance Cython that generates a more efficient code. This generator can be activated by using  'generator': pylbm.generator.CythonGenerator in the dictionary.

```python
import sympy as sp
X, Y, rho, qx, qy, LA = sp.symbols('X, Y, rho, qx, qy, LA')

# parameters
dx = 1./128  # spatial step
la = 1.      # velocity of the scheme
L = 1        # length of the domain
W = 1        # width of the domain
rhoo = 1.    # mean value of the density
mu   = 1.e-3 # shear viscosity
eta = 1.e-1 # bulk viscosity
# initialization
xmin, xmax, ymin, ymax = 0.0, L, -0.5*W, 0.5*W
dummy = 3.0/(la*rhoo*dx)
s_mu = 1.0/(0.5+mu*dummy)
s_eta = 1.0/(0.5+eta*dummy)
s_q = s_eta
s_es = s_mu
s  = [0.,0.,0.,s_mu,s_es,s_q,s_q,s_eta,s_eta]
dummy = 1./(LA**2*rhoo)
qx2 = dummy*qx**2
qy2 = dummy*qy**2
q2  = qx2+qy2
qxy = dummy*qx*qy

dico_sch = {
    'box':{'x':[xmin, xmax], 'y':[ymin, ymax], 'label':0},
    'space_step':dx,
    'scheme_velocity':la,
    'parameters':{LA:la},
    'schemes':[
        {
            'velocities':list(range(9)),
            'conserved_moments':[rho, qx, qy],
            'polynomials':[
                1, LA*X, LA*Y,
                3*(X**2+Y**2)-4,
                (9*(X**2+Y**2)**2-21*(X**2+Y**2)+8)/2,
                3*X*(X**2+Y**2)-5*X, 3*Y*(X**2+Y**2)-5*Y,
                X**2-Y**2, X*Y
            ],
            'relaxation_parameters':s,
            'equilibrium':[
                rho, qx, qy,
                -2*rho + 3*q2,
                rho-3*q2,
                -qx/LA, -qy/LA,
                qx2-qy2, qxy
            ],
            'init':{rho:rhoo, qx:0., qy:0.},
        },
    ],
    'generator': 'cython',
}
sch = pylbm.Scheme(dico_sch)
print(sch)
```

    Scheme informations
    	 spatial dimension: dim=2
    	 number of schemes: nscheme=1
    	 number of velocities:
        Stencil.nv[0]=9
    	 velocities value:
        v[0] = (0: 0, 0), (1: 1, 0), (2: 0, 1), (3: -1, 0), (4: 0, -1), (5: 1, 1), (6: -1, 1), (7: -1, -1), (8: 1, -1),
    	 polynomials:
        P[0] = 1, LA*X, LA*Y, 3*X**2 + 3*Y**2 - 4, -21*X**2/2 - 21*Y**2/2 + 9*(X**2 + Y**2)**2/2 + 4, 3*X*(X**2 + Y**2) - 5*X, 3*Y*(X**2 + Y**2) - 5*Y, X**2 - Y**2, X*Y,
    	 equilibria:
        EQ[0] = rho, qx, qy, -2*rho + 3.0*qx**2/LA**2 + 3.0*qy**2/LA**2, rho - 3.0*qx**2/LA**2 - 3.0*qy**2/LA**2, -qx/LA, -qy/LA, 1.0*qx**2/LA**2 - 1.0*qy**2/LA**2, 1.0*qx*qy/LA**2,
    	 relaxation parameters:
        s[0] = 0.0, 0.0, 0.0, 1.13122171945701, 1.13122171945701, 0.0257069408740360, 0.0257069408740360, 0.0257069408740360, 0.0257069408740360,
    	 moments matrices
    M      = Matrix([[1, 1, 1, 1, 1, 1, 1, 1, 1], [0, 1.0*LA, 0, -1.0*LA, 0, 1.0*LA, -1.0*LA, -1.0*LA, 1.0*LA], [0, 0, 1.0*LA, 0, -1.0*LA, 1.0*LA, 1.0*LA, -1.0*LA, -1.0*LA], [-4, -1.00000000000000, -1.00000000000000, -1.00000000000000, -1.00000000000000, 2.00000000000000, 2.00000000000000, 2.00000000000000, 2.00000000000000], [4, -2.00000000000000, -2.00000000000000, -2.00000000000000, -2.00000000000000, 1.00000000000000, 1.00000000000000, 1.00000000000000, 1.00000000000000], [0, -2.00000000000000, 0, 2.00000000000000, 0, 1.00000000000000, -1.00000000000000, -1.00000000000000, 1.00000000000000], [0, 0, -2.00000000000000, 0, 2.00000000000000, 1.00000000000000, 1.00000000000000, -1.00000000000000, -1.00000000000000], [0, 1.00000000000000, -1.00000000000000, 1.00000000000000, -1.00000000000000, 0, 0, 0, 0], [0, 0, 0, 0, 0, 1.00000000000000, -1.00000000000000, 1.00000000000000, -1.00000000000000]])
    M^(-1) = Matrix([[0.111111111111111, 0, 0, -0.111111111111111, 0.111111111111111, 0, 0, 0, 0], [0.111111111111111, 0.166666666666667/LA, 0, -0.0277777777777778, -0.0555555555555556, -0.166666666666667, 0, 0.250000000000000, 0], [0.111111111111111, 0, 0.166666666666667/LA, -0.0277777777777778, -0.0555555555555556, 0, -0.166666666666667, -0.250000000000000, 0], [0.111111111111111, -0.166666666666667/LA, 0, -0.0277777777777778, -0.0555555555555556, 0.166666666666667, 0, 0.250000000000000, 0], [0.111111111111111, 0, -0.166666666666667/LA, -0.0277777777777778, -0.0555555555555556, 0, 0.166666666666667, -0.250000000000000, 0], [0.111111111111111, 0.166666666666667/LA, 0.166666666666667/LA, 0.0555555555555556, 0.0277777777777778, 0.0833333333333333, 0.0833333333333333, 0, 0.250000000000000], [0.111111111111111, -0.166666666666667/LA, 0.166666666666667/LA, 0.0555555555555556, 0.0277777777777778, -0.0833333333333333, 0.0833333333333333, 0, -0.250000000000000], [0.111111111111111, -0.166666666666667/LA, -0.166666666666667/LA, 0.0555555555555556, 0.0277777777777778, -0.0833333333333333, -0.0833333333333333, 0, 0.250000000000000], [0.111111111111111, 0.166666666666667/LA, -0.166666666666667/LA, 0.0555555555555556, 0.0277777777777778, 0.0833333333333333, -0.0833333333333333, 0, -0.250000000000000]])

### Run the simulation

For the simulation, we take

* The domain $x\in(0, L)$ and $y\in(-W/2,W/2)$, $L=2$, $W=1$,
* the viscosities $\mu=10^{-2}=\eta=10^{-2}$,
* the space step $\dx=1/128$, the scheme velocity $\lambda=1$,
* the mean density $\rho_0=1$.

Concerning the boundary conditions, we impose the velocity on all the edges by a bounce-back condition with a source term that reads
$$q_x(x, y) = \rho_0 v_{\text{max}} \Bigl( 1 - \frac{4y^2}{W^2} \Bigr), \qquad q_y(x, y) = 0,$$
with $v_{\text{max}}=0.1$.

We compute the solution for $t\in(0,50)$ and we plot several slices of the solution during the simulation.

This problem has an exact solution given by
$$ q_x = \rho_0 v_{\text{max}} \Bigl( 1 - \frac{4y^2}{W^2} \Bigr), \qquad q_y = 0, \qquad p = p_0 + K x, $$
where the pressure gradient $K$ reads
$$K = -\frac{8 v_{\text{max}} \eta}{W^2}.$$

We compute the exact and the numerical gradients of the pressure.

```python
X, Y, LA = sp.symbols('X, Y, LA')
rho, qx, qy = sp.symbols('rho, qx, qy')

def bc(f, m, x, y):
    m[qx] = rhoo * vmax * (1.-4.*y**2/W**2)
    m[qy] = 0.

def plot_coupe(sol):
    fig, ax1 = plt.subplots()
    ax2 = ax1.twinx()
    ax1.cla()
    ax2.cla()
    mx = int(sol.domain.shape_in[0]/2)
    my = int(sol.domain.shape_in[1]/2)
    x = sol.domain.x
    y = sol.domain.y
    u = sol.m[qx] / rhoo
    for i in [0,mx,-1]:
        ax1.plot(y+x[i], u[i, :], 'b')
    for j in [0,my,-1]:
        ax1.plot(x+y[j], u[:,j], 'b')
    ax1.set_ylabel('velocity', color='b')
    for tl in ax1.get_yticklabels():
        tl.set_color('b')
    ax1.set_ylim(-.5*rhoo*vmax, 1.5*rhoo*vmax)
    p = sol.m[rho][:,my] * la**2 / 3.0
    p -= np.average(p)
    ax2.plot(x, p, 'r')
    ax2.set_ylabel('pressure', color='r')
    for tl in ax2.get_yticklabels():
        tl.set_color('r')
    ax2.set_ylim(pressure_gradient*L, -pressure_gradient*L)
    plt.title('Poiseuille flow at t = {0:f}'.format(sol.t))
    plt.draw()
    plt.pause(1.e-3)

# parameters
dx = 1./16  # spatial step
la = 1.      # velocity of the scheme
Tf = 50      # final time of the simulation
L = 2        # length of the domain
W = 1        # width of the domain
vmax = 0.1   # maximal velocity obtained in the middle of the channel
rhoo = 1.    # mean value of the density
mu = 1.e-2   # bulk viscosity
eta = 1.e-2  # shear viscosity
pressure_gradient = - vmax * 8.0 / W**2 * eta
# initialization
xmin, xmax, ymin, ymax = 0.0, L, -0.5*W, 0.5*W
dummy = 3.0/(la*rhoo*dx)
s_mu = 1.0/(0.5+mu*dummy)
s_eta = 1.0/(0.5+eta*dummy)
s_q = s_eta
s_es = s_mu
s  = [0.,0.,0.,s_mu,s_es,s_q,s_q,s_eta,s_eta]
dummy = 1./(LA**2*rhoo)
qx2 = dummy*qx**2
qy2 = dummy*qy**2
q2  = qx2+qy2
qxy = dummy*qx*qy

dico = {
    'box':{'x':[xmin, xmax], 'y':[ymin, ymax], 'label':0},
    'space_step':dx,
    'scheme_velocity':la,
    'parameters':{LA:la},
    'schemes':[
        {
            'velocities':list(range(9)),
            'conserved_moments':[rho, qx, qy],
            'polynomials':[
                1, LA*X, LA*Y,
                3*(X**2+Y**2)-4,
                (9*(X**2+Y**2)**2-21*(X**2+Y**2)+8)/2,
                3*X*(X**2+Y**2)-5*X, 3*Y*(X**2+Y**2)-5*Y,
                X**2-Y**2, X*Y
            ],
            'relaxation_parameters':s,
            'equilibrium':[
                rho, qx, qy,
                -2*rho + 3*q2,
                rho-3*q2,
                -qx/LA, -qy/LA,
                qx2-qy2, qxy
            ],
            'init':{rho:rhoo, qx:0., qy:0.},
        },
    ],
    'boundary_conditions':{
        0:{'method':{0: pylbm.bc.BouzidiBounceBack}, 'value':bc}
    },
    'generator': 'cython',
}

sol = pylbm.Simulation(dico)
while (sol.t<Tf):
    sol.one_time_step()
plot_coupe(sol)
ny = int(sol.domain.shape_in[1]/2)
num_pressure_gradient = (sol.m[rho][-2,ny] - sol.m[rho][1,ny]) / (xmax-xmin) * la**2/ 3.0
print("Exact pressure gradient    : {0:10.3e}".format(pressure_gradient))
print("Numerical pressure gradient: {0:10.3e}".format(num_pressure_gradient))
```

```python

```


Copyright (c) 2015, 2016
[Sebastian Raschka](http://sebastianraschka.com/)
[Li-Yi Wei](http://www.liyiwei.org/)

https://github.com/1iyiwei/pyml

[MIT License](https://github.com/1iyiwei/pyml/blob/master/LICENSE.txt)

# Python Machine Learning - Code Examples

# Chapter 2 - Training Machine Learning Algorithms for Classification

Note that the optional watermark extension is a small IPython notebook plugin that I developed to make the code reproducible. You can just skip the following line(s).

```python
%load_ext watermark
%watermark -a '' -u -d -v -p numpy,pandas,matplotlib
```

    last updated: 2016-08-05

    CPython 3.5.1
    IPython 4.1.2

    numpy 1.10.4
    pandas 0.18.0
    matplotlib 1.5.1

*The use of `watermark` is optional. You can install this IPython extension via "`pip install watermark`". For more information, please see: https://github.com/rasbt/watermark.*

# Linear perceptron

A very simple model to illustrate machine learning
* how to perform classification
* how to learn from data via optimization
* connection to biological brains

### Overview

- [Artificial neurons - a brief glimpse into the early history
of machine learning](#Artificial-neurons-a-brief-glimpse-into-the-early-history-of-machine-learning)
- [Implementing a perceptron learning algorithm in Python](#Implementing-a-perceptron-learning-algorithm-in-Python)
    - [Training a perceptron model on the Iris dataset](#Training-a-perceptron-model-on-the-Iris-dataset)
- [Adaptive linear neurons and the convergence of learning](#Adaptive-linear-neurons-and-the-convergence-of-learning)
    - [Minimizing cost functions with gradient descent](#Minimizing-cost-functions-with-gradient-descent)
    - [Implementing an Adaptive Linear Neuron in Python](#Implementing-an-Adaptive-Linear-Neuron-in-Python)
    - [Large scale machine learning and stochastic gradient descent](#Large-scale-machine-learning-and-stochastic-gradient-descent)
- [Summary](#Summary)

```python
from IPython.display import Image
```

# Artificial neurons - a brief glimpse into the early history of machine learning

Biological neurons were the early inspirations for machine learning.

Current neural networks implement neurons via programs, but people are starting to build electronic circuits to more efficiently emulate neurons.

# Perceptron

Proposed by Rosenblatt in 1957 as a very simple model for artifical neurons.

## Representation

A single artificial neuron can be used for binary classification via:
$$
\phi(z) =
\begin{cases}
1 \; z \geq \theta \\
-1 \; z < \theta
\end{cases}
$$
, where $\pm 1$ are the class labels, $\phi$ is the activation function, $\theta$ is some activation threshold, and $z$ is computed from the input $\mathbf{x}$ and neuron weights $\mathbf{w}$
$$
z = \mathbf{w}^T \mathbf{x}
$$

In terms of our general model $f\left(\mathbf{X}, \Theta \right)$, the parameter set are the weights $\Theta = \mathbf{w}$ and the input set $\mathbf{X} = \mathbf{x}$:
$$\phi(z) = f(\mathbf{x}, \mathbf{w}) = 2 \left( \mathbf{w}^T \mathbf{x} \geq \theta \right) - 1$$

Note that both $\mathbf{w}$ and $\mathbf{x}$ are vectors and the above equations is simply computing their inner products.
Specifically,
$$
\begin{align}
\mathbf{w} =
\begin{bmatrix}
w_0 \\
\vdots \\
w_m
\end{bmatrix}
,
\mathbf{x} =
\begin{bmatrix}
x_0 \\
\vdots \\
x_m
\end{bmatrix}
\end{align}
$$

For example $m = 3$:
$$
\begin{align}
\mathbf{w} =
\begin{bmatrix}
1 \\
2 \\
3
\end{bmatrix}
,
\mathbf{x} =
\begin{bmatrix}
4 \\
5 \\
6
\end{bmatrix}
\end{align}
$$

$$
\mathbf{w}^T \mathbf{x} =
\begin{bmatrix}
1 & 2 & 3
\end{bmatrix}
\begin{bmatrix}
4 \\
5 \\
6
\end{bmatrix}
=
1 \times 4 + 2 \times 5 + 3 \times 6
= 32
$$

The convention is to let $x_0 = 1$ and $w_0 = -\theta$ (the bias term), so that we can simplify the model to:
$$
\phi(z) =
\begin{cases}
1 \; z \geq 0 \\
-1 \; z < 0
\end{cases}
$$

## Example

2D case, i.e. $m = 2$

The left visualizes $\phi(z)$, the right illustrates a binary classification:

## Training

How to train a perceptron so that it can learn from new data?

Let the training data be $\left( \mathbf{X}, \mathbf{T} \right)$, where
* $\mathbf{X}$: input vectors
* $\mathbf{T}$: labels, $\pm 1$ for each input vector

The training steps are as follows:
* Initialize $\mathbf{w}$ to $0$ or random
* For each training pair $\left(\mathbf{x}^{(i)}, t^{(i)}\right)$
 1. Compute the predicted output $y^{(i)} = \phi(\mathbf{w}^T \mathbf{x}^{(i)})$
 2. Compute $\delta \mathbf{w} = \eta (t^{(i)} - y^{(i)}) \mathbf{x}^{(i)}$, and update $\mathbf{w} \leftarrow \mathbf{w} + \delta \mathbf{w}$

$\eta$ is called the learning rate.

### Why this works

If the prediction is correct, i.e. $y^{(i)} = t^{(i)}$, $\rightarrow$ $\delta \mathbf{w} = 0$, so no update.

If the prediction is incorrect
 * $t^{(i)} = 1, y^{(i)} = -1$ $\rightarrow$ $\delta \mathbf{w} = \eta(2)\mathbf(x)^{(i)}$
 * $t^{(i)} = -1, y^{(i)} = 1$ $\rightarrow$ $\delta \mathbf{w} = \eta(-2)\mathbf(x)^{(i)}$

$\delta \mathbf{w}$ will move towards the right direction, more positively/negatively correlated with $\mathbf{x}$.

## Linear separability

A single perception is limited to linear classification.

We can handle more general cases via neural networks consisting of multiple perceptrons/neurons.

The binary threshold activation function is not differentiable; we will fix this later via other activation functions.

# Implementing a perceptron learning algorithm in Python

```python
import numpy as np

class Perceptron(object):
    """Perceptron classifier.

    Parameters
    ------------
    eta : float
        Learning rate (between 0.0 and 1.0)
    n_iter : int
        Passes over the training dataset.

    Attributes
    -----------
    w_ : 1d-array
        Weights after fitting.
    errors_ : list
        Number of misclassifications in every epoch.

    """
    def __init__(self, eta=0.01, n_iter=10):
        self.eta = eta
        self.n_iter = n_iter

    def fit(self, X, y):
        """Fit training data.

        Parameters
        ----------
        X : {array-like}, shape = [n_samples, n_features]
            Training vectors, where n_samples is the number of samples and
            n_features is the number of features.
        y : array-like, shape = [n_samples]
            Target values.

        Returns
        -------
        self : object

        """
        self.w_ = np.zeros(1 + X.shape[1])
        self.errors_ = []

        for _ in range(self.n_iter):
            errors = 0
            for xi, target in zip(X, y):
                update = self.eta * (target - self.predict(xi))
                self.w_[1:] += update * xi
                self.w_[0] += update
                errors += int(update != 0.0)
            self.errors_.append(errors)
        return self

    def net_input(self, X):
        """Calculate net input"""
        return np.dot(X, self.w_[1:]) + self.w_[0]

    def predict(self, X):
        """Return class label after unit step"""
        return np.where(self.net_input(X) >= 0.0, 1, -1)
```

### Additional Note (1)

Please note that the learning rate $\eta$ (eta) only has an effect on the classification outcome  if the weights are initialized to non-zero values. If all the weights
are initialized to 0, only the scale of the weight vector, not the direction. To have the learning rate influence the classification outcome, the weights need to be initialized to non-zero values. The respective lines in the code that need to be changed to accomplish that are highlighted on below:

```python
   def __init__(self, eta=0.01, n_iter=50, random_seed=1): # add random_seed=1
        ...
        self.random_seed = random_seed # add this line

    def fit(self, X, y):
        ...
        # self.w_ = np.zeros(1 + X.shape[1]) ## remove this line
        rgen = np.random.RandomState(self.random_seed) # add this line
        self.w_ = rgen.normal(loc=0.0, scale=0.01, size=1 + X.shape[1]) # add this line
```

### Additional Note (2)

I received a note by a reader who asked about the net input function:

>On page 27, you describe the code.

> the net_input method simply calculates the vector product wTx
However, there is more than a simple vector product in the code:

> def net_input(self, X):
    """Calculate net input"""
    return np.dot(X, self.w_[1:]) + self.w_[0]

> In addition to the dot product, there is an addition. The text does not mention anything about what is this + self.w_[0]
> Can you (or anyone) explain why that's there?

-------

Sorry that I went over that so briefly. The `self.w_[0]` is basically the "threshold" or so-called "bias unit." I simply included the bias unit in the weight vector, which makes the math part easier, but on the other hand, it may make the code more confusing as you mentioned.

Let's say we have a 3x2 dimensional dataset `X` (3 training samples with 2 features). Also, let's just assume we have a weight `2` for feature 1 and a weight `3` for feature 2, and we set the bias unit to `4`.

```
import numpy as np
>>> bias = 4.
>>> X = np.array([[2., 3.],
...              [4., 5.],
...              [6., 7.]])
>>> w = np.array([bias, 2., 3.])
```

In order to match the mathematical notation, we would have to add a vector of 1s to compute the dot-product:

```
>>> ones = np.ones((X.shape[0], 1))
>>> X_with1 = np.hstack((ones, X))
>>> X_with1
>>> np.dot(X_with1, w)
array([ 17.,  27.,  37.])
```

However, I thought that adding a vector of 1s to the training array each time we want to make a prediction would be fairly inefficient. So, instead, we can just "add" the bias unit (`w[0]`) to the do product (it's equivalent, since `1.0 * w_0 = w_0`:

```
>>> np.dot(X, w[1:]) + w[0]
array([ 17.,  27.,  37.])
```

Maybe it is helpful to walk through the matrix-vector multiplication by hand. E.g.,

```
| 1  2  3 |   | 4 |   | 1*4 + 2*2 + 3*3 |   | 17 |
| 1  4  5 | x | 2 | = | 1*4 + 4*2 + 5*3 | = | 27 |
| 1  6  7 |   | 3 |   | 1*4 + 6*2 + 7*3 |   | 37 |
```

which is the same as

```
| 2  3 |   | 4 |          | 2*2 + 3*3 |          | 13 + bias |   | 17 |
| 4  5 | x | 2 | + bias = | 4*2 + 5*3 | + bias = | 23 + bias | = | 27 |
| 6  7 |   | 3 |          | 6*2 + 7*3 |          | 33 + bias |   | 37 |
```

### Additional Note (3)

For simplicity at this point, we don't talk about shuffling at this point; I wanted to introduce concepts incrementally so that it's not too overwhelming all at once. Since a reader asked me about this, I wanted to add a note about shuffling, which you may want to use if you are using a Perceptron in practice. I borrowed the code from the `AdalineSGD` section below to modify the Perceptron algorithm accordingly (new lines are marked by trailing "`# new`" inline comment):

```python
class Perceptron(object):
    """Perceptron classifier.

    Parameters
    ------------
    eta : float
        Learning rate (between 0.0 and 1.0)
    n_iter : int
        Passes over the training dataset.
    shuffle : bool (default: True)
        Shuffles training data every epoch if True to prevent cycles.
    random_state : int (default: None)
        Set random state for shuffling and initializing the weights.

    Attributes
    -----------
    w_ : 1d-array
        Weights after fitting.
    errors_ : list
        Number of misclassifications in every epoch.

    """
    def __init__(self, eta=0.01, n_iter=10,
                 shuffle=True, random_state=None):  # new
        self.eta = eta
        self.n_iter = n_iter
        self.shuffle = shuffle  # new
        if random_state:  # new
            np.random.seed(random_state)  # new

    def fit(self, X, y):
        """Fit training data.

        Parameters
        ----------
        X : {array-like}, shape = [n_samples, n_features]
            Training vectors, where n_samples is the number of samples and
            n_features is the number of features.
        y : array-like, shape = [n_samples]
            Target values.

        Returns
        -------
        self : object

        """
        self.w_ = np.zeros(1 + X.shape[1])
        self.errors_ = []

        for _ in range(self.n_iter):
            if self.shuffle:  # new
                X, y = self._shuffle(X, y)  # new
            errors = 0
            for xi, target in zip(X, y):
                update = self.eta * (target - self.predict(xi))
                self.w_[1:] += update * xi
                self.w_[0] += update
                errors += int(update != 0.0)
            self.errors_.append(errors)
        return self

    def _shuffle(self, X, y):  # new
        """Shuffle training data"""  # new
        r = np.random.permutation(len(y))  # new
        return X[r], y[r]  # new

    def net_input(self, X):
        """Calculate net input"""
        return np.dot(X, self.w_[1:]) + self.w_[0]

    def predict(self, X):
        """Return class label after unit step"""
        return np.where(self.net_input(X) >= 0.0, 1, -1)
```

## Training a perceptron model on the Iris dataset

### Reading-in the Iris data

```python
import pandas as pd

data_src = '../datasets/iris/iris.data'
#data_src = 'https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data'

df = pd.read_csv(data_src, header=None)

df.tail()
```

      0
      1
      2
      3
      4

      145
      6.7
      3.0
      5.2
      2.3
      Iris-virginica

      146
      6.3
      2.5
      5.0
      1.9
      Iris-virginica

      147
      6.5
      3.0
      5.2
      2.0
      Iris-virginica

      148
      6.2
      3.4
      5.4
      2.3
      Iris-virginica

      149
      5.9
      3.0
      5.1
      1.8
      Iris-virginica

### Note:

If the link to the Iris dataset provided above does not work for you, you can find a local copy in this repository at [./../datasets/iris/iris.data](./../datasets/iris/iris.data).

Or you could fetch it via https://raw.githubusercontent.com/1iyiwei/pyml/master/code/datasets/iris/iris.data

### Plotting the Iris data

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

# select setosa and versicolor
y = df.iloc[0:100, 4].values
y = np.where(y == 'Iris-setosa', -1, 1)

# extract sepal length and petal length
X = df.iloc[0:100, [0, 2]].values

# plot data
plt.scatter(X[:50, 0], X[:50, 1],
            color='red', marker='o', label='setosa')
plt.scatter(X[50:100, 0], X[50:100, 1],
            color='blue', marker='x', label='versicolor')

plt.xlabel('sepal length [cm]')
plt.ylabel('petal length [cm]')
plt.legend(loc='upper left')

plt.tight_layout()
#plt.savefig('./images/02_06.png', dpi=300)
plt.show()
```

### Training the perceptron model

```python
ppn = Perceptron(eta=0.1, n_iter=10)

ppn.fit(X, y)

plt.plot(range(1, len(ppn.errors_) + 1), ppn.errors_, marker='o')
plt.xlabel('Epochs')
plt.ylabel('Number of misclassifications')

plt.tight_layout()
# plt.savefig('./perceptron_1.png', dpi=300)
plt.show()
```

### A function for plotting decision regions

```python
from matplotlib.colors import ListedColormap

def plot_decision_regions(X, y, classifier, resolution=0.02):

    # setup marker generator and color map
    markers = ('s', 'x', 'o', '^', 'v')
    colors = ('red', 'blue', 'lightgreen', 'gray', 'cyan')
    cmap = ListedColormap(colors[:len(np.unique(y))])

    # plot the decision surface
    x1_min, x1_max = X[:, 0].min() - 1, X[:, 0].max() + 1
    x2_min, x2_max = X[:, 1].min() - 1, X[:, 1].max() + 1
    xx1, xx2 = np.meshgrid(np.arange(x1_min, x1_max, resolution),
                           np.arange(x2_min, x2_max, resolution))
    Z = classifier.predict(np.array([xx1.ravel(), xx2.ravel()]).T)
    Z = Z.reshape(xx1.shape)
    plt.contourf(xx1, xx2, Z, alpha=0.4, cmap=cmap)
    plt.xlim(xx1.min(), xx1.max())
    plt.ylim(xx2.min(), xx2.max())

    # plot class samples
    for idx, cl in enumerate(np.unique(y)):
        plt.scatter(x=X[y == cl, 0], y=X[y == cl, 1],
                    alpha=0.8, c=cmap(idx),
                    marker=markers[idx], label=cl)
```

```python
plot_decision_regions(X, y, classifier=ppn)
plt.xlabel('sepal length [cm]')
plt.ylabel('petal length [cm]')
plt.legend(loc='upper left')

plt.tight_layout()
# plt.savefig('./perceptron_2.png', dpi=300)
plt.show()
```

#### Additional Note (3)

The `plt.scatter` function in the `plot_decision_regions` plot may raise errors if you have matplotlib <= 1.5.0 installed if you use this function to plot more than 4 classes as a reader pointed out: "[...] if there are four items to be displayed as the RGBA tuple is mis-interpreted as a list of colours".

```python
plt.scatter(x=X[y == cl, 0], y=X[y == cl, 1],
            alpha=0.8, c=cmap(idx),
            marker=markers[idx], label=cl)
```

To address this problem in older matplotlib versions, you can replace `c=cmap(idx)` by `c=colors[idx]`.

# Adaptive linear neurons and the convergence of learning

Let's change the activation function of the perceptron from binary to linear, i.e.
$$
\begin{align}
\phi(z) &= z \\
\phi(\mathbf{w}^T \mathbf{x}) &= \mathbf{w}^T \mathbf{x}
\end{align}
$$
And separate out the quantizer as another unit, so that we can optimize via gradient descent.

This is called Adaline for Adaptive linear neuron.

Adaline:

Perceptron:

## Minimizing cost functions with gradient descent

Let's define the loss/objective function via SSE (sum of squared errors) as follows:
$$
\begin{align}
L(\mathbf{X}, \mathbf{T}, \Theta) &= J(\mathbf{w}) = \frac{1}{2} \sum_i \left(t^{(i)} - z^{(i)}  \right)^2 \\
z^{(i)} &= \phi\left(\mathbf{w}^T \mathbf{x}^{(i)}\right) = \mathbf{w}^T \mathbf{x}
\end{align}
$$

Note that we are optimizing for $z$, the output of the linear activation $\phi$, instead of the final output $y$, which is:
$$
y =
\begin{cases}
+1 \; z \geq 0 \\
-1 \; z < 0
\end{cases}
$$

However, this would work, as $y$ and $z$ are proportional, so increasing/decreasing $z$ will also increase/decrease $y$.
Moreover, unlike the binary threshold activation function, the linear activation function is differentiable, so we can use calculus to optimize the above equation.

In particular, we can use gradient descent.
$$
\delta J = \frac{\partial J}{\partial \mathbf{w}} = \sum_i \mathbf{x}^{(i) }\left( \mathbf{w}^T \mathbf{x}^{(i)} - \mathbf{t}^{(i)} \right)
$$

$$
\begin{align}
\delta \mathbf{w} &= -\eta \delta J \\
\mathbf{w} & \leftarrow \mathbf{w} + \delta \mathbf{w}
\end{align}
$$

## Implementing an adaptive linear neuron in Python

```python
class AdalineGD(object):
    """ADAptive LInear NEuron classifier.

    Parameters
    ------------
    eta : float
        Learning rate (between 0.0 and 1.0)
    n_iter : int
        Passes over the training dataset.

    Attributes
    -----------
    w_ : 1d-array
        Weights after fitting.
    errors_ : list
        Number of misclassifications in every epoch.

    """
    def __init__(self, eta=0.01, n_iter=50):
        self.eta = eta
        self.n_iter = n_iter

    def fit(self, X, y):
        """ Fit training data.

        Parameters
        ----------
        X : {array-like}, shape = [n_samples, n_features]
            Training vectors, where n_samples is the number of samples and
            n_features is the number of features.
        y : array-like, shape = [n_samples]
            Target values.

        Returns
        -------
        self : object

        """
        self.w_ = np.zeros(1 + X.shape[1])
        self.cost_ = []

        for i in range(self.n_iter):
            net_input = self.net_input(X)
            # Please note that the "activation" method has no effect
            # in the code since it is simply an identity function. We
            # could write `output = self.net_input(X)` directly instead.
            # The purpose of the activation is more conceptual, i.e.,
            # in the case of logistic regression, we could change it to
            # a sigmoid function to implement a logistic regression classifier.
            output = self.activation(X)
            errors = (y - output)
            self.w_[1:] += self.eta * X.T.dot(errors)
            self.w_[0] += self.eta * errors.sum()
            cost = (errors**2).sum() / 2.0
            self.cost_.append(cost)
        return self

    def net_input(self, X):
        """Calculate net input"""
        return np.dot(X, self.w_[1:]) + self.w_[0]

    def activation(self, X):
        """Compute linear activation"""
        return self.net_input(X)

    def predict(self, X):
        """Return class label after unit step"""
        return np.where(self.activation(X) >= 0.0, 1, -1)
```

```python
fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(8, 4))

ada1 = AdalineGD(n_iter=10, eta=0.01).fit(X, y)
ax[0].plot(range(1, len(ada1.cost_) + 1), np.log10(ada1.cost_), marker='o')
ax[0].set_xlabel('Epochs')
ax[0].set_ylabel('log(Sum-squared-error)')
ax[0].set_title('Adaline - Learning rate 0.01')

ada2 = AdalineGD(n_iter=10, eta=0.0001).fit(X, y)
ax[1].plot(range(1, len(ada2.cost_) + 1), ada2.cost_, marker='o')
ax[1].set_xlabel('Epochs')
ax[1].set_ylabel('Sum-squared-error')
ax[1].set_title('Adaline - Learning rate 0.0001')

plt.tight_layout()
# plt.savefig('./adaline_1.png', dpi=300)
plt.show()
```

### Learning rate

We need to choose the right learning rate $\eta$
* too small will take many steps to reach minimum
* too large will oscillate and miss the minimum

### Scaling

For better performance, it is often a good idea to normalize/scale the dimensions of the data so that they have similar orders of magnitude:
$$
\mathbf{x}_j = \frac{\mathbf{x}_j - \mu_j}{\sigma_j}
$$
, where $\mu$ and $\sigma$ are the mean and std.

```python
# standardize features
X_std = np.copy(X)
X_std[:, 0] = (X[:, 0] - X[:, 0].mean()) / X[:, 0].std()
X_std[:, 1] = (X[:, 1] - X[:, 1].mean()) / X[:, 1].std()
```

```python
ada = AdalineGD(n_iter=15, eta=0.01)
ada.fit(X_std, y)

plot_decision_regions(X_std, y, classifier=ada)
plt.title('Adaline - Gradient Descent')
plt.xlabel('sepal length [standardized]')
plt.ylabel('petal length [standardized]')
plt.legend(loc='upper left')
plt.tight_layout()
# plt.savefig('./adaline_2.png', dpi=300)
plt.show()

plt.plot(range(1, len(ada.cost_) + 1), ada.cost_, marker='o')
plt.title('Adaline - Learning rate 0.01')
plt.xlabel('Epochs')
plt.ylabel('Sum-squared-error')

plt.tight_layout()
# plt.savefig('./adaline_3.png', dpi=300)
plt.show()
```

## Large scale machine learning and stochastic gradient descent

Recall for gradient descent, we compute $\delta J$ from all $N$ data samples:
$$
\begin{align}
\delta J &= \frac{\partial J}{\partial \mathbf{w}} = \sum_{i = 1}^{N} \mathbf{x}^{(i) }\left( \mathbf{w}^T \mathbf{x}^{(i)} - \mathbf{t}^{(i)} \right) \\
\delta \mathbf{w} &= -\eta \delta J \\
\mathbf{w} & \leftarrow \mathbf{w} + \delta \mathbf{w}
\end{align}
$$

This can be slow if $N$ is large.
One solution is to compute $\delta J$ from only a small subset of all data, and iterate over the subsets.
This is called stochastic gradient descent.
Each subset is often called a mini-batch.

For each $batch_j \subset{N}$:
$$
\begin{align}
\delta J &= \frac{\partial J}{\partial \mathbf{w}} = \sum_{i \in batch_j} \mathbf{x}^{(i) }\left( \mathbf{w}^T \mathbf{x}^{(i)} - \mathbf{t}^{(i)} \right) \\
\delta \mathbf{w} &= -\eta \delta J \\
\mathbf{w} & \leftarrow \mathbf{w} + \delta \mathbf{w}
\end{align}
$$

Machine learning libraries usually let you tune the subset, or mini-batch, size, between two extremes:
* just one sample
* entire data set $\rightarrow$ traditional gradient descent

### Example

Input data: 10 samples total with indices
* $[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]$

Random shuffle them for better performance at the beginning of each iteration/epoch:
* $[7, 1, 5, 6, 0, 3, 8, 2, 9, 4]$

Say the mini-batch size is 2, then we train the network with the following mini-batches:
* $[7, 1]$
* $[5, 6]$
* $[0, 3]$
* $[8, 2]$
* $[9, 4]$

Continue for more iterations/epochs, over the entire dataset, until convergence
* i.e. go back to the random shuffle step above with a different permutation, e.g. $[9, 1, 4, 7, 2, 5, 6, 0, 3, 8]$

## More background information

https://en.wikipedia.org/wiki/Stochastic_gradient_descent

```python
from numpy.random import seed

class AdalineSGD(object):
    """ADAptive LInear NEuron classifier.

    Parameters
    ------------
    eta : float
        Learning rate (between 0.0 and 1.0)
    n_iter : int
        Passes over the training dataset.

    Attributes
    -----------
    w_ : 1d-array
        Weights after fitting.
    errors_ : list
        Number of misclassifications in every epoch.
    shuffle : bool (default: True)
        Shuffles training data every epoch if True to prevent cycles.
    random_state : int (default: None)
        Set random state for shuffling and initializing the weights.

    """
    def __init__(self, eta=0.01, n_iter=10, shuffle=True, random_state=None):
        self.eta = eta
        self.n_iter = n_iter
        self.w_initialized = False
        self.shuffle = shuffle
        if random_state:
            seed(random_state)

    def fit(self, X, y):
        """ Fit training data.

        Parameters
        ----------
        X : {array-like}, shape = [n_samples, n_features]
            Training vectors, where n_samples is the number of samples and
            n_features is the number of features.
        y : array-like, shape = [n_samples]
            Target values.

        Returns
        -------
        self : object

        """
        self._initialize_weights(X.shape[1])
        self.cost_ = []
        for i in range(self.n_iter):
            if self.shuffle:
                X, y = self._shuffle(X, y)
            cost = []
            for xi, target in zip(X, y):
                cost.append(self._update_weights(xi, target))
            avg_cost = sum(cost) / len(y)
            self.cost_.append(avg_cost)
        return self

    def partial_fit(self, X, y):
        """Fit training data without reinitializing the weights"""
        if not self.w_initialized:
            self._initialize_weights(X.shape[1])
        if y.ravel().shape[0] > 1:
            for xi, target in zip(X, y):
                self._update_weights(xi, target)
        else:
            self._update_weights(X, y)
        return self

    def _shuffle(self, X, y):
        """Shuffle training data"""
        r = np.random.permutation(len(y))
        return X[r], y[r]

    def _initialize_weights(self, m):
        """Initialize weights to zeros"""
        self.w_ = np.zeros(1 + m)
        self.w_initialized = True

    def _update_weights(self, xi, target):
        """Apply Adaline learning rule to update the weights"""
        output = self.net_input(xi)
        error = (target - output)
        self.w_[1:] += self.eta * xi.dot(error)
        self.w_[0] += self.eta * error
        cost = 0.5 * error**2
        return cost

    def net_input(self, X):
        """Calculate net input"""
        return np.dot(X, self.w_[1:]) + self.w_[0]

    def activation(self, X):
        """Compute linear activation"""
        return self.net_input(X)

    def predict(self, X):
        """Return class label after unit step"""
        return np.where(self.activation(X) >= 0.0, 1, -1)
```

```python
ada = AdalineSGD(n_iter=15, eta=0.01, random_state=1)
ada.fit(X_std, y)

plot_decision_regions(X_std, y, classifier=ada)
plt.title('Adaline - Stochastic Gradient Descent')
plt.xlabel('sepal length [standardized]')
plt.ylabel('petal length [standardized]')
plt.legend(loc='upper left')

plt.tight_layout()
#plt.savefig('./adaline_4.png', dpi=300)
plt.show()

plt.plot(range(1, len(ada.cost_) + 1), ada.cost_, marker='o')
plt.xlabel('Epochs')
plt.ylabel('Average Cost')

plt.tight_layout()
# plt.savefig('./adaline_5.png', dpi=300)
plt.show()
```

```python
_ = ada.partial_fit(X_std[0, :], y[0])
```

# Reading
* PML Chapter 2
* IML Chapter 10.1-10.4, linear discrimination

# Assignment
* [ex02](http://nbviewer.jupyter.org/github/1iyiwei/pyml/blob/master/code/ch02/ex02.ipynb)

# Appendix

The code below (not in the book) is a simplified, example implementation of a logistic regression classifier trained via gradient descent. The AdalineGD classifier was used as template and I commented the respective lines that were changed to turn it into a logistic regression classifier (as briefly mentioned in the "logistic regression" sections of Chapter 3).

```python
class LogisticRegressionGD(object):
    """Logistic regression classifier via gradient descent.

    Parameters
    ------------
    eta : float
        Learning rate (between 0.0 and 1.0)
    n_iter : int
        Passes over the training dataset.

    Attributes
    -----------
    w_ : 1d-array
        Weights after fitting.
    errors_ : list
        Number of misclassifications in every epoch.

    """
    def __init__(self, eta=0.01, n_iter=50):
        self.eta = eta
        self.n_iter = n_iter

    def fit(self, X, y):
        """ Fit training data.

        Parameters
        ----------
        X : {array-like}, shape = [n_samples, n_features]
            Training vectors, where n_samples is the number of samples and
            n_features is the number of features.
        y : array-like, shape = [n_samples]
            Target values.

        Returns
        -------
        self : object

        """
        self.w_ = np.zeros(1 + X.shape[1])
        self.cost_ = []

        for i in range(self.n_iter):
            net_input = self.net_input(X)
            output = self.activation(X)
            errors = (y - output)
            self.w_[1:] += self.eta * X.T.dot(errors)
            self.w_[0] += self.eta * errors.sum()

            # note that we compute the logistic `cost` now
            # instead of the sum of squared errors cost
            cost = -y.dot(np.log(output)) - ((1 - y).dot(np.log(1 - output)))
            self.cost_.append(cost)
        return self

    def net_input(self, X):
        """Calculate net input"""
        return np.dot(X, self.w_[1:]) + self.w_[0]

    def predict(self, X):
        """Return class label after unit step"""
        # We use the more common convention for logistic
        # regression returning class labels 0 and 1
        # instead of -1 and 1. Also, the threshold then
        # changes from 0.0 to 0.5
        return np.where(self.activation(X) >= 0.5, 1, 0)

    # The Content of `activation` changed
    # from linear (Adaline) to sigmoid.
    # Note that this method is now returning the
    # probability of the positive class
    # also "predict_proba" in scikit-learn
    def activation(self, X):
        """ Compute sigmoid activation."""
        z = self.net_input(X)
        sigmoid = 1.0 / (1.0 + np.exp(-z))
        return sigmoid
```

```python
from sklearn.datasets import load_iris

iris = load_iris()
X, y = iris.data[:100, [0, 2]], iris.target[:100]

X_std = np.copy(X)
X_std[:, 0] = (X[:, 0] - X[:, 0].mean()) / X[:, 0].std()
X_std[:, 1] = (X[:, 1] - X[:, 1].mean()) / X[:, 1].std()
```

```python
lr = LogisticRegressionGD(n_iter=25, eta=0.15)
lr.fit(X_std, y)

plot_decision_regions(X_std, y, classifier=lr)
plt.title('Logistic Regression - Gradient Descent')
plt.xlabel('sepal length [standardized]')
plt.ylabel('sepal width [standardized]')
plt.legend(loc='upper left')
plt.tight_layout()

plt.show()

plt.plot(range(1, len(lr.cost_) + 1), lr.cost_, marker='o')
plt.xlabel('Epochs')
plt.ylabel('Logistic Cost')

plt.tight_layout()
# plt.savefig('./adaline_3.png', dpi=300)
plt.show()
```


# Laplace 2D

This example shows how to declare a bilinear form using sympde, then create and evaluate a GLT symbol.

We first start by importing what is needed from sympde and gelato:

```python
# imports from sympde, to write bilinear forms
from sympde.core     import Constant
from sympde.calculus import grad, dot
from sympde.topology import ScalarFunctionSpace
from sympde.topology import Domain
from sympde.topology import elements_of
from sympde.expr     import BilinearForm
from sympde.expr     import integral
```

```python
# imports from gelato
from gelato import gelatize, GltExpr
```

A domain is created as follows

```python
domain = Domain('Omega', dim=2)
```

Then we declare a space of function over our domain

```python
V = ScalarFunctionSpace('V', domain)
```

and define dummy test functions living in the space $V$

```python
u,v = elements_of(V, names='u,v')
```

Finaly, we declare a blinear form, as a lambda expression

```python
# declaring a constant from sympde
c = Constant('c')

expr = dot(grad(v), grad(u)) + c*v*u
a = BilinearForm((u,v), integral(domain, expr))
```

Now we can create the associated GLT expression to the bilinear form $a$

```python
glt = GltExpr(a)
```

The following instruction inspects what is a GltExpr: it is a lambda expression, that has two tuples as inputs: fourier space variables denoted by $(t_x, t_y)$ and no space variables in this case.

```python
print(glt)
```

    GltExpr([tx, ty], [], BilinearForm(((u,), (v,)), DomainIntegral(Dot(Grad(u), Grad(v)), Omega) + DomainIntegral(c*u*v, Omega)))

We can use a partial evaluation of the GLT expression, by providing the spline degrees

```python
print(glt(degrees=[2,2]))
```

    c*(13*cos(tx)/30 + cos(2*tx)/60 + 11/20)*(13*cos(ty)/30 + cos(2*ty)/60 + 11/20)/(nx*ny) + nx*(-2*cos(tx)/3 - cos(2*tx)/3 + 1)*(13*cos(ty)/30 + cos(2*ty)/60 + 11/20)/ny + ny*(13*cos(tx)/30 + cos(2*tx)/60 + 11/20)*(-2*cos(ty)/3 - cos(2*ty)/3 + 1)/nx

Or numerically, evaluate it, although we are relaying on sympy to perform the evaluation, which is not the right way to proceed. One may use the *lambdify* function from sympy, or rely on *PsyDac* to generate automaticaly the discrete GltExpr, when having more complicated expressions (involving a mpping or fields)

```python
print(glt(tx=0.1, ty=0.2, degrees=[2,2], n_elements=[16,16]))
```

    0.00385771212059162*c + 0.0493788050561308

```python
# css style
from IPython.core.display import HTML
def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>
<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
    }
    div.cell{
        width:600px;
        margin-left:16% !important;
        margin-right:auto;
    }
    h1 {
        font-family: 'Alegreya Sans', sans-serif;
    }
    h2 {
        font-family: 'Fenix', serif;
    }
    h3{
		font-family: 'Fenix', serif;
        margin-top:12px;
        margin-bottom: 3px;
       }
	h4{
		font-family: 'Fenix', serif;
       }
    h5 {
        font-family: 'Alegreya Sans', sans-serif;
    }
    div.text_cell_render{
        font-family: 'Alegreya Sans',Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 135%;
        font-size: 120%;
        width:600px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro";
			font-size: 90%;
    }
/*    .prompt{
        display: None;
    }*/
    .text_cell_render h1 {
        font-weight: 200;
        font-size: 50pt;
		line-height: 100%;
        color:#054BCD;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
        display: block;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 16pt;
        color: #054BCD;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

```python

```


```python
"""
Explore algebraic forms arising from the integral f(u)*v in the finite
element method.

   |              phi_im1            phi_i            phi_ip1
   +1                /\               /\                /\
   |                /  \             /  \              /  \
   |               /    \           /    \            /    \
   |              /      \         /      \          /      \
   |             /        \       /        \        /        \
   |            /          \     /          \      /          \
   |           /            \   /            \    /            \
   |          /              \ /              \  /              \
   |         /                /                \/                \
   |        /                / \               /\                 \
   |       /                /   \             /  \                 \
   |      /                /     \           /    \                 \
   |     /                /       \         /      \                 \
   |    /                /         \       /        \                 \
   |   /                /           \     /          \                 \
   |  /                /             \   /            \                 \
   | /                /               \ /              \                 \
--------------------------------------------------------------------------
                     i-1               i               i+1

                          cell L             cell R
"""
from sympy import *
import sys

x, u_im1, u_i, u_ip1, u, h, x_i = symbols('x u_im1 u_i u_ip1 u h x_i')

# Left cell:  [x_im1, x_i]
# Right cell: [x_i, x_ip1]
x_im1 = x_i - h
x_ip1 = x_i + h

phi = {'L':  # Left cell
       {'im1': 1 - (x - x_im1)/h,
        'i': (x - x_im1)/h},
       'R':  # Right cell
       {'i': 1 - (x-x_i)/h,
        'ip1': (x - x_i)/h}}

u = {'L': u_im1*phi['L']['im1'] + u_i  *phi['L']['i'],
     'R': u_i  *phi['R']['i']   + u_ip1*phi['R']['ip1']}

f = lambda u: eval(sys.argv[1])

integral_L = integrate(f(u['L'])*phi['L']['i'], (x, x_im1, x_i))
integral_R = integrate(f(u['R'])*phi['R']['i'], (x, x_i, x_ip1))
expr_i = simplify(expand(integral_L + integral_R))
print(expr_i)
latex_code = latex(expr_i, mode='plain')
# Replace u_im1 sympy symbol name by latex symbol u_{i-1}
latex_code = latex_code.replace('im1', '{i-1}')
# Replace u_ip1 sympy symbol name by latex symbol u_{i+1}
latex_code = latex_code.replace('ip1', '{i+1}')
print(latex_code)
# Escape (quote) latex_code so it can be sent as HTML text
import cgi
html_code = cgi.escape(latex_code)
print(html_code)
# Make a file with HTML code for displaying the LaTeX formula
f = open('tmp.html', 'w')
# Include an image that can be clicked on to yield a new
# page with an interactive editor and display area where the
# formula can be further edited
text = """
<a href="http://www.codecogs.com/eqnedit.php?latex=%(html_code)s"
 target="_blank">

</a>
 """ % vars()
f.write(text)
f.close()
# load tmp.html into a browser

```

```python

```

Data Science and Machine Learning/Machine-Learning-In-Python-THOROUGH/EXAMPLES/FINITE_ELEMENTS/INTRO/EXERCICES/16_FU_FEM_INT.ipynb


$$ \newcommand{\pd}[2]{ \frac{\partial #1}{\partial #2} }
\newcommand{\od}[2]{\frac{d #1}{d #2}}
\newcommand{\td}[2]{\frac{D #1}{D #2}}
\newcommand{\ab}[1]{\langle #1 \rangle}
\newcommand{\bss}[1]{\textsf{\textbf{#1}}}
\newcommand{\ol}{\overline}
\newcommand{\olx}[1]{\overline{#1}^x}
$$

# Hydrostatic and Geostrophic Balances

In the previous lecture, we obtained the local-tangent-plane form of the Boussinesq equations of motion. We repeat the final equations, in component form, below:

$$ \begin{align}
\td{u}{t} - f v &= -\pd{\phi}{x} + \nu \nabla^2 u \\
\td{v}{t} + f u &= -\pd{\phi}{y} + \nu \nabla^2 v \\
\td{w}{t}       &= -\pd{\phi}{z} + b + \nu \nabla^2 w  \ .
\end{align} $$

In this lecture, we ask _what are the dominant balances in these equations under common oceanographic conditions_.

## Hydrostatic Balance

We already saw hydrostatic balance for the _background pressure and density field_. For the large-scale flow, the dynamic pressure is also in hydrostatic balance:

$$ \pd{\phi}{z} = b \ . $$

To remind ourselves of the underlying physics, we can write out $\phi$ and $b$ explicitly and drop the common factor of $1/\rho_0$:

$$ \pd{}{z} \delta p = - g \delta \rho \ . $$

Hydrostatic balance can be used to define $\phi$ at any point. In order to do so, however, we must think a bit more about sea-surface height and its relation to pressure.

## Sea Surface Height, Dynamic Height, and Pressure

What is the dynamic pressure $\phi$ at an arbitrary depth $z$ for a flow in hydrostatic balance? The dynamc sea-surface $\eta$ is defined relative to the mean geoid (i.e. a surface of constant geopotential) at $z=0$.

We go back to the full hydrostatic balance with both backround and dynamic pressure:

$$ \pd{}{z} (p) = - g \rho \ . $$

We now integrate this equation in the vertical from an arbitrary $z$ up to the sea surface $\eta$:

$$ \int_z^\eta \pd{}{z'} p dz'
= p(\eta) - p(z)
= - g \int_z^\eta  \rho dz' \ . $$

$p(\eta)$ is the pressure right at the sea surface, which is given by the atmospheric pressure. Although atmospheric pressure loading can have a small effect on ocean circulation, it is generally negligible compared to the huge pressures generated internally by the ocean. We will now take $p(\eta)=0$ to simplify the bookkeeping, this gives.

$$ p(z) = g \int_z^\eta  \rho dz' \ . $$

Now let's subtract the reference pressure. It given integrating the hydrostatic balance for the background density up to $z=0$. This upper limit is important: the reference pressure is defined for a flat sea surface.

$$ p_{ref}(z) = g \int_z^0  \rho_0 dz' \ . $$

Subtracting the two equations, we obtain

$$ \begin{align}
\delta p(z) = p(z) - p_{ref}(z) &= g \int_z^\eta  \rho dz' \ - g \int_z^0  \rho_0 dz' \\
&= g \int_0^\eta  \rho  dz' + g \int_z^0 \delta \rho dz'
\end{align} $$

We see there is a contribution from the density fluctuations in the interior (second term), plus the variations in the sea-surface height (first term). We can, to the usual precision of the Boussinesq approximation, neglect the density fluctuations within the depths 0 to $\eta$. Dividing through by $\rho_0$, we obtain

$$ \begin{align}
\phi(z) = g \eta  + \int_z^0 b dz'
\end{align} $$

It is common in oceanography to define a quanity known as _dynamic height_, which expresses dynamic pressure variations in terms of their effective sea-surface height. In our notation, with the Boussinesq approximation, dynamic height is defined as

$$ H = \frac{\phi}{g} \ .$$

## Rossby Number

Let's estimate the relative size of the acceleration to Coriolis terms on the left-hand side of the horizontal momentum equations.

The ratio of these terms defines the _Rossby number_.

$$ R_O =  \frac{ \left | \td{u}{t} \right | }{ | f v |} $$

The magnitude of the acceleration term can be estimated as $U^2 / L$, where $U$ is a representitive velocity scale of the flow and $L$ is a representative length scale. So we can estimate the Rossby number as

$$ R_O =  \frac{U}{f L} \ . $$

What are representative values? For the large-scale ocean circulation, U = 0.01 m/s, L = 1000 km, and f = 10$^{-4}$ s$^{-1}$. This gives $R_O = 10^{-4}$. So the _acceleration terms are often totally negligible_!

For low Rossby number conditions, we can neglect the acceleration and write the horizontal momentum equations as

$$ \begin{align} - f v &= -\pd{\phi}{x} + \nu \nabla^2 u \\
 f u &= -\pd{\phi}{y} + \nu \nabla^2 v
\end{align} $$

The _geostrophic flow_ is defined as the flow determined by the balance between the Coriolis term and and the pressure gradient:

$$ \begin{align} - f v_g &= -\pd{\phi}{x} \\
  f u_g &= -\pd{\phi}{y}
\end{align} $$

while the ageostrophic flow is defined via the balance between the Coriolis term and the friction term:

$$ \begin{align} - f v_a &= \nu \nabla^2 u \\
  f u_a &= \nu \nabla^2 v
\end{align} $$

The total flow is given by the sum of the geostrophic and ageostrophic components:

$$ \mathbf{u} = \mathbf{u}_a + \mathbf{u}_g \ .$$

## Geostrophic Flow

Away from the boundaries, Friction is weak, and the flow is, to a good approximation geostrophic. Geostrophic (or "balanced") flow is a ubiquitous feature of flows in the ocean and atmosphere. Geostrophic flow is characterized by flow along the pressure contours (i.e. _isobars_), as illustrated below.

Rotational flow around pressure minimum is called _cyclonic_. Cyclonc flow is counterclockwise in the northern hemisphere and (due to the change of sign of $f$) clockwise in the southern hemisphere.

_ [AVISO mean dynamic topography](https://www.aviso.altimetry.fr/en/applications/ocean/large-scale-circulation/mean-dynamic-topography.html) _

### Thermal Wind

The geostrophic flow is determined by the pressure and the pressure is determined by the density (via hydrostatic balance). We can see this relationship more clearly if we take the derivative in $z$ of the geostrophic equations:

$$ \begin{align} - f \pd{v_g}{z} &= -\pd{}{x}\pd{\phi}{z} = -\pd{b}{x}\\
f \pd{u_g}{z} &= -\pd{}{y} \pd{\phi}{z} = -\pd{b}{y} \ .
\end{align} $$

These equations, called _thermal wind_ relate the _vertical shear of the geostrophic flow_ to the _horizontal gradients of buoyancy_. They are very useful for interpreting hydrographic data.

[WOCE Atlantic Atlas Vol. 3](http://whp-atlas.ucsd.edu/whp_atlas/atlantic/a03/sections/printatlas/printatlas.htm)

```python
import xarray as xr
from matplotlib import pyplot as plt
%matplotlib inline
```

```python
! curl -O ftp://ftp.spacecenter.dk/pub/DTU10/2_MIN/DTU10MDT_2min.nc
```

      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  111M  100  111M    0     0   227k      0  0:08:20  0:08:20 --:--:--  190k02  0:09:39  196k:00:15  0:05:16  335k5:53  0:00:20  0:05:33  262k5:21  350k17  0:05:06  262k:24  0:04:11  288k:41  0:04:04  309k  281k      0  0:06:45  0:02:44  0:04:01  304k 0:03:51  310k     0  0:07:47  0:05:18  0:02:29  188k  120kk

```python
ds = xr.open_dataset('DTU10MDT_2min.nc')
ds
```

    <xarray.Dataset>
    Dimensions:  (lat: 5400, lon: 10801)
    Coordinates:
      * lon      (lon) float64 0.0 0.03333 0.06667 0.1 0.1333 0.1667 0.2 0.2333 ...
      * lat      (lat) float64 -89.98 -89.95 -89.92 -89.88 -89.85 -89.82 -89.78 ...
    Data variables:
        mdt      (lat, lon) float64 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ...
    Attributes:
        Conventions:  COARDS/CF-1.0
        title:        DTU10MDT_2min.mdt.nc
        source:       Danish National Space Center
        node_offset:  1

```python
fig, ax = plt.subplots(figsize=(18,12))
ds.mdt.plot(ax=ax)
```

```python

```


# 10 Ordinary differential equations – Integrators
More about integrators to solve ODEs.

* Euler's Rule (and it's shortcomings)
* Runge-Kutta methods

## Euler's Rule

From the last lecture: Given the $n$-dimensional vectors from the ODE standard form

$$
\frac{d\mathbf{y}}{dt} = \mathbf{f}(t, \mathbf{y})
$$
the **Euler rule** amounts to

\begin{align}
\mathbf{f}(t, \mathbf{y}) = \frac{d\mathbf{y}(t)}{dt} &\approx \frac{\mathbf{y}(t_{n+1}) - \mathbf{y}(t_n)}{h}\\
\mathbf{y}_{n+1} &\approx \mathbf{y}_n + h \mathbf{f}(t_n, \mathbf{y}_n) \quad \text{with} \quad \mathbf{y}_n := \mathbf{y}(t_n)
\end{align}

Euler's rule cannot obtain high precision without very small $h$ but this leads to higher round-off error accumulation.

### Euler's Rule applied to the 1D harmonic oscillator (theory)

\begin{alignat}{3}
f^{(0)}(t, \mathbf{y}) &= y^{(1)},
    &\quad y^{(0)}_0 &= x_0,\\
f^{(1)}(t, \mathbf{y}) &=   - \frac{k}{m} y^{(0)},
    &\quad y^{(1)}_0 &= v_0.
\end{alignat}

with $k=1$; $x_0 = 0$ and $v_0 = +1$.

First step from $t=0$ to $t=h$:

\begin{align}
y^{(0)}_1 &= y^{(0)}_0 + h y^{(1)}_0 = x_0 + v_0 h\\
y^{(1)}_1 &= y^{(1)}_0 + h \left(-\frac{k}{m}y^{(0)}_0\right) = v_0 + \frac{-k x_0}{m} h
\end{align}

Compare to the equations of movement under a constant force (*ballistic motion*):

$$
x = x_0 + v_0 h + \frac{1}{2} a h^2, \quad v = v_0 + a h
$$

In Euler, the position does not even contain the $h^2$ term, i.e., the acceleration only contributes *indirectly* via the velocity of the *next* step. (Remember: Euler's error is $\mathcal{O}(h^2)$!)

### Euler's Rule: Application to 3 oscillators (numerical)
`git pull` new resources.

Put the notebook `10-ODE_integrators-students.ipynb` *and* the module `integrators.py` into the same directory, for instance a work directory `~/PHY494/10_ODEs`.

Open [integrators.py](https://github.com/ASU-CompMethodsPhysics-PHY494/PHY494-resources/blob/master/10_ODEs/integrators.py) to have a look at it: it contains code to integrate Newton's equations of motions for different forces and with different integrators (based on the work done in the last lecture).

```python
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
matplotlib.style.use('ggplot')

import integrators_solution as integrators
```

```python
t_max = 20
t_harm, y_harm = integrators.integrate_newton(t_max=t_max)
t_anharm, y_anharm = integrators.integrate_newton(t_max=t_max,
                                                  force=integrators.F_anharmonic)
t_power, y_power = integrators.integrate_newton(t_max=t_max,
                                                force=integrators.F_power)
```

```python
plt.plot(t_harm, y_harm[:, 0], label=r"harmonic $\frac{k}{2} x^2$")
plt.plot(t_anharm, y_anharm[:, 0], label=r"anharmonic $\frac{k}{2} x^2(1 - \frac{2}{3} \alpha x)$")
plt.plot(t_power, y_power[:, 0], label=r"power $\frac{k}{6} x^6$")
plt.legend(loc="best")
```

#### Phase-space portrait
Plot position against momentum ($x, p$):

```python
m = 1
x = y_harm[:, 0]
p = m * y_harm[:, 1]
ax = plt.subplot(1,1,1)
ax.plot(x, p)
ax.set_aspect(1)
```

```python
t, y = integrators.integrate_newton(t_max=40,
                                    force=integrators.F_harmonic,
                                    h=0.01)
t2, y2 = integrators.integrate_newton(t_max=40, h=0.001,
                                      force=integrators.F_harmonic)
```

```python
ax = plt.subplot(1,1,1)
ax.plot(y[:, 0], m * y[:, 1], 'b-', label=r"$h=0.01$")
ax.plot(y2[:, 0], m * y2[:, 1], 'r--', label="$h=0.001$")
ax.set_xlabel("position $x$")
ax.set_ylabel("momentum $p$")
ax.set_aspect(1)
ax.legend(loc="best")
```

#### Energy
Energy is conserved: stringent test for the integrator!

$$
E = T_\text{kin} + U\\
\frac{dE}{dt} = 0
$$
with $T_\text{kin} = \frac{1}{2} m v^2$.

Calculate and plot the three energy terms:

```python
KE = 0.5 * m * y_harm[:, 1]**2
PE = integrators.U_harmonic(y_harm[:, 0])
energy = KE + PE
plt.plot(t_harm, KE, 'r-', t_harm, PE, 'b-', t_harm, energy, 'k--')
```

Harmonic oscillator with larger timestep $h=0.01$

```python
m = 1
t, y = integrators.integrate_newton(t_max=500, force=integrators.F_harmonic, h=0.01, mass=m)
x, v = y.T
KE = 1/2 * m * v**2
PE = integrators.U_harmonic(x)
energy = KE + PE
```

```python
ax = plt.subplot(1,1,1)
ax.plot(t, KE, 'r-', t, PE, 'b-', t, energy, 'k--')
ax.set_xlim(0, 20)
ax.set_ylim(0, 0.7)
ax.set_xlabel("time")
ax.set_ylabel("energy")
ax.set_title("Euler: harmonic oscillator")
```

Note how energy conservation becomes worse when $h$ is increased.

For longer time scales, the energy just keeps on increasing — clearly, we are not solving the equations of motions in a physically meaningful way.

```python
ax.set_ylim(0, 80)
ax.set_xlim(0,500)
ax.figure
```

##### Quantify energy drift
Measure the relative departure from energy conservation

$$
\epsilon_E(t) = \log_{10}\left|\frac{E(t) - E(t=0)}{E(t=0)}\right|
$$

(By using the decadic logarithm, $-\epsilon_E$ is a direct measure of the precision.)

###### Harmonic oscillator
Analyze energy drift for the last harmonic oscillator example ($h=0.01$):

```python
energy_precision = np.log10(np.abs(energy/energy[0] - 1))
plt.plot(t, energy_precision)
plt.ylabel("log(relative error in $E$)")
plt.xlabel("time");
```

As we already know, with such a large time step, energy conservation is very bad with the *Euler* integrator: The relative (!) error quickly becomes 100% (after $t=200$).

##### Code re-use: energy analysis as a function
The module `integrators` contains the function `integrators.analyze_energies(t, y, U)` to automate the task of analysing the energy. You *must* provide the appropriate potential energy function `U`.

##### Problem: Make `analyze_energies()` work and analyze the power law oscillator
The code is *incomplete*.
1. Open `integrators.py` in your editor.
2. Complete the `kinetic_energy()` function (remember: $T_\text{kin} = \frac{1}{2} m v^2$)

Then `reload()` the module:

```python
# when working on the integrator module you need to reload() whenever you make changes
from importlib import reload   # for Python 3
reload(integrators);
```

and try it out for power law oscillator with a large time step $h = 0.01$. Can you find a value of $h$ that will give you energy precision to 4th decimal over 40 time units?

###### Energy stability analysis of the 6-th power oscillator
$x^6$ potential with $h=0.01$:

```python
t, y = integrators.integrate_newton(t_max=40, force=integrators.F_power,
                                    h=0.01)
```

```python
integrators.analyze_energies(t, y, integrators.U_power)
```

Trying 100 times smaller $h$ (takes 100 times longer!!)

```python
t, y = integrators.integrate_newton(t_max=40, force=integrators.F_power,
                                    h=0.0001)
integrators.analyze_energies(t, y, integrators.U_power)
```

*Reducing $h$ further does not seem to help and takes order of magnitude longer to run*. **We need a better integrator!**

## Runge-Kutta methods
Explicit RK methods propagate with the general form

$$
y_{n+1} = y_n + h \sum_{i=1}^s b_i k_i, \quad t_{n+1} = t_n + h
$$

where $s$ is the number of *stages* and the coefficients $a_{ij}$, $b_i$ and $c_i$ define the method.

\begin{align}
k_1 &= f(t_n, y_n)\\
k_2 &= f(t_n + c_2 h, y_n + h(a_{21} k_1))\\
k_3 &= f(t_n + c_3 h, y_n + h(a_{31} k_1 + a_{32} k_2))\\
 \vdots& \\
k_s &= f(t_n + c_s h, y_n + h(a_{s1} k_1 + a_{s2} k_2 + \cdots + a_{s,s-1} k_{s-1})).
\end{align}

Basic idea: improve estimate for the slope by using weighted averages of slopes over the interval.

### RK4
General purpose ODE solver.

$$
y_{n+1} = y_n + \frac{h}{6}\Big(k_1 + 2 k_2 + 2 k_3 + k_4\Big), \quad t_{n+1} = t_n + h
$$

and

\begin{align}
k_1 &= f(t_n, y_n)\\
k_2 &= f(t_n + \frac{1}{2} h, y_n + \frac{1}{2} h k_1)\\
k_3 &= f(t_n + \frac{1}{2} h, y_n + \frac{1}{2} h k_2)\\
k_4 &= f(t_n + h, y_n + h k_3)
\end{align}

`rk4` can obtain $\mathcal{O}(h^4)$ precision.

### RK2
`rk2` is simpler to derive and understand than `rk4` (which we will use more often).

Start with the exact solution

$$
\frac{dy}{dt} = f(t, y) \quad\text{and}\quad y(t) = \int f(t, y)dt\\
y_{n+1} = y_n + \int_{t_n}^{t_{n+1}} f(t, y)dt
$$

Expand $f(t, y)$ around *midpoint* $t_{n+1/2} = t + \frac{h}{2}$ of integration interval

\begin{gather}
f(t, y) \approx f(t_{n+1/2}, y_{n+1/2}) + (t - t_{n+1/2})\,  f^{(1)}(t_{n+1/2}) +
   \mathcal{O}(h^2)
\end{gather}

Integrate the *expansion* (Taylor series around midpoint):
\begin{align}
\int_{t_n}^{t_{n+1}} f(t, y) \, dt &\approx h\, f(t_{n+1/2}, y_{n+1/2}) + \mathcal{O}(h^3)\\
y_{n+1} &= y_n + h\, f(t_{n+1/2}, y_{n+1/2}) + \mathcal{O}(h^3)
\end{align}

I.e., use **slope at midpoint**! (The $\mathcal{O}(h^2)$ term in the integral is zero because $[(h/2)^2 - (h/2)^2] = 0$.)

But we don't have $y_{n+1/2}$ so we use Euler's rule to estimate it to $\mathcal{O}(h^2)$:

$$
y_{n+1/2} \approx y_n + \frac{1}{2} h\, f(t_n, y_n)
$$

and insert into $y_{n+1} = y_n + h\, f(t_{n+1/2}, y_{n+1/2})$ (so it is still $\mathcal{O}(h^3)$ accurate).

Altogether: **RK2** algorithm

$$
\mathbf{y}_{n+1} = \mathbf{y}_n + h \mathbf{k}_2, \quad t_{n+1} = t_n + h
$$

and

\begin{align}
\mathbf{k}_1 &= \mathbf{f}(t_n, \mathbf{y}_n)\\
\mathbf{k}_2 &= \mathbf{f}(t_n + \frac{1}{2} h, \mathbf{y}_n + \frac{1}{2} h \mathbf{k}_1)
\end{align}

with $\mathcal{O}(h^3)$ accuracy.

#### RK2 applied to the 1D harmonic oscillator (theory)

\begin{alignat}{3}
f^{(0)}(t, \mathbf{y}) &= y^{(1)},
    &\quad y^{(0)}_0 &= x_0,\\
f^{(1)}(t, \mathbf{y}) &=   - \frac{k}{m} y^{(0)},
    &\quad y^{(1)}_0 &= v_0.
\end{alignat}

with $k=1$; $x_0 = 0$ and $v_0 = +1$.

First step from $t=0$ to $t=h$:

\begin{align}
  y^{(0)}_{1} &=
    y^{(0)} _{0} + hf^{(0)} \Big(\frac{h}{2}, y^{(0)}_{0} + hk_{1} \Big)\\
    &= x_{0} + h\left[v_{0} + \frac{h}{2} \left(-\frac{k}{m} x_{0}\right)  \right]\\
    &= x_0 + v_0 h + \frac{1}{2} \left(-\frac{k}{m} x_{0}\right) h^2\\
  y^{(1)}_{1} &= y^{(1)}_{0} +
                h f^{(1)}\left(\frac{h}{2},
                     \mathbf{y}_{0} + \frac{h}{2} \mathbf{f}(0, \mathbf{y}_{0})\right) \\
             &= v_{0} + \frac{h}{m} \left[-k\left(x_{0} - \frac{h}{2} k x_{0}\right)\right]
\end{align}

Note that `rk2` contains a $h^2$ term in the position update.

#### RK2 applied in practice
Let's implement `rk2`

\begin{align}
\mathbf{y}_{n+1} &= \mathbf{y}_n + h \mathbf{k}_2\\
\mathbf{k}_1 &= \mathbf{f}(t_n, \mathbf{y}_n)\\
\mathbf{k}_2 &= \mathbf{f}(t_n + \frac{1}{2} h, \mathbf{y}_n + \frac{1}{2} h \mathbf{k}_1)
\end{align}

in our `integrators.py`.

```python
from importlib import reload
reload(integrators);
```

##### RK2 for the power potential

```python
t, y = integrators.integrate_newton(t_max=40, force=integrators.F_power,
                                    integrator=integrators.rk2, h=0.01)
```

###### phase-space portrait

```python
plt.plot(y[:, 0], m*y[:, 1])
```

###### energy drift

```python
integrators.analyze_energies(t, y, integrators.U_power)
```

#### RK4 applied in practice
Implement `rk4` in `integrator.py`:

\begin{align}
y_{n+1} &= y_n + \frac{h}{6}\Big(k_1 + 2 k_2 + 2 k_3 + k_4\Big)\\
k_1 &= f(t_n, y_n)\\
k_2 &= f(t_n + \frac{1}{2} h, y_n + \frac{1}{2} h k_1)\\
k_3 &= f(t_n + \frac{1}{2} h, y_n + \frac{1}{2} h k_2)\\
k_4 &= f(t_n + h, y_n + h k_3)
\end{align}

```python
reload(integrators)
```

    <module 'integrators_solution' from '/Volumes/Data/oliver/Documents/Teaching/ASU/CompPhys_PHY494/2020/PHY494-resources/10_ODEs/integrators_solution.py'>

and test the 6th power potential again. What level of energy precision can you obtain over 40 time units?

##### RK4 for the 6th power potential
Note that we can easily run `rk4` with a $h=0.01$!

```python
t, y = integrators.integrate_newton(t_max=40, force=integrators.F_power,
                                    integrator=integrators.rk4, h=0.01)
```

###### phase-space protrait

```python
plt.plot(y[:, 0], m*y[:, 1])
```

###### energy drift

```python
integrators.analyze_energies(t, y, integrators.U_power)
```

You should be able to get 8 - 11 decimals precision from `rk4` with a smaller $h$.

##### High precision RK4 for the 6th power potential
Try a 10 times smaller step $h=0.001$

```python
t, y = integrators.integrate_newton(t_max=40, force=integrators.F_power,
                                    integrator=integrators.rk4, h=0.001)
integrators.analyze_energies(t, y, integrators.U_power)
```

This looks like stable integration with an error of not worse than $10^{-12}$: *Looking very good!* (Especially when you compare it to *Euler* with $h=0.001$!)

The error spikes when the potential energy is dominant, which happens near the classical turning points where (in this case) the potential is the steepest and the forces are highest.

```python

```