# Task 1 - Simple Linear Classification

In this assignment you will practice putting together a simple image classification pipeline, based on the SVM and Softmax classifier.

The goals of this assignment are as follows:
* understand the basic Image Classification pipeline and the data-driven approach (train/predict stages)
* understand the train / val / test splits and the use of validation data for hyperparameter tuning.
* develop proficiency in writing efficient vectorized code with numpy

* implement and apply a Softmax classifier
* implement and apply a Multiclass Support Vector Machine (SVM) classifier

* understand the differences and tradeoffs between these classifiers

---
## 0 - Import necessary libraries and informations

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

np.set_printoptions(precision=8)
```

Write down your Name and Student ID

```python
## --- start your code here ----

NIM = 1301174682
Nama = "Muhammad Ihza Mahendra"

## --- end your code here ----
```

---
---
## 1 - Load CIFAR-10 Dataset

* First, Obtain Cifar-10 dataset.
  There are from various source in Internet like [Keras](https://keras.io/datasets/), [Tensorflow](https://www.tensorflow.org/api_docs/python/tf/keras/datasets), or any other source
* Next you will prepare the dataset by first:
 * visualizing data
 * split into training, validation, and testing set
 * normalize data

---
### a. Import Data ***CIFAR-10***

```python
import tensorflow as tf

(X_train, y_train), (X_test, y_test) = tf.keras.datasets.cifar10.load_data()
classes = ['plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'forse', 'ship', 'truck']

```

    Downloading data from https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz
    170500096/170498071 [==============================] - 6s 0us/step

Check your implementation

```python
print('X_train.shape =',X_train.shape)
print('y_train.shape =',y_train.shape)
print('X_test.shape  =',X_test.shape)
print('y_test.shape  =',y_test.shape)
```

    X_train.shape = (50000, 32, 32, 3)
    y_train.shape = (50000, 1)
    X_test.shape  = (10000, 32, 32, 3)
    y_test.shape  = (10000, 1)

**Expected Output**:

X_train.shape = (50000, 32, 32, 3)
y_train.shape = (50000, 1)
X_test.shape  = (10000, 32, 32, 3)
y_test.shape  = (10000, 1)

---
### b. Visualizing Data

Show the first 20 images from X_train

```python
fig, ax = plt.subplots(2,10,figsize=(15,4.5))
fig.subplots_adjust(hspace=0.1, wspace=0.1)
for j in range(0,2):
    for i in range(0, 10):
        ax[j,i].imshow(X_train[i+j*10])
        ax[j,i].set_title(classes[y_train[i+j*10,0]])
        ax[j,i].axis('off')
plt.show()
```

---
### c. Split Training Data

#####**EXERCISE**:
* Cut the `last 10000 data` from `Training Set`, and save it as `Validation Set`

```python
X_train = X_train.astype('float32')
X_test  = X_test.astype('float32')

X_val = X_train[-X_train.shape[0]//5:, :, :]
y_val = y_train[-y_train.shape[0]//5:, :]

X_train = X_train[:-X_train.shape[0]//5, :, :]
y_train = y_train[:-y_train.shape[0]//5, :]

```

Check your implementation

```python
print('X_val.shape   =',X_val.shape)
print('y_val.shape   =',y_val.shape)
print('X_train.shape =',X_train.shape)
print('y_train.shape =',y_train.shape)
```

    X_val.shape   = (10000, 32, 32, 3)
    y_val.shape   = (10000, 1)
    X_train.shape = (40000, 32, 32, 3)
    y_train.shape = (40000, 1)

**Expected Output**:

X_val.shape   = (10000, 32, 32, 3)
y_val.shape   = (10000, 1)
X_train.shape = (40000, 32, 32, 3)
y_train.shape = (40000, 1)

---
### d. Normalizing Data

##### **EXERCISE**:
* Normalize `X_train`, `X_val`, and `X_test` by *zero-centering* them:
    1. calculate the `mean` of training data `X_train`
    * subtract `X_train`, `X_val`, and `X_test` using mean of `X_train`

```python
mean_image = np.mean(X_train)

X_train = (X_train - mean_image)/np.std(X_train)
X_val   = X_val - mean_image/np.std(X_train)
X_test  = X_test - mean_image/np.std(X_train)
```

```python
mean_image
```

    120.67766

Check your implementation

```python
print('np.mean(X_train) =',np.mean(X_train))
print('np.mean(X_val)   =',np.mean(X_val))
print('np.mean(X_test)  =',np.mean(X_test))
```

    np.mean(X_train) = 4.9464265e-06
    np.mean(X_val)   = 0.14816807
    np.mean(X_test)  = 0.8513755

**Expected Output**:

np.mean(X_train) = -9.2315673e-07
np.mean(X_val)   = 0.14794703
np.mean(X_test)  = 0.8511902

---
### e. Reshape Data

#####**EXERCISE**:
* Reshape each data in `X_train`, `X_val`, and `X_test` into 1-dimensional matrix

*Hint: use `np.reshape()`*

```python
X_train = np.reshape(X_train, (X_train.shape[0], -1))
X_val   = np.reshape(X_val, (X_val.shape[0], -1))
X_test  = np.reshape(X_test, (X_test.shape[0], -1))
```

Check your implementation

```python
print('X_train.shape =',X_train.shape)
print('X_val.shape   =',X_val.shape)
print('X_test.shape  =',X_test.shape)
```

    X_train.shape = (40000, 3072)
    X_val.shape   = (10000, 3072)
    X_test.shape  = (10000, 3072)

**Expected Output**:

X_train.shape = (40000, 3072)
X_val.shape   = (10000, 3072)
X_test.shape  = (10000, 3072)

#####**EXERCISE**:
* Reshape `y_train`, `y_val`, and `y_test` into a vector

*Hint: use `np.ravel()`*

```python
y_train = np.ravel(y_train)
y_val   = np.ravel(y_val)
y_test  = np.ravel(y_test)
```

Check your implementation

```python
print('y_train.shape =',y_train.shape)
print('y_val.shape   =',y_val.shape)
print('y_test.shape  =',y_test.shape)
```

    y_train.shape = (40000,)
    y_val.shape   = (10000,)
    y_test.shape  = (10000,)

**Expected Output**:

y_train.shape = (40000,)
y_val.shape   = (10000,)
y_test.shape  = (10000,)

---
---
## 2 - Linear Function
* Complete the forward and backward function of basic linear regression function

---
### a. Forward Function

#####**EXERCISE**:
* Implement `forward function` for Linear Classifier as follow:

$$
\begin{align}
f(x, W, b) = x.W + b
\end{align}
$$

```python
def forward(x, W, b):
    v = np.dot(x,W)+b             # x dot W + b

    return v
```

Check your implementation

```python
np.random.seed(2020)

X = np.array([[0, 0, 1], [0, 1, 1], [1, 0, 1], [1, 1, 1]])
W = np.random.random((3, 2))
b = np.zeros((1, 2))

v = forward(X, W, b)

print('v.shape =', v.shape)
print('v =')
print(v)

```

    v.shape = (4, 2)
    v =
    [[0.33691873 0.21695427]
     [0.84666425 0.48878998]
     [1.32319556 1.09034621]
     [1.83294108 1.36218193]]

**Expected Output**:

v.shape = (4, 2)
v =
[[0.33691873 0.21695427]
 [0.84666425 0.48878998]
 [1.32319556 1.09034621]
 [1.83294108 1.36218193]]

---
### b. Backward Function

#####**EXERCISE**:
* Implement `backward function` for Linear Classifier as follow:

$$
\begin{align*}
\partial W & = x^T.\partial out \\
\partial b & = \sum \partial out \\
\partial x & = \partial out.W^T \\
\end{align*}
$$

*Hint: use `axis=0` and `keepdims=True` to calculate $\partial b$*

```python
def backward(dout, x, W, b):
    dW = np.dot(x.T,dout)                        # x.T dot dout
    db = np.sum(dout, axis=0, keepdims=True)                        # sum dout, axis=0, keepdims=True
    dx = np.dot(dout,W.T)               # dout dot W.T

    return dW, db, dx
```

Check your implementation

```python
np.random.seed(2020)

dout = np.random.random((4, 2))
dW, db, dX = backward(dout, X, W, b)

```

```python
print('dW.shape =', dW.shape)
print('dW =')
print(dW)
```

    dW.shape = (3, 2)
    dW =
    [[0.61339587 0.56026986]
     [0.78622267 0.61515131]
     [2.10941822 1.70549752]]

**Expected Output**:

dW.shape = (3, 2)
dW =
[[0.61339587 0.56026986]
 [0.78622267 0.61515131]
 [2.10941822 1.70549752]]

```python
print('db.shape =', db.shape)
print('db =', db)
```

    db.shape = (1, 2)
    db = [[2.10941822 1.70549752]]

**Expected Output**:

db.shape = (1, 2)
db = [[2.10941822 1.70549752]]

```python
print('dX.shape =', dX.shape)
print('dX =')
print(dX)
```

    dX.shape = (4, 3)
    dX =
    [[1.73555547 0.74016932 0.52178124]
     [0.74016932 0.33373516 0.23071873]
     [0.52178124 0.23071873 0.16058338]
     [0.57253207 0.23425843 0.16763411]]

**Expected Output**:

dX.shape = (4, 3)
dX =
[[1.73555547 0.74016932 0.52178124]
 [0.74016932 0.33373516 0.23071873]
 [0.52178124 0.23071873 0.16058338]
 [0.57253207 0.23425843 0.16763411]]

---
---
## 3 - Softmax Function

* A very important concept to understand in numpy is `"broadcasting"`.
* It is very useful for performing mathematical operations between arrays of different shapes.
* For the full details on broadcasting, you can read the official [broadcasting documentation](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html).

---
### a. Softmax Score

#####**EXERCISE**:
* Implement a `softmax score function` using numpy.

for a matrix $x \in \mathbb{R}^{m\times n}$
- $ n $   : number of data
- $ m $  : number of label
$x_{ij}$ maps to the element in the $i^{th}$ row and $j^{th}$ column of $x$

thus we have
$$softmax(x) = softmax\begin{bmatrix}
    x_{11} & x_{12} & x_{13} & \dots  & x_{1n} \\
    x_{21} & x_{22} & x_{23} & \dots  & x_{2n} \\
    \vdots & \vdots & \vdots & \ddots & \vdots \\
    x_{m1} & x_{m2} & x_{m3} & \dots  & x_{mn}
\end{bmatrix} = \begin{bmatrix}
    \frac{e^{x_{11}}}{\sum_{j}e^{x_{1j}}} & \frac{e^{x_{12}}}{\sum_{j}e^{x_{1j}}} & \frac{e^{x_{13}}}{\sum_{j}e^{x_{1j}}} & \dots  & \frac{e^{x_{1n}}}{\sum_{j}e^{x_{1j}}} \\
    \frac{e^{x_{21}}}{\sum_{j}e^{x_{2j}}} & \frac{e^{x_{22}}}{\sum_{j}e^{x_{2j}}} & \frac{e^{x_{23}}}{\sum_{j}e^{x_{2j}}} & \dots  & \frac{e^{x_{2n}}}{\sum_{j}e^{x_{2j}}} \\
    \vdots & \vdots & \vdots & \ddots & \vdots \\
    \frac{e^{x_{m1}}}{\sum_{j}e^{x_{mj}}} & \frac{e^{x_{m2}}}{\sum_{j}e^{x_{mj}}} & \frac{e^{x_{m3}}}{\sum_{j}e^{x_{mj}}} & \dots  & \frac{e^{x_{mn}}}{\sum_{j}e^{x_{mj}}}
\end{bmatrix}$$

it is equal to
$$softmax(x)  = \begin{pmatrix}
    softmax\text{(first row of x)}  \\
    softmax\text{(second row of x)} \\
    ...  \\
    softmax\text{(last row of x)} \\
\end{pmatrix} $$

```python
def softmax(x):
    """Calculates the softmax for each row of the input x.

Argument:
    x -- A numpy matrix of shape (n,m)

    Returns:
    score -- A numpy matrix equal to the softmax of x, of shape (n,m)
          -- normalized log probabilities score
    """

    # 1. COMPUTE THE CLASS PROBABILITIES

    # shift x by subtracting with its maximum value . Use np.max(...)
    x = x - np.max(x)

    # Apply exp() element-wise to x. Use np.exp(...).
    x_exp = np.exp(x)

    # Create a vector X_sum that sums each row of X_exp.
    # Use np.sum(..., axis = 1, keepdims = True).
    x_sum = np.sum(x_exp, axis = 1, keepdims = True )

    # Compute softmax(x) score by dividing X_exp by X_sum.
    # It should automatically use numpy broadcasting.
    score = x_exp / x_sum

    return score
```

Check your implementation

```python
np.random.seed(2020)

x = np.array([
    [1, 3, 0, 1 ,1],
    [0, 2, 0, 1, 9],
    [0, 1, 0, 1, 0]])

W = np.random.random((5, 2))
b = np.zeros((1, 2))

score = forward(x, W, b)

print("score = " )
print(score,'\n')
print("softmax(score) = " )
print(softmax(score))
```

    score =
    [[3.65414948 2.18891435]
     [9.05539861 2.29728405]
     [0.78622267 0.61515131]]

    softmax(score) =
    [[0.81233207 0.18766793]
     [0.99883993 0.00116007]
     [0.54266384 0.45733616]]

**Expected Output**:

score =
[[3.65414948 2.18891435]
 [9.05539861 2.29728405]
 [0.78622267 0.61515131]]

softmax(score) =
[[0.81233207 0.18766793]
 [0.99883993 0.00116007]
 [0.54266384 0.45733616]]

loss   = 2.526128866233045

dscore =
[[-0.06255598  0.06255598]
 [ 0.33294664 -0.33294664]
 [-0.15244539  0.15244539]]

---
### b. Softmax Loss

#####**EXERCISE**:
* Implement a `softmax loss` function using numpy.

- Loss of the `i-th data` is the normalized log probability of the score at the class should be

  $$L_i =  -\log(\frac{e^{X_{y_i}}}{\sum_{j}e^{X_{j}}})$$

- The Softmax Loss is the average of all data loss

  $$L = \frac{1}{N}\sum^N_{i=1}L_i$$

```python
def softmax_loss(score, y):
    """Calculates the softmax loss for each row of the input x.

    Argument:
    score -- A numpy matrix equal to the softmax of x, of shape (n,m)
          -- normalized log probabilities score

    y     -- A numpy vector of shape(n,)
          -- containing training labels;
          -- y[i] = c means that X[i] has label c, where 0 <= c < C.

    Returns:
    dscore -- A numpy matrix equal to the softmax of x, of shape (n,m)
           -- gradient score of softmax
    """

    num_examples = score.shape[0]

    # 2. COMPUTE THE LOSS : average cross-entropy loss
    #make a number list containing [1 2 3 ... n]
#     number_list = list(range(num_examples))
    number_list = [i for i in range(1, num_examples+1)]

    # calculate the correct log probability of score[number_list,y] by applycing -np.log(...)
    corect_logprobs = - np.log(score[range(num_examples), y])

    # average the correct log probability, use np.sum then divide it by num_examples
    loss = np.sum(corect_logprobs) / num_examples

    # 3. COMPUTE THE GRADIENT ON SCORES
    dscores = score
    dscores[range(num_examples),y] -= 1
    dscores /= num_examples

    return loss, dscores
```

Check your implementation

```python
np.random.seed(2020)

x = np.array([
    [1, 3, 0, 1 ,1],
    [0, 2, 0, 1, 9],
    [0, 1, 0, 1, 0]])

y = np.array([
    0, 1, 0
])

W = np.random.random((5, 2))
b = np.zeros((1, 2))

score = forward(x, W, b)

print("score = " )
print(score, '\n')

score = softmax(score)
print("softmax(score) = " )
print(score,'\n')

loss, dscore  = softmax_loss(score, y)
print('loss   =',loss, '\n')
print('dscore =')
print(dscore)

```

    score =
    [[3.65414948 2.18891435]
     [9.05539861 2.29728405]
     [0.78622267 0.61515131]]

    softmax(score) =
    [[0.81233207 0.18766793]
     [0.99883993 0.00116007]
     [0.54266384 0.45733616]]

    loss   = 2.526128866233045

    dscore =
    [[-0.06255598  0.06255598]
     [ 0.33294664 -0.33294664]
     [-0.15244539  0.15244539]]

**Expected Output**:

score =
[[3.65414948 2.18891435]
 [9.05539861 2.29728405]
 [0.78622267 0.61515131]]

softmax(score) =
[[0.81233207 0.18766793]
 [0.99883993 0.00116007]
 [0.54266384 0.45733616]]

loss   = 2.526128866233045

dscore =
[[-0.06255598  0.06255598]
 [ 0.33294664 -0.33294664]
 [-0.15244539  0.15244539]]

---
---
## 4 - Linear Classifier: Softmax Classifier
* Combine linear function and softmax function that you have completed to train a multiclass softmax classifier

---
### a. Weight Initialization

#####**EXERCISE**:
* Implement `Weight initialization function`
    - receive input size `n`, number of neuron `d`, and standard deviation `std`
    - generate normal random matrix `W` of size `(n x d)` and multiply it by `std`
    - generate zeros matrix `b` of size `(1 x d)`

```python
def initialize_weights(n, d, std):
    W = np.random.random(size=(n,d)) * std                           # np.random.random * std
#     print(W)
    b = np.zeros(shape=(1,d))

    return W, b
```

```python
W, b = initialize_weights(20, 10, 0.001)
print('shape W:', W.shape)
print('shape b:', b.shape)
```

    shape W: (20, 10)
    shape b: (1, 10)

**Expected Output**:

shape W: (20, 10)
shape b: (1, 10)

---
### b. Training Function

#####**EXERCISE**:
* Implement **Training Function**
    * call `forward function`
    * call `softmax function`
    * call `softmax_loss function`
    * call `backward function`
    * implement `weight update`

```python
def train(X, y, W=None, b=None, learning_rate=1e-6, reg=1e4, num_iters=100, batch_size=200, verbose=False):

    num_classes = np.max(y) + 1 # assume y takes values 0...K-1 where K is number of classes

    # get number of training data and its dimension from X
    num_train, dim = X.shape

    # initialize new weights if W matrix is not provided
    # call initialize_weights function using dim, num_classes, and std=0.001
    if W is None:
        W, b = initialize_weights(dim, num_classes, std=0.001)

    # Run stochastic gradient descent to optimize W
    loss_history = []

    for it in range(num_iters):
        X_batch = None
        y_batch = None

        # Randomly select indices from training examples
        train_rows = np.arange(num_train)
        idxs = np.random.choice(train_rows, batch_size, replace=False)

        X_batch = X[idxs]
        y_batch = y[idxs]

        # calculate class score by calling forward function using X_batch, W, and b
        scores = forward(X_batch, W, b)

        # calculate softmax score by calling softmax function using scores
        softmaX_score = softmax(scores)

        # evaluate loss and gradient by calling softmax_loss function using softmax_score and y_batch
        loss, dout = softmax_loss(softmaX_score, y_batch)

        # append the loss history
        loss_history.append(loss)

        # calculate weights gradient by calling backward function using dout, X_batch, W, and b
        dW, db, _ = backward(dout, X_batch, W, b)

        # perform regulatization gradient
        dW += reg*W

        # perform parameter update by subtracting W and b with a fraction of dW and db
        # according to the learning rate
        W -= dW * learning_rate
        b -= db * learning_rate

        if verbose and it % 100 == 0:
            print ('iteration', it,'/',num_iters, ': loss =', loss)
    return loss_history, W, b
```

---
### c. Train the Softmax Classifier

Try the training Function using the initial parameter

```python
print(X_train.shape, y_train.shape)
```

    (40000, 3072) (40000,)

```python
loss, W, b = train(X_train, y_train)
```

Visualize the loss

```python
plt.plot(loss)
plt.show()
```

**Expected Output**:
The graph should show that the loss is generally decreased as the training goes

---
### d. Predict Function

#####**EXERCISE**:
* Implement **Predict Function**
    * call `forward function`

```python
def predict(X, W, b):
    y_pred = np.zeros(X.shape[1])

    # calculate class score by calling forward function using X, W, and b
    y_pred = forward(X, W, b)

    # take the maximum prediction and use that column to get the class
    y_pred = y_pred.argmax(axis=-1)

    return y_pred
```

Check your implementation

```python
import sklearn
from sklearn.metrics import accuracy_score

y_pred = predict(X_train, W, b)
accuracy = sklearn.metrics.accuracy_score(y_train, y_pred)

print('Training Accuracy =', accuracy*100,'%')

print('Training label  =', y_test[:15])
print('Predicted label =', y_pred[:15])
```

    Training Accuracy = 15.1425 %
    Training label  = [3 8 8 0 6 6 1 6 3 1 0 9 5 7 9]
    Predicted label = [1 0 9 4 0 4 8 2 3 6 4 6 9 3 0]

**Expected Output**:

You should get about **`27-29%`** accuracy on training set using the initial run

---

```python
y_pred = predict(X_val, W, b)
accuracy = sklearn.metrics.accuracy_score(y_val, y_pred)
print('Validation Accuracy =', accuracy*100,'%')

print('Validation label =',y_test[:15])
print('Predicted label  =',y_pred[:15])
```

    Validation Accuracy = 16.07 %
    Validation label = [3 8 8 0 6 6 1 6 3 1 0 9 5 7 9]
    Predicted label  = [0 8 4 8 4 0 4 4 7 4 7 4 0 7 0]

**Expected Output**:

You should also get about **`27-29%`** accuracy on validation set

You can retrain further the weights by adding the pre-trained `W` and `b` to the arguments when calling training function

---
---
## 5 - Hyperparameter Tuning
* After you find that the classifier is working properly, next
* Find the best Learning Rate and Regularization Strength

---
###a. Try Another Hyperparameter Combination

Try to add another combination of learning rates and regularization strength

#####**EXERCISE**:
* Use the `validation set` to tune hyperparameters (regularization strength and learning rate).
     * Greedily loop over `learning_rates` and `regulatization` to get all combination

* You **should** experiment with different ranges for the learning rates and regularization strengths;

```python
import warnings
warnings.filterwarnings('ignore')
np.random.seed(None)

results = {}
best_val = -1
best_W = None
best_b = None
learning_rates = [1e-7, 5e-7]
regularization_strengths = [9e4, 1e5]

iterations = 2000

# Greedily loop over learning_rates and regularization_strengths
for lr in learning_rates:
    for reg in regularization_strengths:
        print('Running {} iterations, rate = {}, reg = {}'.format(iterations, lr, reg))
        # call train function using the learning rate and regularization selected
        loss, W, b = train(X_train, y_train,
                              learning_rate=lr, reg=reg,
                              num_iters=iterations, verbose=False)

        # call predict function using pretrained W and b on X_train and X_val to evaluate
        y_train_pred = predict(X_train, W, b)
        y_val_pred = predict(X_val, W, b)

        # calculate the accuracy
        train_accuracy = np.mean(y_train == y_train_pred)
        val_accuracy   = np.mean(y_val == y_val_pred)

        print ('lr = {}, reg = {}, test accuracy = {}, validation accuracy = {}'.format(
            lr, reg, train_accuracy, val_accuracy))

        # store the result accuracy combination
        results[(lr, reg)] = (train_accuracy, val_accuracy)

        # store the best Weight and Bias
        if val_accuracy > best_val:
            best_W   = W
            best_b   = b
            best_val = val_accuracy

# Print out results.
for lr, reg in sorted(results):
    train_accuracy, val_accuracy = results[(lr, reg)]
    print('lr',lr,',reg',reg,', train accuracy: ',train_accuracy*100,'%, val accuracy: ', val_accuracy ,'%')

print()
print('best validation accuracy achieved during cross-validation:',best_val*100,'%')
print('best Weights and bias are stored in `best_W` and `best_b`')
```

    Running 2000 iterations, rate = 1e-07, reg = 90000.0
    lr = 1e-07, reg = 90000.0, test accuracy = 0.245125, validation accuracy = 0.2461
    Running 2000 iterations, rate = 1e-07, reg = 100000.0
    lr = 1e-07, reg = 100000.0, test accuracy = 0.248275, validation accuracy = 0.2492
    Running 2000 iterations, rate = 5e-07, reg = 90000.0
    lr = 5e-07, reg = 90000.0, test accuracy = 0.2419, validation accuracy = 0.2448
    Running 2000 iterations, rate = 5e-07, reg = 100000.0
    lr = 5e-07, reg = 100000.0, test accuracy = 0.245875, validation accuracy = 0.2481
    lr 1e-07 ,reg 90000.0 , train accuracy:  24.5125 %, val accuracy:  0.2461 %
    lr 1e-07 ,reg 100000.0 , train accuracy:  24.8275 %, val accuracy:  0.2492 %
    lr 5e-07 ,reg 90000.0 , train accuracy:  24.19 %, val accuracy:  0.2448 %
    lr 5e-07 ,reg 100000.0 , train accuracy:  24.587500000000002 %, val accuracy:  0.2481 %

    best validation accuracy achieved during cross-validation: 24.92 %
    best Weights and bias are stored in `best_W` and `best_b`

**Expected Output**:

if you are careful you should be able to get a classification accuracy of **over 35%** on the validation set

---
### b. Test the Trained Weights

#####**EXERCISE**:
* Test the **Best Weights and Bias** to the `X_test`

```python
y_pred = predict(X_test, best_W, best_b)

accuracy = sklearn.metrics.accuracy_score(y_test, y_pred)

print('Testing Accuracy =', accuracy*100,'%')
print('Test label       =', y_test[:15])
print('Predicted label  =', y_pred[:15])
```

    Testing Accuracy = 25.040000000000003 %
    Test label       = [3 8 8 0 6 6 1 6 3 1 0 9 5 7 9]
    Predicted label  = [6 8 8 8 6 6 6 6 0 9 0 9 6 6 9]

**Expected Output**:

You should reach at least  **32%** accuracy on test set

---
---
## 6 - Vizualize the Trained Weights

Visualize the learned weights for each class for each class

---
### a. Retrieve one class weights
Try to visualize one trained weights.

**Note:** You can change the `id`

```python
id = 0

# get the id-th column from best weight
w = best_W[:,id]

print('w.shape =',w.shape)
```

    w.shape = (3072,)

**Expected Output**:

w.shape = (3072,)

---
### b. Normalize the weight for visualization
Image plot can only receive input range [0..1] for float type, or [0..255] for integer type, and the weights are not in those range.

So we need to normalize. Here we use MinMaxScaler from sklearn toolkit

```python
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler((0.05,.95))
w = scaler.fit_transform(w.reshape(-1, 1))

print('w.shape =',w.shape)
```

    w.shape = (3072, 1)

**Expected Output**:

w.shape = (3072, 1)

---
### c. Visualize the weight
Now reshape the weight back to (32,32,3), and plot the image. Use `np.reshape()`

```python
w = np.reshape(w,(32,32,3))

plt.figure(figsize=(2,2))
plt.imshow(w)
plt.title(classes[id])
plt.show()
```

---
### d. Do it to the other weights

#####**EXERCISE**:
* complete the implementation to show the trained weight visualizations

```python
fig, ax = plt.subplots(1,10,figsize=(18,5))
fig.subplots_adjust(hspace=0.1, wspace=0.1)
for i in range(0, 10):
    # get the i-th weight from best_W
    w = best_W[:,i]

    # normalize the weight (see previous example)
    w = scaler.fit_transform(w.reshape(-1, 1))

    # reshape the weight
    w = np.reshape(w,(32,32,3))

    ax[i].imshow(w)
    ax[i].axis('off')
    ax[i].set_title(classes[i])
plt.show()
```

**Expected Output**:

You should see that the weights visualization should  match the ilustration below

---
---
## 7 - Linear Classifier: Multiclass SVM
Similar to before, let's build a Linear classifier to classify cifar-10 dataset, but now using Multiclass SVM loss, or Hinge Loss

---
### a. SVM Loss
Calculates the SVM loss for each row of the input x

#####**EXERCISE**:
* Implement `Multiclass SVM Loss` or `Hinge Loss`

```python
def svm_loss(score, y):
    """
    Argument:
    score -- A numpy matrix equal to the softmax of x, of shape (n,m)
          -- normalized log probabilities score

    y     -- A numpy vector of shape(n,)
          -- containing training labels;
          -- y[i] = c means that X[i] has label c, where 0 <= c < C.

    Returns:
    dscore -- A numpy matrix equal to the softmax of x, of shape (n,m)
           -- gradient score of softmax
    """

    num_examples = score.shape[0]

    # 1. COMPUTE THE SVM LOSS

    # get the correct class score. Use np.choose( , ) using y and score.T
    correct_scores = np.choose(y,score.T)

    # calculate margin: score.T subtracted by correct_score, then add by 1
    margin = score.T - correct_scores + 1

    # remove all margin below 0. Use np.maximum(0, margin)
    margin = np.maximum(0, margin)

    # COMPUTE THE LOSS
    # calculate SVM loss of each data: sum of margin of non-target class
    # simply total all the margin (use np.sum()), and subtract it by 1
    loss_i = np.sum(margin) - 1

    # average loss_i by num_examples
    loss = loss_i/num_examples

    # 2. COMPUTE THE GRADIENT ON SCORES
    # margins for the correct class have already been set to 0
    dscores = (margin.T > 0).astype(float)
    marginsSum = dscores.sum(1) - 1
    dscores[range(dscores.shape[0]), y] = -marginsSum

    return loss, dscores
```

Check your implementation

```python
np.random.seed(2020)

x = np.array([
    [9, 2, 5, 0, 0],
    [7, 5, 0, 0 ,0],
    [1, 0, 3, 5 ,0]])

y = np.array([
    0, 0, 1
])

W = np.random.random((5, 2))
b = np.zeros((1, 2))

score = forward(x, W, b)

print("score = " )
print(score)
print()

loss, dscore  = svm_loss(score, y)
print('loss   =',loss)
print('dscore =')
print(dscore)

```

    score =
    [[11.58057615  9.48897027]
     [ 9.45266543  7.47292219]
     [ 3.37941873  3.2408327 ]]

    loss   = 1.0461953410332576
    dscore =
    [[-0.  0.]
     [-0.  0.]
     [ 1. -1.]]

**Expected Output**:

score =
[[11.58057615  9.48897027]
 [ 9.45266543  7.47292219]
 [ 3.37941873  3.2408327 ]]

loss   = 0.37952867436659093
dscore =
[[-0.  0.]
 [-0.  0.]
 [ 1. -1.]]

---

Notice the loss gradients resulted from SVM Loss.

What's the difference between those gradients and Softmax Gradients?

What are the effects?

---
### b. Training Function

#####**EXERCISE**:
* The same as the training function using Softmax before, complete the train function below,
* but rather than using `softmax_loss`, use `svm_loss`

```python
def train_svm(X, y, W=None, b=None, learning_rate=1e-6, reg=1e4, num_iters=100, batch_size=200, verbose=False):
    num_train, dim = X.shape
    num_classes = np.max(y) + 1 # assume y takes values 0...K-1 where K is number of classes

    if W is None:
        W = 0.001 * np.random.randn(dim, num_classes)
    if b is None:
        b = np.zeros((1,num_classes))

    # Run stochastic gradient descent to optimize W
    loss_history = []

    for it in range(num_iters):
        X_batch = None
        y_batch = None

        # Randomly select indices from training examples
        train_rows = np.arange(num_train)
        idxs = np.random.choice(train_rows, batch_size, replace=False)

        X_batch = X[idxs]
        y_batch = y[idxs]

        # calculate class score by calling forward function using X_batch, W, and b
        scores = forward(X_batch, W, b)

        # evaluate loss and gradient by calling svm_loss function using score and y_batch
        loss, dout = svm_loss(scores, y_batch)

        # append the loss history
        loss_history.append(loss)

        # calculate weights gradient by calling backward function using dout, X_batch, W, and b
        dW, db, _ = backward(dout, X_batch, W, b)

        # perform regulatization gradient
        dW += reg*W

        # perform parameter update by subtracting W and b with a fraction of dW and db
        # according to the learning rate
        W -= dW * learning_rate
        b -= db * learning_rate

        if verbose and it % 100 == 0:
            print ('iteration', it,'/',num_iters, ': loss =', loss)
    return loss_history, W, b
```

---
### c. Hyperparameter Tuning

#####**EXERCISE**:
* Again, complete the codes below similar to the Hyperparameter tuning for `Softmax Classifier`

```python
import warnings
warnings.filterwarnings('ignore')
np.random.seed(None)

results = {}
best_val = -1
best_W = None
best_b = None
learning_rates = [1e-7, 5e-7]
regularization_strengths = [1e5, 2e5]

iterations = 2000

# Greedily loop over learning_rates and regularization_strengths
for lr in learning_rates:
    for reg in regularization_strengths:
        print('Running {} iterations, rate = {}, reg = {}'.format(iterations, lr, reg))
        # call train function using the learning rate and regularization selected
        loss, W, b = train_svm(X_train, y_train,
                              learning_rate=lr, reg=reg,
                              num_iters=iterations, verbose=False)

        # call predict function using pretrained W and b on X_train and X_val to evaluate
        y_train_pred = predict(X_train, W, b)
        y_val_pred   = predict(X_val, W, b)

        # calculate the accuracy
        train_accuracy = np.mean(y_train == y_train_pred)
        val_accuracy   = np.mean(y_val == y_val_pred)

        print ('rate = {}, reg = {}, test accuracy = {}, validation accuracy = {}'.format(
            lr, reg, train_accuracy, val_accuracy))

        # store the result accuracy combination
        results[(lr, reg)] = (train_accuracy, val_accuracy)

        # store the best Weight and Bias
        if val_accuracy > best_val:
            best_W   = W
            best_b   = b
            best_val = val_accuracy

# Print out results.
for lr, reg in sorted(results):
    train_accuracy, val_accuracy = results[(lr, reg)]
    print('lr',lr,',reg',reg,', train accuracy: ', train_accuracy*100,'%, val accuracy: ', val_accuracy ,'%')

print()
print('best validation accuracy achieved during cross-validation:', best_val*100, '%')
```

    Running 2000 iterations, rate = 1e-07, reg = 100000.0
    rate = 1e-07, reg = 100000.0, test accuracy = 0.278175, validation accuracy = 0.2753
    Running 2000 iterations, rate = 1e-07, reg = 200000.0
    rate = 1e-07, reg = 200000.0, test accuracy = 0.2612, validation accuracy = 0.2625
    Running 2000 iterations, rate = 5e-07, reg = 100000.0
    rate = 5e-07, reg = 100000.0, test accuracy = 0.278175, validation accuracy = 0.2763
    Running 2000 iterations, rate = 5e-07, reg = 200000.0
    rate = 5e-07, reg = 200000.0, test accuracy = 0.2596, validation accuracy = 0.2573
    lr 1e-07 ,reg 100000.0 , train accuracy:  27.8175 %, val accuracy:  0.2753 %
    lr 1e-07 ,reg 200000.0 , train accuracy:  26.119999999999997 %, val accuracy:  0.2625 %
    lr 5e-07 ,reg 100000.0 , train accuracy:  27.8175 %, val accuracy:  0.2763 %
    lr 5e-07 ,reg 200000.0 , train accuracy:  25.96 %, val accuracy:  0.2573 %

    best validation accuracy achieved during cross-validation: 27.63 %

**Expected Output**:

You should reach `at least`  **`30%`** accuracy on cross-validation

---
### d. Visualize the Trained Weights

#####**EXERCISE**:
* complete the implementation to show the trained weight visualizations

```python
fig, ax = plt.subplots(1,10,figsize=(18,5))
fig.subplots_adjust(hspace=0.1, wspace=0.1)
for i in range(0, 10):
    # get the i-th weight
    w = best_W[:,i]

    w = scaler.fit_transform(w.reshape(-1, 1))

    # reshape the weight
    w = np.reshape(w,(32,32,3))

    ax[i].imshow(w)
    ax[i].axis('off')
    ax[i].set_title(classes[i])
plt.show()
```

### Notice the difference in weight visualization
Describe what your visualized SVM weights look like. Think on why they look they way that they do.

---

---

# Congratulation, You've Completed Linear Classifier Exercise

All best regard to the one who've had assignment me to complete this hands-on lab Exercise series which is Mas Andit - Telkom University Lecturer

```python

```

Linear Classifier - CV (Hands-on Lab).ipynb


Intuitive Understanding of Randomized Singular Value Decomposition

A Python Implementation of SVD with Randomized Linear Algebra

Matrix decomposition is a powerful tool for many machine learning problems and which has been widely used in data compression, dimensionality reduction, and sparsity learning, to name but a few. In many cases, for purposes of approximating a data matrix by a low-rank structure, Singular Value Decomposition (SVD) is often verified as the best choice. However, the accurate and efficient SVD of large data matrices (e.g., 8k-by-10k matrix) is computationally challenging. To resolve the SVD in this situation, there are many algorithms have been developed by applying randomized linear algebra methods. One of the most important algorithms is randomized SVD, which is competitively efficient for decomposing any large matrix with a relatively low rank.

Figure 1: A timeline of major SVD developments. (The picture is from [2]).

This post will introduce the preliminary and essential idea of the randomized SVD. To help readers gain a better understanding of randomized SVD, we also provide the corresponding Python implementation in this post.

> For reproducing this notebook, please clone or download the **tensor-learning** repository ([https://github.com/xinychen/tensor-learning](https://github.com/xinychen/tensor-learning)) on your computer first.

## Preliminary

### SVD Formula

We start by recalling the concept of SVD. As you may already know, SVD is one of the most important decomposition formula in linear algebra. For any given matrix $\boldsymbol{A}$, SVD has the form of
\begin{equation}
\boldsymbol{A}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^\top
\end{equation}
where the matrices $\boldsymbol{U}$ and $\boldsymbol{V}$ consist of left and right singular vectors, respectively. The diagonal entries of $\boldsymbol{\Sigma}$ are singular values.

### A Small Matrix Example

We will give a sufficiently detailed understanding with a small worked example. The problem is a simple SVD of 3-by-3 matrix, i.e.,
$$\boldsymbol{A}=\left(\begin{array}{cccc}
1 & 3 & 2 \\
5 & 3 & 1 \\
3 & 4 & 5 \\
\end{array}\right)\in\mathbb{R}^{3\times 3}.$$

Take this 3-by-3 matrix for example, we can compute the SVD by using `numpy.linalg.svd()` in Python. Let us take a look:

```python
import numpy as np

A = np.array([[1, 3, 2],
              [5, 3, 1],
              [3, 4, 5]])
u, s, v = np.linalg.svd(A, full_matrices = 0)
print('Left singular vectors:')
print(u)
print()
print('Singular values:')
print(s)
print()
print('Right singular vectors:')
print(v)
print()
```

    Left singular vectors:
    [[-0.37421754  0.28475648 -0.88253894]
     [-0.56470638 -0.82485997 -0.02669705]
     [-0.7355732   0.48838486  0.46948087]]

    Singular values:
    [9.34265841 3.24497827 1.08850813]

    Right singular vectors:
    [[-0.57847229 -0.61642675 -0.53421706]
     [-0.73171177  0.10269066  0.67383419]
     [ 0.36051032 -0.78068732  0.51045041]]

In this case, the singular values are **9.3427**, **3.2450**, and **1.0885**.

## Randomized SVD

### Essential Idea

Randomized SVD can be broken into three steps. For any given $m$-by-$n$ matrix $\boldsymbol{A}$, if we impose a target rank $k$ with $k < \min\{m, n\}$, then the first step is to

- 1) generate a Gaussian random matrix $\boldsymbol{Ω}$ with size of $n$-by-$k$,
- 2) compute a new $m$-by-$k$ matrix $\boldsymbol{Y}$,
- and 3) apply QR decomposition to the matrix $\boldsymbol{Y}$.

Note that the first step needs to return the $m$-by-$k$ matrix $\boldsymbol{Q}$.

Figure 2: The first step of randomized SVD. (The picture is from [2]).

Then, the second step as shown in Figure 3 is to

- 4) derive a $k$-by-$n$ matrix $\boldsymbol{B}$ by multiplying $\boldsymbol{Q}^\top$ and the matrix $\boldsymbol{A}$ together, i.e., $\boldsymbol{B}=\boldsymbol{Q}^\top\boldsymbol{A}$,
- and 5) compute the SVD of the matrix $\boldsymbol{B}$. Here, instead of computing the SVD of the original matrix $\boldsymbol{A}$, $\boldsymbol{B}$ is a smaller matrix to work with.

Note that the singular values (i.e., $\boldsymbol{\Sigma}$)and right singular vectors (i.e., $\boldsymbol{V}$) of the matrix $\boldsymbol{B}$ are also the singular values and right singular vectors of the matrix $\boldsymbol{A}$.

Figure 3: The second and third steps of randomized SVD. (The picture is from [2]).

As shown in Figure 3, if we combine the matrix $\boldsymbol{Q}$ derived in the first step with the left singular vectors of $\boldsymbol{B}$, we can get the left singular vectors (i.e., $\boldsymbol{U}$) of the matrix $\boldsymbol{A}$ in the third step.

### A Small Matrix Example

Even though we have learned the essential idea of randomized SVD in above, it would not be really clear if there is no intuitive example. To this end, we follow the aforementioned small matrix SVD.

First, let us try to write a Python function for randomized SVD. Here, we will use two Numpy functions, i.e., `np.linalg.qr()` and `np.linalg.svd()`.

```python
import numpy as np

def rsvd(A, Omega):
    Y = A @ Omega
    Q, _ = np.linalg.qr(Y)
    B = Q.T @ A
    u_tilde, s, v = np.linalg.svd(B, full_matrices = 0)
    u = Q @ u_tilde
    return u, s, v
```

Now, let us test it with 3-by-3 matrix (`rank = 2` for indicating $k$ with $k < \min(m, n)$):

```python
np.random.seed(1000)
A = np.array([[1, 3, 2],
              [5, 3, 1],
              [3, 4, 5]])
rank = 2
Omega = np.random.randn(A.shape[1], rank)
u, s, v = rsvd(A, Omega)
print('Left singular vectors:')
print(u)
print()
print('Singular values:')
print(s)
print()
print('Right singular vectors:')
print(v)
print()
```

    Left singular vectors:
    [[ 0.38070859  0.60505354]
     [ 0.56830191 -0.74963644]
     [ 0.72944767  0.26824507]]

    Singular values:
    [9.34224023 3.02039888]

    Right singular vectors:
    [[ 0.57915029  0.61707064  0.53273704]
     [-0.77420021  0.21163814  0.59650929]]

Recall that the singular values of this matrix are **9.3427**, **3.2450**, and 1.0885. In this case, randomized SVD has the first two singular values as **9.3422** and **3.0204**.

We can see that the first singular values computed by these two SVD algorithms are extremely close. However, the second singular value of randomized SVD has a slight bias. Is there any other method to improve this result? And how?

The answer is yes!

## Randomized SVD with Power Iteration

To improve the quality of randomized SVD, power iteration method can be used directly. For more detail about power iteration, please see the page 39 in [1] and there is also a Matlab implementation in the page 40.

In the following Python codes, `power_iteration()` is the function for computing the $m$-by-$k$ matrix $\boldsymbol{Y}$ iteratively (the default `power_iter` is 3) and then derive the $m$-by-$k$ matrix $\boldsymbol{Q}$ by QR decomposition.

```python
import numpy as np

def power_iteration(A, Omega, power_iter = 3):
    Y = A @ Omega
    for q in range(power_iter):
        Y = A @ (A.T @ Y)
    Q, _ = np.linalg.qr(Y)
    return Q

def rsvd(A, Omega):
    Q = power_iteration(A, Omega)
    B = Q.T @ A
    u_tilde, s, v = np.linalg.svd(B, full_matrices = 0)
    u = Q @ u_tilde
    return u, s, v
```

Let us test our new `rsvd()` function:

```python
np.random.seed(1000)

A = np.array([[1, 3, 2],
              [5, 3, 1],
              [3, 4, 5]])
rank = 2
Omega = np.random.randn(A.shape[1], rank)
u, s, v = rsvd(A, Omega)
print('Left singular vectors:')
print(u)
print()
print('Singular values:')
print(s)
print()
print('Right singular vectors:')
print(v)
print()
```

    Left singular vectors:
    [[ 0.37421757  0.28528579]
     [ 0.56470638 -0.82484381]
     [ 0.73557319  0.48810317]]

    Singular values:
    [9.34265841 3.24497775]

    Right singular vectors:
    [[ 0.57847229  0.61642675  0.53421706]
     [-0.73178429  0.10284774  0.67373147]]

Recall that:

- Singular values of SVD are: **9.3427**, **3.2450**, and 1.0885.
- Singular values of randomized SVD without power iteration are: **9.3422** and **3.0204**.
- Singular values of randomized SVD with power iteration are: **9.3427** and **3.2450**.

As you can see, the randomized SVD with power iteration provides extremely accurate singular values.

## Image Compression

As mentioned above, it is possible to compress (low-rank) signal matrix using the SVD or randomized SVD. In fact, the way to compress an image using the SVD is rather simple: taking the SVD of the image directly and only keeping the dominant singular values and left/right singular vectors. In terms of randomized SVD, we can predefine the number of dominant singular values first, and then obtain the singular values and left/right singular vectors by the randomized SVD.

For our evaluation, we choose the color image of **Lena** as our data. The size of this image is $256\times 256\times 3$. Here, we build a matrix $\boldsymbol{A}$ of size $256\times 256$ by only selecting the green chanel.

- **Using SVD directly**

```python
import numpy as np
import imageio

image = imageio.imread('../datasets/color-images/lena.bmp')
A = image[:, :, 1]
%time u, s, v = np.linalg.svd(A, full_matrices = 0)
```

    CPU times: user 28.3 ms, sys: 3.24 ms, total: 31.5 ms
    Wall time: 16.3 ms

```python
import matplotlib.pyplot as plt
%matplotlib inline

plt.subplots(1, 2, figsize=(10, 10))
plt.subplot(1, 2, 1)
plt.imshow(A)
plt.title('The original Lena at the green chanel.')
plt.axis('off')

plt.subplot(1, 2, 2)
rank = 50
plt.imshow(u[:, : rank] @ np.diag(s[: rank]) @ v[: rank, :])
plt.title('The rank-50 Lena at the green chanel.')
plt.axis('off')
plt.show()
```

- **Using randomized SVD instead**

```python
plt.subplots(1, 2, figsize=(10, 10))
plt.subplot(1, 2, 1)
plt.imshow(u[:, : rank] @ np.diag(s[: rank]) @ v[: rank, :])
plt.title('The rank-50 Lena using SVD.')
plt.axis('off')

rank = 50
Omega = np.random.randn(A.shape[1], rank)
%time u, s, v = rsvd(A, Omega)

plt.subplot(1, 2, 2)
plt.imshow(u[:, : rank] @ np.diag(s[: rank]) @ v[: rank, :])
plt.title('The rank-50 Lena using randomized SVD.')
plt.axis('off')
plt.show()
```

We can see from this image compression experiment that:

- 1) By comparing to the SVD, the randomized SVD can also produce accurate compression with a prescribed low rank (here, we set `rank = 50`).

- 2) The randomized SVD is computational friendly. By specifying `rank = 50`, the total CPU times of the randomized SVD is **11.6 ms**, while the total CPU times of the SVD is **31.5 ms**.

### Summary

In this post, you discovered the randomized linear algebra method for SVD. Specifically, you learned:

- The essential idea of randomized SVD.
- How to implement randomized SVD in Python.

Do you have any question? Ask your question by creating an issue at the **tensor-learning** repository ([https://github.com/xinychen/tensor-learning](https://github.com/xinychen/tensor-learning)) and I will do my best to answer. If you find these codes useful, please star (★) this repository.

### References

[1] Steven L. Brunton, J. Nathan Kutz (2019). Data-Driven Science and Engineering: Machine Learning, Dynamical Systems, and Control. Page 37–41.

[2] N. Benjamin Erichson, Sergey Voronin, Steven L. Brunton, J. Nathan Kutz (2016). Randomized Matrix Decompositions Using R. arXiv:1608.02148. [[PDF](https://arxiv.org/pdf/1608.02148.pdf)]

### License

This work is released under the MIT license.


# Rootfinding, Newton's Method, and Dynamical Systems

In the Continued Fractions unit, we met with the sequence
\begin{equation}
	1, \frac{3}{2}, \frac{17}{12}, \frac{577}{408}, \frac{665857}{470832}, \ldots
\end{equation}
which was generated by $x_{n+1} = \frac12{\left(x_{n} + \frac{2}{x_n}\right)}$; in words, the average of the number and two divided by the number. This unit explores where that sequence came from, and its relationship to $\sqrt{2}$. We'll approach this algebraically, as Newton did. Consider the equation
\begin{equation}
	x^2 - 2 = 0.
\end{equation}
Clearly the solutions to this equation are $x = \sqrt{2}$ and $x = -\sqrt{2}$. Let us _shift the origin_ by putting $x = 1 + s$; so $s = 0$ corresponds to $x = 1$.

We draw the vertical part of the new axis that we will shift to, in red. Notice that we use the labels and tickmarks from the old axis, in black.

```python
import numpy as np
from matplotlib import pyplot as plt

fig = plt.figure(figsize=(6, 6))
ax = fig.add_axes([0,0,1,1])
n = 501
x = np.linspace(-1,3,n)
y = x*x-2;

plt.plot(x,y,'b')   # x^2-2 is in blue
ax.grid(True, which='both')
ax.axhline(y=0, color='k')
ax.axvline(x=0, color='k')
ax.axvline(x=1, color='r') # The new axis is in red
plt.show()
```

Then
\begin{equation}
	\left(1 + s\right)^2 - 2 = 1 + 2s + s^2 - 2 = -1 + 2s + s^2 = 0.
\end{equation}
we now make the surprising assumption that $s$ is so small that we may ignore $s^2$ in comparison to $2s$. If it turned out that $s = 10^{-6}$, then $s^2 = 10^{-12}$, very much smaller than $2s = 2\cdot10^{-6}$; so there are small numbers $s$ for which this is true; but we don't know that this is true, here. We just hope.

```python
fig = plt.figure(figsize=(6, 6))
ax = fig.add_axes([0,0,1,1])
n = 501
s = np.linspace(-1,1,n)
y = -1+2*s+s*s;  # Newton would have written s^2 as s*s, too

plt.plot(s,y,'b')   # equation in blue again
ax.grid(True, which='both')
ax.axhline(y=0, color='r')
ax.axvline(x=0, color='r')
ax.axvline(x=1/2, color='k') # The new axis is in black
plt.show()
```

Then if $s^2$ can be ignored, our equation becomes
\begin{equation}
	-1 + 2s = 0
\end{equation}
or $s = \frac{1}{2}$. This means $x = 1 + s = 1 + \frac{1}{2} = \frac{3}{2}$.

We now repeat the process: shift the origin to $\frac{3}{2}$, not $1$: put now
\begin{equation}
	x = \frac{3}{2} +t
\end{equation}
which is equivalent to $s = 1/2 + t$, so
\begin{equation}
	\left(\frac{3}{2} + t\right)^2 = \frac{9}{4} + 3t + t^2 - 2 = \frac{1}{4} + 3t + t^2 = 0.
\end{equation}

```python
fig = plt.figure(figsize=(6, 6))
ax = fig.add_axes([0,0,1,1])
n = 501
t = np.linspace(-0.2,0.2,n)
y = 1/4+3*t+t*t;  # Newton would have written t^2 as t*t, too

plt.plot(t,y,'b')   # equation in blue again
ax.grid(True, which='both')
ax.axhline(y=0, color='k')
ax.axvline(x=0, color='k')
ax.axvline(x=-1/12, color='r') # The new axis will be red again
plt.show()
```

This gives $3t + t^2 + \frac{1}{4} = 0$ and again we ignore $t^2$ and hope it's smaller than $3t$. This gives
\begin{equation}
	3t + \frac{1}{4} = 0
\end{equation}
or $t = -\frac{1}{12}$. This means $x = \frac{3}{2} - \frac{1}{12}$ or $x = \frac{17}{12}$. Now we see the process.

Again, shift the origin: $x = \frac{17}{12} + u$. Now
\begin{equation}
	\left(\dfrac{17}{12} + u\right)^2 = \dfrac{289}{144} + \dfrac{17}{6}u + u^2 - 2 = 0.
\end{equation}
Ignoring $u^2$,
\begin{equation}
	\dfrac{17}{6}u + \dfrac{1}{144} = 0
\end{equation}
or
\begin{equation}
	u = \dfrac{-6}{17\cdot144} = \dfrac{-1}{17\cdot24} = \dfrac{-1}{408}.
\end{equation}
Thus,
\begin{equation}
	x = \dfrac{17}{12} - \dfrac{1}{408} = \dfrac{577}{408}.
\end{equation}
As we saw in the Continued Fractions unit, these are the exact square roots of numbers ever more close to 2. For instance,
\begin{equation}
	\dfrac{577}{408} = \sqrt{2 + \dfrac{1}{408^2}}.
\end{equation}

## Euler again

It was Euler who took Newton's "shift the origin" strategy and made a general method—which we call Newton's method—out of it. In modern notation, Euler considered solving $f(x) = 0$ for a differentiable function $f(x)$, and used the tangent line approximation near an initial approximation $x_0$: if $x = x_0 + s$ then, using $f'(x_0)$ to denote the slope at $x_0$, $0 = f(x) = f(x_0 + s) \approx f(x_0) + f'(x_0)s$ ignoring terms of order $s^2$ or higher. Then
\begin{equation}
	s = -\dfrac{f(x_0)}{f'(x_0)}
\end{equation}
so
\begin{equation}
	x \approx x_0 + s = x - \dfrac{f(x_0)}{f'(x_0)}.
\end{equation}
The fundamental idea of Newton's method is that, if it worked once, we can do it again: pass the parcel! Put

$$
\begin{align}
	x_1 &= x_0 - \dfrac{f(x_0)}{f'(x_0)} \\
	x_2 &= x_1 - \dfrac{f(x_1)}{f'(x_1)} \\
	x_3 &= x_2 - \dfrac{f(x_2)}{f'(x_2)}
\end{align}
$$

and keep going, until $f(x_k)$ is so small that you're happy to stop.

Notice that each $x_k$ solves
\begin{equation}
	f(x) - f(x_k) = 0
\end{equation}
not $f(x) = 0$. But if $f(x_k)$ is really small, you've solved "almost as good" an equation, like finding $\sqrt{2 + \frac{1}{408^2}}$ instead of $\sqrt{2}$. So where did $\frac12{\left(x_n + \frac{2}{x_n}\right)}$ come from?
\begin{equation}
	x_{n+1} = x_n - \dfrac{f(x_n)}{f'(x_n)} = x_n - \dfrac{\left(x_n^2 - 2\right)}{2x_n}
\end{equation}
because if $f(x) = x^2 - 2$, $f'(x) = 2x - 0 = 2x$. Therefore,

$$
\begin{align}
	x_{n+1} &= x_n - \dfrac{\left(x_n^2 - 2\right)}{2x_n} \\
    &= \dfrac{2x_n^2 - x_n^2 + 2}{2x_n} \\
	&= \dfrac{x_n^2 + 2}{2x_n} \\
    &= \dfrac{1}{2}\left(x_n + \dfrac{2}{x_n}\right)
\end{align}
$$

as claimed. (For more complicated functions one _shouldn't_ simplify. But for $x^2 - a$, it's okay.)

Executing this process in decimals, using a calculator (our handy HP48G+ again), with $x_0 = 1$, we get

$$
\begin{align}
	x_0 &= 1 \nonumber \\
	x_1 &= \underline{1}.5 \nonumber \\
	x_2 &= \underline{1.4}1666\ldots \nonumber \\
	x_3 &= \underline{1.41421}568628 \nonumber \\
	x_4 &= \underline{1.41421356238} \nonumber \\
	x_5 &= x_4 \text{ to all 11 places in the calculator}
\end{align}
$$

Now $\sqrt{2} = 1.41421356237$ on this calculator. We see (approximately) 1, 2, 5 then 10 correct digits. The convergence behaviour is clearer in the continued fraction representation:
\begin{equation}
	1, 1 + \left[2\right], 1 + \left[2, 2, 2\right], 1 + \left[2, 2, 2, 2, 2, 2, 2 \right],
	1 + \left[2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2\right]
\end{equation}

with 0, 1, 3, 7, 15 twos in the fraction part: each time doubling the previous plus 1, giving $2^0 - 1$, $2^1 - 1$, $2^2 - 1$, $2^3 - 1$, $2^4 - 1$ correct entries. This "almost doubling the number of correct digits with each iteration" is quite characteristic of Newton's method.

## Newton's Method

In the [Continued Fractions](continued-fractions.ipynb) unit, we saw Newton's method to extract square roots: $x_{n+1} = (x_n + a/x_n)/2$. That is, we simply took the average of our previous approximation with the result of dividing our approximation into what we wanted to extract the square root of.  We saw rapid convergence, in that the number of correct entries in the continued fraction for $\sqrt{a}$ roughly doubled with each iteration.  This simple iteration comes from a more general technique for finding zeros of nonlinear equations, such as $f(x) = x^5 + 3x + 2 = 0$. As we saw above, for general $f(x)$, Newton's method is
\begin{equation}
x_{n+1} = x_n - \frac{f(x_n)}{f'(x_n)}\>,
\end{equation}
where the notation $f'(x_n)$ means the _derivative_ of $f(x)$ evaluated at $x=x_n$.  In the special case $f(x) = x^2-a$, whose roots are $\pm \sqrt{a}$, the general Newton iteration above reduces to the simple formula we used before, because $f'(x) = 2x$.

### Derivatives
We're not assuming that you know so much calculus that you can take derivatives of general functions, although if you do, great.  We _could_ show a Python method to get the computer to do it, and we might (or we might leave it to the exercises; it's kind of fun).  But we do assume that you know that derivatives of polynomials are easy: for instance, if $f(x) = x^3$ then $f'(x) = 3x^2$, and similarly if $f(x) = x^n$ for some integer $n$ then $f'(x) = n x^{n-1}$ (which holds true even if $n$ is zero or a negative integer---or, for that matter, if $n$ is any constant).  Then by the _linearity_ of the derivative, we can compute the derivative of a polynomial expressed in the monomial basis.  For instance, if $f(x) = x^5 + 3x + 2$ then $f'(x) = 5x^4 + 3$. It turns out that the Python implementation of polynomials knows how to do this as well (technically, polynomials _know how to differentiate themselves_), as we will see.  For this example we can use this known derivative in the Newton formula to get the iteration
\begin{equation}
x_{n+1} = x_n - \frac{x^5 + 3x + 2}{5x^4+3}\>.
\end{equation}

#### Exercises
Write Newton's method iterations for the following functions, and estimate the desired root(s).
1. $f(x) = x^2 - 3$ with $x_0 = 1$
2. $f(x) = x^3 - 2$ with $x_0 = 1$
3. Newton's original example, $f(x) = x^3 - 2x - 5 = 0$. Since $f(0) = -5$, $f(1) = -6$, $f(2) = -1$, and $f(3) = 16$, we see there is a root between $x=2$ and $x=3$.  Use that knowledge to choose your initial estimate.

### The Railway Pranksters Problem

Late one night, some pranksters weld a $2$cm piece of steel into a train track $2$km long, sealing the gaps meant to allow for heat expansion.  In the morning as the temperature rises, the train track expands and bows up into a perfectly circular arc. How high is the arc in the middle?

The fun part of this problem is setting it up and drawing it. We should let you do it, so we will hide our drawing and set-up.  The solution, using Newton's method, is below.  Our solution won't make sense until you do your own drawing.  We used a symbol $R$ for the radius of the circular arc, and $s$ for the small increment in the length of the track.

```python
s = 1.0e-5
f = lambda R: np.sin((1+s)/R) - 1/R
df = lambda R: -(1 + s) / R ** 2 * np.cos((1 + s) / R) + 1 / R ** 2
n = 5
Ar = [1000.0/9.0]
for k in range(n):
    nxt = Ar[k] - f(Ar[k])/df(Ar[k])
    Ar.append(nxt)
print( Ar, [ f(rho) for rho in Ar] )
```

    [111.11111111111111, 123.86244904780746, 128.59835439622927, 129.09631810259108, 129.10118725925943, 129.10118771897493] [-3.1503152937012446e-08, -6.973718736161261e-09, -6.092985586556021e-10, -5.843914761827218e-12, -5.516420653606247e-16, 8.673617379884035e-19]

The convergence there was a _bit_ slow; the residual was already pretty small with our initial estimate (we explain below where we got it) but Newton's method got us there in the end.  The difficulty is that the root is _nearly_ a multiple root.  We talk a little about that, below.

```python
height = 1/(Ar[-1] + np.sqrt(Ar[-1]**2-1))
print( "Height in is {} km or {} feet".format(height,3280.84*height))
```

    Height in is 0.0038729891556916357 km or 12.706657741559347 feet

That might seem ridiculous.  Consider the related problem, where instead of bowing up in a perfect circular arc, the track bows up into two straight lines meeting in the middle, making an isosceles triangle with base $2$km long and sides $1+s$.  The height satisfies $h^2 +1 = (1+s)^2$ by Pythagoras; since $s=10^{-5}$km, we find pretty rapidly that $h = \sqrt{(1+s)^2-1} = \sqrt{2s + s^2}$ or $0.00447$km or about $14.7$ feet.

Okay, then.  Inserting one extra centimeter in a kilometer gives a height of 4.47m or 14.7 feet if we make a triangle; or 3.9 meters (12.7 feet) if it makes a semicircular arc.  This is kind of wild, but would be true if our crazed assumptions were true.  By the way, we used the height estimate from the triangle problem to give us an $R$ estimate for the circular arc problem; that $1000/9$ didn't come out of nowhere!

### A more serious application

[Kepler's equation](https://en.wikipedia.org/wiki/Kepler's_equation) is
\begin{equation}
M = E - e\sin E
\end{equation}
where the _mean anomaly_ $M = n(t-t_0)$ where $t$ is time, $t_0$ is the starting time, $n = 2\pi/T$ is the sweep speed for a circular orbit with period $T$, and the _eccentricity_ $e$ of the (elliptical) orbit are all known, and one wants to compute the _eccentric anomaly_ $E$ by solving that nonlinear equation for $E$.  Since we typically want to do this for a succession of times $t$, initial estimates are usually handy to the purpose (just use the $E$ from the previous instant of time), and typically only one Newton correction is needed.

```python
T = 1 # Period of one Jupiter year, say (equal to 11.8618 earth years)
n = 2*np.pi/T
# At t=t_0, which we may as well take to be zero, M=0 and so E = 0 as well.
nsamples = 12 # Let's take 13 sample positions including 0.
e = 0.0487 # Jupiter has a slightly eccentric orbit; not really visible to the eye here
E = [0.0]
f = lambda ee: ee-e*np.sin(ee)-M
df = lambda ee: 1 - e*np.cos(ee)
newtonmax = 5
for k in range(nsamples):
    M = n*(k+1)/nsamples # measuring time in Jupiter years
    EE = E[k]
    newtoniters = 0
    for j in range(newtonmax):
        residual = f(EE);
        if abs(residual) <= 1.0e-12:
            break
        EE = EE - residual/df(EE)
    E.append(EE)

x = [np.cos(ee)-e for ee in E]
y = [np.sqrt(1-e*e)*np.sin(ee) for ee in E]
fig = plt.figure(figsize=(6, 6))
ax = fig.add_axes([0,0,1,1])
plt.scatter(x,y,s=200,color='orange')
ax.set_aspect('equal')
ax.grid(True, which='both')
plt.show()
```

### A useful interpretation

When we were looking at square roots, we noticed that each iteration (say $17/12$) could be considered the _exact_ square root of something near to what we wanted to extract the root of; this kind of interpretation is possible for general functions as well.  Each iterate $x_n$, when we are trying to solve $f(x) = 0$, can be considered to be the _exact_ zero of $g(x) = f(x)-r_n$, where $r_n = f(x_n)$ is the so-called _residual_.  The equation is so simple that it's slippery: $x_n$ solves $f(x) - f(x_n) = 0$.  Of course it does.  No matter what you put in for $x_n$, so long as $f(x)$ was defined there, you would have a zero.  The key of course is the question of whether or not the residual is small enough to ignore.  There are two issues with that idea.

1. The function $f(x)$ might be sensitive to changes (think of $f(x) = (x-\pi)^2$, for instance; adding just a bit to it makes both roots complex, and the change is quite sudden).
2. The change might not make sense in the physical/chemical/biological context in which $f(x)$ arose.

This second question is not mathematical in nature: it depends on the context of the problem.  Nevertheless, as we saw for the square roots, this is _often_ a very useful look at our computations.

To answer the question of sensitivity, we need calculus, and the general notion of derivative:

$f(x)+s = 0$ makes some kind of change to the zero of $f(x)=0$, say $x=r$.  If the new root is $x=r+t$ where $t$ is tiny (just like $s$ is small) then calculus says $0 = f(r+t) + s = f(r)+f'(r)t+s + O(t^2)$ where we have used the tangent-line approximation (which is, in fact, where Newton iteration comes from).  Solving for this we have that $t = -s/f'(r)$, because of course $f(r)=0$.

This is nothing more (or less!) than the calculus derivation of Newton's method, but that's not what we are using it for here.  We are using it to _estimate_ the sensitivity of the root to changes in the function.  We see that if $f'(r)$ is small, then $t$ will be much larger than $s$; in this case we say that the zero of $f(x)$ is _sensitive_.  If $f'(r)=0$ we are kind of out of luck with Newton's method, as it is; it needs to be fixed up.

### Things that can go wrong

Newton's method is a workhorse of science, there's no question.  There are multidimensional versions, approximate versions, faster versions, slower versions, applications to just about everything, especially optimization.  But it is not a perfect method.  Solving nonlinear equations is _hard_. Here are some kinds of things that can go wrong.

1. The method can converge to the wrong root.  If there's more than one root, then this can happen.
2. The method can divide by zero (if you happen to hit a place where the derivative is zero).  Even just with $f(x)=x^2-2$ this can happen if you choose your initial guess as $x_0 = 0$. (See if you can either find other $x_0$ that would lead to zero, or else prove it couldn't happen for this function).
3. If the derivative is _small_ but not zero the convergence can be very, very slow.  This is related to the sensitivity issue.
4. You might get caught in a _cycle_ (see the animated GIF as the header of this chapter).  That is, like with the game of pass the parcel with the Gauss map, we may eventually hit the initial point we started with.
5. The iteration may take a very long time to settle down before converging.
6. The iteration may go off to infinity.

Let's take Cleve Moler's "perverse example", from section 4.3 of [Numerical Computing with Matlab](https://www.mathworks.com/moler/chapters.html), namely

\begin{align}
   f(x) &= \sqrt{x-r} \mathrm{\ if\ } x \ge r \nonumber\\
        &= -\sqrt{r-x} \mathrm{\ if\ } x \le r
\end{align}

We will draw this in the next cell, for some harmless $r$.  This peculiar function was chosen so that _every_ initial estimate (that wasn't exactly right, i.e. $x=r$) would create a two-cycle:  pretty mean.

Let's check.  The derivative is

\begin{align}
   f(x) &= \frac12\left(x-r\right)^{-1/2} \mathrm{\ if\ } x > r \nonumber\\
        &= -\frac12\left(r-x\right)^{-1/2} \mathrm{\ if\ } x < r
\end{align}

So the Newton iteration is, if $x_n > r$,
\begin{equation}
x_{n+1} = x_n - \frac{ (x_n-r)^{1/2} }{ (x_n-r)^{-1/2}/2 } = x_n - 2(x_n-r) = 2r - x_n
\end{equation}
which can be rewritten to be $x_{n+1}-r = r -x_n$, so the distance to $r$ is exactly the same but now we are on the other side. If instead $x_n < r$,
\begin{equation}
x_{n+1} = x_n - \frac{ -(r-x_n)^{1/2} }{ +(r-x_n)^{-1/2}/2 } = x_n + 2(r-x_n) = 2r - x_n
\end{equation}
which again says $x_{n+1}-r = r -x_n$.

In particular, take $x_0 = 0$.  Then $x_1 = 2r$.  Then $x_2 = 0$ again.

Obviously this example was contrived to show peculiar behaviour; but these things really can happen.

```python
r = 0.73  # Some more or less random number
n = 501
x = np.linspace(-1+r,1+r,n)
y = np.sign(x-r)*np.sqrt(np.abs(x-r))

fig = plt.figure(figsize=(6, 6))
ax = fig.add_axes([0,0,1,1])

plt.plot(x,y,'k')
ax.grid(True, which='both')
ax.axhline(y=0, color='k')
ax.axvline(x=0, color='k')
plt.title("Cleve Moler's really mean example")
plt.show()
```

## A _Really Useful_ trick to find initial guesses

One of our "generic good questions" was "can you find an easier question to solve?" and "does answering that easier question help you to to solve this one?"  This gives an amazing method, called _continuation_ or _homotopy continuation_ for solving many equations, such as polynomial equations.  The trick is easily stated.  Suppose you want to solve $H(x) = 0$ (H for "Hard") and you have no idea where the roots are.  You might instead be able to solve $E(x) = 0$, a similar equation but one you know the roots of already (E for "Easy", of course).  Then consider blending them: $F(x,t) = (1-t)E(x) + tH(x)$ for some parameter $t$, which we take to be a real number between $0$ and $1$.

You start by solving the Easy problem, at $t=0$, because $F(x,0) = E(x)$.  Then increase $t$ a little bit, say to $t=0.1$; use the roots of $E(x)$ as your initial estimate of the roots of $F(x,0.1) = 0.9E(x) + 0.1H(x)$.  If it works, great; increase $t$ again, and use those recently-computed roots of $F(x,0.1)$ as the initial estimates for (say) $F(x,0.2)$.  Continue on until either the method breaks down (it can, the root paths can cross, which is annoying, or go off to infinity, which is worse) or you reach the end.

As an example, suppose you want to solve $h(x) = x e^x - 2.0 = 0$.  (This isn't a very hard problem at all, but it's a nice example).  If we were trying to solve $e(x) = e^x - 2.0 = 0$, then we already know the answer: $x=\ln(2.0) = 0.69314718056\ldots$. So $f(x,t) = (1-t)(e^x - 2) + t(xe^x -2) = (1-t+tx)e^x - 2 = (1+t(x-1))e^x - 2$.  Newton's method for this is straightforward (this is the first non-polynomial function we've used, but we bet that you know that the derivative of $e^x$ is $e^x$, and that you know the product rule as well).  So our iteration is (for a fixed $t$)

\begin{equation}
x_{n+1} = x_n - \frac{ (1+t(x_n-1))e^{x_n} - 2 }{(1-t+t(x_n+1))e^{x_n}}
\end{equation}

```python
nt = 10
newtonmax = 3
t = np.linspace(0,1,nt)
x = np.zeros(nt+1)
x[0] = np.log(2.0) # Like all snobs, Python insists on log rather than ln
for k in range(1,nt):
    xi = x[k-1]
    for i in range(newtonmax):
        y = np.exp(xi)
        ft = (1+t[k]*(xi-1))*y-2;
        dft = (1-t[k] + t[k]*(xi+1))*y
        xi = xi - ft/dft
    x[k] = xi

print( 'The solution is {} and the residual is {}'.format(xi, xi*np.exp(xi)-2.0) )
```

    The solution is 0.8526055020137254 and the residual is -2.220446049250313e-16

```python
from scipy import special
reference = special.lambertw(2.0)
print("The reference value is {}".format(reference))
```

    The reference value is (0.8526055020137254+0j)

You can learn more about the [Lambert W function here](https://orcca.on.ca/LambertW/) or at the [Wikipedia link](https://en.wikipedia.org/wiki/Lambert_W_function).

### The problem with multiple roots

Newton's iteration divides by $f'(x_n)$, and if the root $r$ we are trying to find happens to be a _multiple_ root, that is, both $f(r) = 0$ and $f'(r)=0$, then the $f'(x_n)$ we are dividing by will get smaller and smaller the closer we get to $r$. This slows Newton iteration to a crawl.  Consider $f(x) = W(x^3)$ where $W(s)$ is the [Lambert W function](https://orcca.on.ca/LambertW/).  Then since $W(0)=0$ we have $f'(x) = 3x^2 W'(x^3)$ so $f'(0)=0$ as well.  Let's look at how this slows us down.  Notice that $W(s)$ is itself _defined_ to be the root of the equation $y\exp(y)-s = 0$, and it itself is usually evaluated by an iterative method (in Maple, Halley's method is used because the derivatives are cheap).  But let's just let python evaluate it for us here. We can evaluate $W'(x)$ by implicit differentiation: $W(x)\exp W(x) = x$ so $W'(x)\exp W(x) + W(x) W'(x) \exp W(x) = 1$ and therefore $W'(x) = 1/(\exp W(x)(1+ W(x)))$.

```python
f = lambda x: special.lambertw( x**3 )
df = lambda x: 3*x**2/(np.exp(f(x))*(1+ f(x)))
SirIsaac = lambda x: x - f(x)/df(x)
hex = [0.1] # We pretend that we don't know the root is 0
n = 10
for k in range(n):
    nxt = SirIsaac( hex[k] )
    hex.append(nxt)
print( hex )

```

    [0.1, (0.0666333666167554+0j), (0.044415675138000696+0j), (0.02960915295524054+0j), (0.019739179108047577+0j), (0.013159402133893345+0j), (0.008772924760017682+0j), (0.005848614532183306+0j), (0.0038990759647654794+0j), (0.002599383899468682+0j), (0.001732922584427688+0j)]

We can see that the iterates are _decreasing_ but the turtle imitation is absurd.

There is an _almost useless_ trick to speed this up.  If we "just happen to know" the multiplicity $\mu$ of the root, then we can speed things up.  Here the multiplicity is $\mu=3$.  Then we can _modify_ Newton's iteration to speed it up, like so:
\begin{equation}
x_{n+1} = x_n - \mu\frac{f(x_n)}{f'(x_n)}
\end{equation}
Let's try it out.

```python
mu = 3
ModSirIsaac = lambda x: x - mu*f(x)/df(x)
n = 3
hex = [0.2]
for k in range(n):
    nxt = ModSirIsaac( hex[k] )
    hex.append(nxt)
print( hex )
```

    [0.2, (-0.0015873514490433727+0j), (-6.348810149478523e-12+0j), (8.077935669463161e-28+0j)]

So, it _does_ work; instead of hardly getting anywhere in ten iterations, it got the root in three.  __NB: The first time we tried this, we had a bug in our derivative and the iterates did not approach zero with the theoretical speed: so we deduced that there must have been a bug in the derivative, and indeed there was.__

But this trick is (as we said) almost useless; because if we don't know the root, how do we know its multiplicity?

### What happens if we get the derivative wrong?

Let's suppose that we get the derivative wrong—maybe deliberately, to save some computation, and we only use the original estimate $f'(x_0)$.  This can be useful in some circumstances.  In this case, we don't get such rapid approach to the zero, but we can get the iterates to approach the root, if not very fast.  Let's try.  Take $f(x) = x^2-2$, and our initial estimate $x_0 = 1$ so $f'(x_0) = 2$. This "approximate Newton" iteration then becomes
\begin{equation}
x_{n+1} = x_n - \frac{1}{2}f(x_n)
\end{equation}
or $x_{n+1} = x_n - (x_n^2-2)/2$.

```python
n = 10
hex = [1.0]
f = lambda x: x*x-2
df = 2
quasi = lambda x: x - f(x)/df;
for k in range(n):
    nxt = quasi( hex[k] )
    hex.append(nxt)
print( hex )
```

    [1.0, 1.5, 1.375, 1.4296875, 1.407684326171875, 1.4168967450968921, 1.4130985519638084, 1.4146747931827024, 1.4140224079494415, 1.4142927228578732, 1.4141807698935047]

We see that the iterates are getting closer to the root, but quite slowly.

To be honest, this happens more often when someone codes the derivative incorrectly than it does by deliberate choice to save the effort of computing the derivative.  It has to be one ridiculously costly derivative before this slow approach is considered worthwhile (it looks like we get about one more digit of accuracy after two iterations).

### What if there is more than one root?

A polynomial of degree $n$ has $n$ roots (some or all of which may be complex, and some of which may be multiple).  It turns out to be important in practice to solve polynomial equations.  John McNamee wrote a bibliography in the late nineties that had about _ten thousand_ entries in it—that is, the bibliography listed ten thousand published works on methods for the solution of polynomials. It was later published as a book, but would be best as a computer resource.  Unfortunately, we can't find this online anywhere now, which is a great pity.  But in any case ten thousand papers is a bit too much to expect anyone to digest.  So we will content ourselves with a short discussion.

First, Newton's method is not very satisfactory for solving polynomials.  It only finds one root at a time; you need to supply an initial estimate; and then you need to "deflate" each root as you find it, so you don't find it again by accident.  This turns out to introduce numerical instability (sometimes).  This all _can_ be done but it's not so simple.  We will see better methods in the Mandelbrot unit.

But we really don't have to do anything: we can use Maple's `fsolve`, which is robust and fast enough for most purposes. In Python, we can use the similarly-named routine `fsolve` from SciPy, if we only want one root: there are other facilities in NumPy for polynomial rootfinding, which we will meet in a later unit.

We do point out that the "World Champion" polynomial solver is a program called MPSolve, written by Dario Bini and Leonardo Robol.  It is freely available at [this GitHub link](https://github.com/robol/MPSolve). The paper describing it is [here](https://www.sciencedirect.com/science/article/pii/S037704271300232X).

```python
from scipy.optimize import fsolve
f = lambda x: x**2+x*np.exp(x)-2
oneroot = fsolve( f, 0.5 )
print( oneroot, oneroot**2 + oneroot*np.exp(oneroot)-2 )
```

    [0.72048399] [-2.44249065e-15]

### Complex initial approximations, fractal boundaries, Julia sets, and chaos
Using Newton's method to extract square roots (when you have a calculator, or Google) is like growing your own wheat, grinding it to flour by hand, and then baking bread, when you live a block away from a good bakery. It's kind of fun, but faster to do it the modern way. But even for _cube_ roots, the story gets more interesting when complex numbers enter the picture.

Consider finding all $z$ with
\begin{equation}
	z^3 - 8 = 0 .
\end{equation}
The results are $z = 2$, $z = 2\cdot e^{\frac{i\cdot2\pi}{3}}$, and $z = 2\cdot e^{\frac{-i2\pi}{3}}$.

See our [Appendix on complex numbers](../Appendix/complex-numbers.ipynb).

## Exercises
1. Write down as many questions as you can, about this section.

2. Sometimes Newton iteration is "too expensive"; a cheaper alternative is the so-called _secant iteration_, which goes as follows: $z_{n+1} = z_n - f(z_n)(z_{n}-z_{n-1})/(f(z_n) - f(z_{n-1}))$. You need not one, but _two_ initial approximations for this.  Put $f(z) = z^2-2$ and start with the two initial approximations $z_0  = 1$, $z_1 = 3/2$.  Carry out several steps of this (in exact arithmetic is better).  Convert each rational $z_n$ to continued fraction form.  Discuss what you find.

3. Try Newton and secant iteration on some functions of your own choosing.  You should see that Newton iteration usually takes fewer iterations to converge, but since it needs a derivative evaluation while the secant method does not, each iteration is "cheaper" in terms of computational cost (if $f(z)$ is at all expensive to evaluate, $f'(z)$ usually is too; there are exceptions, of course).  The consensus seems to be that the secant method is a bit more practical; but in some sense it is just a variation on Newton's method.

4. Both the Newton iteration and the secant iteration applied to $f(z) = z^2-a^2$ can be _solved analytically_ by the transformation $z = a\coth \theta$.  [Hyperbolic functions](https://en.wikipedia.org/wiki/Hyperbolic_functions)  The iteration $z_{n+1} = (z_n + a^2/z_n)/2$ becomes (you can check this) $\coth \theta_{n+1} = \cosh 2\theta_n/\sinh 2\theta_n = \coth 2\theta_n$, and so we may take $\theta_{n+1} = 2\theta_n$.  This can be solved to get $\theta_n = 2^n\theta_0$ and so we have an analytical formula for each $z_n = a \coth( 2^n \theta_0 )$.  Try this on $a^2=2$; you should find that $\theta_0 = \mathrm{invcoth}(1/\sqrt{2})$.  By "invcoth" we mean the functional inverse of coth, i.e.: $\coth\theta_0 = 1/\sqrt{2}$. It may surprise you that that number is complex.  Nevertheless, you will find that all subsequent iterates are real, and $\coth 2^n\theta_0$ goes to $1$ very quickly.
NB This was inadvertently difficult.  Neither numpy nor scipy has an invcoth (or arccoth) function. The Digital Library of Mathematical Functions says (equation 4.37.6) that arccoth(z) = arctanh(1/z).  Indeed we had to go to Maple to find out that invcoth$(1/\sqrt{2}) = \ln(1+\sqrt{2}) - i\pi/2$.

5. Try the above with $a^2=-1$.  NB the initial estimate $z_0 = 1$ fails! Try $z_0 = e = \exp(1) = 2.71828...$ instead. For this, the $\theta_0 = 1j\arctan(e^{-1})$. Then you might enjoy reading Gil Strang's lovely article [A Chaotic Search for $i$](https://www.jstor.org/stable/2686733).

6. Try to solve the _secant_ iteration for $z^2-a^2$ analytically.  You should eventually find a connection to Fibonacci numbers.

7. People keep inventing new rootfinding iterations.  Usually they are reinventions of methods that others have invented before, such as so-called _Schroeder_ iteration and _Householder_ iteration.  One step along the way is the method known as _Halley iteration_, which looks like this:
    \begin{equation*}
    z_{n+1} = z_n - \frac{f(z_n)}{f'(z_n) - \frac{f(z_n)f''(z_n)}{2f'(z_n)}}
    \end{equation*}
    which, as you can see, also involves the _second_ derivative of $f$.  When it works, it works quickly, typically converging in fewer iterations than Newton (although, typically, each step is more expensive computationally).  Try the method out on some examples.  It may help you to reuse your code (or Maple's code) if you are told that Newton iteration on $F(z) = f(z)/\sqrt{f'(z)}$ turns out to be identical to Halley iteration on $f(z)$. __NB: this trick helps you to "re-use" code, but it doesn't generate a particularly efficient iteration. In particular, the square roots muck up the formula for the derivatives, and simplification beforehand makes a big difference to program speed.  So if you want speed, you should program Halley's method directly.__

8. Try to solve Halley's iteration  for $x^2-a$ analytically.  Then you might enjoy reading [Revisiting Gilbert Strang's "A Chaotic Search for i"](https://doi.org/10.1145/3363520.3363521) by Ao Li and Rob Corless; Ao was a (graduate) student in the first iteration of this course at Western, and she solved—in class!—what was then an _open_ problem (this problem!).

9. Let's revisit question 4.  It turns out that we don't need to use hyperbolic functions. In the OEIS when searching for the numerators of our original sequence $1$, $3/2$, $17/12$ and so on, and also in the paper [What Newton Might Have Known](https://doi.org/10.1080/00029890.2021.1964274), we find the formulas $x_n = r_n/s_n$ where
    \begin{align*}
    r_n &= \frac{1}{2}\left( (1+\sqrt2)^{2^n} + (1-\sqrt2)^{2^n}\right) \\
    s_n &= \frac{1}{2\sqrt2}\left( (1+\sqrt2)^{2^n} - (1-\sqrt2)^{2^n}\right)
    \end{align*}
    Verify that this formula gives the same answers (when $a=2$) as the formula in question 4.  Try to generalize this formula for other integers $a$.  Discuss the growth of $r_n$ and $s_n$: it is termed _doubly exponential_. Show that the error $x_n - \sqrt2$ goes to zero like $1/(3+2\sqrt2)^{2^n}$.  How many iterations would you need to get ten thousand digits of accuracy? Do you need to calculate the $(1-\sqrt2)^{2^n}$ part?

10. Do the other results of [Revisiting Gilbert Strang's "A Chaotic Search for i"](https://doi.org/10.1145/3363520.3363521) on secant iteration, Halley's iteration, Householder iteration, and so on, translate to a form like that of question 9?  (We have not tried this ourselves yet).

11. Solve the Schroeder iteration problem of the paper [Revisiting Gilbert Strang's "A Chaotic Search for i"](https://doi.org/10.1145/3363520.3363521).  This iteration generates the image of the "infinite number of infinity symbols" used in the Preamble, by the way.  We don't know how to solve this problem (we mean, analytically, the way Newton, Secant, Halley, and Householder iterations were solved).  We'd be interested in your solution.

12. A farmer has a goat, a rope, a circular field, and a pole fixed firmly at the _edge_ of the field.  How much rope should the farmer allow so that the goat, tied to the pole, can eat the grass on exactly half the field? Discuss your assumptions a bit, and indicate how much precision in your solution is reasonable.


```julia
using Distributions
using PyPlot
using VinDsl
# include("lognormal.jl")
import VinDsl.value
srand(12345);
```

Set up some parameters

```julia
K = 3  # number of latent states
M = 2  # number of values of each latent state
T = 100  # number of time steps
U = 10  # units
```

    10

# Make firing rates

```julia
baseline = 10.
baseline_sd = 0.10
log_bl = log(baseline)
unit_bl = log_bl + baseline_sd * randn(U);
```

```julia
PyPlot.plt[:hist](unit_bl, 20);
```

```julia
fr_log = unit_bl' .+ zeros(T, U)
eta = fr_log + 0.001 * ones(size(fr_log)...)
fr = exp(eta);
```

```julia
PyPlot.plt[:hist](fr[:], 100);
```

# Make spikes

```julia
spikes = Array{Int}(size(fr)...)
for i in eachindex(fr)
    spikes[i] = rand(Poisson(fr[i]))
end
```

```julia
matshow(spikes', aspect="auto", cmap="gray")
```

# Initialize priors

## Baseline

```julia
μ_a = log(15.)
σ_a = 0.1
τ_a = 1/σ_a^2
a_prior = NormalCanon(μ_a * τ_a, τ_a)
```

    Distributions.NormalCanon{Float64,Float64}(η=270.805020110221, λ=99.99999999999999, μ=2.7080502011022105)

```julia
xx = linspace(0, 5, 500)
plot(xx, pdf(a_prior, xx));
```

## Noise

```julia
τ_prior = Gamma(2, 1/20)
```

    Distributions.Gamma{Float64}(α=2.0, θ=0.05)

```julia
xx = linspace(0, 1, 500)
plot(xx, pdf(τ_prior, xx));
```

# Make VinDsl model

We want to make the model

$$\begin{align}
N_{tu} &\sim \mathrm{Poisson}(e^{\eta_{tu}}) \\
\eta_{tu} &\sim \mathcal{N}(a_u, \tau_\eta) \\
a_u &\sim \mathcal{N}(m_a, t_a) \\
\tau_\eta &\sim \mathrm{Gamma}(s_\tau, l_\tau)
\end{align}$$

To do this, we first need to define a factor for the Poisson distribution parameterized in terms of log firing rate:

```julia
@deffactor LogPoissonAltFactor [n, x] begin
    E(n) * E(x) - lgamma(E(n) + 1) - mgf(x, 1)
end
```

    value (generic function with 11 methods)

Now let's define a posterior model:

$$\begin{align}
\eta_{tu} &\sim \mathcal{N}((\mu_\eta)_{tu}, (\sigma_\eta)_{tu}) \\
a_u &\sim \mathcal{N}((\mu_a)_u, (\sigma_a)_u) \\
\tau_\eta &\sim \mathrm{Gamma}(\alpha_\tau, \theta_\tau)
\end{align}$$

```julia
n[t, u] ~ Const(spikes)
η[t, u] ~ Normal(zeros(T, U), 0.5 * ones(T, U))
a[u] ~ Normal(zeros(U), 0.5 * ones(U))
τ ~ Gamma(1.1, 1/10)
m_a[u] ~ Const(zeros(U))
t_a[u] ~ Const(1 * ones(U))
s_τ ~ Const(1.1)
l_τ ~ Const(1/10)
```

    VinDsl.ConstantNode{Float64}(:l_τ,Symbol[],[:scalar],[0.1])

along with some factors...

```julia
a_prior = @factor LogNormalFactor a m_a t_a
a_entropy = @factor EntropyFactor a
τ_prior = @factor LogGammaFactor τ s_τ l_τ
τ_entropy = @factor EntropyFactor τ
η_entropy = @factor EntropyFactor η
η_lik = @factor LogNormalFactor η a τ
obs = @factor LogPoissonAltFactor n η
m = VBModel([n, η, a, τ, m_a, t_a, s_τ, l_τ], [a_prior, a_entropy, τ_prior, τ_entropy, η_entropy, η_lik, obs]);
```

```julia
ELBO(m::VBModel) = sum([value(f) for f in m.factors])
```

    ELBO (generic function with 1 method)

```julia
[(n.name, s) for (n, s) in m.update_strategy]
```

    8-element Array{Tuple{Any,Any},1}:
     (:a,:conjugate)
     (:l_τ,:constant)
     (:n,:constant)
     (:τ,:conjugate)
     (:t_a,:constant)
     (:m_a,:constant)
     (:s_τ,:constant)
     (:η,:undefined)

```julia
m.update_strategy[η] = :l_bfgs
```

    :l_bfgs

ELBO(m)update!(a, m, Val{m.update_strategy[a]})
ELBO(m)update!(τ, m, Val{m.update_strategy[τ]})
ELBO(m)update!(η, m, Val{m.update_strategy[η]})
ELBO(m)

```julia
niter = 10
display(ELBO(m))
for _ in 1:niter
    update!(m)
    display(ELBO(m))
end
```

    -19604.303584733316

    -3265.7516584427913

    -2957.433222863406

    -2877.5827311914823

    -2856.4826982347195

    -2850.14756593828

    -2848.1305868552895

    -2847.4718277108905

    -2847.2492707397414

    -2847.1734585485774

    -2847.1475372136974

```julia
E(a.data)
```

    10-element Array{Float64,1}:
     2.36172
     2.38031
     2.30471
     2.31935
     2.31475
     2.32662
     2.18501
     2.28043
     2.40414
     2.25594

```julia
unit_bl
```

    10-element Array{Float64,1}:
     2.41982
     2.38786
     2.34414
     2.35423
     2.37116
     2.33081
     2.15488
     2.31526
     2.44923
     2.28677

```julia

```


```python
%matplotlib inline
import matplotlib.pyplot as plt
from sympy import init_session
from IPython.display import HTML
#HTML('')
init_session()
rho, sigma, b = symbols('rho sigma b')
```

    IPython console for SymPy 1.0 (Python 3.5.2-64-bit) (ground types: python)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at http://docs.sympy.org/1.0/

# Chapter 9: Lorenz Equations

p.309 of Strogatz, S (2015)

```python
# Youtube
#HTML('')
```

```python
#The basic equations
x_dot, y_dot, z_dot = sigma*(y-x), rho*x - y - x*z, x*y-b*z
x_dot, y_dot, z_dot
```

```python
#Example 9.2.2: Find fixed points
#In a future version of sympy there is a plan to be able to add conditions like rho>1
fixed_points = solve([x_dot, y_dot, z_dot] ,[x,y,z])
fixed_points
#These can be indexed for use later...
```

```python
#Linear stability of the origin: p.322.
tau, Delta = symbols('tau Delta')
A = Matrix([[-sigma, sigma],[rho,-1]])
xx = Matrix([x,y])
# A = Matrix([[-sigma, sigma,0],[rho,-1,0],[0,0,-b]])
# xx = Matrix([x,y,z])
determinant, trace = A.det(), A.trace()
stable_matrix = A*xx
stable_matrix, Eq(Delta,determinant),Eq(tau, trace)
```

```python
factor(trace**2-4*determinant)
```

```python
# Calculate:
V_dot = (1/sigma*x_dot*x + y_dot*y + z_dot*z)
V_dot
```

```python
ans = solve([V_dot],[x,y,z])
ans
```

```python
solve([ans[0][x].subs(rho,0)],[y,z])

```

```python
#With some more fiddling around, one can see that the solution becomes x = 0 when b = below for soln #1
rh = symbols('rho', positive=True)
eq = Eq(y/2*(rh+1),Rational(1,2)*sqrt(-4*b*z**2+rh**2*y**2+2*rh*y**2-3*y**2))
ans1 = solve(eq)
ans1
```

```python
factor(eq.subs(ans1[0]))
```

### 9.3 Chaos on a strange attractor.

This section helps visualise what plotting one axis, the y-axis, looks like over time. Of course one has to integrate against some starting values.

In this instance, the code loads a script plotting the trajectory of the velocity over time, from which the values of $(x,y,z)$ are derived from $(x_0,y_0,z_0)$.

```python
# %load Lorenz\ attractor\ mayavi.py

# Please attribute to Llewelyn Richards-Ward,
#llewelyn62@icloud.com
#Use and distribute as you want.
from mayavi import mlab
import numpy as np
import numba as nb
from scipy.integrate import odeint

#Create figure
mlab.figure(size=(900,800), bgcolor=(0,0,0), fgcolor=(1,1,1))
#For a very basic  approach which shows the steps, this commented
#section can be used. The preferred method is to
#use scipy's odeint and an array/function combination, as below.
#Note the use of the Euler integration method.
#=========================================
#   # Integration time step
# dt = 0.001
#   # Lorenz ODE parameters
# sigma = 10.0
# r     = 28.0
# b     = 8.0/3.0
#   # Initial conditions
# x = -.5
# y =  .2
# z =  2.17
#   # Store the trajectory
# TrajX = []
# TrajY = []
# TrajZ = []
# Time  = []
#
#   # Integrate the Lorenz ODEs
#   #Pretty basic approach -- in real examples would
#   #use odeint and an array/function combination.
# for t in np.arange(0.,50.,dt):
# 	dxdt = sigma*(y - x)
# 	dydt = r * x - y - x * z
# 	dzdt = x * y - b * z
# 	x = x + dxdt * dt
# 	y = y + dydt * dt
# 	z = z + dzdt * dt
#
# 	TrajX.append(x)
# 	TrajY.append(y)
# 	TrajZ.append(z)
# 	Time.append(t)
#=========================================

# Lorenz paramters and initial conditions
sigma, beta, rho = 10, 8/3, 28
u0, v0, w0 = 0, 1, 0

# Maximum time point and total number of time points
tmax, n = 50, 10000
@nb.jit(nopython=False) #optimises for speed.
def deriv_lorenz(X, t, sigma, beta, rho):
    """The Lorenz equations."""
    x, y, z = X
    dx_dt = -sigma*(x - y)
    dy_dt = rho*x - y - x*z
    dz_dt = -beta*z + x*y
    return dx_dt, dy_dt, dz_dt

# Integrate the Lorenz equations on the time grid t
#Implements a detailed use of memory space for optimised accuracy, not
#so much speed.
@nb.jit('float64(float64,float64,float64,float64,float64,float64,float64,float64)',nopython=False)
def int_f(u0=u0,v0=v0,w0=w0,sigma=sigma,beta=beta,rho=rho,tma=tmax,n=n):
    t = np.linspace(0, tmax, n)
    f = odeint(deriv_lorenz, (u0, v0, w0), t, args=(sigma, beta, rho))
    return f
x1, y1, z1 = int_f().T
Time = np.linspace(0,tmax,n)
def plot_mayavi():
    mlab.plot3d(x1,y1,z1,Time,colormap='Spectral',tube_radius=0.3)
    mlab.outline(opacity=.4)
    mlab.points3d(0,0,0,opacity=.5)
    mlab.text3d(0,0,0,'Origin')
    mlab.points3d(np.sqrt(beta*(rho-1)),np.sqrt(beta*(rho-1)),rho-1,opacity=.5)
    mlab.text3d(np.sqrt(beta*(rho-1))*1.2,np.sqrt(beta*(rho-1)),rho-1,'C+')
    mlab.points3d(-np.sqrt(beta*(rho-1)),-np.sqrt(beta*(rho-1)),rho-1,opacity=.5)
    mlab.text3d(-np.sqrt(beta*(rho-1))*1.2,-np.sqrt(beta*(rho-1)),rho-1,'C-')
    mlab.orientation_axes(xlabel='x',ylabel='y',zlabel='z')
    plt.show()

def animate_the_mayavi_plot():
    @mlab.animate(delay=50, ui=False)
    def anim():
        f = mlab.gcf()
        while 1:
            f.scene.camera.azimuth(2)
            f.scene.render()
            yield

    a = anim() # Starts the animation.
# mlab.savefig('Lorenz.png')

```

```python
#Now to plot y(t), as in Figure 9.3.1
%matplotlib inline
fig, (ax0,ax1) = plt.subplots(1,2,figsize=(12,6))
fig.set_facecolor('lightgrey')
ax0.set_title('Fig. 9.3.1: Lorenz plot of one axis against time')
ax0.set_xlabel('Time evolution')
ax0.set_ylabel(r'$y(t)$')
ax0.grid()
ax0.plot(y1)
ax1.set_title('Fig. 9.3.2: Lorenz plot of x(t) v z(t)')
ax1.set_xlabel(r'$x(t)$')
ax1.set_ylabel(r'$z(t)$')
ax1.grid()
ax1.plot(x1,z1,'r,')
```

### 9.4 Lorenz Map
This is about finding the local maximum.

```python
#From http://pythonfiddle.com/return-local-maxima/
def FindMaxima(numbers):
    maxima = []
    length = len(numbers)
    if length >= 2:
        if numbers[0] > numbers[1]:
              maxima.append(numbers[0])

    if length > 3:
        for i in range(1, length-1):
            if numbers[i] > numbers[i-1] and numbers[i] > numbers[i+1]:
                maxima.append(numbers[i])

    if numbers[length-1] > numbers[length-2]:
        maxima.append(numbers[length-1])
    return maxima

zmax2 = FindMaxima(z1)
```

```python
fig, (ax2,ax3) = plt.subplots(1,2,figsize=(12,6))
fig.set_facecolor('lightgrey',)
ax2.set_title('Fig. 9.4.2: Lorenz plot of one axis against time')
ax2.set_xlabel('Time evolution')
ax2.set_ylabel(r'$z(t)$')
ax2.grid()
ax2.plot(y1)
ax3.set_title(r'Fig. 9.4.3: Does $z_n$ predict $z_{n + 1}$?')
ax3.set_xlabel(r'$z_n$')
ax3.set_ylabel(r'$z_{n+1}$')
ax3.grid()
ax3.plot(zmax2[1:-2],zmax2[2:-1],'r+') #Dump the first point set.
```

```python
ax3.set_facecolor?
```

```python

```