# Assignment 1

The goal of this assignment is to supply you with machine learning models and algorithms. In this notebook, we will cover linear and nonlinear models, the concept of loss functions and some optimization techniques. All mathematical operations should be implemented in **NumPy** only.

## Table of contents
* [1. Logistic Regression](#1.-Logistic-Regression)
    * [1.1 Linear Mapping](#1.1-Linear-Mapping)
    * [1.2 Sigmoid](#1.2-Sigmoid)
    * [1.3 Negative Log Likelihood](#1.3-Negative-Log-Likelihood)
    * [1.4 Model](#1.4-Model)
    * [1.5 Simple Experiment](#1.5-Simple-Experiment)
* [2. Decision Tree](#2.-Decision-Tree)
    * [2.1 Gini Index & Data Split](#2.1-Gini-Index-&-Data-Split)
    * [2.2 Terminal Node](#2.2-Terminal-Node)
    * [2.3 Build the Decision Tree](#2.3-Build-the-Decision-Tree)
* [3. Experiments](#3.-Experiments)
    * [3.1 Decision Tree for Heart Disease Prediction](#3.1-Decision-Tree-for-Heart-Disease-Prediction)
    * [3.2 Logistic Regression for Heart Disease Prediction](#3.2-Logistic-Regression-for-Heart-Disease-Prediction)

### Note
Some of the concepts below have not (yet) been discussed during the lecture. These will be discussed further during the next lectures.

### Before you begin

To check whether the code you've written is correct, we'll use **automark**. For this, we created for each of you an account with the username being your student number.

```python
import automark as am

# fill in you student number as your username
username = '10677186'

# to check your progress, you can run this function
am.get_progress(username)
```

    ---------------------------------------------
    | Harm Manders                              |
    | harmmanders@gmail.com                     |
    ---------------------------------------------
    | W2: linear_forward       | completed      |
    | W2: linear_grad_W        | completed      |
    | W2: linear_grad_b        | completed      |
    | W2: nll_forward          | completed      |
    | W2: nll_grad_input       | completed      |
    | W2: sigmoid_forward      | completed      |
    | W2: sigmoid_grad_input   | completed      |
    | W2: tree_gini_index      | not attempted  |
    | W2: tree_split_data_left | not attempted  |
    | W2: tree_split_data_right| not attempted  |
    | W2: tree_to_terminal     | not attempted  |
    | W3: box_blur             | not attempted  |
    | W3: conv_matrix          | not attempted  |
    | W3: dense_forward        | not attempted  |
    | W3: dense_grad_W         | not attempted  |
    | W3: dense_grad_b         | not attempted  |
    | W3: dense_grad_input     | not attempted  |
    | W3: flatten_forward      | not attempted  |
    | W3: flatten_grad_input   | not attempted  |
    | W3: l2_regularizer       | not attempted  |
    | W3: maxpool_forward      | not attempted  |
    | W3: relu_forward         | not attempted  |
    | W3: relu_grad_input      | not attempted  |
    ---------------------------------------------

So far all your tests are 'not attempted'. At the end of this notebook you'll need to have completed all test. The output of `am.get_progress(username)` should at least match the example below. However, we encourage you to take a shot at the 'not attempted' tests!

```
---------------------------------------------
| Your name / student number                |
| your_email@your_domain.whatever           |
---------------------------------------------
| linear_forward           | not attempted  |
| linear_grad_W            | not attempted  |
| linear_grad_b            | not attempted  |
| nll_forward              | not attempted  |
| nll_grad_input           | not attempted  |
| sigmoid_forward          | not attempted  |
| sigmoid_grad_input       | not attempted  |
| tree_data_split_left     | not attempted  |
| tree_data_split_right    | not attempted  |
| tree_gini_index          | not attempted  |
| tree_to_terminal         | not attempted  |
---------------------------------------------
```

```python
from __future__ import print_function, absolute_import, division # You don't need to know what this is.
import numpy as np # this imports numpy, which is used for vector- and matrix calculations
```

```python
import time
start = time.time()
```

```python
def call30(f, args):
    global start
    ttw = 30 - (time.time() - start)
    if ttw > 0:
        print("Time to wait: {}s".format(round(ttw, 2)))
        time.sleep(ttw)
    f( *args )
    start = time.time()
```

This notebook makes use of **classes** and their **instances** that we have already implemented for you. It allows us to write less code and make it more readable. If you are interested in it, here are some useful links:
* The official [documentation](https://docs.python.org/3/tutorial/classes.html)
* Video by *sentdex*: [Object Oriented Programming Introduction](https://www.youtube.com/watch?v=ekA6hvk-8H8)
* Antipatterns in OOP: [Stop Writing Classes](https://www.youtube.com/watch?v=o9pEzgHorH0)

# 1. Logistic Regression

We start with a very simple algorithm called **Logistic Regression**. It is a generalized linear model for 2-class classification.
It can be generalized to the case of many classes and to non-linear cases as well. However, here we consider only the simplest case.

Let us consider a data with 2 classes. Class 0 and class 1. For a given test sample, logistic regression returns a value from $[0, 1]$ which is interpreted as a probability of belonging to class 1. The set of points for which the prediction is $0.5$ is called a *decision boundary*. It is a line on a plane or a hyper-plane in a space.

Logistic regression has two trainable parameters: a weight $W$ and a bias $b$. For a vector of features $X$, the prediction of logistic regression is given by

$$
f(X) = \frac{1}{1 + \exp(-[XW + b])} = \sigma(h(X))
$$
where $\sigma(z) = \frac{1}{1 + \exp(-z)}$ and $h(X)=XW + b$.

Parameters $W$ and $b$ are fitted by maximizing the log-likelihood (or minimizing the negative log-likelihood) of the model on the training data. For a training subset $\{X_j, Y_j\}_{j=1}^N$ the normalized negative log likelihood (NLL) is given by

$$
\mathcal{L} = -\frac{1}{N}\sum_j \log\Big[ f(X_j)^{Y_j} \cdot (1-f(X_j))^{1-Y_j}\Big]
= -\frac{1}{N}\sum_j \Big[ Y_j\log f(X_j) + (1-Y_j)\log(1-f(X_j))\Big]
$$

There are different ways of fitting this model. In this assignment we consider Logistic Regression as a one-layer neural network. We use the following algorithm for the **forward** pass:

1. Linear mapping: $h=XW + b$
2. Sigmoid activation function: $f=\sigma(h)$
3. Calculation of NLL: $\mathcal{L} = -\frac{1}{N}\sum_j \Big[ Y_j\log f_j + (1-Y_j)\log(1-f_j)\Big]$

In order to fit $W$ and $b$ we perform Gradient Descent ([GD](https://en.wikipedia.org/wiki/Gradient_descent)). We choose a small learning rate $\gamma$ and after each computation of forward pass, we update the parameters

$$W_{\text{new}} = W_{\text{old}} - \gamma \frac{\partial \mathcal{L}}{\partial W}$$

$$b_{\text{new}} = b_{\text{old}} - \gamma \frac{\partial \mathcal{L}}{\partial b}$$

We use Backpropagation method ([BP](https://en.wikipedia.org/wiki/Backpropagation)) to calculate the partial derivatives of the loss function with respect to the parameters of the model.

$$
\frac{\partial\mathcal{L}}{\partial W} =
\frac{\partial\mathcal{L}}{\partial h} \frac{\partial h}{\partial W} =
\frac{\partial\mathcal{L}}{\partial f} \frac{\partial f}{\partial h} \frac{\partial h}{\partial W}
$$

$$
\frac{\partial\mathcal{L}}{\partial b} =
\frac{\partial\mathcal{L}}{\partial h} \frac{\partial h}{\partial b} =
\frac{\partial\mathcal{L}}{\partial f} \frac{\partial f}{\partial h} \frac{\partial h}{\partial b}
$$

## 1.1 Linear Mapping
First of all, you need to implement the forward pass of a linear mapping:
$$
h(X) = XW +b
$$

**Note**: here we use `n_out` as the dimensionality of the output. For logisitc regression `n_out = 1`. However, we will work with cases of `n_out > 1` in next assignments. You will **pass** the current assignment even if your implementation works only in case `n_out = 1`. If your implementation works for the cases of `n_out > 1` then you will not have to modify your method next week. All **numpy** operations are generic. It is recommended to use numpy when is it possible.

```python
def linear_forward(x_input, W, b):
    """Perform the mapping of the input
    # Arguments
        x_input: input of the linear function - np.array of size `(n_objects, n_in)`
        W: np.array of size `(n_in, n_out)`
        b: np.array of size `(n_out,)`
    # Output
        the output of the linear function
        np.array of size `(n_objects, n_out)`
    """
    return x_input @ W + b # @ is operator for matrix multiplication
```

Let's check your first function. We set the matrices $X, W, b$:
$$
X = \begin{bmatrix}
1 & -1 \\
-1 & 0 \\
1 & 1 \\
\end{bmatrix} \quad
W = \begin{bmatrix}
4 \\
2 \\
\end{bmatrix} \quad
b = \begin{bmatrix}
3 \\
\end{bmatrix}
$$

And then compute
$$
XW = \begin{bmatrix}
1 & -1 \\
-1 & 0 \\
1 & 1 \\
\end{bmatrix}
\begin{bmatrix}
4 \\
2 \\
\end{bmatrix} =
\begin{bmatrix}
2 \\
-4 \\
6 \\
\end{bmatrix} \\
XW + b =
\begin{bmatrix}
5 \\
-1 \\
9 \\
\end{bmatrix}
$$

```python
X_test = np.array([[1, -1],
                   [-1, 0],
                   [1, 1]])

W_test = np.array([[4],
                   [2]])

b_test = np.array([3])

h_test = linear_forward(X_test, W_test, b_test)
print(h_test)
```

    [[ 5]
     [-1]
     [ 9]]

```python
call30(am.test_student_function, (username, linear_forward, ['x_input', 'W', 'b']))
```

    Time to wait: 26.79s
    Running local tests...
    linear_forward successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!

Now you need to implement the calculation of the partial derivative of the loss function with respect to the parameters of the model. As this expressions are used for the updates of the parameters, we refer to them as gradients.
$$
\frac{\partial \mathcal{L}}{\partial W} =
\frac{\partial \mathcal{L}}{\partial h}
\frac{\partial h}{\partial W} \\
\frac{\partial \mathcal{L}}{\partial b} =
\frac{\partial \mathcal{L}}{\partial h}
\frac{\partial h}{\partial b} \\
$$

```python
def linear_grad_W(x_input, grad_output, W, b):
    """Calculate the partial derivative of
        the loss with respect to W parameter of the function
        dL / dW = (dL / dh) * (dh / dW)
    # Arguments
        x_input: input of a dense layer - np.array of size `(n_objects, n_in)`
        grad_output: partial derivative of the loss functions with
            respect to the ouput of the dense layer (dL / dh)
            np.array of size `(n_objects, n_out)`
        W: np.array of size `(n_in, n_out)`
        b: np.array of size `(n_out,)`
    # Output
        the partial derivative of the loss
        with respect to W parameter of the function
        np.array of size `(n_in, n_out)`
    """
    # dh/dW = X
    return x_input.T @ grad_output
```

```python
call30(am.test_student_function, (username, linear_grad_W, ['x_input', 'grad_output', 'W', 'b']))
```

    Running local tests...
    linear_grad_W successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!

```python
def linear_grad_b(x_input, grad_output, W, b):
    """Calculate the partial derivative of
        the loss with respect to b parameter of the function
        dL / db = (dL / dh) * (dh / db)
    # Arguments
        x_input: input of a dense layer - np.array of size `(n_objects, n_in)`
        grad_output: partial derivative of the loss functions with
            respect to the ouput of the linear function (dL / dh)
            np.array of size `(n_objects, n_out)`
        W: np.array of size `(n_in, n_out)`
        b: np.array of size `(n_out,)`
    # Output
        the partial derivative of the loss
        with respect to b parameter of the linear function
        np.array of size `(n_out,)`
    """
    # dh/db = [1] * n_out
    return grad_output.T @ np.ones((np.shape(x_input)[0],))
```

```python
call30(am.test_student_function, (username, linear_grad_b, ['x_input', 'grad_output', 'W', 'b']))
```

    Time to wait: 15.5s
    Running local tests...
    linear_grad_b successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!

```python
am.get_progress(username)
```

    ---------------------------------------------
    | Harm Manders                              |
    | harmmanders@gmail.com                     |
    ---------------------------------------------
    | W2: linear_forward       | completed      |
    | W2: linear_grad_W        | completed      |
    | W2: linear_grad_b        | completed      |
    | W2: nll_forward          | completed      |
    | W2: nll_grad_input       | completed      |
    | W2: sigmoid_forward      | completed      |
    | W2: sigmoid_grad_input   | completed      |
    | W2: tree_gini_index      | not attempted  |
    | W2: tree_split_data_left | not attempted  |
    | W2: tree_split_data_right| not attempted  |
    | W2: tree_to_terminal     | not attempted  |
    | W3: box_blur             | not attempted  |
    | W3: conv_matrix          | not attempted  |
    | W3: dense_forward        | not attempted  |
    | W3: dense_grad_W         | not attempted  |
    | W3: dense_grad_b         | not attempted  |
    | W3: dense_grad_input     | not attempted  |
    | W3: flatten_forward      | not attempted  |
    | W3: flatten_grad_input   | not attempted  |
    | W3: l2_regularizer       | not attempted  |
    | W3: maxpool_forward      | not attempted  |
    | W3: relu_forward         | not attempted  |
    | W3: relu_grad_input      | not attempted  |
    ---------------------------------------------

## 1.2 Sigmoid
$$
f = \sigma(h) = \frac{1}{1 + e^{-h}}
$$

Sigmoid function is applied element-wise. It does not change the dimensionality of the tensor and its implementation is shape-agnostic in general.

```python
def sigmoid_forward(x_input):
    """sigmoid nonlinearity
    # Arguments
        x_input: np.array of size `(n_objects, n_in)`
    # Output
        the output of relu layer
        np.array of size `(n_objects, n_in)`
    """
    return 1 / (1+np.exp(-x_input))
```

```python
call30(am.test_student_function, (username, sigmoid_forward, ['x_input']))
```

    Time to wait: 29.89s
    Running local tests...
    sigmoid_forward successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!

Now you need to implement the calculation of the partial derivative of the loss function with respect to the input of sigmoid.

$$
\frac{\partial \mathcal{L}}{\partial h} =
\frac{\partial \mathcal{L}}{\partial f}
\frac{\partial f}{\partial h}
$$

Tensor $\frac{\partial \mathcal{L}}{\partial f}$ comes from the loss function. Let's calculate $\frac{\partial f}{\partial h}$

$$
\frac{\partial f}{\partial h} =
\frac{\partial \sigma(h)}{\partial h} =
\frac{\partial}{\partial h} \Big(\frac{1}{1 + e^{-h}}\Big)
= \frac{e^{-h}}{(1 + e^{-h})^2}
= \frac{1}{1 + e^{-h}} \frac{e^{-h}}{1 + e^{-h}}
= f(h) (1 - f(h))
$$

Therefore, in order to calculate the gradient of the loss with respect to the input of sigmoid function you need
to
1. calculate $f(h) (1 - f(h))$
2. multiply it element-wise by $\frac{\partial \mathcal{L}}{\partial f}$

```python
def sigmoid_grad_input(x_input, grad_output):
    """sigmoid nonlinearity gradient.
        Calculate the partial derivative of the loss
        with respect to the input of the layer
    # Arguments
        x_input: np.array of size `(n_objects, n_in)`
        grad_output: np.array of size `(n_objects, n_in)`
            dL / df
    # Output
        the partial derivative of the loss
        with respect to the input of the function
        np.array of size `(n_objects, n_in)`
        dL / dh
    """
    return sigmoid_forward(x_input) * (1 - sigmoid_forward(x_input)) * grad_output
```

```python
call30(am.test_student_function, (username, sigmoid_grad_input, ['x_input', 'grad_output']))
```

    Time to wait: 29.99s
    Running local tests...
    sigmoid_grad_input successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!

## 1.3 Negative Log Likelihood

$$
\mathcal{L}
= -\frac{1}{N}\sum_j \Big[ Y_j\log \dot{Y}_j + (1-Y_j)\log(1-\dot{Y}_j)\Big]
$$

Here $N$ is the number of objects. $Y_j$ is the real label of an object and $\dot{Y}_j$ is the predicted one.

```python
def nll_forward(target_pred, target_true):
    """Compute the value of NLL
        for a given prediction and the ground truth
    # Arguments
        target_pred: predictions - np.array of size `(n_objects, 1)`
        target_true: ground truth - np.array of size `(n_objects, 1)`
    # Output
        the value of NLL for a given prediction and the ground truth
        scalar
    """
    N = np.shape(target_pred)[0]
    s = np.sum(target_true * np.log(target_pred) + (1-target_true) * np.log(1-target_pred))
    return - 1 / N * s
```

```python
call30(am.test_student_function, (username, nll_forward, ['target_pred', 'target_true']))
```

    Time to wait: 29.99s
    Running local tests...
    nll_forward successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!

Now you need to calculate the partial derivative of NLL with with respect to its input.

$$
\frac{\partial \mathcal{L}}{\partial \dot{Y}}
=
\begin{pmatrix}
\frac{\partial \mathcal{L}}{\partial \dot{Y}_0} \\
\frac{\partial \mathcal{L}}{\partial \dot{Y}_1} \\
\vdots \\
\frac{\partial \mathcal{L}}{\partial \dot{Y}_N}
\end{pmatrix}
$$

Let's do it step-by-step

\begin{equation}
\begin{split}
\frac{\partial \mathcal{L}}{\partial \dot{Y}_0}
&= \frac{\partial}{\partial \dot{Y}_0} \Big(-\frac{1}{N}\sum_j \Big[ Y_j\log \dot{Y}_j + (1-Y_j)\log(1-\dot{Y}_j)\Big]\Big) \\
&= -\frac{1}{N} \frac{\partial}{\partial \dot{Y}_0} \Big(Y_0\log \dot{Y}_0 + (1-Y_0)\log(1-\dot{Y}_0)\Big) \\
&= -\frac{1}{N} \Big(\frac{Y_0}{\dot{Y}_0} - \frac{1-Y_0}{1-\dot{Y}_0}\Big)
= \frac{1}{N} \frac{\dot{Y}_0 - Y_0}{\dot{Y}_0 (1 - \dot{Y}_0)}
\end{split}
\end{equation}

And for the other components it can be done in exactly the same way. So the result is the vector where each component is given by
$$\frac{1}{N} \frac{\dot{Y}_j - Y_j}{\dot{Y}_j (1 - \dot{Y}_j)}$$

Or if we assume all multiplications and divisions to be done element-wise the output can be calculated as
$$
\frac{\partial \mathcal{L}}{\partial \dot{Y}} = \frac{1}{N} \frac{\dot{Y} - Y}{\dot{Y} (1 - \dot{Y})}
$$

```python
def nll_grad_input(target_pred, target_true):
    """Compute the partial derivative of NLL
        with respect to its input
    # Arguments
        target_pred: predictions - np.array of size `(n_objects, 1)`
        target_true: ground truth - np.array of size `(n_objects, 1)`
    # Output
        the partial derivative
        of NLL with respect to its input
        np.array of size `(n_objects, 1)`
    """
    N = np.shape(target_pred)[0]
    return 1 / N * (target_pred - target_true) / (target_pred * (1 - target_pred))
```

```python
call30(am.test_student_function, (username, nll_grad_input, ['target_pred', 'target_true']))
```

    Time to wait: 29.99s
    Running local tests...
    nll_grad_input successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!

```python
am.get_progress(username)
```

    ---------------------------------------------
    | Harm Manders                              |
    | harmmanders@gmail.com                     |
    ---------------------------------------------
    | W2: linear_forward       | completed      |
    | W2: linear_grad_W        | completed      |
    | W2: linear_grad_b        | completed      |
    | W2: nll_forward          | completed      |
    | W2: nll_grad_input       | completed      |
    | W2: sigmoid_forward      | completed      |
    | W2: sigmoid_grad_input   | completed      |
    | W2: tree_gini_index      | not attempted  |
    | W2: tree_split_data_left | not attempted  |
    | W2: tree_split_data_right| not attempted  |
    | W2: tree_to_terminal     | not attempted  |
    | W3: box_blur             | not attempted  |
    | W3: conv_matrix          | not attempted  |
    | W3: dense_forward        | not attempted  |
    | W3: dense_grad_W         | not attempted  |
    | W3: dense_grad_b         | not attempted  |
    | W3: dense_grad_input     | not attempted  |
    | W3: flatten_forward      | not attempted  |
    | W3: flatten_grad_input   | not attempted  |
    | W3: l2_regularizer       | not attempted  |
    | W3: maxpool_forward      | not attempted  |
    | W3: relu_forward         | not attempted  |
    | W3: relu_grad_input      | not attempted  |
    ---------------------------------------------

## 1.4 Model

Here we provide a model for your. It consist of the function which you have implmeneted above

```python
class LogsticRegressionGD(object):

    def __init__(self, n_in, lr=0.05):
        super().__init__()
        self.lr = lr
        self.b = np.zeros(1, )
        self.W = np.random.randn(n_in, 1)

    def forward(self, x):
        self.h = linear_forward(x, self.W, self.b)
        y = sigmoid_forward(self.h)
        return y

    def update_params(self, x, nll_grad):
        # compute gradients
        grad_h = sigmoid_grad_input(self.h, nll_grad)
        grad_W = linear_grad_W(x, grad_h, self.W, self.b)
        grad_b = linear_grad_b(x, grad_h, self.W, self.b)
        # update params
        self.W = self.W - self.lr * grad_W
        self.b = self.b - self.lr * grad_b
```

## 1.5 Simple Experiment

```python
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
# Generate some data
def generate_2_circles(N=100):
    phi = np.linspace(0.0, np.pi * 2, 100)
    X1 = 1.1 * np.array([np.sin(phi), np.cos(phi)])
    X2 = 3.0 * np.array([np.sin(phi), np.cos(phi)])
    Y = np.concatenate([np.ones(N), np.zeros(N)]).reshape((-1, 1))
    X = np.hstack([X1,X2]).T
    return X, Y

def generate_2_gaussians(N=100):
    phi = np.linspace(0.0, np.pi * 2, 100)
    X1 = np.random.normal(loc=[1, 2], scale=[2.5, 0.9], size=(N, 2))
    X1 = X1.dot(np.array([[0.7, -0.7], [0.7, 0.7]]))
    X2 = np.random.normal(loc=[-2, 0], scale=[1, 1.5], size=(N, 2))
    X2 = X2.dot(np.array([[0.7, 0.7], [-0.7, 0.7]]))
    Y = np.concatenate([np.ones(N), np.zeros(N)]).reshape((-1, 1))
    X = np.vstack([X1,X2])
    return X, Y

def split(X, Y, train_ratio=0.7):
    size = len(X)
    train_size = int(size * train_ratio)
    indices = np.arange(size)
    np.random.shuffle(indices)
    train_indices = indices[:train_size]
    test_indices = indices[train_size:]
    return X[train_indices], Y[train_indices], X[test_indices], Y[test_indices]

f, (ax1, ax2) = plt.subplots(1, 2, figsize=(8, 4))

X, Y = generate_2_circles()
ax1.scatter(X[:,0], X[:,1], c=Y.ravel(), edgecolors= 'none')
ax1.set_aspect('equal')

X, Y = generate_2_gaussians()
ax2.scatter(X[:,0], X[:,1], c=Y.ravel(), edgecolors= 'none')
ax2.set_aspect('equal')

```

### Fit on gaussian data

```python
X_train, Y_train, X_test, Y_test = split(*generate_2_gaussians(), 0.7)
```

```python
# let's train our model
model = LogsticRegressionGD(2, 0.05)

for step in range(30):
    Y_pred = model.forward(X_train)

    loss_value = nll_forward(Y_pred, Y_train)
    accuracy = ((Y_pred > 0.5) == Y_train).mean()
    print('Step: {} \t Loss: {:.3f} \t Acc: {:.1f}%'.format(step, loss_value, accuracy * 100))

    loss_grad = nll_grad_input(Y_pred, Y_train)
    model.update_params(X_train, loss_grad)

print('\n\nTesting...')
Y_test_pred = model.forward(X_test)
test_accuracy = ((Y_test_pred > 0.5) == Y_test).mean()
print('Acc: {:.1f}%'.format(test_accuracy * 100))
```

    Step: 0 	 Loss: 0.662 	 Acc: 65.7%
    Step: 1 	 Loss: 0.627 	 Acc: 68.6%
    Step: 2 	 Loss: 0.595 	 Acc: 70.7%
    Step: 3 	 Loss: 0.565 	 Acc: 72.1%
    Step: 4 	 Loss: 0.537 	 Acc: 72.9%
    Step: 5 	 Loss: 0.512 	 Acc: 75.0%
    Step: 6 	 Loss: 0.488 	 Acc: 76.4%
    Step: 7 	 Loss: 0.466 	 Acc: 77.1%
    Step: 8 	 Loss: 0.446 	 Acc: 79.3%
    Step: 9 	 Loss: 0.427 	 Acc: 80.0%
    Step: 10 	 Loss: 0.410 	 Acc: 82.9%
    Step: 11 	 Loss: 0.393 	 Acc: 84.3%
    Step: 12 	 Loss: 0.378 	 Acc: 84.3%
    Step: 13 	 Loss: 0.364 	 Acc: 84.3%
    Step: 14 	 Loss: 0.351 	 Acc: 85.7%
    Step: 15 	 Loss: 0.339 	 Acc: 86.4%
    Step: 16 	 Loss: 0.328 	 Acc: 87.1%
    Step: 17 	 Loss: 0.318 	 Acc: 87.1%
    Step: 18 	 Loss: 0.308 	 Acc: 88.6%
    Step: 19 	 Loss: 0.299 	 Acc: 89.3%
    Step: 20 	 Loss: 0.290 	 Acc: 90.0%
    Step: 21 	 Loss: 0.282 	 Acc: 90.7%
    Step: 22 	 Loss: 0.275 	 Acc: 92.1%
    Step: 23 	 Loss: 0.268 	 Acc: 92.1%
    Step: 24 	 Loss: 0.261 	 Acc: 92.1%
    Step: 25 	 Loss: 0.255 	 Acc: 93.6%
    Step: 26 	 Loss: 0.249 	 Acc: 93.6%
    Step: 27 	 Loss: 0.244 	 Acc: 94.3%
    Step: 28 	 Loss: 0.238 	 Acc: 94.3%
    Step: 29 	 Loss: 0.234 	 Acc: 94.3%

    Testing...
    Acc: 93.3%

```python
def plot_model_prediction(prediction_func, X, Y, hard=True):
    u_min = X[:, 0].min()-1
    u_max = X[:, 0].max()+1
    v_min = X[:, 1].min()-1
    v_max = X[:, 1].max()+1

    U, V = np.meshgrid(np.linspace(u_min, u_max, 100), np.linspace(v_min, v_max, 100))
    UV = np.stack([U.ravel(), V.ravel()]).T
    c = prediction_func(UV).ravel()
    if hard:
        c = c > 0.5
    plt.scatter(UV[:,0], UV[:,1], c=c, edgecolors= 'none', alpha=0.15)
    plt.scatter(X[:,0], X[:,1], c=Y.ravel(), edgecolors= 'black')
    plt.xlim(left=u_min, right=u_max)
    plt.ylim(bottom=v_min, top=v_max)
    plt.axes().set_aspect('equal')
    plt.show()

plot_model_prediction(lambda x: model.forward(x), X_train, Y_train, False)

plot_model_prediction(lambda x: model.forward(x), X_train, Y_train, True)
```

### Fit on circle data

```python
X_train, Y_train, X_test, Y_test = split(*generate_2_circles(), 0.7)
```

```python
# let's train our model
model = LogsticRegressionGD(2, 0.05)

for step in range(30):
    Y_pred = model.forward(X_train)

    loss_value = nll_forward(Y_pred, Y_train)
    accuracy = ((Y_pred > 0.5) == Y_train).mean()
    print('Step: {} \t Loss: {:.3f} \t Acc: {:.1f}%'.format(step, loss_value, accuracy * 100))

    loss_grad = nll_grad_input(Y_pred, Y_train)
    model.update_params(X_train, loss_grad)

print('\n\nTesting...')
Y_test_pred = model.forward(X_test)
test_accuracy = ((Y_test_pred > 0.5) == Y_test).mean()
print('Acc: {:.1f}%'.format(test_accuracy * 100))
```

    Step: 0 	 Loss: 1.371 	 Acc: 49.3%
    Step: 1 	 Loss: 1.357 	 Acc: 49.3%
    Step: 2 	 Loss: 1.343 	 Acc: 49.3%
    Step: 3 	 Loss: 1.329 	 Acc: 49.3%
    Step: 4 	 Loss: 1.315 	 Acc: 49.3%
    Step: 5 	 Loss: 1.301 	 Acc: 49.3%
    Step: 6 	 Loss: 1.287 	 Acc: 49.3%
    Step: 7 	 Loss: 1.273 	 Acc: 49.3%
    Step: 8 	 Loss: 1.260 	 Acc: 49.3%
    Step: 9 	 Loss: 1.246 	 Acc: 49.3%
    Step: 10 	 Loss: 1.233 	 Acc: 49.3%
    Step: 11 	 Loss: 1.220 	 Acc: 49.3%
    Step: 12 	 Loss: 1.207 	 Acc: 49.3%
    Step: 13 	 Loss: 1.194 	 Acc: 49.3%
    Step: 14 	 Loss: 1.181 	 Acc: 49.3%
    Step: 15 	 Loss: 1.168 	 Acc: 49.3%
    Step: 16 	 Loss: 1.156 	 Acc: 49.3%
    Step: 17 	 Loss: 1.144 	 Acc: 50.0%
    Step: 18 	 Loss: 1.131 	 Acc: 50.0%
    Step: 19 	 Loss: 1.119 	 Acc: 50.0%
    Step: 20 	 Loss: 1.107 	 Acc: 50.0%
    Step: 21 	 Loss: 1.096 	 Acc: 50.7%
    Step: 22 	 Loss: 1.084 	 Acc: 50.7%
    Step: 23 	 Loss: 1.073 	 Acc: 50.7%
    Step: 24 	 Loss: 1.061 	 Acc: 50.0%
    Step: 25 	 Loss: 1.050 	 Acc: 50.0%
    Step: 26 	 Loss: 1.039 	 Acc: 50.0%
    Step: 27 	 Loss: 1.028 	 Acc: 50.0%
    Step: 28 	 Loss: 1.018 	 Acc: 50.0%
    Step: 29 	 Loss: 1.007 	 Acc: 50.0%

    Testing...
    Acc: 50.0%

```python
plot_model_prediction(lambda x: model.forward(x), X_train, Y_train, False)

plot_model_prediction(lambda x: model.forward(x), X_train, Y_train, True)
```

# 2. Decision Tree
The next model we look at is called **Decision Tree**. This type of model is non-parametric, meaning in contrast to **Logistic Regression** we do not have any parameters here that need to be trained.

Let us consider a simple binary decision tree for deciding on the two classes of "creditable" and "Not creditable".

Each node, except the leafs, asks a question about the the client in question. A decision is made by going from the root node to a leaf node, while considering the clients situation. The situation of the client, in this case, is fully described by the features:
1. Checking account balance
2. Duration of requested credit
3. Payment status of previous loan
4. Length of current employment

In order to build a decision tree we need training data. To carry on the previous example: we need a number of clients for which we know the properties 1.-4. and their creditability.
The process of building a decision tree starts with the root node and involves the following steps:
1. Choose a splitting criteria and add it to the current node.
2. Split the dataset at the current node into those that fullfil the criteria and those that do not.
3. Add a child node for each data split.
4. For each child node decide on either A. or B.:
    1. Repeat from 1. step
    2. Make it a leaf node: The predicted class label is decided by the majority vote over the training data in the current split.

## 2.1 Gini Index & Data Split
Deciding on how to split your training data at each node is dominated by the following two criterias:
1. Does the rule help me make a final decision?
2. Is the rule general enough such that it applies not only to my training data, but also to new unseen examples?

When considering our previous example, splitting the clients by their handedness would not help us deciding on their creditability. Knowning if a rule will generalize is usually a hard call to make, but in practice we rely on [Occam's razor](https://en.wikipedia.org/wiki/Occam%27s_razor) principle. Thus the less rules we use, the better we believe it to generalize to previously unseen examples.

One way to measure the quality of a rule is by the [**Gini Index**](https://en.wikipedia.org/wiki/Gini_coefficient).
Since we only consider binary classification, it is calculated by:
$$
Gini = \sum_{n\in\{L,R\}}\frac{|S_n|}{|S|}\left( 1 - \sum_{c \in C} p_{S_n}(c)^2\right)\\
p_{S_n}(c) = \frac{|\{\mathbf{x}_{i}\in \mathbf{X}|y_{i} = c, i \in S_n\}|}{|S_n|}, n \in \{L, R\}
$$
with $|C|=2$ being your set of class labels and $S_L$ and $S_R$ the two splits determined by the splitting
criteria.
The lower the gini score, the better the split. In the extreme case, where all class labels are the same in each split respectively, the gini index takes the value of $0$.

```python
def tree_gini_index(Y_left, Y_right, classes):
    """Compute the Gini Index.
    # Arguments
        Y_left: class labels of the data left set
            np.array of size `(n_objects, 1)`
        Y_right: class labels of the data right set
            np.array of size `(n_objects, 1)`
        classes: list of all class values
    # Output
        gini: scalar `float`
    """
    Splits = (Y_left, Y_right)
    T_len = np.sum([len(S) for S in Splits])

    gini = 0.0
    for S in Splits:
        p_sum = 0.0
        for c in classes:
            p = np.sum(np.array(S) == c) / len(S)
            p_sum += p**2
        gini += len(S) / T_len * (1 - p_sum)

    return gini
```

```python
am.test_student_function(username, tree_gini_index, ['Y_left', 'Y_right', 'classes'])
```

    Running local tests...
    tree_gini_index successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!

At each node in the tree, the data is split according to a split criterion and each split is passed onto the left/right child respectively.
Implement the following function to return all rows in `X` and `Y` such that the left child gets all examples that are less than the split value and vice versa.

```python
def tree_split_data_left(X, Y, feature_index, split_value):
    """Split the data `X` and `Y`, at the feature indexed by `feature_index`.
    If the value is less than `split_value` then return it as part of the left group.

    # Arguments
        X: np.array of size `(n_objects, n_in)`
        Y: np.array of size `(n_objects, 1)`
        feature_index: index of the feature to split at
        split_value: value to split between
    # Output
        (XY_left): np.array of size `(n_objects_left, n_in + 1)`
    """
    assert(feature_index <= np.shape(X)[1])
    assert(feature_index >= 0)

    Z = np.append(X,Y, axis=1)
    return Z[Z[:,feature_index] < split_value]

def tree_split_data_right(X, Y, feature_index, split_value):
    """Split the data `X` and `Y`, at the feature indexed by `feature_index`.
    If the value is greater or equal than `split_value` then return it as part of the right group.

    # Arguments
        X: np.array of size `(n_objects, n_in)`
        Y: np.array of size `(n_objects, 1)`
        feature_index: index of the feature to split at
        split_value: value to split between
    # Output
        (XY_left): np.array of size `(n_objects_left, n_in + 1)`
    """
    assert(feature_index <= np.shape(X)[1])
    assert(feature_index >= 0)

    Z = np.append(X,Y, axis=1)
    return Z[Z[:,feature_index] >= split_value]
```

```python
X = np.array([[ 4.44443155,  9.84755411],
             [ 0.87669203,  0.23400127],
             [ 6.93641601, -3.14280346],
             [ 4.54863768,  5.0348375 ],
             [-1.14587177, -2.46124326],
             [-6.47480422, -4.74631328],
             [-3.16880904,  3.71041515],
             [ 4.74583194, -1.22801333],
             [-6.37401965,  8.73366778]])

Y = [[1.], [0.], [0.], [1.], [0.], [0.], [1.], [0.], [0.]]
feature_index = 1
split_value = -1.1136209970632827

tree_split_data_left(X, Y, feature_index, split_value)
```

    [[ 6.93641601 -3.14280346  0.        ]
     [-1.14587177 -2.46124326  0.        ]
     [-6.47480422 -4.74631328  0.        ]
     [ 4.74583194 -1.22801333  0.        ]]

```python
call30(am.test_student_function, (username, tree_split_data_left, ['X', 'Y', 'feature_index', 'split_value']))
```

    Running local tests...
    tree_split_data_left successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!

```python
call30(am.test_student_function, (username, tree_split_data_right, ['X', 'Y', 'feature_index', 'split_value']))
```

    Running local tests...
    tree_split_data_right successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!

```python
am.get_progress(username)
```

    ---------------------------------------------
    | Harm Manders                              |
    | harmmanders@gmail.com                     |
    ---------------------------------------------
    | W2: linear_forward       | completed      |
    | W2: linear_grad_W        | completed      |
    | W2: linear_grad_b        | completed      |
    | W2: nll_forward          | completed      |
    | W2: nll_grad_input       | completed      |
    | W2: sigmoid_forward      | completed      |
    | W2: sigmoid_grad_input   | completed      |
    | W2: tree_gini_index      | completed      |
    | W2: tree_split_data_left | completed      |
    | W2: tree_split_data_right| completed      |
    | W2: tree_to_terminal     | not attempted  |
    | W3: box_blur             | not attempted  |
    | W3: conv_matrix          | not attempted  |
    | W3: dense_forward        | not attempted  |
    | W3: dense_grad_W         | not attempted  |
    | W3: dense_grad_b         | not attempted  |
    | W3: dense_grad_input     | not attempted  |
    | W3: flatten_forward      | not attempted  |
    | W3: flatten_grad_input   | not attempted  |
    | W3: l2_regularizer       | not attempted  |
    | W3: maxpool_forward      | not attempted  |
    | W3: relu_forward         | not attempted  |
    | W3: relu_grad_input      | not attempted  |
    ---------------------------------------------

Now to find the split rule with the lowest gini score, we brute-force search over all features and values to split by.

```python
def tree_best_split(X, Y):
    class_values = list(set(Y.flatten().tolist()))
    r_index, r_value, r_score =  float("inf"),  float("inf"), float("inf")
    r_XY_left, r_XY_right = (X,Y), (X,Y)
    for feature_index in range(X.shape[1]):
        for row in X:
            XY_left = tree_split_data_left(X, Y, feature_index, row[feature_index])
            XY_right = tree_split_data_right(X, Y, feature_index, row[feature_index])
            XY_left, XY_right = (XY_left[:,:-1], XY_left[:,-1:]), (XY_right[:,:-1], XY_right[:,-1:])
            gini = tree_gini_index(XY_left[1], XY_right[1], class_values)
            if gini < r_score:
                r_index, r_value, r_score = feature_index, row[feature_index], gini
                r_XY_left, r_XY_right = XY_left, XY_right
    return {'index':r_index, 'value':r_value, 'XY_left': r_XY_left, 'XY_right':r_XY_right}
```

## 2.2 Terminal Node
The leaf nodes predict the label of an unseen example, by taking a majority vote over all training class labels in that node.

```python
def tree_to_terminal(Y):
    """The most frequent class label, out of the data points belonging to the leaf node,
    is selected as the predicted class.

    # Arguments
        Y: np.array of size `(n_objects)`

    # Output
        label: most frequent label of `Y.dtype`
    """
    counts = np.asarray(np.unique(Y, return_counts=True))
    return counts[0, np.argmax(counts[1])]

```

```python
call30(am.test_student_function, (username, tree_to_terminal, ['Y']))
```

    Running local tests...
    tree_to_terminal successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!

```python
am.get_progress(username)
```

    ---------------------------------------------
    | Harm Manders                              |
    | harmmanders@gmail.com                     |
    ---------------------------------------------
    | W2: linear_forward       | completed      |
    | W2: linear_grad_W        | completed      |
    | W2: linear_grad_b        | completed      |
    | W2: nll_forward          | completed      |
    | W2: nll_grad_input       | completed      |
    | W2: sigmoid_forward      | completed      |
    | W2: sigmoid_grad_input   | completed      |
    | W2: tree_gini_index      | completed      |
    | W2: tree_split_data_left | completed      |
    | W2: tree_split_data_right| completed      |
    | W2: tree_to_terminal     | completed      |
    | W3: box_blur             | not attempted  |
    | W3: conv_matrix          | not attempted  |
    | W3: dense_forward        | not attempted  |
    | W3: dense_grad_W         | not attempted  |
    | W3: dense_grad_b         | not attempted  |
    | W3: dense_grad_input     | not attempted  |
    | W3: flatten_forward      | not attempted  |
    | W3: flatten_grad_input   | not attempted  |
    | W3: l2_regularizer       | not attempted  |
    | W3: maxpool_forward      | not attempted  |
    | W3: relu_forward         | not attempted  |
    | W3: relu_grad_input      | not attempted  |
    ---------------------------------------------

## 2.3 Build the Decision Tree
Now we recursively build the decision tree, by greedily splitting the data at each node according to the gini index.
To prevent the model from overfitting, we transform a node into a terminal/leaf node, if:
1. a maximum depth is reached.
2. the node does not reach a minimum number of training samples.

```python
def tree_recursive_split(X, Y, node, max_depth, min_size, depth):
    XY_left, XY_right = node['XY_left'], node['XY_right']
    del(node['XY_left'])
    del(node['XY_right'])
    # check for a no split
    if XY_left[0].size <= 0 or XY_right[0].size <= 0:
        node['left_child'] = node['right_child'] = tree_to_terminal(np.concatenate((XY_left[1], XY_right[1])))
        return
    # check for max depth
    if depth >= max_depth:
        node['left_child'], node['right_child'] = tree_to_terminal(XY_left[1]), tree_to_terminal(XY_right[1])
        return
    # process left child
    if XY_left[0].shape[0] <= min_size:
        node['left_child'] = tree_to_terminal(XY_left[1])
    else:
        node['left_child'] = tree_best_split(*XY_left)
        tree_recursive_split(X, Y, node['left_child'], max_depth, min_size, depth+1)
    # process right child
    if XY_right[0].shape[0] <= min_size:
        node['right_child'] = tree_to_terminal(XY_right[1])
    else:
        node['right_child'] = tree_best_split(*XY_right)
        tree_recursive_split(X, Y, node['right_child'], max_depth, min_size, depth+1)

def build_tree(X, Y, max_depth, min_size):
    root = tree_best_split(X, Y)
    tree_recursive_split(X, Y, root, max_depth, min_size, 1)
    return root
```

By printing the split criteria or the predicted class at each node, we can visualise the decising making process.
Both the tree and a a prediction can be implemented recursively, by going from the root to a leaf node.

```python
def print_tree(node, depth=0):
    if isinstance(node, dict):
        print('%s[X%d < %.3f]' % ((depth*' ', (node['index']+1), node['value'])))
        print_tree(node['left_child'], depth+1)
        print_tree(node['right_child'], depth+1)
    else:
        print('%s[%s]' % ((depth*' ', node)))

def tree_predict_single(x, node):
    if isinstance(node, dict):
        if x[node['index']] < node['value']:
            return tree_predict_single(x, node['left_child'])
        else:
            return tree_predict_single(x, node['right_child'])

    return node

def tree_predict_multi(X, node):
    Y = np.array([tree_predict_single(row, node) for row in X])
    return Y[:, None]  # size: (n_object,) -> (n_object, 1)
```

Let's test our decision tree model on some toy data.

```python
X_train, Y_train, X_test, Y_test = split(*generate_2_circles(), 0.7)

tree = build_tree(X_train, Y_train, 4, 1)
Y_pred = tree_predict_multi(X_test, tree)
test_accuracy = (Y_pred == Y_test).mean()
print('Test Acc: {:.1f}%'.format(test_accuracy * 100))
```

    Test Acc: 100.0%

    /home/harm/.local/lib/python3.6/site-packages/ipykernel_launcher.py:20: RuntimeWarning: invalid value encountered in long_scalars

We print the decision tree in [pre-order](https://en.wikipedia.org/wiki/Tree_traversal#Pre-order_(NLR)).

```python
print_tree(tree)
```

    [X2 < -1.099]
     [X1 < 0.474]
      [X1 < -2.598]
       [0.0]
       [X1 < -1.203]
        [0.0]
        [0.0]
      [X1 < 1.375]
       [X1 < 1.203]
        [0.0]
        [0.0]
       [X1 < 2.688]
        [0.0]
        [0.0]
     [X2 < 1.246]
      [X1 < 2.802]
       [X1 < -1.100]
        [0.0]
        [1.0]
       [X1 < 2.936]
        [0.0]
        [0.0]
      [X1 < -2.444]
       [X1 < -2.549]
        [0.0]
        [0.0]
       [X1 < -1.115]
        [0.0]
        [0.0]

```python
plot_model_prediction(lambda x: tree_predict_multi(x, tree), X_test, Y_test)
```

# 3. Experiments
The [Cleveland Heart Disease](https://archive.ics.uci.edu/ml/datasets/Heart+Disease) dataset aims at predicting the presence of heart disease based on other available medical information of the patient.

Although the whole database contains 76 attributes, we focus on the following 14:
1. Age: age in years
2. Sex:
    * 0 = female
    * 1 = male
3. Chest pain type:
    * 1 = typical angina
    * 2 = atypical angina
    * 3 = non-anginal pain
    * 4 = asymptomatic
4. Trestbps: resting blood pressure in mm Hg on admission to the hospital
5. Chol: serum cholestoral in mg/dl
6. Fasting blood sugar: > 120 mg/dl
    * 0 = false
    * 1 = true
7. Resting electrocardiographic results:
    * 0 = normal
    * 1 = having ST-T wave abnormality (T wave inversions and/or ST elevation or depression of > 0.05 mV)
    * 2 = showing probable or definite left ventricular hypertrophy by Estes' criteria
8. Thalach: maximum heart rate achieved
9. Exercise induced angina:
    * 0 = no
    * 1 = yes
10. Oldpeak: ST depression induced by exercise relative to rest
11. Slope: the slope of the peak exercise ST segment
    * 1 = upsloping
    * 2 = flat
    * 3 = downsloping
12. Ca: number of major vessels (0-3) colored by flourosopy
13. Thal:
    * 3 = normal
    * 6 = fixed defect
    * 7 = reversable defect
14. Target: diagnosis of heart disease (angiographic disease status)
    * 0 = < 50% diameter narrowing
    * 1 = > 50% diameter narrowing

The 14. attribute is the target variable that we would like to predict based on the rest.

We have prepared some helper functions to download and pre-process the data in `heart_disease_data.py`

```python
import heart_disease_data
```

```python
X, Y = heart_disease_data.download_and_preprocess()
X_train, Y_train, X_test, Y_test = split(X, Y, 0.7)
```

Let's have a look at some examples

```python
print(X_train[0:2])
print(Y_train[0:2])

# TODO feel free to explore more examples and see if you can predict the presence of a heart disease
```

    [[ 71.    0.    4.  112.  149.    0.    0.  125.    0.    1.6   2.    0.
        3. ]
     [ 60.    1.    4.  130.  206.    0.    2.  132.    1.    2.4   2.    2.
        7. ]]
    [[0.]
     [1.]]

## 3.1 Decision Tree for Heart Disease Prediction
Let's build a decision tree model on the training data and see how well it performs

```python
# TODO: you are free to make use of code that we provide in previous cells
# TODO: play around with different hyper parameters and see how these impact your performance

tree = build_tree(X_train, Y_train, 5, 4)
Y_pred = tree_predict_multi(X_test, tree)
test_accuracy = (Y_pred == Y_test).mean()
print('Test Acc: {:.1f}%'.format(test_accuracy * 100))
```

    /home/harm/.local/lib/python3.6/site-packages/ipykernel_launcher.py:20: RuntimeWarning: invalid value encountered in long_scalars

    Test Acc: 81.1%

How did changing the hyper parameters affect the test performance? Usually hyper parameters are tuned using a hold-out [validation set](https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets#Validation_dataset) instead of the test set.

## 3.2 Logistic Regression for Heart Disease Prediction

Instead of manually going through the data to find possible correlations, let's try training a logistic regression model on the data.

```python
# TODO: you are free to make use of code that we provide in previous cells
# TODO: play around with different hyper parameters and see how these impact your performance
```

How well did your model perform? Was it actually better then guessing? Let's look at the empirical mean of the target.

```python
Y_train.mean()
```

So what is the problem? Let's have a look at the learned parameters of our model.

```python
print(model.W, model.b)
```

If you trained sufficiently many steps you'll probably see how some weights are much larger than others. Have a look at what range the parameters were initialized and how much change we allow per step (learning rate). Compare this to the scale of the input features. Here an important concept arises, when we want to train on real world data:
[Feature Scaling](https://en.wikipedia.org/wiki/Feature_scaling).

Let's try applying it on our data and see how it affects our performance.

```python
# TODO: Rescale the input features and train again
```

Notice that we did not need any rescaling for the decision tree. Can you think of why?


```python
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()
```

# The frequency of a Ricker wavelet

We often use Ricker wavelets to model seismic, for example when making a synthetic seismogram with which to help tie a well. One simple way to guesstimate the peak or central frequency of the wavelet that will model a particlar seismic section is to count the peaks per unit time in the seismic. But this tends to overestimate the actual frequency because the maximum [frequency](http://www.subsurfwiki.org/wiki/Frequency) of [a Ricker wavelet](http://subsurfwiki.org/wiki/Ricker_wavelet) is more than the peak frequency. The question is, how much more?

To investigate, let's make a Ricker wavelet and see what it looks like in the time and frequency domains.

```python
T, dt, f = 0.256, 0.001, 25

import bruges
w, t = bruges.filters.ricker(T, dt, f, return_t=True)

import scipy.signal
f_W, W = scipy.signal.welch(w, fs=1/dt, nperseg=256)
```

```python
fig, axs = plt.subplots(figsize=(15,5), ncols=2)
axs[0].plot(t, w)
axs[0].set_xlabel("Time [s]")
axs[1].plot(f_W[:25], W[:25], c="C1")
axs[1].set_xlabel("Frequency [Hz]")
plt.show()
```

When we count the peaks in a section, the assumption is that this apparent frequency — that is, the reciprocal of apparent period or distance between the extrema — tells us the dominant or peak frequency.

To help see why this assumption is wrong, let's compare the Ricker with a signal whose apparent frequency does match its peak frequency: a pure cosine:

```python
c = np.cos(2*25*np.pi*t)

f_C, C = scipy.signal.welch(c, fs=1/dt, nperseg=256)
```

```python
fig, axs = plt.subplots(figsize=(15,5), ncols=2)
axs[0].plot(t, c, c="C2")
axs[0].set_xlabel("Time [s]")
axs[1].plot(f_C[:25], C[:25], c="C1")
axs[1].set_xlabel("Frequency [Hz]")
plt.show()
```

Notice that the signal is much narrower in bandwidth. If we allowed more oscillations, it would be even narrower. If it lasted forever, it would be a spike in the frequency domain.

Let's overlay the signals to get a picture of the difference in the relative periods:

```python
plt.figure(figsize=(15, 5))
plt.plot(t, c, c='C2')
plt.plot(t, w)
plt.xlabel("Time [s]")
plt.show()
```

The practical consequence of this is that if we estimate the peak frequency to be $f\ \mathrm{Hz}$, then we need to reduce $f$ by some factor if we want to design a wavelet to match the data. To get this factor, we need to know the apparent period of the Ricker function, as given by the time difference between the two minima.

Let's look at a couple of different ways to find those minima: numerically and analytically.

## Find minima numerically

We'll use [`scipy.optimize.minimize`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html#scipy.optimize.minimize) to find a numerical solution. In order to use it, we'll need a slightly different expression for the Ricker function — casting it in terms of a time basis `t`. We'll also keep `f` as a variable, rather than hard-coding it in the expression, to give us the flexibility of computing the minima for different values of `f`.

Here's the equation we're implementing:

$$w(t, f) = (1 - 2\pi^2 f^2 t^2)\ e^{-\pi^2 f^2 t^2}$$

```python
def ricker(t, f):
    return (1 - 2*(np.pi*f*t)**2) * np.exp(-(np.pi*f*t)**2)
```

Check that the wavelet looks like it did before, by comparing the output of this function when `f` is 25 with the wavelet `w` we were using before:

```python
f = 25
np.allclose(w, ricker(t, f=25))
```

    True

```python
plt.figure(figsize=(15, 5))
plt.plot(w, lw=3)
plt.plot(ricker(t, f), '--', c='C4', lw=3)
plt.show()
```

Now we call SciPy's `minimize` function on our `ricker` function. It itertively searches for a minimum solution, then gives us the `x` (which is really `t` in our case) at that minimum:

```python
import scipy.optimize

f = 25

scipy.optimize.minimize(ricker, x0=0, args=(f))
```

          fun: -0.4462603202963996
     hess_inv: array([[1]])
          jac: array([-2.19792128e-07])
      message: 'Optimization terminated successfully.'
         nfev: 30
          nit: 1
         njev: 10
       status: 0
      success: True
            x: array([0.01559393])

So the minimum amplitude, given by `fun`, is $-0.44626$ and it occurs at an `x` (time) of $\pm 0.01559\ \mathrm{s}$.

In comparison, the minima of the cosine function occur at a time of $\pm 0.02\ \mathrm{s}$. In other words, the period appears to be $0.02 - 0.01559 = 0.00441\ \mathrm{s}$ shorter than the pure waveform, which is...

```python
(0.02 - 0.01559) / 0.02
```

    0.22050000000000003

...about 22% shorter. This means that if we naively estimate frequency by counting peaks or zero crossings, we'll tend to overestimate the peak frequency of the wavelet by about 22% — assuming it is approximately Ricker-like; if it isn't we can use the same method to estimate the error for other functions.

This is good to know, but it would be interesting to know if this parameter depends on frequency, and also to have a more precise way to describe it than a decimal. To get at these questions, we need an analytic solution.

## Find minima analytically

Python's [SymPy package](http://sympy.org/) is a bit like Maple — it understands math symbolically. We'll use [`sympy.solve`](http://docs.sympy.org/latest/modules/solvers/solvers.html) to find an analytic solution. It turns out that it needs the Ricker function writing in yet another way, using SymPy symbols and expressions for $\mathrm{e}$ and $\pi$.

```python
import sympy as sp

t = sp.Symbol('t')
f = sp.Symbol('f')

r = (1 - 2*(sp.pi*f*t)**2) * sp.exp(-(sp.pi*f*t)**2)
```

Now we can easily find the solutions to the Ricker equation, that is, the times at which the function is equal to zero:

```python
sp.solvers.solve(r, t)
```

    [-sqrt(2)/(2*pi*f), sqrt(2)/(2*pi*f)]

But this is not quite what we want. We need the minima, not the zero-crossings.

Maybe there's a better way to do this, but here's one way. Note that the gradient (slope or derivative) of the Ricker function is zero at the minima, so let's just solve the first time derivative of the Ricker function. That will give us the three times at which the function has a gradient of zero.

```python
dwdt = sp.diff(r, t)
sp.solvers.solve(dwdt, t)
```

    [0, -sqrt(6)/(2*pi*f), sqrt(6)/(2*pi*f)]

In other words, the non-zero minima of the Ricker function are at:

$$\pm \frac{\sqrt{6}}{2\pi f}$$

Let's just check that this evaluates to the same answer we got from `scipy.optimize`, which was 0.01559.

```python
np.sqrt(6) / (2 * np.pi * 25)
```

    0.015593936024673521

The solutions agree.

While we're looking at this, we can also compute the analytic solution to the amplitude of the minima, which SciPy calculated as -0.446. We just substitute one of the expressions for the minimum time into the expression for `r`:

```python
r.subs({t: sp.sqrt(6)/(2*sp.pi*f)})
```

    -2*exp(-3/2)

## Apparent frequency

So what's the result of all this? What's the correction we need to make?

The minima of the Ricker wavelet are $\sqrt{6}\ /\ \pi f_\mathrm{actual}\ \mathrm{s}$ apart — this is the apparent period. If we're assuming a pure tone, this period corresponds to an apparent frequency of $\pi f_\mathrm{actual}\ /\ \sqrt{6}\ \mathrm{Hz}$. For $f = 25\ \mathrm{Hz}$, this apparent frequency is:

```python
(np.pi * 25) / np.sqrt(6)
```

    32.06374575404661

If we were to try to model the data with a Ricker of 32 Hz, the frequency will be too high. We need to multiply the frequency by a factor of $\sqrt{6} / \pi$, like so:

```python
32.064 * np.sqrt(6) / (np.pi)
```

    25.00019823475659

This gives the correct frequency of 25 Hz.

To sum up, rearranging the expression above:

$$f_\mathrm{actual} = f_\mathrm{apparent} \frac{\sqrt{6}}{\pi}$$

Expressed as a decimal, the factor we were seeking is therefore $\sqrt{6}\ /\ \pi$:

```python
np.sqrt(6) / np.pi
```

    0.779696801233676

That is, the reduction factor is 22%.

----

Curious coincidence: in [the recent Pi Day post](https://agilescientific.com/blog/2018/3/14/happy-pi-day-einstein), I mentioned the Riemann zeta function of 2 as a way to compute pi. It evaluates to $(\pi / \sqrt{6})^2$. Is there a connection between the Ricker wavelet and the Riemann hypothesis?

I doubt it.


# Introduction to `numpy`, `sympy` and `matplotlib`

Kevin J. Walchko
created 5 July 2017

---

Matplotlib, NumPy and pysym are open-source add-on modules to Python that provide common mathematical and numerical routines in pre-compiled, fast functions. These are growing into highly mature packages that provide functionality that meets, or perhaps exceeds, that associated with common commercial software like MatLab. There is little that Matlab can do that python, numpy and other modules can't do too.

Matplotlib is a plotting library modeled after Matlab's plotting commands. We will use `matplotlib` to produce some graphs, but really use it heavily when we get to image processing. Again, we will delay the image specific functions until we get to that point. Here we are just going to learn the basics.

## Objectives

- Understand how to do very basic linear algebra in python
- Understand how to do symbolic manipulation in python
- Understand how to use matplotlib
   - How to insert LaTeX math symbols into plots
- Understand how to plot engineering data in a Jupyter notebook

## References

- [numpy tutorial](https://docs.scipy.org/doc/numpy-dev/user/quickstart.html)
- [numpy for matlab users](https://docs.scipy.org/doc/numpy-dev/user/numpy-for-matlab-users.html)
- [sympy](http://www.sympy.org/en/index.html)
- [sympy tutorials](http://docs.sympy.org/latest/tutorial/index.html)
- [matplotlib examples](https://matplotlib.org/users/pyplot_tutorial.html)
- [matplotlib summary](https://matplotlib.org/api/pyplot_summary.html)
- [matplotlib gallery of plots](https://matplotlib.org/gallery.html)
- [LaTEX math symbols](http://web.ift.uib.no/Teori/KURS/WRK/TeX/symALL.html)

## Setup

```python
%matplotlib inline
from __future__ import division
from __future__ import print_function
import numpy as np  # common to rename numpy to np ... because programmers are lazy :)
from matplotlib import pyplot as plt
```

# How Does the Jupyter Notebook Work?

Well, I probably should have covered this last lesson, but there already was a lot of material in that lesson. This one is shorter. When you run the command `jupyter notebook` from a bash terminal, it lauches a webserver and a web browser that shows the contents of the current directory. When you click on a jupyter file, `lsn4.ipynb`, the webserver reads the file and launches a kernel to run the notebook. Kernels are processes that run interactive code in a particular programming language and return output to the user.

Jupyter supports many different languages, we are only using python. However, there are a [bunch of kernels](https://github.com/jupyter/jupyter/wiki/Jupyter-kernels) for other languages you can use if you want, like: Go, Ruby, Julia, R, etc.

## How To Shut Things Down

Since you are launching both a web server and browser, you need to close the browser and shutdown the server. The server can be shutdown in one of two ways.

1. In the bash window, press contro-C
1. Or, just close the bash window

Either of these will work.

# Numpy

First let's do some simple matrix/vector operations, since we will need this for our serial manipulator work. We will also use `numpy` later for OpenCV imagery work, but we will push that off to later when we need it. There are some specific things we need  to understand when working with images that would be best covered right before we need it.

```python
# lets create an array
a = np.array([1,2,3])  # you could call this a 3 dimensional vector if you want
print(a)
```

    [1 2 3]

```python
# let's create a matrix with rows: [1,2,3], [4,5,6], and [6,7,8]
m = np.array([
    [1,2,3],
    [4,5,6],
    [6,7,8]
])
print('m =', m)
print('size', m.size)
print('shape', m.shape)
print('type', m.dtype)
```

    m = [[1 2 3]
     [4 5 6]
     [6 7 8]]
    size 9
    shape (3L, 3L)
    type int32

Now, generally we don't think about data types with python, but for math (and later working with images in OpenCV) we will. *Note:* you will probably have to prefix these with a namespace like `np`: `np.uint8`. This assumes you did the `import numpy as np`.

| Data type	| Description |
|---|:---|
| bool_	| Boolean (True or False) stored as a byte
| int_	| Default integer type (same as C long; normally either int64 or int32)
| intc	| Identical to C int (normally int32 or int64)
| intp	| Integer used for indexing (same as C ssize_t; normally either int32 or int64)
| int8	| Byte (-128 to 127)
| int16	| Integer (-32768 to 32767)
| int32	| Integer (-2147483648 to 2147483647)
| int64	| Integer (-9223372036854775808 to 9223372036854775807)
| uint8	| Unsigned integer (0 to 255)
| uint16 | Unsigned integer (0 to 65535)
| uint32 | Unsigned integer (0 to 4294967295)
| uint64 | Unsigned integer (0 to 18446744073709551615)
| float_ | Shorthand for float64.
| float16 | Half precision float: sign bit, 5 bits exponent, 10 bits mantissa
| float32 | Single precision float: sign bit, 8 bits exponent, 23 bits mantissa
| float64 | Double precision float: sign bit, 11 bits exponent, 52 bits mantissa
| complex_ | Shorthand for complex128.
| complex64	| Complex number, represented by two 32-bit floats (real and imaginary components)
| complex128 | Complex number, represented by two 64-bit floats (real and imaginary components)

Honestely, I generally only use `np.float` for math or `np.unit8` for OpenCV. However, if you are calculating 3-phase power from ENGR 311, you *might* need `np.complex`, but I am not an AC power guy.

```python
# there are functions for special matrices
# zeros matrix
z = np.zeros((3,3))
print(z)

# Identity matrix
o = np.eye(3)
print(o)
```

    [[0. 0. 0.]
     [0. 0. 0.]
     [0. 0. 0.]]
    [[1. 0. 0.]
     [0. 1. 0.]
     [0. 0. 1.]]

```python
# numpy even works in multiple dimension beyond standard MxN matrices we will do for this class.
# numpy actually lets you do n dimensional tensor math, and if you are Einstein, that is awesome!
np.zeros((3,4,5)) # go on and add some more dimentions like: (3,3,3,3)
```

    array([[[0., 0., 0., 0., 0.],
            [0., 0., 0., 0., 0.],
            [0., 0., 0., 0., 0.],
            [0., 0., 0., 0., 0.]],

           [[0., 0., 0., 0., 0.],
            [0., 0., 0., 0., 0.],
            [0., 0., 0., 0., 0.],
            [0., 0., 0., 0., 0.]],

           [[0., 0., 0., 0., 0.],
            [0., 0., 0., 0., 0.],
            [0., 0., 0., 0., 0.],
            [0., 0., 0., 0., 0.]]])

```python
# let's multiply them together
# NOTE: you can NOT do v = M * a
a = np.array([1,2,3])
v = m.dot(a)
print('m * a = m.dot(a) =', v)
```

    m * a = m.dot(a) = [14 32 44]

```python
# let's remember what v and m are
print('v = ', v, '\n')
print('m = ', m, '\n')

# let's access the number of each ... all zero based arrays
print('first v:', v[0])   # this is a vector (or array) so only 1 number given to get the value
print('first m:', m[0,0]) # this is a matrix, so you need 2 numbers to get the value

# now let's do the last number
print('last v:', v[2])
print('last m:', m[2,2])

# remember: m[row, column]
print('\nm[0,1]', m[0,1])
print('m[2]', m[2], ' this gives me the entire row')

# let's get their dimensions
print('\nlen(a):', len(v))
print('v.shape:', v.shape)
print('m.shape:', m.shape)
```

    v =  [14 32 44]

    m =  [[1 2 3]
     [4 5 6]
     [6 7 8]]

    first v: 14
    first m: 1
    last v: 44
    last m: 8

    m[0,1] 2
    m[2] [6 7 8]  this gives me the entire row

    len(a): 3
    v.shape: (3L,)
    m.shape: (3L, 3L)

**Remeber for later:** OpenCV uses numpy to represent images. We access things by Matrix[row, col]. During image processing we have the same matrix interface, but we access a pixel by Image[height, width]. Here, height and row are the same while column and width are the same. This is backwards from how we usually think about images or screen resolutions, width x height like: 640 x 480.

```python
n = np.array([
    [1,0,0],
    [0,2,0],
    [0,0,3]
])

# inverse
nn = np.linalg.inv(n)
print('nn:\n', nn)

# calculate the determinate
nn = np.linalg.det(n)
print('det(n) =',nn)

# calculate the norm
nn = np.linalg.norm(n)
print('norm(n) =', nn)
```

    nn:
     [[1.         0.         0.        ]
     [0.         0.5        0.        ]
     [0.         0.         0.33333333]]
    det(n) = 6.0
    norm(n) = 3.7416573867739413

```python
# too much typing ... let's shorten it
# just becareful not to alias another python function ... python for example
# already has a built in abs (absolute) function, but so numpy
from numpy.linalg import inv, det, norm

# inverse
nn = inv(n)
print('nn', nn)

# calculate the determinate
nn = det(n)
print('det(n)', nn)

# calculate the norm
nn = norm(n)
print('norm(n)', nn)
```

    nn [[1.         0.         0.        ]
     [0.         0.5        0.        ]
     [0.         0.         0.33333333]]
    det(n) 6.0
    norm(n) 3.7416573867739413

```python
# let's iterate over a matrix and print each value out
# remember, m.shape returns (rows, cols) tuple
m = np.array([
    [1,2,3],
    [4,5,6],
    [6,7,8]
])

# remember, shape gives rows, cols
mm, nn = m.shape
for i in range(mm):
    for j in range(nn):
        print('m[{}, {}] = {}'.format(i,j,m[i,j]))

print('') # add a blank line between the 2 prints
print(m)
```

    m[0, 0] = 1
    m[0, 1] = 2
    m[0, 2] = 3
    m[1, 0] = 4
    m[1, 1] = 5
    m[1, 2] = 6
    m[2, 0] = 6
    m[2, 1] = 7
    m[2, 2] = 8

    [[1 2 3]
     [4 5 6]
     [6 7 8]]

```python
# we can do it another way
# also, enumerate is a nice way to get a counter
for i, r in enumerate(m):
    for j, c in enumerate(r):
        print('m[{}, {}] = {}'.format(i,j,c))
```

    m[0, 0] = 1
    m[0, 1] = 2
    m[0, 2] = 3
    m[1, 0] = 4
    m[1, 1] = 5
    m[1, 2] = 6
    m[2, 0] = 6
    m[2, 1] = 7
    m[2, 2] = 8

## Documentation

If you don't understand what a function does, Google it or (like all python libraries) use the built in help system: `help(numpy.inv)`.

You can also find out what is availble in a  package by: `dir(numpy)`. This would list everything in `numpy`. Or if you wanted to know what was in the linear algebra sub-package: `dir(numpy.linalg)`.

# Sympy

You can use sympy to manipulate equations symbolically. This will be useful later when we do forward/inverse kinematics.

1. get library and functions you need: from sympy import symbols, simplify
   1. you may also need other things like trig functions: from sympy import sin, cos, pi
1. define your symbols()
1. build your equations
1. then run simplify()
1. finally substitute in values for variables if needed

```python
from sympy import symbols, sin, cos, pi, simplify
```

```python
# create some symbols and a symbolic equation
a, b = symbols('a b')
eqn = sin(a)*sin(a)+cos(a)**2+sin(b)/cos(b) + a
print(eqn)
```

    a + sin(a)**2 + sin(b)/cos(b) + cos(a)**2

```python
# simplify the equations
# sin^2 + cos^2 = 1
# sin/cos = tan
e = simplify(eqn)
print(e)
```

    a + tan(b) + 1

```python
# e is a numpy object, uncomment and see all of the amazing function (methods in CompSci lingo) that are associated with it
dir(e)
```

    ['__abs__',
     '__add__',
     '__class__',
     '__complex__',
     '__delattr__',
     '__div__',
     '__doc__',
     '__eq__',
     '__float__',
     '__floordiv__',
     '__format__',
     '__ge__',
     '__getattribute__',
     '__getnewargs__',
     '__getstate__',
     '__gt__',
     '__hash__',
     '__init__',
     '__int__',
     '__le__',
     '__long__',
     '__lt__',
     '__mod__',
     '__module__',
     '__mul__',
     '__ne__',
     '__neg__',
     '__new__',
     '__pos__',
     '__pow__',
     '__radd__',
     '__rdiv__',
     '__reduce__',
     '__reduce_ex__',
     '__repr__',
     '__rfloordiv__',
     '__rmod__',
     '__rmul__',
     '__rpow__',
     '__rsub__',
     '__rtruediv__',
     '__setattr__',
     '__setstate__',
     '__sizeof__',
     '__slots__',
     '__str__',
     '__sub__',
     '__subclasshook__',
     '__truediv__',
     '_args',
     '_assumptions',
     '_combine_inverse',
     '_compare_pretty',
     '_constructor_postprocessor_mapping',
     '_diff_wrt',
     '_eval_adjoint',
     '_eval_as_leading_term',
     '_eval_conjugate',
     '_eval_derivative',
     '_eval_difference_delta',
     '_eval_evalf',
     '_eval_expand_complex',
     '_eval_interval',
     '_eval_is_algebraic',
     '_eval_is_algebraic_expr',
     '_eval_is_antihermitian',
     '_eval_is_commutative',
     '_eval_is_complex',
     '_eval_is_finite',
     '_eval_is_hermitian',
     '_eval_is_imaginary',
     '_eval_is_integer',
     '_eval_is_irrational',
     '_eval_is_negative',
     '_eval_is_nonnegative',
     '_eval_is_nonpositive',
     '_eval_is_odd',
     '_eval_is_polynomial',
     '_eval_is_positive',
     '_eval_is_rational',
     '_eval_is_rational_function',
     '_eval_is_real',
     '_eval_is_zero',
     '_eval_lseries',
     '_eval_nseries',
     '_eval_power',
     '_eval_rewrite',
     '_eval_subs',
     '_eval_transpose',
     '_evalf',
     '_exec_constructor_postprocessors',
     '_expand_hint',
     '_explicit_class_assumptions',
     '_from_args',
     '_from_mpmath',
     '_has',
     '_has_matcher',
     '_hashable_content',
     '_matches_commutative',
     '_matches_simple',
     '_mhash',
     '_mpc_',
     '_new_rawargs',
     '_op_priority',
     '_parse_order',
     '_prop_handler',
     '_random',
     '_recursive_call',
     '_sage_',
     '_sorted_args',
     '_subs',
     '_to_mpmath',
     '_xreplace',
     'adjoint',
     'apart',
     'args',
     'args_cnc',
     'as_base_exp',
     'as_coeff_Add',
     'as_coeff_Mul',
     'as_coeff_add',
     'as_coeff_exponent',
     'as_coeff_mul',
     'as_coefficient',
     'as_coefficients_dict',
     'as_content_primitive',
     'as_expr',
     'as_independent',
     'as_leading_term',
     'as_numer_denom',
     'as_ordered_factors',
     'as_ordered_terms',
     'as_poly',
     'as_powers_dict',
     'as_real_imag',
     'as_terms',
     'as_two_terms',
     'assumptions0',
     'atoms',
     'cancel',
     'canonical_variables',
     'class_key',
     'coeff',
     'collect',
     'combsimp',
     'compare',
     'compute_leading_term',
     'conjugate',
     'copy',
     'could_extract_minus_sign',
     'count',
     'count_ops',
     'default_assumptions',
     'diff',
     'doit',
     'dummy_eq',
     'equals',
     'evalf',
     'expand',
     'extract_additively',
     'extract_branch_factor',
     'extract_leading_order',
     'extract_multiplicatively',
     'factor',
     'find',
     'flatten',
     'fourier_series',
     'fps',
     'free_symbols',
     'fromiter',
     'func',
     'getO',
     'getn',
     'has',
     'identity',
     'integrate',
     'invert',
     'is_Add',
     'is_AlgebraicNumber',
     'is_Atom',
     'is_Boolean',
     'is_Derivative',
     'is_Dummy',
     'is_Equality',
     'is_Float',
     'is_Function',
     'is_Indexed',
     'is_Integer',
     'is_Matrix',
     'is_Mul',
     'is_Not',
     'is_Number',
     'is_NumberSymbol',
     'is_Order',
     'is_Piecewise',
     'is_Point',
     'is_Poly',
     'is_Pow',
     'is_Rational',
     'is_Relational',
     'is_Symbol',
     'is_Vector',
     'is_Wild',
     'is_algebraic',
     'is_algebraic_expr',
     'is_antihermitian',
     'is_commutative',
     'is_comparable',
     'is_complex',
     'is_composite',
     'is_constant',
     'is_even',
     'is_finite',
     'is_hermitian',
     'is_hypergeometric',
     'is_imaginary',
     'is_infinite',
     'is_integer',
     'is_irrational',
     'is_negative',
     'is_noninteger',
     'is_nonnegative',
     'is_nonpositive',
     'is_nonzero',
     'is_number',
     'is_odd',
     'is_polar',
     'is_polynomial',
     'is_positive',
     'is_prime',
     'is_rational',
     'is_rational_function',
     'is_real',
     'is_symbol',
     'is_transcendental',
     'is_zero',
     'leadterm',
     'limit',
     'lseries',
     'make_args',
     'match',
     'matches',
     'n',
     'normal',
     'nseries',
     'nsimplify',
     'powsimp',
     'primitive',
     'radsimp',
     'ratsimp',
     'rcall',
     'refine',
     'removeO',
     'replace',
     'rewrite',
     'round',
     'separate',
     'series',
     'simplify',
     'sort_key',
     'subs',
     'taylor_term',
     'together',
     'transpose',
     'trigsimp',
     'xreplace']

```python
# substitute in some numbers for a and b
ee=e.subs([(b, 2.0), (a, 3.0)])
print(ee)
```

    1.81496013673848

```python
# this should give us the same answer
import math
3.0+math.tan(2.0)+1.0
```

    1.8149601367384811

# Matplotlib

This will allow us to plot sensor data, robot arm positions, a path through a map, images, etc. We will cover a few commands here, but revisit this later when we are doing image processing. The key commands are:

- plt.plot()
- plt.subplot()
- plt.legend()
- plt.title()
- plt.ylabel() and plt.xlabel()
- plt.grit()
- plt.text()

There really isn't anything hard here, just examples of the plotting commands we will use the most. See the [Matplotlib gallery](https://matplotlib.org/gallery.html) for more examples of what is possible.

```python
# Prepare the data
x = np.linspace(0, 100, 10)  # linspace(start, stop, number_of_points)
y = np.linspace(0, 10, 10)

# Plot the data
plt.plot(x, y, label='linear')

# make it look nice
plt.grid(True)
plt.legend()
plt.title('title')
plt.ylabel('y $\Omega$')  # you can also include latex math symbols
plt.xlabel('x');  # note, if you don't put the ';' at the end, it prints
                  # the object address. Not an issue, but it annoys me
```

```python
from math import cos

# Prepare the data
x = np.linspace(0, 5, 100)  # linspace(start, stop, how_many_inbetween)
y1 = x                      # simple stupid y = x linear line
y2 = [n*cos(n) for n in y1] # remember this fancy way to use a for loop and create an array?

# similar to matlab
plt.plot(x,y1, 'r-', x,y2, 'b^')  # if you only put 1 array, plot assumes it is the y-axis

# make it look nice
plt.grid(True)
plt.title('title')
plt.ylabel('y1 and y2')
plt.xlabel('x');
```

```python
# let's put more than one plot together ... every use subplot in Matlab?
plt.subplot(1,2,1)
plt.plot(x, label='linear')
plt.title('no grid')

plt.subplot(1,2,2)
plt.plot(x, label='linear')
plt.title('grid');
plt.grid(True)
```

# Quiz Next Class

You can do the entire thing ahead of time if you want ... take a look at the next lesson. I suggest you come into class with the answers done. **Do not work with anyone else, come to me if you have questions!** I want this to be a learning oppertunity! You are now considered to know python, jupyter, matplotlib, and sympy.

# Exercises

- Run back through this notebook and play with things so you understand it. This will lay the foundation for the Quiz and Lab 2.

# Questions

1. How do you create a 2x3 matrix?
1. How do you create a 3x3 zero or ones (identity) matrix?
1. How do you add, multiply or divide 2 matrices together? How about a matrix and a vector?
1. How do you plot y = 3x-5 between [-5,5]? Make sure you turn on the grid, label the axes and give it title.

-----------

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

```python

```


# Number Theory and a Google Recruitment Puzzle

The main goal of this project is to find the first 10-digit prime in the decimal expansion of 17$\pi$.

This blog post is going to solve this question by using three helper functions.

This blog can be retrieved in here: [Pu's Blog](https://puzeng.github.io/BIOSTAT823_Blog_PuZeng/).

In addition, this notebook is kept in here: [Pu's Repo](https://github.com/puzeng/BIOSTAT823_Blog_PuZeng) under the folder `_notebooks`.

Let's load all the required packages first.

```python
import math
import numpy as np
import re
import sympy
```

## Expand the mathematical expreesion

First, we are going to generate an arbitrary large expression of a mathematical expression like $\pi$. For that, a helper function called `expansion_expression` will help to do that.

As the input will be a string that contains the mathematical expression and its coefficient if has, we are going to use regular expression to extract the coefficient of the mathematical expression if there is one.

Next, we will expand the mathematical expression first to have a number of self-defined digits before generate the multiple of the mathematical expression.

Then, the multiple of the mathematical expression is generated by multiplying the coefficient with the mathematical expression.

The following is how the helper function is programmed.

```python
def expansion_mathematical_expression(math_expression, num_digits):
    """Genearate an arbitrary expression of mathematical expression."""
    #Coefficient are the any digits in numeric before the mathematical expression.
    pattern = re.compile(r'[0-9]+')

    if "pi" in math_expression:
        #Generate the mathematical expression in self-defined digits
        expanded_expression = sympy.N(sympy.pi, num_digits)
        if len(pattern.findall(math_expression)) == 1:
            times = float(pattern.match(math_expression)[0])
            expanded_expression = str(expanded_expression * times)

        else:
            expanded_expression = str(expanded_expression)

    if "e" in math_expression:
        #Generate the mathematical expression in self-defined digits
        expanded_expression = sympy.N(sympy.E, num_digits)
        if len(pattern.findall(math_expression)) == 1:
            times = float(pattern.match(math_expression)[0])
            expanded_expression = str(expanded_expression * times)

        else:
            expanded_expression = str(expanded_expression)

    #Ignore the period
    expanded_expression = re.sub(r'\.' , "" ,expanded_expression)
    return expanded_expression
```

## Check prime number

The second helper function that we are going to implement is the function to check whether the number is a prime or not, called `is_prime`.

We are not going to solve by brute force method which is to use all numbers from 2 to itself to divide that number. However, we can analyze this problem first and solve with the analytical method.

The product combinations of a number can be divided into two halves where they are mirroring to each other. For example, number 36 has the following combinations:
1. 1 * 36
2. 2 * 18
3. 3 * 12
4. 4 * 9
5. 6 * 6
6. 9 * 4
7. 12 * 3
8. 18 * 2
9. 36 * 1

We can notice that the first 4 pairs are mirroring with the last 4 pairs while the fifth pair is the mirroring line. Thus, we can generalize this question by checking whether the number can be divided by the factors up to its square root so that we don't need to check all the factors.

The codes below show how the analytical method is implemented:

```python
def is_prime(check_window):
    """Check whether the number is a prime or not."""
    check_window = int(check_window)
    # Check whether this number can be divided by any factors up to its sqrt(check_window)
    for i in range(2, int(math.sqrt(check_window))+1):
        if check_window % i == 0:
            return False

    return True
```

## Slide window

The third helper function is to generate all the sliding windows on the mathematical expression through looping through the expression. The length of the window is depending on the number of digits that the user is trying to check for the prime.

This is accomplished by appending all the sliding windows to the other list while sliding down the string.

Below shows how the above arguments are implemented:

```python
def slide_window(width, input_text):
    """Generate all sliding windows for a mathematical expression based on the customized length."""
    slide_windows = list()
    while len(input_text) >= width:
        slide_windows.append(input_text[:width])
        input_text = input_text[1:]

    return slide_windows
```

## Assemble to answer the question

As we go through all the helper functions that are used in answering the question, we now can assemble them as in one function to help solve the puzzle.

The main function will take two arguments:
1. `num_digits`: an integer for defining the number of digits to expand on the mathematical expression.
2. `math_expression`: a string that are going to be expanded based on which mathematical expression is input.
3. `width_side_window`: an integer for indicating how many digits that the prime is going to be.

Within the big function called `ten_digits_primes_in_expression`, we will first generate an arbitrary large expression of the corresponding mathematical expression by self definition through the function, `expansion_mathematical_expression`.

Then, we are generating all the sliding windows based on a self-defined length which is also equivalent to the number of digits in the prime, by calling the function `slide_window`.

To tell whether the slide window is a prime a not, we need to call function `is_prime`. If it is a prime, this function will return True and otherwise.

Once, we detect that if the slide window is the prime, we will immediatelly return as the digits in the slide window. If not, we will go to the next slide window.

Below shows how the above arguments are implemented and how to use the function:

```python
def ten_digits_primes_in_expression(num_digits, math_expression, width_slide_window):
    """Extrat first 10 prime digits in a mathematical expression."""
    #Expand the expression in self-defined digits
    expanded_expression = expansion_mathematical_expression(math_expression, num_digits)

    #Generate all the slide windows by self-defined width
    all_slide_windows = slide_window(10, expanded_expression)

    #Check whether the current window is prime or not
    for ele in all_slide_windows:
        if is_prime(ele) == True:
            return ele
```

```python
first_prime = ten_digits_primes_in_expression(300, "17pi", 10)

print(f"The first 10-digit prime in the decimal expansion of 17\u03C0 is {first_prime}.")
```

    The first 10-digit prime in the decimal expansion of 17π is 8649375157.

## For contributors: Testing

```python
import unittest

class TestNotebook(unittest.TestCase):
    def test_expansion_mathematical_expression(self):
        self.assertEqual(expansion_mathematical_expression("pi", 10), '3141592654')
        self.assertEqual(expansion_mathematical_expression("e", 10), '2718281828')

    def test_is_prime(self):
        self.assertEqual(is_prime(2), True)
        self.assertEqual(is_prime(111), False)
        self.assertEqual(is_prime(5), True)
        self.assertEqual(is_prime(596), False)

    def test_slide_window(self):
        self.assertEqual(slide_window(5, "12345"), ["12345"])
        self.assertEqual(slide_window(1, "12345"), ['1', '2', '3', '4', '5'])

    def test_ten_digits_primes_in_expression(self):
        self.assertEqual(ten_digits_primes_in_expression(300, "e", 10),
                        '7427466391')

unittest.main(argv = [''], verbosity = 2, exit = False)
```

    test_expansion_mathematical_expression (__main__.TestNotebook) ... ok
    test_is_prime (__main__.TestNotebook) ... ok
    test_slide_window (__main__.TestNotebook) ... ok
    test_ten_digits_primes_in_expression (__main__.TestNotebook) ... ok

    ----------------------------------------------------------------------
    Ran 4 tests in 0.015s

    OK

    <unittest.main.TestProgram at 0x7fd1e715e850>

## Supplementary materials

Besides to check whether a number is a prime or not by the above analytical method, we can also solve this question by another analytical approach, Sieve of Eratosthenes.

In general, the mechanism follows that within a range of number, the multiples of 2, 3, 4, and all the way up to the square root of the number will be crossed out.

The implementation then will follow:
1. Initially, generate a boolean list to cast those primes by the length of that number.
2. We will then cross out the multiples of 2 at the beginning and stop untill the multiples of 2 is greater that number.
3. Then, cross out the multiples of 3.
4. Cross out the multiples of factors that are all the way up till the square root of that number.
5. Generate an array contains a range of that number.
6. Map the boolean list to the array to get all the primes of the range.
7. If that number itself is within the list, then return True or otherwise.

This method based on Siev of Eratosthene is much faster than the method used above since the complexity is O(N* log(log(N))) compared to O(sqrt(N)).

```python
def is_prime_sieve_eratosthenes(check_window):
    """Check whether the number is a prime or not based on Sieve Of Eratosthenes."""
    #Create a list of True initially
    #Once find the multiples, update True as False
    check_window = int(check_window)
    prime_list = [True for num in range(check_window+1)]
    iterate_num = 2

    while iterate_num * iterate_num <= check_window:
        if prime_list[iterate_num] == True:
            for i in range(iterate_num * iterate_num, check_window+1, 2):
                prime_list[i] = False
        iterate_num += 1

    target_list = np.array(range(0, check_window+1))
    target_list = target_list[prime_list]
    return check_window in target_list

print(f"5 is a prime? {is_prime_sieve_eratosthenes(5)}")
```

    5 is a prime? True