# Assignment 1

The goal of this assignment is to supply you with machine learning models and algorithms. In this notebook, we will cover linear and nonlinear models, the concept of loss functions and some optimization techniques. All mathematical operations should be implemented in **NumPy** only.

## Table of contents
* [1. Logistic Regression](#1.-Logistic-Regression)
    * [1.1 Linear Mapping](#1.1-Linear-Mapping)
    * [1.2 Sigmoid](#1.2-Sigmoid)
    * [1.3 Negative Log Likelihood](#1.3-Negative-Log-Likelihood)
    * [1.4 Model](#1.4-Model)
    * [1.5 Simple Experiment](#1.5-Simple-Experiment)
* [2. Decision Tree](#2.-Decision-Tree)
    * [2.1 Gini Index & Data Split](#2.1-Gini-Index-&-Data-Split)
    * [2.2 Terminal Node](#2.2-Terminal-Node)
    * [2.3 Build the Decision Tree](#2.3-Build-the-Decision-Tree)
* [3. Experiments](#3.-Experiments)
    * [3.1 Decision Tree for Heart Disease Prediction](#3.1-Decision-Tree-for-Heart-Disease-Prediction)
    * [3.2 Logistic Regression for Heart Disease Prediction](#3.2-Logistic-Regression-for-Heart-Disease-Prediction)

### Note
Some of the concepts below have not (yet) been discussed during the lecture. These will be discussed further during the next lectures.

### Before you begin

To check whether the code you've written is correct, we'll use **automark**. For this, we created for each of you an account with the username being your student number.

```python
import automark as am

# fill in you student number as your username
username = '13060775'

# to check your progress, you can run this function
am.get_progress(username)
```

    ---------------------------------------------
    | Carlo Harprecht                           |
    | carlo.harprecht@student.uva.nl            |
    ---------------------------------------------
    | linear_forward           | completed      |
    | linear_grad_W            | completed      |
    | linear_grad_b            | completed      |
    | nll_forward              | completed      |
    | nll_grad_input           | completed      |
    | sigmoid_forward          | completed      |
    | sigmoid_grad_input       | completed      |
    | tree_gini_index          | completed      |
    | tree_split_data_left     | completed      |
    | tree_split_data_right    | completed      |
    | tree_to_terminal         | completed      |
    ---------------------------------------------

So far all your tests are 'not attempted'. At the end of this notebook you'll need to have completed all test. The output of `am.get_progress(username)` should at least match the example below. However, we encourage you to take a shot at the 'not attempted' tests!

```
---------------------------------------------
| Your name / student number                |
| your_email@your_domain.whatever           |
---------------------------------------------
| linear_forward           | not attempted  |
| linear_grad_W            | not attempted  |
| linear_grad_b            | not attempted  |
| nll_forward              | not attempted  |
| nll_grad_input           | not attempted  |
| sigmoid_forward          | not attempted  |
| sigmoid_grad_input       | not attempted  |
| tree_data_split_left     | not attempted  |
| tree_data_split_right    | not attempted  |
| tree_gini_index          | not attempted  |
| tree_to_terminal         | not attempted  |
---------------------------------------------
```

```python
from __future__ import print_function, absolute_import, division # You don't need to know what this is.
import numpy as np # this imports numpy, which is used for vector- and matrix calculations
```

This notebook makes use of **classes** and their **instances** that we have already implemented for you. It allows us to write less code and make it more readable. If you are interested in it, here are some useful links:
* The official [documentation](https://docs.python.org/3/tutorial/classes.html)
* Video by *sentdex*: [Object Oriented Programming Introduction](https://www.youtube.com/watch?v=ekA6hvk-8H8)
* Antipatterns in OOP: [Stop Writing Classes](https://www.youtube.com/watch?v=o9pEzgHorH0)

# 1. Logistic Regression

We start with a very simple algorithm called **Logistic Regression**. It is a generalized linear model for 2-class classification.
It can be generalized to the case of many classes and to non-linear cases as well. However, here we consider only the simplest case.

Let us consider a data with 2 classes. Class 0 and class 1. For a given test sample, logistic regression returns a value from $[0, 1]$ which is interpreted as a probability of belonging to class 1. The set of points for which the prediction is $0.5$ is called a *decision boundary*. It is a line on a plane or a hyper-plane in a space.

Logistic regression has two trainable parameters: a weight $W$ and a bias $b$. For a vector of features $X$, the prediction of logistic regression is given by

$$
f(X) = \frac{1}{1 + \exp(-[XW + b])} = \sigma(h(X))
$$
where $\sigma(z) = \frac{1}{1 + \exp(-z)}$ and $h(X)=XW + b$.

Parameters $W$ and $b$ are fitted by maximizing the log-likelihood (or minimizing the negative log-likelihood) of the model on the training data. For a training subset $\{X_j, Y_j\}_{j=1}^N$ the normalized negative log likelihood (NLL) is given by

$$
\mathcal{L} = -\frac{1}{N}\sum_j \log\Big[ f(X_j)^{Y_j} \cdot (1-f(X_j))^{1-Y_j}\Big]
= -\frac{1}{N}\sum_j \Big[ Y_j\log f(X_j) + (1-Y_j)\log(1-f(X_j))\Big]
$$

There are different ways of fitting this model. In this assignment we consider Logistic Regression as a one-layer neural network. We use the following algorithm for the **forward** pass:

1. Linear mapping: $h=XW + b$
2. Sigmoid activation function: $f=\sigma(h)$
3. Calculation of NLL: $\mathcal{L} = -\frac{1}{N}\sum_j \Big[ Y_j\log f_j + (1-Y_j)\log(1-f_j)\Big]$

In order to fit $W$ and $b$ we perform Gradient Descent ([GD](https://en.wikipedia.org/wiki/Gradient_descent)). We choose a small learning rate $\gamma$ and after each computation of forward pass, we update the parameters

$$W_{\text{new}} = W_{\text{old}} - \gamma \frac{\partial \mathcal{L}}{\partial W}$$

$$b_{\text{new}} = b_{\text{old}} - \gamma \frac{\partial \mathcal{L}}{\partial b}$$

We use Backpropagation method ([BP](https://en.wikipedia.org/wiki/Backpropagation)) to calculate the partial derivatives of the loss function with respect to the parameters of the model.

$$
\frac{\partial\mathcal{L}}{\partial W} =
\frac{\partial\mathcal{L}}{\partial h} \frac{\partial h}{\partial W} =
\frac{\partial\mathcal{L}}{\partial f} \frac{\partial f}{\partial h} \frac{\partial h}{\partial W}
$$

$$
\frac{\partial\mathcal{L}}{\partial b} =
\frac{\partial\mathcal{L}}{\partial h} \frac{\partial h}{\partial b} =
\frac{\partial\mathcal{L}}{\partial f} \frac{\partial f}{\partial h} \frac{\partial h}{\partial b}
$$

## 1.1 Linear Mapping
First of all, you need to implement the forward pass of a linear mapping:
$$
h(X) = XW +b
$$

**Note**: here we use `n_out` as the dimensionality of the output. For logisitc regression `n_out = 1`. However, we will work with cases of `n_out > 1` in next assignments. You will **pass** the current assignment even if your implementation works only in case `n_out = 1`. If your implementation works for the cases of `n_out > 1` then you will not have to modify your method next week. All **numpy** operations are generic. It is recommended to use numpy when is it possible.

```python
def linear_forward(x_input, W, b):
    """Perform the mapping of the input
    # Arguments
        x_input: input of the linear function - np.array of size `(n_objects, n_in)`
        W: np.array of size `(n_in, n_out)`
        b: np.array of size `(n_out,)`
    # Output
        the output of the linear function
        np.array of size `(n_objects, n_out)`
    """
    output = np.dot(x_input,W) + b

    return output
```

Let's check your first function. We set the matrices $X, W, b$:
$$
X = \begin{bmatrix}
1 & -1 \\
-1 & 0 \\
1 & 1 \\
\end{bmatrix} \quad
W = \begin{bmatrix}
4 \\
2 \\
\end{bmatrix} \quad
b = \begin{bmatrix}
3 \\
\end{bmatrix}
$$

And then compute
$$
XW = \begin{bmatrix}
1 & -1 \\
-1 & 0 \\
1 & 1 \\
\end{bmatrix}
\begin{bmatrix}
4 \\
2 \\
\end{bmatrix} =
\begin{bmatrix}
2 \\
-4 \\
6 \\
\end{bmatrix} \\
XW + b =
\begin{bmatrix}
5 \\
-1 \\
9 \\
\end{bmatrix}
$$

```python
X_test = np.array([[1, -1],
                   [-1, 0],
                   [1, 1]])

W_test = np.array([[4],
                   [2]])

b_test = np.array([3])

h_test = linear_forward(X_test, W_test, b_test)
print(h_test)
```

    [[ 5]
     [-1]
     [ 9]]

```python
am.test_student_function(username, linear_forward, ['x_input', 'W', 'b'])
```

    Running local tests...
    linear_forward successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!

Now you need to implement the calculation of the partial derivative of the loss function with respect to the parameters of the model. As this expressions are used for the updates of the parameters, we refer to them as gradients.
$$
\frac{\partial \mathcal{L}}{\partial W} =
\frac{\partial \mathcal{L}}{\partial h}
\frac{\partial h}{\partial W} \\
\frac{\partial \mathcal{L}}{\partial b} =
\frac{\partial \mathcal{L}}{\partial h}
\frac{\partial h}{\partial b} \\
$$

```python
def linear_grad_W(x_input, grad_output, W, b):
    """Calculate the partial derivative of
        the loss with respect to W parameter of the function
        dL / dW = (dL / dh) * (dh / dW)
    # Arguments
        x_input: input of a dense layer - np.array of size `(n_objects, n_in)`
        grad_output: partial derivative of the loss functions with
            respect to the ouput of the dense layer (dL / dh)
            np.array of size `(n_objects, n_out)`
        W: np.array of size `(n_in, n_out)`
        b: np.array of size `(n_out,)`
    # Output
        the partial derivative of the loss
        with respect to W parameter of the function
        np.array of size `(n_in, n_out)`
    """
    grad_W = np.dot(x_input.T,grad_output)

    return grad_W
```

```python
am.test_student_function(username, linear_grad_W, ['x_input', 'grad_output', 'W', 'b'])
```

    Running local tests...
    linear_grad_W successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!

```python
def linear_grad_b(x_input, grad_output, W, b):
    """Calculate the partial derivative of
        the loss with respect to b parameter of the function
        dL / db = (dL / dh) * (dh / db)
    # Arguments
        x_input: input of a dense layer - np.array of size `(n_objects, n_in)`
        grad_output: partial derivative of the loss functions with
            respect to the ouput of the linear function (dL / dh)
            np.array of size `(n_objects, n_out)`
        W: np.array of size `(n_in, n_out)`
        b: np.array of size `(n_out,)`
    # Output
        the partial derivative of the loss
        with respect to b parameter of the linear function
        np.array of size `(n_out,)`
    """
    grad_b = np.sum(grad_output)

    return grad_b
```

```python
am.test_student_function(username, linear_grad_b, ['x_input', 'grad_output', 'W', 'b'])
```

    Running local tests...
    linear_grad_b successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!

```python
am.get_progress(username)
```

    ---------------------------------------------
    | Carlo Harprecht                           |
    | carlo.harprecht@student.uva.nl            |
    ---------------------------------------------
    | linear_forward           | completed      |
    | linear_grad_W            | completed      |
    | linear_grad_b            | completed      |
    | nll_forward              | completed      |
    | nll_grad_input           | completed      |
    | sigmoid_forward          | completed      |
    | sigmoid_grad_input       | completed      |
    | tree_gini_index          | completed      |
    | tree_split_data_left     | completed      |
    | tree_split_data_right    | completed      |
    | tree_to_terminal         | not attempted  |
    ---------------------------------------------

## 1.2 Sigmoid
$$
f = \sigma(h) = \frac{1}{1 + e^{-h}}
$$

Sigmoid function is applied element-wise. It does not change the dimensionality of the tensor and its implementation is shape-agnostic in general.

```python
def sigmoid_forward(x_input):
    """sigmoid nonlinearity
    # Arguments
        x_input: np.array of size `(n_objects, n_in)`
    # Output
        the output of relu layer
        np.array of size `(n_objects, n_in)`
    """
    output = 1/(1+np.e**(-x_input))
    return output
```

```python
am.test_student_function(username, sigmoid_forward, ['x_input'])
```

    Running local tests...
    sigmoid_forward successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!

Now you need to implement the calculation of the partial derivative of the loss function with respect to the input of sigmoid.

$$
\frac{\partial \mathcal{L}}{\partial h} =
\frac{\partial \mathcal{L}}{\partial f}
\frac{\partial f}{\partial h}
$$

Tensor $\frac{\partial \mathcal{L}}{\partial f}$ comes from the loss function. Let's calculate $\frac{\partial f}{\partial h}$

$$
\frac{\partial f}{\partial h} =
\frac{\partial \sigma(h)}{\partial h} =
\frac{\partial}{\partial h} \Big(\frac{1}{1 + e^{-h}}\Big)
= \frac{e^{-h}}{(1 + e^{-h})^2}
= \frac{1}{1 + e^{-h}} \frac{e^{-h}}{1 + e^{-h}}
= f(h) (1 - f(h))
$$

Therefore, in order to calculate the gradient of the loss with respect to the input of sigmoid function you need
to
1. calculate $f(h) (1 - f(h))$
2. multiply it element-wise by $\frac{\partial \mathcal{L}}{\partial f}$

```python
def sigmoid_grad_input(x_input, grad_output):
    """sigmoid nonlinearity gradient.
        Calculate the partial derivative of the loss
        with respect to the input of the layer
    # Arguments
        x_input: np.array of size `(n_objects, n_in)`
        grad_output: np.array of size `(n_objects, n_in)`
            dL / df
    # Output
        the partial derivative of the loss
        with respect to the input of the function
        np.array of size `(n_objects, n_in)`
        dL / dh
    """
    f_h = (1/(1+np.e**(-x_input)))*(1-(1/(1+np.e**(-x_input))))
    grad_input = f_h * grad_output
    return grad_input
```

```python
am.test_student_function(username, sigmoid_grad_input, ['x_input', 'grad_output'])
```

    Running local tests...
    sigmoid_grad_input successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!

## 1.3 Negative Log Likelihood

$$
\mathcal{L}
= -\frac{1}{N}\sum_j \Big[ Y_j\log \dot{Y}_j + (1-Y_j)\log(1-\dot{Y}_j)\Big]
$$

Here $N$ is the number of objects. $Y_j$ is the real label of an object and $\dot{Y}_j$ is the predicted one.

```python
def nll_forward(target_pred, target_true):
    """Compute the value of NLL
        for a given prediction and the ground truth
    # Arguments
        target_pred: predictions - np.array of size `(n_objects, 1)`
        target_true: ground truth - np.array of size `(n_objects, 1)`
    # Output
        the value of NLL for a given prediction and the ground truth
        scalar
    """
    output = -(1/len(target_pred))*np.sum(target_true*np.log(target_pred) + (1-target_true)*np.log(1-target_pred))
    return output
```

```python
am.test_student_function(username, nll_forward, ['target_pred', 'target_true'])
```

    Running local tests...
    nll_forward successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!

Now you need to calculate the partial derivative of NLL with with respect to its input.

$$
\frac{\partial \mathcal{L}}{\partial \dot{Y}}
=
\begin{pmatrix}
\frac{\partial \mathcal{L}}{\partial \dot{Y}_0} \\
\frac{\partial \mathcal{L}}{\partial \dot{Y}_1} \\
\vdots \\
\frac{\partial \mathcal{L}}{\partial \dot{Y}_N}
\end{pmatrix}
$$

Let's do it step-by-step

\begin{equation}
\begin{split}
\frac{\partial \mathcal{L}}{\partial \dot{Y}_0}
&= \frac{\partial}{\partial \dot{Y}_0} \Big(-\frac{1}{N}\sum_j \Big[ Y_j\log \dot{Y}_j + (1-Y_j)\log(1-\dot{Y}_j)\Big]\Big) \\
&= -\frac{1}{N} \frac{\partial}{\partial \dot{Y}_0} \Big(Y_0\log \dot{Y}_0 + (1-Y_0)\log(1-\dot{Y}_0)\Big) \\
&= -\frac{1}{N} \Big(\frac{Y_0}{\dot{Y}_0} - \frac{1-Y_0}{1-\dot{Y}_0}\Big)
= \frac{1}{N} \frac{\dot{Y}_0 - Y_0}{\dot{Y}_0 (1 - \dot{Y}_0)}
\end{split}
\end{equation}

And for the other components it can be done in exactly the same way. So the result is the vector where each component is given by
$$\frac{1}{N} \frac{\dot{Y}_j - Y_j}{\dot{Y}_j (1 - \dot{Y}_j)}$$

Or if we assume all multiplications and divisions to be done element-wise the output can be calculated as
$$
\frac{\partial \mathcal{L}}{\partial \dot{Y}} = \frac{1}{N} \frac{\dot{Y} - Y}{\dot{Y} (1 - \dot{Y})}
$$

```python
def nll_grad_input(target_pred, target_true):
    """Compute the partial derivative of NLL
        with respect to its input
    # Arguments
        target_pred: predictions - np.array of size `(n_objects, 1)`
        target_true: ground truth - np.array of size `(n_objects, 1)`
    # Output
        the partial derivative
        of NLL with respect to its input
        np.array of size `(n_objects, 1)`
    """
    grad_input = (1/len(target_pred))*((target_pred-target_true)/(target_pred*(1-target_pred)))
    return grad_input
```

```python
am.test_student_function(username, nll_grad_input, ['target_pred', 'target_true'])
```

    Running local tests...
    nll_grad_input successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!

```python
am.get_progress(username)
```

    ---------------------------------------------
    | Carlo Harprecht                           |
    | carlo.harprecht@student.uva.nl            |
    ---------------------------------------------
    | linear_forward           | completed      |
    | linear_grad_W            | completed      |
    | linear_grad_b            | completed      |
    | nll_forward              | completed      |
    | nll_grad_input           | completed      |
    | sigmoid_forward          | not attempted  |
    | sigmoid_grad_input       | completed      |
    | tree_gini_index          | not attempted  |
    | tree_split_data_left     | not attempted  |
    | tree_split_data_right    | not attempted  |
    | tree_to_terminal         | not attempted  |
    ---------------------------------------------

## 1.4 Model

Here we provide a model for your. It consist of the function which you have implmeneted above

```python
class LogsticRegressionGD(object):

    def __init__(self, n_in, lr=0.05):
        super().__init__()
        self.lr = lr
        self.b = np.zeros(1, )
        self.W = np.random.randn(n_in, 1)

    def forward(self, x):
        self.h = linear_forward(x, self.W, self.b)
        y = sigmoid_forward(self.h)
        return y

    def update_params(self, x, nll_grad):
        # compute gradients
        grad_h = sigmoid_grad_input(self.h, nll_grad)
        grad_W = linear_grad_W(x, grad_h, self.W, self.b)
        grad_b = linear_grad_b(x, grad_h, self.W, self.b)
        # update params
        self.W = self.W - self.lr * grad_W
        self.b = self.b - self.lr * grad_b
```

## 1.5 Simple Experiment

```python
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
# Generate some data
def generate_2_circles(N=100):
    phi = np.linspace(0.0, np.pi * 2, 100)
    X1 = 1.1 * np.array([np.sin(phi), np.cos(phi)])
    X2 = 3.0 * np.array([np.sin(phi), np.cos(phi)])
    Y = np.concatenate([np.ones(N), np.zeros(N)]).reshape((-1, 1))
    X = np.hstack([X1,X2]).T
    return X, Y

def generate_2_gaussians(N=100):
    phi = np.linspace(0.0, np.pi * 2, 100)
    X1 = np.random.normal(loc=[1, 2], scale=[2.5, 0.9], size=(N, 2))
    X1 = X1 @ np.array([[0.7, -0.7], [0.7, 0.7]])
    X2 = np.random.normal(loc=[-2, 0], scale=[1, 1.5], size=(N, 2))
    X2 = X2 @ np.array([[0.7, 0.7], [-0.7, 0.7]])
    Y = np.concatenate([np.ones(N), np.zeros(N)]).reshape((-1, 1))
    X = np.vstack([X1,X2])
    return X, Y

def split(X, Y, train_ratio=0.7):
    size = len(X)
    train_size = int(size * train_ratio)
    indices = np.arange(size)
    np.random.shuffle(indices)
    train_indices = indices[:train_size]
    test_indices = indices[train_size:]
    return X[train_indices], Y[train_indices], X[test_indices], Y[test_indices]

f, (ax1, ax2) = plt.subplots(1, 2, figsize=(8, 4))

X, Y = generate_2_circles()
ax1.scatter(X[:,0], X[:,1], c=Y.ravel(), edgecolors= 'none')
ax1.set_aspect('equal')

X, Y = generate_2_gaussians()
ax2.scatter(X[:,0], X[:,1], c=Y.ravel(), edgecolors= 'none')
ax2.set_aspect('equal')

```

```python
X_train, Y_train, X_test, Y_test = split(*generate_2_gaussians(), 0.7)
```

```python
# let's train our model
model = LogsticRegressionGD(2, 0.05)

for step in range(30):
    Y_pred = model.forward(X_train)

    loss_value = nll_forward(Y_pred, Y_train)
    accuracy = ((Y_pred > 0.5) == Y_train).mean()
    print('Step: {} \t Loss: {:.3f} \t Acc: {:.1f}%'.format(step, loss_value, accuracy * 100))

    loss_grad = nll_grad_input(Y_pred, Y_train)
    model.update_params(X_train, loss_grad)

print('\n\nTesting...')
Y_test_pred = model.forward(X_test)
test_accuracy = ((Y_test_pred > 0.5) == Y_test).mean()
print('Acc: {:.1f}%'.format(test_accuracy * 100))
```

    Step: 0 	 Loss: 0.546 	 Acc: 73.6%
    Step: 1 	 Loss: 0.528 	 Acc: 73.6%
    Step: 2 	 Loss: 0.511 	 Acc: 75.0%
    Step: 3 	 Loss: 0.495 	 Acc: 75.7%
    Step: 4 	 Loss: 0.480 	 Acc: 75.7%
    Step: 5 	 Loss: 0.465 	 Acc: 77.1%
    Step: 6 	 Loss: 0.452 	 Acc: 77.9%
    Step: 7 	 Loss: 0.439 	 Acc: 77.9%
    Step: 8 	 Loss: 0.426 	 Acc: 80.0%
    Step: 9 	 Loss: 0.415 	 Acc: 81.4%
    Step: 10 	 Loss: 0.403 	 Acc: 81.4%
    Step: 11 	 Loss: 0.393 	 Acc: 81.4%
    Step: 12 	 Loss: 0.383 	 Acc: 82.1%
    Step: 13 	 Loss: 0.373 	 Acc: 84.3%
    Step: 14 	 Loss: 0.364 	 Acc: 84.3%
    Step: 15 	 Loss: 0.355 	 Acc: 84.3%
    Step: 16 	 Loss: 0.347 	 Acc: 84.3%
    Step: 17 	 Loss: 0.339 	 Acc: 85.7%
    Step: 18 	 Loss: 0.331 	 Acc: 85.7%
    Step: 19 	 Loss: 0.324 	 Acc: 86.4%
    Step: 20 	 Loss: 0.317 	 Acc: 86.4%
    Step: 21 	 Loss: 0.310 	 Acc: 87.1%
    Step: 22 	 Loss: 0.303 	 Acc: 87.9%
    Step: 23 	 Loss: 0.297 	 Acc: 87.9%
    Step: 24 	 Loss: 0.291 	 Acc: 89.3%
    Step: 25 	 Loss: 0.286 	 Acc: 89.3%
    Step: 26 	 Loss: 0.280 	 Acc: 89.3%
    Step: 27 	 Loss: 0.275 	 Acc: 89.3%
    Step: 28 	 Loss: 0.270 	 Acc: 89.3%
    Step: 29 	 Loss: 0.265 	 Acc: 90.7%

    Testing...
    Acc: 88.3%

```python
def plot_model_prediction(prediction_func, X, Y, hard=True):
    u_min = X[:, 0].min()-1
    u_max = X[:, 0].max()+1
    v_min = X[:, 1].min()-1
    v_max = X[:, 1].max()+1

    U, V = np.meshgrid(np.linspace(u_min, u_max, 100), np.linspace(v_min, v_max, 100))
    UV = np.stack([U.ravel(), V.ravel()]).T
    c = prediction_func(UV).ravel()
    if hard:
        c = c > 0.5
    plt.scatter(UV[:,0], UV[:,1], c=c, edgecolors= 'none', alpha=0.15)
    plt.scatter(X[:,0], X[:,1], c=Y.ravel(), edgecolors= 'black')
    plt.xlim(left=u_min, right=u_max)
    plt.ylim(bottom=v_min, top=v_max)
    plt.axes().set_aspect('equal')
    plt.show()

plot_model_prediction(lambda x: model.forward(x), X_train, Y_train, False)

plot_model_prediction(lambda x: model.forward(x), X_train, Y_train, True)
```

```python
X_train, Y_train, X_test, Y_test = split(*generate_2_circles(), 0.7)
```

```python
# let's train our model
model = LogsticRegressionGD(2, 0.05)

for step in range(30):
    Y_pred = model.forward(X_train)

    loss_value = nll_forward(Y_pred, Y_train)
    accuracy = ((Y_pred > 0.5) == Y_train).mean()
    print('Step: {} \t Loss: {:.3f} \t Acc: {:.1f}%'.format(step, loss_value, accuracy * 100))

    loss_grad = nll_grad_input(Y_pred, Y_train)
    model.update_params(X_train, loss_grad)

print('\n\nTesting...')
Y_test_pred = model.forward(X_test)
test_accuracy = ((Y_test_pred > 0.5) == Y_test).mean()
print('Acc: {:.1f}%'.format(test_accuracy * 100))
```

    Step: 0 	 Loss: 1.135 	 Acc: 53.6%
    Step: 1 	 Loss: 1.123 	 Acc: 53.6%
    Step: 2 	 Loss: 1.111 	 Acc: 53.6%
    Step: 3 	 Loss: 1.099 	 Acc: 53.6%
    Step: 4 	 Loss: 1.087 	 Acc: 53.6%
    Step: 5 	 Loss: 1.075 	 Acc: 53.6%
    Step: 6 	 Loss: 1.064 	 Acc: 53.6%
    Step: 7 	 Loss: 1.053 	 Acc: 53.6%
    Step: 8 	 Loss: 1.041 	 Acc: 53.6%
    Step: 9 	 Loss: 1.031 	 Acc: 53.6%
    Step: 10 	 Loss: 1.020 	 Acc: 53.6%
    Step: 11 	 Loss: 1.009 	 Acc: 53.6%
    Step: 12 	 Loss: 0.999 	 Acc: 53.6%
    Step: 13 	 Loss: 0.989 	 Acc: 53.6%
    Step: 14 	 Loss: 0.978 	 Acc: 53.6%
    Step: 15 	 Loss: 0.969 	 Acc: 53.6%
    Step: 16 	 Loss: 0.959 	 Acc: 53.6%
    Step: 17 	 Loss: 0.949 	 Acc: 53.6%
    Step: 18 	 Loss: 0.940 	 Acc: 53.6%
    Step: 19 	 Loss: 0.931 	 Acc: 53.6%
    Step: 20 	 Loss: 0.922 	 Acc: 53.6%
    Step: 21 	 Loss: 0.913 	 Acc: 53.6%
    Step: 22 	 Loss: 0.905 	 Acc: 53.6%
    Step: 23 	 Loss: 0.897 	 Acc: 53.6%
    Step: 24 	 Loss: 0.889 	 Acc: 53.6%
    Step: 25 	 Loss: 0.881 	 Acc: 53.6%
    Step: 26 	 Loss: 0.873 	 Acc: 53.6%
    Step: 27 	 Loss: 0.866 	 Acc: 53.6%
    Step: 28 	 Loss: 0.858 	 Acc: 53.6%
    Step: 29 	 Loss: 0.851 	 Acc: 53.6%

    Testing...
    Acc: 40.0%

```python
def plot_model_prediction(prediction_func, X, Y, hard=True):
    u_min = X[:, 0].min()-1
    u_max = X[:, 0].max()+1
    v_min = X[:, 1].min()-1
    v_max = X[:, 1].max()+1

    U, V = np.meshgrid(np.linspace(u_min, u_max, 100), np.linspace(v_min, v_max, 100))
    UV = np.stack([U.ravel(), V.ravel()]).T
    c = prediction_func(UV).ravel()
    if hard:
        c = c > 0.5
    plt.scatter(UV[:,0], UV[:,1], c=c, edgecolors= 'none', alpha=0.15)
    plt.scatter(X[:,0], X[:,1], c=Y.ravel(), edgecolors= 'black')
    plt.xlim(left=u_min, right=u_max)
    plt.ylim(bottom=v_min, top=v_max)
    plt.axes().set_aspect('equal')
    plt.show()

plot_model_prediction(lambda x: model.forward(x), X_train, Y_train, False)

plot_model_prediction(lambda x: model.forward(x), X_train, Y_train, True)
```

# 2. Decision Tree
The next model we look at is called **Decision Tree**. This type of model is non-parametric, meaning in contrast to **Logistic Regression** we do not have any parameters here that need to be trained.

Let us consider a simple binary decision tree for deciding on the two classes of "creditable" and "Not creditable".

Each node, except the leafs, asks a question about the the client in question. A decision is made by going from the root node to a leaf node, while considering the clients situation. The situation of the client, in this case, is fully described by the features:
1. Checking account balance
2. Duration of requested credit
3. Payment status of previous loan
4. Length of current employment

In order to build a decision tree we need training data. To carry on the previous example: we need a number of clients for which we know the properties 1.-4. and their creditability.
The process of building a decision tree starts with the root node and involves the following steps:
1. Choose a splitting criteria and add it to the current node.
2. Split the dataset at the current node into those that fullfil the criteria and those that do not.
3. Add a child node for each data split.
4. For each child node decide on either A. or B.:
    1. Repeat from 1. step
    2. Make it a leaf node: The predicted class label is decided by the majority vote over the training data in the current split.

## 2.1 Gini Index & Data Split
Deciding on how to split your training data at each node is dominated by the following two criterias:
1. Does the rule help me make a final decision?
2. Is the rule general enough such that it applies not only to my training data, but also to new unseen examples?

When considering our previous example, splitting the clients by their handedness would not help us deciding on their creditability. Knowning if a rule will generalize is usually a hard call to make, but in practice we rely on [Occam's razor](https://en.wikipedia.org/wiki/Occam%27s_razor) principle. Thus the less rules we use, the better we believe it to generalize to previously unseen examples.

One way to measure the quality of a rule is by the [**Gini Index**](https://en.wikipedia.org/wiki/Gini_coefficient).
Since we only consider binary classification, it is calculated by:
$$
Gini = \sum_{n\in\{L,R\}}\frac{|S_n|}{|S|}\left( 1 - \sum_{c \in C} p_{S_n}(c)^2\right)\\
p_{S_n}(c) = \frac{|\{\mathbf{x}_{i}\in \mathbf{X}|y_{i} = c, i \in S_n\}|}{|S_n|}, n \in \{L, R\}
$$
with $|C|=2$ being your set of class labels and $S_L$ and $S_R$ the two splits determined by the splitting criteria.
While we only consider two class problems for decision trees, the method can also be applied when $|C|>2$.
The lower the gini score, the better the split. In the extreme case, where all class labels are the same in each split respectively, the gini index takes the value of $0$.

```python
def tree_gini_index(Y_left, Y_right, classes):
    """Compute the Gini Index.
    # Arguments
        Y_left: class labels of the data left set
            np.array of size `(n_objects, 1)`
        Y_right: class labels of the data right set
            np.array of size `(n_objects, 1)`
        classes: list of all class values
    # Output
        gini: scalar `float`
    """
    gini = 0
    all_inst = len(Y_left)+len(Y_right)

    gini += (len(Y_left)/all_inst)*(1-((len(Y_left[Y_left == classes[0]])/len(Y_left))**2 +
                                       (len(Y_left[Y_left == classes[1]])/len(Y_left))**2))
    gini += (len(Y_right)/all_inst)*(1-((len(Y_right[Y_right == classes[0]])/len(Y_right))**2 +
                                       (len(Y_right[Y_right == classes[1]])/len(Y_right))**2))

    return gini
```

```python
am.test_student_function(username, tree_gini_index, ['Y_left', 'Y_right', 'classes'])
```

    Running local tests...
    tree_gini_index successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!

At each node in the tree, the data is split according to a split criterion and each split is passed onto the left/right child respectively.
Implement the following function to return all rows in `X` and `Y` such that the left child gets all examples that are less than the split value and vice versa.

```python
def tree_split_data_left(X, Y, feature_index, split_value):
    """Split the data `X` and `Y`, at the feature indexed by `feature_index`.
    If the value is less than `split_value` then return it as part of the left group.

    # Arguments
        X: np.array of size `(n_objects, n_in)`
        Y: np.array of size `(n_objects, 1)`
        feature_index: index of the feature to split at
        split_value: value to split between
    # Output
        (XY_left): np.array of size `(n_objects_left, n_in + 1)`
    """
    X_left, Y_left = None, None
    X_left = X[X[:,feature_index] < split_value]
    Y_left = Y[X[:,feature_index] < split_value]

    XY_left = np.concatenate([X_left, Y_left], axis=-1)
    return XY_left

def tree_split_data_right(X, Y, feature_index, split_value):
    """Split the data `X` and `Y`, at the feature indexed by `feature_index`.
    If the value is greater or equal than `split_value` then return it as part of the right group.

    # Arguments
        X: np.array of size `(n_objects, n_in)`
        Y: np.array of size `(n_objects, 1)`
        feature_index: index of the feature to split at
        split_value: value to split between
    # Output
        (XY_left): np.array of size `(n_objects_left, n_in + 1)`
    """
    X_right, Y_right = None, None

    X_right = X[X[:,feature_index] >= split_value]
    Y_right = Y[X[:,feature_index] >= split_value]

    XY_right = np.concatenate([X_right, Y_right], axis=-1)
    return XY_right
```

```python
am.test_student_function(username, tree_split_data_left, ['X', 'Y', 'feature_index', 'split_value'])
```

    Running local tests...
    tree_split_data_left successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!

```python
am.test_student_function(username, tree_split_data_right, ['X', 'Y', 'feature_index', 'split_value'])
```

    Running local tests...
    tree_split_data_right successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!

```python
am.get_progress(username)
```

    ---------------------------------------------
    | Carlo Harprecht                           |
    | carlo.harprecht@student.uva.nl            |
    ---------------------------------------------
    | linear_forward           | completed      |
    | linear_grad_W            | completed      |
    | linear_grad_b            | completed      |
    | nll_forward              | completed      |
    | nll_grad_input           | completed      |
    | sigmoid_forward          | completed      |
    | sigmoid_grad_input       | completed      |
    | tree_gini_index          | completed      |
    | tree_split_data_left     | completed      |
    | tree_split_data_right    | completed      |
    | tree_to_terminal         | not attempted  |
    ---------------------------------------------

Now to find the split rule with the lowest gini score, we brute-force search over all features and values to split by.

```python
def tree_best_split(X, Y):
    class_values = list(set(Y.flatten().tolist()))
    r_index, r_value, r_score =  float("inf"),  float("inf"), float("inf")
    r_XY_left, r_XY_right = (X,Y), (X,Y)
    for feature_index in range(X.shape[1]):
        for row in X:
            XY_left = tree_split_data_left(X, Y, feature_index, row[feature_index])
            XY_right = tree_split_data_right(X, Y, feature_index, row[feature_index])
            XY_left, XY_right = (XY_left[:,:-1], XY_left[:,-1:]), (XY_right[:,:-1], XY_right[:,-1:])
            gini = tree_gini_index(XY_left[1], XY_right[1], class_values)
            if gini < r_score:
                r_index, r_value, r_score = feature_index, row[feature_index], gini
                r_XY_left, r_XY_right = XY_left, XY_right
    return {'index':r_index, 'value':r_value, 'XY_left': r_XY_left, 'XY_right':r_XY_right}
```

## 2.2 Terminal Node
The leaf nodes predict the label of an unseen example, by taking a majority vote over all training class labels in that node.

```python
def tree_to_terminal(Y):
    """The most frequent class label, out of the data points belonging to the leaf node,
    is selected as the predicted class.

    # Arguments
        Y: np.array of size `(n_objects)`

    # Output
        label: most frequent label of `Y.dtype`
    """
    (values,counts) = np.unique(Y,return_counts=True)
    ind = np.argmax(counts)
    label = values[ind]

    return label
```

```python
am.test_student_function(username, tree_to_terminal, ['Y'])
```

    Running local tests...
    tree_to_terminal successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!

```python
am.get_progress(username)
```

    ---------------------------------------------
    | Carlo Harprecht                           |
    | carlo.harprecht@student.uva.nl            |
    ---------------------------------------------
    | linear_forward           | completed      |
    | linear_grad_W            | completed      |
    | linear_grad_b            | completed      |
    | nll_forward              | completed      |
    | nll_grad_input           | completed      |
    | sigmoid_forward          | completed      |
    | sigmoid_grad_input       | completed      |
    | tree_gini_index          | completed      |
    | tree_split_data_left     | completed      |
    | tree_split_data_right    | completed      |
    | tree_to_terminal         | completed      |
    ---------------------------------------------

## 2.3 Build the Decision Tree
Now we recursively build the decision tree, by greedily splitting the data at each node according to the gini index.
To prevent the model from overfitting, we transform a node into a terminal/leaf node, if:
1. a maximum depth is reached.
2. the node does not reach a minimum number of training samples.

```python
def tree_recursive_split(X, Y, node, max_depth, min_size, depth):
    XY_left, XY_right = node['XY_left'], node['XY_right']
    del(node['XY_left'])
    del(node['XY_right'])
    # check for a no split
    if XY_left[0].size <= 0 or XY_right[0].size <= 0:
        node['left_child'] = node['right_child'] = tree_to_terminal(np.concatenate((XY_left[1], XY_right[1])))
        return
    # check for max depth
    if depth >= max_depth:
        node['left_child'], node['right_child'] = tree_to_terminal(XY_left[1]), tree_to_terminal(XY_right[1])
        return
    # process left child
    if XY_left[0].shape[0] <= min_size:
        node['left_child'] = tree_to_terminal(XY_left[1])
    else:
        node['left_child'] = tree_best_split(*XY_left)
        tree_recursive_split(X, Y, node['left_child'], max_depth, min_size, depth+1)
    # process right child
    if XY_right[0].shape[0] <= min_size:
        node['right_child'] = tree_to_terminal(XY_right[1])
    else:
        node['right_child'] = tree_best_split(*XY_right)
        tree_recursive_split(X, Y, node['right_child'], max_depth, min_size, depth+1)

def build_tree(X, Y, max_depth, min_size):
    root = tree_best_split(X, Y)
    tree_recursive_split(X, Y, root, max_depth, min_size, 1)
    return root
```

By printing the split criteria or the predicted class at each node, we can visualise the decising making process.
Both the tree and a a prediction can be implemented recursively, by going from the root to a leaf node.

```python
def print_tree(node, depth=0):
    if isinstance(node, dict):
        print('%s[X%d < %.3f]' % ((depth*' ', (node['index']+1), node['value'])))
        print_tree(node['left_child'], depth+1)
        print_tree(node['right_child'], depth+1)
    else:
        print('%s[%s]' % ((depth*' ', node)))

def tree_predict_single(x, node):
    if isinstance(node, dict):
        if x[node['index']] < node['value']:
            return tree_predict_single(x, node['left_child'])
        else:
            return tree_predict_single(x, node['right_child'])

    return node

def tree_predict_multi(X, node):
    Y = np.array([tree_predict_single(row, node) for row in X])
    return Y[:, None]  # size: (n_object,) -> (n_object, 1)
```

Let's test our decision tree model on some toy data.

```python
X_train, Y_train, X_test, Y_test = split(*generate_2_circles(), 0.7)

tree = build_tree(X_train, Y_train, 4, 1)
Y_pred = tree_predict_multi(X_test, tree)
test_accuracy = (Y_pred == Y_test).mean()
print('Test Acc: {:.1f}%'.format(test_accuracy * 100))
```

We print the decision tree in [pre-order](https://en.wikipedia.org/wiki/Tree_traversal#Pre-order_(NLR)).

```python
print_tree(tree)
```

```python
plot_model_prediction(lambda x: tree_predict_multi(x, tree), X_test, Y_test)
```

# 3. Experiments
The [Cleveland Heart Disease](https://archive.ics.uci.edu/ml/datasets/Heart+Disease) dataset aims at predicting the presence of heart disease based on other available medical information of the patient.

Although the whole database contains 76 attributes, we focus on the following 14:
1. Age: age in years
2. Sex:
    * 0 = female
    * 1 = male
3. Chest pain type:
    * 1 = typical angina
    * 2 = atypical angina
    * 3 = non-anginal pain
    * 4 = asymptomatic
4. Trestbps: resting blood pressure in mm Hg on admission to the hospital
5. Chol: serum cholestoral in mg/dl
6. Fasting blood sugar: > 120 mg/dl
    * 0 = false
    * 1 = true
7. Resting electrocardiographic results:
    * 0 = normal
    * 1 = having ST-T wave abnormality (T wave inversions and/or ST elevation or depression of > 0.05 mV)
    * 2 = showing probable or definite left ventricular hypertrophy by Estes' criteria
8. Thalach: maximum heart rate achieved
9. Exercise induced angina:
    * 0 = no
    * 1 = yes
10. Oldpeak: ST depression induced by exercise relative to rest
11. Slope: the slope of the peak exercise ST segment
    * 1 = upsloping
    * 2 = flat
    * 3 = downsloping
12. Ca: number of major vessels (0-3) colored by flourosopy
13. Thal:
    * 3 = normal
    * 6 = fixed defect
    * 7 = reversable defect
14. Target: diagnosis of heart disease (angiographic disease status)
    * 0 = < 50% diameter narrowing
    * 1 = > 50% diameter narrowing

The 14. attribute is the target variable that we would like to predict based on the rest.

We have prepared some helper functions to download and pre-process the data in `heart_disease_data.py`

```python
import heart_disease_data
```

```python
X, Y = heart_disease_data.download_and_preprocess()
X_train, Y_train, X_test, Y_test = split(X, Y, 0.7)
```

Let's have a look at some examples

```python
print(X_train[0:2])
print(Y_train[0:2])

# TODO feel free to explore more examples and see if you can predict the presence of a heart disease
```

## 3.1 Decision Tree for Heart Disease Prediction
Let's build a decision tree model on the training data and see how well it performs

```python
# TODO: you are free to make use of code that we provide in previous cells
# TODO: play around with different hyper parameters and see how these impact your performance

tree = build_tree(X_train, Y_train, 5, 4)
Y_pred = tree_predict_multi(X_test, tree)
test_accuracy = (Y_pred == Y_test).mean()
print('Test Acc: {:.1f}%'.format(test_accuracy * 100))
```

How did changing the hyper parameters affect the test performance? Usually hyper parameters are tuned using a hold-out [validation set](https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets#Validation_dataset) instead of the test set.

## 3.2 Logistic Regression for Heart Disease Prediction

Instead of manually going through the data to find possible correlations, let's try training a logistic regression model on the data.

```python
# TODO: you are free to make use of code that we provide in previous cells
# TODO: play around with different hyper parameters and see how these impact your performance
```

How well did your model perform? Was it actually better then guessing? Let's look at the empirical mean of the target.

```python
Y_train.mean()
```

So what is the problem? Let's have a look at the learned parameters of our model.

```python
print(model.W, model.b)
```

If you trained sufficiently many steps you'll probably see how some weights are much larger than others. Have a look at what range the parameters were initialized and how much change we allow per step (learning rate). Compare this to the scale of the input features. Here an important concept arises, when we want to train on real world data:
[Feature Scaling](https://en.wikipedia.org/wiki/Feature_scaling).

Let's try applying it on our data and see how it affects our performance.

```python
# TODO: Rescale the input features and train again
```

Notice that we did not need any rescaling for the decision tree. Can you think of why?


# Gram-Schmidt process

The Gram-Schmidt procedure takes a list of (column) vectors and forms an orthonormal basis from this set.
As a corollary, the procedure allows us to determine the dimension of the space spanned by the basis vectors, which is equal to or less than the space which the vectors sit.

```python
# !pip install sympy
```

```python
import numpy as np
import numpy.linalg as la

verySmallNumber = 1e-14 # That's 1×10⁻¹⁴ = 0.00000000000001
```

```python
# sympy converts numpy as laTex
# from IPython.display import display, Math, Latex
import sympy as sp
from sympy import Matrix as spm
sp.init_printing(use_unicode=True)
```

### Gram-Schmidt procedure
Take 4 basis vectors as a list of vectors as the columns of a matrix, A.
Go through the vectors one at a time and set them to be orthogonal to all the vectors that came before it and normalise it.

```python
def gsBasis4(A):
    """Gram-Schmidt for 4 vectors"""

    # Work with a copy - vectors are mutable
    B = np.array(A, dtype=np.float_)

    # Column(vector) 0: Normalise(divide by its modulus or norm)
    B[:, 0] = B[:, 0] / la.norm(B[:, 0])

    # Column 1: - subtract any overlap with the zeroth vector
    B[:, 1] = B[:, 1] - B[:, 1] @ B[:, 0] * B[:, 0]

    # If there's anything left after that subtraction, then B[:, 1] is linearly independant of B[:, 0]
    # Normalise - norm(indepent)=1, norm(dependent)=0
    if la.norm(B[:, 1]) > verySmallNumber :
        B[:, 1] = B[:, 1] / la.norm(B[:, 1])
    else :
        B[:, 1] = np.zeros_like(B[:, 1])

    # Column 2: - subtract the overlap with the zeroth vector
    #           - subtract the overlap with the first
    B[:, 2] = B[:, 2] - B[:, 2] @ B[:, 0] * B[:, 0] - B[:, 2] @ B[:, 1] * B[:, 1]

    # Normalise - norm(indepent)=1, norm(dependent)=0
    if la.norm(B[:, 2]) > verySmallNumber :
        B[:, 2] = B[:, 2] / la.norm(B[:, 2])
    else :
        B[:, 2] = np.zeros_like(B[:, 2])

    # Column 2: - subtract the overlap with the zeroth vector
    #           - subtract the overlap with the first
    #           - subtract the overlap with the second
    B[:, 3] = B[:, 3] - (B[:, 3] @ B[:, 0] * B[:, 0]) - (B[:, 3] @ B[:, 1] * B[:, 1]) - (B[:, 3] @ B[:, 2] * B[:, 2])

    # Normalise - norm(indepent)=1, norm(dependent)=0
    if la.norm(B[:, 3]) > verySmallNumber :
        B[:, 3] = B[:, 3] / la.norm(B[:, 3])
    else :
        B[:, 3] = np.zeros_like(B[:, 3])

    return B
```

```python
def gsBasis(A):
    """Gram-Schmidt for n vectors"""

    # Work with a copy - vectors are mutable
    B = np.array(A, dtype=np.float_)

    # Loop over all vectors
    for i in range(B.shape[1]):
        # Loop over all previous vectors, j, to subtract.
        for j in range(i):
            # Subtract the overlap with previous vectors
            # you'll need the current vector B[:, i] and a previous vector B[:, j]
            B[:, i] -= (B[:, i] @ B[:, j] * B[:, j])

        # Normalise - norm(indepent)=1, norm(dependent)=0
        if la.norm(B[:, i]) > verySmallNumber :
            B[:, i] = B[:, i] / la.norm(B[:, i])
        else :
            B[:, i] = np.zeros_like(B[:, i])

    return B

def dimensions(A):
    """Gram-schmidt process to calculate the dimension spanned by a list of vectors.
    Independent vectors are normalised to one and dependent vectors to zero
    Thus the sum of all the norms will be the dimension"""
    return np.sum(la.norm(gsBasis(A), axis=0))
```

## Test your code before submission
To test the code you've written above, run the cell (select the cell above, then press the play button [ ▶| ] or press shift-enter).
You can then use the code below to test out your function.
You don't need to submit this cell; you can edit and run it as much as you like.

Try out your code on tricky test cases!

### 4 vector function

```python
V = np.array([[1,0,2,6],
              [0,1,8,2],
              [2,8,3,1],
              [1,-6,2,3]], dtype=np.float_)
spm(gsBasis4(V).round(2))
```

```python
# Once you've done Gram-Schmidt once, doing it again should give you the same result.
U = gsBasis4(V)
spm(U.round(2)), spm(gsBasis4(U).round(2))
np.testing.assert_almost_equal(gsBasis4(U), gsBasis4(V))
```

### Generic function

```python
spm(gsBasis(V).round(2))
```

### Non-square matrices

```python
A = np.array([[3,2,3],
              [2,5,-1],
              [2,4,8],
              [12,2,1]], dtype=np.float_)
spm(gsBasis(A).round(2)), dimensions(A)
```

### Dependent vectors - a linear combination of the others

```python
B = np.array([[6,2,1,7,5],
              [2,8,5,-4,1],
              [1,-6,3,2,8]], dtype=np.float_)
spm(gsBasis(B).round(2)), dimensions(B)
```

```python
C = np.array([[1,0,2],
              [0,1,-3],
              [1,0,2]], dtype=np.float_)
spm(gsBasis(C).round(2)), dimensions(C)
```

```python

```

_math/Imperial College London - Math for ML/GramSchmidtProcess.ipynb


```python
%pylab inline
import sympy as sym
```

    Populating the interactive namespace from numpy and matplotlib

We'll need to compute

$$
A_{ij} := \int_a^b v_j' v_i'
$$

and

$$
f_i := \int_a^b v_i f
$$

We split the interval in $M$ elements (segments), with $M+1$ vertices (element boundaries). Define the element boundaries $q := [a, q_1, q_2, ..., q_{M-1}, b]$, and the elements sizes $h_k := v_{k+1}-q_k$.
In $[a,b]$ we have a total of $N$ basis functions, which are piecewise polynomials of degree $d$, so that in each segment there are always at most $d+1$ non-zero basis functions, and $d+1$ support points.

We assume that each support point can be interpreted as the image of reference support points $\hat a_\alpha$, where $\alpha \in [0,d]$, throught the mapping $F_k(s) := q_k+h_k s$ which maps $[0,1]$ to $T_k := [q_k, q_{k+1}]$.

Similarly, every global basis function $v_j$ can be seen as the composition of a *reference* basis function $\hat v_\alpha$ defined on the reference interval $[0,1]$, and the inverse of the element transformation $F_k(s) := q_k+h_k s$ which maps $[0,1]$ to $[q_k, q_{k+1}]$, that is:

$$
v_i(F_k(s)) = P_{ki\alpha} \hat v_\alpha(s)
$$

Where $P_{i\alpha}$ represents the numbering of the local basis function $\alpha \in [0,d]$, i.e., given the $(d+1)$ reference basis functions in $0,1$ of degree $d$, $P_{ki\alpha}$.

We will implement the numbering as a matrix $P \in R^{M,d+1}$, which returns the global index $i$, given the local element index $k$ and the local basis function index $i$, i.e.,

$$
v_{P_{k\alpha}}(F_k(s)) = \hat v_\alpha(s)
$$

Notice that, if we want to compute the derivative w.r.t. $x$ of $v_i$, computed in $F_k(s)$, as a function of the derivative of $\hat v_\alpha$ w.r.t. $s$, we need to take into account also the derivative of $F_k$, i.e., since $(v\circ F_k)' = (v' \circ F_k) F_k'$, we have

$$
v'_{P_{k\alpha}}(F_k(s)) = \hat v'_\alpha(s)/F'_k(s) = \hat v'_\alpha(s)/h_k
$$

```python
# Let's define the domain discretisazion. In 1D, this is just a set of vertices, identifying edges
a = 0
b = 1
M = 4 # Number of elements
degree = 3
# Make sure we don't choose degree = 0. This won't work. Piecewise constants cannot be continuous
assert degree > 0

# We now choose the number of quadrature points, in order to integrate *exactly*
# both (v_i, v_j) and (v'_i, v'_j)
n_quadrature_points = 2*degree+1

# To get a continuous space, we construct piecewise polynomials with support points on the boundary of the
# elements. If the degree is greater than 1, then we pick (d-1) equispaced points in the interior of the
# elements as additional support points.

# Notice that the total number of degrees of freedom is equal to the number of vertices (M+1) plus the
# number of *interior*  basis functions (d+1-2), that is: (M+1) + (d-1)*M = M*d+1
N = M*degree+1

ref_vertices = linspace(0,1,degree+1)

vertices = linspace(a,b,M+1) # Vertices of our triangulation
```

```python
# The reference element is [0,1]. We construct the mappings, the determinant of their Jacobians, and the
# reference Basis functions

def mapping(q, i):
    """
    Returns the mapping from [0,1] to T_k := [q[k], q[k+1]]
    """
    assert i < len(q)-1
    assert i >= 0
    return lambda x: q[i]+x*(q[i+1]-q[i])

def mapping_J(q,i):
    assert i < len(q)-1
    assert i >= 0
    return (q[i+1]-q[i])

def lagrange_basis(q, i):
    assert i < len(q)
    assert i >= 0
    return lambda x: prod([(x-q[j])/(q[i]-q[j]) for j in range(len(q)) if i!=j], axis=0)

# Workaround, to allow lambdify to work also on constant expressions
def np_lambdify(varname, func):
    lamb = sym.lambdify(varname, func, modules=['numpy'])
    if func.is_constant():
        return lambda t: full_like(t, lamb(t))
    else:
        return lambda t: lamb(np.array(t))

def lagrange_basis_derivative(q,i,order=1):
    t = sym.var('t')
    return np_lambdify(t, lagrange_basis(q,i)(t).diff(t,order))
```

```python
# Let's check that, on each element, F_k(0) = q[k] and F_k(1) = q[k+1]
assert abs(array([mapping(vertices, i)(0) for i in range(M)]) - vertices[:-1]).max() < 1e-16
assert abs(array([mapping(vertices, i)(1) for i in range(M)]) - vertices[1:]).max() < 1e-16
```

```python
x = linspace(0,1,51)

V = array([lagrange_basis(ref_vertices,i)(x) for i in range(degree+1)]).T
Vp = array([lagrange_basis_derivative(ref_vertices,i)(x) for i in range(degree+1)]).T
_ = [plot(x, V)]
show()
_ = [plot(x, Vp)]
show()
```

```python
# Now construct an interpolatory quadrature formula on [0,1]
q, w = numpy.polynomial.legendre.leggauss(n_quadrature_points)

q = (q+1)/2
w = w/2
```

```python
# And build a global numbering of the basis functions i = P[k,alpha]. Keep in mind that, to ensure continuity,
# we identify the global index the first basis function of each element, with the global index of the
# last basis function of the previous element

P = zeros((M,degree+1), dtype=int)

for k in range(M):
    start = k*degree
    P[k] = array(range(start,start+degree+1))

assert P.max() == N-1
print(P)
```

    [[ 0  1  2  3]
     [ 3  4  5  6]
     [ 6  7  8  9]
     [ 9 10 11 12]]

```python
# Now we build, for each segment, the transformation of quadrature points and weights, so that we can
# integrate the rhs and the matrices

Q = array([mapping(vertices,k)(q) for k in range(M)])
JxW = array([mapping_J(vertices,k)*w for k in range(M)])
```

```python
# Let's test that everything works: the integral between 0 and 1 of the function f(x) = x should return 0.5.
# Then sum_k q[k].dot(w_k) should be 0.5.

integral = 0
for k in range(M):
    integral = integral + Q[k].dot(JxW[k])

assert abs(integral - .5) < 1e-16

#same as
einsum('kq,kq', Q, JxW)
```

    0.49999999999999994

```python
# Construct the matrix B[k,j,i], defined as the value of v_i(F_k(x[j]))
B = zeros((M, len(x), N))

for k in range(M):
    B[k,:,P[k]] = V.T

# To evaluate functions and to do some plotting, also gather together all F_k(x[j]) in X[k,j]
X = array([mapping(vertices,k)(x) for k in range(M)])
```

```python
X.shape, B.shape
```

    ((4, 51), (4, 51, 13))

```python
# Reshaping X and B, we can use them to compute piecewise interpolation
X = X.flatten()
B = B.reshape((len(X),-1))

X.shape, B.shape
```

    ((204,), (204, 13))

```python
_ = plot(X, B[:,0:3])
_ = plot(vertices, 0*vertices,'ro')
```

```python
# Notice that the global support points are the image through F_k of the reference support points,
# Numbered according to the matrix P, i.e..

support_points = zeros((N,))

for k in range(M):
    support_points[P[k]] = mapping(vertices,k)(ref_vertices)

# If we chose equispaced vertices, they should be identical to linspace(a,b,N):
abs(support_points - linspace(a,b,N)).max()
```

    1.1102230246251565e-16

```python
# Let's use the runge function as an example, and compute its *piecewise* polynomial interpolation
def runge(x):
    return 1/(1+50*(x-.5)**2)
```

```python
plot(X, runge(X))
plot(support_points, runge(support_points), 'ro')
plot(X, B.dot(runge(support_points)),'r')
```

```python
# Construct a arrays Bq and Bprimeq: Bq[k,j,i] is v_i(T_k(q[j])),
# and Bprimeq[k,j,i] is v'_i(T_k(q[j]))/T'_k(q[j])
Bq = zeros((M, n_quadrature_points, N))
Bprimeq = zeros((M, n_quadrature_points, N))

Vq = array([lagrange_basis(ref_vertices,i)(q) for i in range(degree+1)]).T
Vprimeq = array([lagrange_basis_derivative(ref_vertices,i)(q) for i in range(degree+1)]).T

for k in range(M):
    Bq[k,:,P[k]] = Vq.T
    Bprimeq[k,:,P[k]] = Vprimeq.T/mapping_J(vertices,k)

Xq = Q.flatten()
Bq = Bq.reshape((len(Xq),-1))
Bprimeq = Bprimeq.reshape((len(Xq),-1))
JxWq = JxW.flatten()
```

```python
# Now compute the integrals for the mass and stiffness matrices
mass_matrix = einsum('qi,qj,q',Bq,Bq,JxWq)
stiffness_matrix = einsum('qi,qj,q',Bprimeq,Bprimeq,JxWq)
```

```python
# Notice that the stiffness matrix is non-invertible (Pure neumann problem in H1 is not uniquely solvable)
# In fact we are not imposing the boundary conditions on the space, and we need to enforce them numerically
linalg.cond(stiffness_matrix)
```

    7.89617984234469e+16

```python
# We do the same thing we did in the finite difference case: eliminate the rows corresponding to the boundary
# points, set the diagonal to 1, and set the rhs to the desired boundary condition
stiffness_matrix[0,:] = stiffness_matrix[-1,:] = 0
stiffness_matrix[0,0] = stiffness_matrix[-1,-1] = 1

# Let's check now if the matrix is well conditioned
linalg.cond(stiffness_matrix)
```

    250.25346068360452

```python
# Now let's first compute an L2 projection of the runge function, and compare with the interpolation
mass_rhs = einsum('qi,q,q', Bq, runge(Xq), JxWq)
u_projection = linalg.solve(mass_matrix, mass_rhs)
```

```python
plot(X, runge(X))
u_interpolate = runge(support_points)
plot(support_points, u_interpolate, 'ro')
plot(X, B.dot(runge(support_points)),'r')
plot(X, B.dot(u_projection),'g')
```

```python
# Finally, let's solve non trivial problem: -u'' = sin(2 pi x) with zero bc.

def rhs_function(x):
    return sin(2*pi*x)

def exact(x):
    return sin(2*pi*x)/(4*pi**2)

# Assemble the rhs, and make sure we set to zero the boundary conditions
rhs = einsum('qi,q,q', Bq, rhs_function(Xq), JxWq)
rhs[0] = rhs[-1] = 0
```

```python
u = linalg.solve(stiffness_matrix, rhs)
```

```python
plot(X, exact(X))
plot(X, B.dot(u))
plot(support_points, exact(support_points), 'o')
```

```python
# Let's compute also the L2 error:
error = sqrt(einsum('q,q', (Bq.dot(u)-exact(Xq))**2, JxWq))
error
```

    3.5160616773129974e-05

slides/Lecture 12 - LH - LAB - Introduction to PDEs - Finite Elements in 1D - Implementation.ipynb


# Timeseries Prediction

** EVERYTHING IN THIS TUTORIAL IS VERY NEW! CONSIDER IT ON AN ALPHA PHASE!**

**APIs are likely to change!**

**Examples shown in this tutorial are also in the repo `TimeseriesPrediction.jl/examples`**

Topics:

* Nature of prediction models of **DynamicalSystems.jl**
* Local model prediction
* Multi-variate local model Prediction
* Spatio-temporal Timeseries Prediction
* Docstrings

## Nature of prediction models
Suppose you have a scalar or multi-variate timeseries and you want to predict its future behaviour.

You can either take your *neural-network/machine-learning hammer* and lots of computing power **or** you can use methods from nonlinear dynamics and chaos.

**DynamicalSystems.jl** follows the second approach. This road is not only surprisingly powerful, but also much, **much** simpler.

---

# Local Model Prediction

Local model prediction does something very simple: it makes a prediction of a state, by finding the future of similar (*neighboring*) states! Then it uses the predicted state as a new state from which other predictions can be made!

Yeap, that simple.

Let's see how well this method fares in a simple system, the Roessler system (3D & chaotic):

$$
\begin{aligned}
\dot{x} &= -y-z \\
\dot{y} &= x+ay \\
\dot{z} &= b + z(x-c)
\end{aligned}
$$

```julia
using DynamicalSystems

# This initial condition gives a good prediction:
u0_good = [0.065081, 0.917503, 0.300242]

ross = Systems.roessler(u0_good)
```

    3-dimensional continuous dynamical system
     state:     [0.065081, 0.917503, 0.300242]
     e.o.m.:    DynamicalSystemsBase.Systems.roessler_eom
     in-place?  false
     jacobian:  DynamicalSystemsBase.Systems.roessler_jacob

Let's get a "measurement" from the roessler system

```julia
dt = 0.1 # sampling rate
tf = 1000.0 # final time
tr = trajectory(ross, tf; dt = dt)

# This is the measurement
s = tr[50:end, 2] # we skip the first points, they are transient
# This is the accompanying time vector:
timevec = collect(0:dt:tf)[50:end];
```

How does this timeseries look?

```julia
using PyPlot; figure(figsize = (8,4))
plot(timevec, s, lw = 1.0);
```

Please note: these are chaotic oscillations, the system is *not* periodic for the chosen (default) parameter values!

Alright, so we have a recorded some timeseries of length:

```julia
length(s)
```

9952

And now we want to predict!

Let's see the prediction function in action! The function to use is
```julia
localmodel_tsp(s, D::Int, τ, p::Int; kwargs...)
```
Here `s` is the timeseries to be predicted. `D, τ` are the values of the `Reconstruction` that has to be made from `s`. The last argument `p` is simply the amount of points to predict!

The `Reconstruction` idea and functions were introduced in the tutorial "Delay Coordinates Embedding".

This local model prediction method assumes that the system is on some kind of chaotic attractor. This is why it is crucial to reconstruct a signal before using the method!

Let's use only a first part of the timeseries as a "training set"

```julia
N = length(s)
N_train = 1000
s_train = s[1:N_train];
```

and then use the rest of the timeseries to compare with the prediction

```julia
s_test = s[N_train+1:end];
```

Here we define the parameters to make a prediction:

```julia
τ = 17
D = 3
p = 500

s_pred = localmodel_tsp(s_train, D, τ, p)
# prediction always includes last point of `s_train`
```

    501-element Array{Float64,1}:
     -1.01244
     -0.618513
     -0.25985
      0.107845
      0.481012
      0.855973
      1.22897
      1.59618
      1.95381
      2.29806
      2.62522
      2.93168
      3.21399
      ⋮
      6.92972
      7.25001
      7.49616
      7.66678
      7.76051
      7.77608
      7.71248
      7.56916
      7.34617
      7.04425
      6.66491
      6.21044

Let's plot!

```julia
figure(figsize=(8,4))
past = 100
plot(timevec[N_train-past:N_train+1], s[N_train-past:N_train+1], color = "C1", label = "timeseries")
plot(timevec[N_train:N_train+p], s[N_train:N_train+p], color = "C3", label = "real future")
plot(timevec[N_train:N_train+p], s_pred, color = "C0", linestyle = "dashed", alpha = 0.5, label = "prediction")
legend(); xlabel("\$t\$"); ylabel("\$y\$")
println("Prediction of $(p) points from $(N_train) points. i.c.: $(get_state(ross))")
```

Of course the prediction depends strongly on:

* Choosing proper `Reconstruction` parameters
* The initial condition

How did I know that the value of `τ=17` was good?

```julia
estimate_delay(s, "first_zero")
```

17

The function `localmodel_tsp` also accepts some keyword arguments which I did not discuss. These are:

  * `method = AverageLocalModel(2)` : Subtype of [`AbstractLocalModel`](@ref).
  * `ntype = FixedMassNeighborhood(2)` : Subtype of [`AbstractNeighborhood`](@ref).
  * `stepsize = 1` : Prediction step size.

We already know what does the `ntype` keyword does: it chooses a neighborhood type.

The `method` keyword chooses the method of the local prediction. There are two methods, the `AverageLocalModel`, which we already used by default, as well as the `LinearLocalModel`. Their docstrings are at the end of this tutorial.

Without explanations, their call signatures are:
```julia
AverageLocalModel(n::Int)
LinearLocalModel(n::Int, μ::Real)
LinearLocalModel(n::Int, s_min::Real, s_max::Real)
```

```julia
using BenchmarkTools
@btime localmodel_tsp($s_train, $D, $τ, $p)
println("Time for predicting $(p) points from $(N_train) points.")
```

      686.934 μs (6029 allocations: 455.09 KiB)
    Time for predicting 500 points from 1000 points.

Let's bundle all the production-prediction-plotting process into one function and play around!

```julia
function predict_roessler(N_train, p, method, u0 = rand(3); ntype = FixedMassNeighborhood(5))

    ds = Systems.roessler(u0)
    dt = 0.1
    tr = trajectory(ds, (N_train+p)÷dt; dt = dt)

    s = tr[:, 2] # actually, any of the 3 variables of the Roessler work well

    s_train = s[1:N_train]
    s_test = s[N_train+1:end]

    # parameters to predict:
    τ = 17
    D = 3

    s_pred = localmodel_tsp(s_train, D, τ, p; method = method, ntype = ntype)

    figure(figsize=(8,4))
    past = 100
    plot(timevec[N_train-past:N_train+1], s[N_train-past:N_train+1], color = "C1", label = "timeseries")
    plot(timevec[N_train:N_train+p], s[N_train:N_train+p], color = "C3", label = "real future")
    plot(timevec[N_train:N_train+p], s_pred, color = "C0", linestyle = "dashed", alpha = 0.5, label = "prediction")
    legend(); xlabel("\$t\$"); ylabel("\$y\$")
    mprint = Base.datatype_name(typeof(method))
    println("N_train = $(N_train), p = $(p), method = $(mprint), u0 = $(u0)")
    return
end

predict_roessler(3000, 500, LinearLocalModel(2, 5.0))
# Linear Local model is slower than Average local model, and in general not that
# much more powerful.
```

# Multi-Variate Prediction

On purpose I was always referring to `s` as "timeseries". There is no reason for `s` to be scalar though, this prediction method works just as well when predicting multiple timeseries. And the call signature does not change at all!

The following example demonstrates the prediction of the Lorenz96 model

$$
\frac{dx_i}{dt} = (x_{i+1}-x_{i-2})x_{i-1} - x_i + F
$$

a system that displays high-dimensional chaos and is thus very difficult to predict!

```julia
using DynamicalSystems, PyPlot

#Generate timeseries set
ds = Systems.lorenz96(5; F=8.)
ic = get_state(ds)
Δt = 0.05
s = trajectory(ds, 2100; dt=Δt)[:,1:2]

#Set Training and Test Set
N_train = 40000
p = 200
s_train = s[1:N_train,1:2]
s_test  = s[N_train:N_train+p,1:2]

#Embedding Parameters
D = 5; # total dimension of reconstruction is D*2 ! ! !
x = s[:, 1]
τ = estimate_delay(x, "first_zero")
println("Delay time estimation: $(τ)")

#Prediction
method = LinearLocalModel(2, 2.5)
method = AverageLocalModel(2)
ntype = FixedMassNeighborhood(5)
s_pred  = localmodel_tsp(s_train, D, τ, p; method = method, ntype = ntype)

figure(figsize=(12,4))
ax = subplot(121)
plot((N_train:N_train+p)*Δt, s_test[:,1], label="signal")
plot((N_train:N_train+p)*Δt, s_pred[:,1], label="prediction")
ylabel("\$x_1(n)\$")
xlabel("\$t\$")
legend()
ax = subplot(122)
plot((N_train:N_train+p)*Δt, s_test[:,2], label="signal")
plot((N_train:N_train+p)*Δt, s_pred[:,2], label="prediction")
ylabel("\$x_2(n)\$")
xlabel("\$t\$")
legend()
tight_layout();
println("Prediction p=$p of Lorenz96 Model (5 Nodes) from $N_train points")
println("i.c.: $ic")
```

# Spatio Temporal Timeseries prediction

Spatio-temporal systems are systems that depend on both space and time, i.e. *fields* (like Partial Differential Equations). These systems can also be predicted using these local model methods!

In the following sections we will see 3 examples, but there won't be any code shown for the last example (because its humongus).

See `TimeseriesPrediction.jl/examples` repository for more examples!

## Barkley Model
The Barkley model consists of 2 coupled fields each having 2 spatial dimensions, and is considered one of the simplest spatio-temporal systems

$$
\begin{align}
\frac{\partial u }{\partial t} =& \frac{1}{\epsilon} u (1-u)\left(u-\frac{v+b}{a}\right) + \nabla^2 u\nonumber \\
\frac{\partial v }{\partial t} =& u - v
\end{align}
$$

```julia
# This Algorithm of evolving the Barkley model is taken from
# http://www.scholarpedia.org/article/Barkley_model
function barkley(T, Nx, Ny)
    a = 0.75; b = 0.02; ε = 0.02

    u = zeros(Nx, Ny); v = zeros(Nx, Ny)
    U = Vector{Array{Float64,2}}()
    V = Vector{Array{Float64,2}}()

    #Initial state that creates spirals
    u[40:end,34] = 0.1; u[40:end,35] = 0.5
    u[40:end,36] = 5; v[40:end,34] = 1
    u[1:10,14] = 5; u[1:10,15] = 0.5
    u[1:10,16] = 0.1; v[1:10,17] = 1
    u[27:36,20] = 5; u[27:36,19] = 0.5
    u[27:36,18] = 0.1; v[27:36,17] = 1

    h = 0.75; Δt = 0.1; δ = 0.001
    Σ = zeros(Nx, Ny, 2)
    r = 1; s = 2

    function F(u, uth)
        if u < uth
            u/(1-(Δt/ε)*(1-u)*(u-uth))
        else
            (u + (Δt/ε)*u*(u-uth))/(1+(Δt/ε)*u*(u-uth))
        end
    end

    for m=1:T
        for i=1:Nx, j=1:Ny
            if u[i,j] < δ
                u[i,j] = Δt/h^2 * Σ[i,j,r]
                v[i,j] = (1 - Δt)* v[i,j]
            else
                uth = (v[i,j] + b)/a
                v[i,j] = v[i,j] + Δt*(u[i,j] - v[i,j])
                u[i,j] = F(u[i,j], uth) + Δt/h^2 *Σ[i,j,r]
                Σ[i,j,s] -= 4u[i,j]
                i > 1  && (Σ[i-1,j,s] += u[i,j])
                i < Nx && (Σ[i+1,j,s] += u[i,j])
                j > 1  && (Σ[i,j-1,s] += u[i,j])
                j < Ny && (Σ[i,j+1,s] += u[i,j])
            end
            Σ[i,j,r] = 0
        end
        r,s = s,r
        #V[:,:,m] .= v
        #U[:,:,m] .= u
        push!(U,copy(u))
        push!(V,copy(v))
    end
    return U,V
end
```

    barkley (generic function with 1 method)

```julia
Nx = 50
Ny = 50
Tskip = 100
Ttrain = 500
p = 20
T = Tskip + Ttrain + p

U,V = barkley(T, Nx, Ny);

Vtrain = V[Tskip + 1:Tskip + Ttrain]
Vtest  = V[Tskip + Ttrain :  T]

D = 2
τ = 1
B = 2
k = 1
c = 200

Vpred = localmodel_stts(Vtrain, D, τ, p, B, k; boundary=c)
err = [abs.(Vtest[i]-Vpred[i]) for i=1:p+1];
println("Maximum error: $(maximum(maximum.(err)))")
```

    Reconstructing
    Creating Tree
    Working on Frame 1/20
    Working on Frame 2/20
    Working on Frame 3/20
    Working on Frame 4/20
    Working on Frame 5/20
    Working on Frame 6/20
    Working on Frame 7/20
    Working on Frame 8/20
    Working on Frame 9/20
    Working on Frame 10/20
    Working on Frame 11/20
    Working on Frame 12/20
    Working on Frame 13/20
    Working on Frame 14/20
    Working on Frame 15/20
    Working on Frame 16/20
    Working on Frame 17/20
    Working on Frame 18/20
    Working on Frame 19/20
    Working on Frame 20/20
    Maximum error: 0.018341198402581194

Plotting the real evolution, prediction, and error side by side (plotting takes a *lot* of time) with `Tskip = 200, Ttrain = 1000, p = 200` produces:

## Kuramoto-Sivashinsky

This system consists of only a single field and a single spatial dimension

$$
y_t =  −y y_x − y_{xx} − y_{xxxx}
$$
where the subscripts denote partial derivatives.

We will not show code for this example, but only the results. The code is located at `TimeseriesPrediction.jl/examples/KSprediction.jl` and is quite large for a Jupyter notebook.

### Prediction

A typical prediction with parameters `D = 2, τ = 1, Β = 5, k = 1` and system parameters `L = 150` (see file for more) looks like this:

The vertical axis, which is *time*, is measured within units of the maximal Lyapunov exponent of the system Λ, which is around 0.1.

### Mean Squared Error of prediction

We now present a measure of the error of the prediction, by averaging the error values across all timesteps and across all spatial values, and then normalizing properly

The above curves are also *averaged* over 10 different initial conditions and subsequent predictions!

The parameters used for the prediction

You can compare it with e.g. figure 5 of this [Physical Review Letters article](https://doi.org/10.1103/PhysRevLett.120.024102).

# Docstrings

```julia
?localmodel_tsp
```

    search: localmodel_tsp localmodel_stts AverageLocalModel AbstractLocalModel

```
localmodel_tsp(s, D::Int, τ, p::Int; method, ntype, stepsize)
localmodel_tsp(s, p::Int; method, ntype, stepsize)
```

Perform a timeseries prediction for `p` points, using local weighted modeling [1]. The function always returns an object of the same type as `s`, which can be either a timeseries (vector) or an `AbstractDataset` (trajectory), and the returned data always contains the final point of `s` as starting point. This means that the returned data has length of `p + 1`.

If given `(s, D, τ)`, then a [`Reconstruction`](@ref) is performed on `s` with dimension `D` and delay `τ`. If given only `s` then no [`Reconstruction`](@ref) is done. Keep in mind that the intented behavior of the algorithm is to work with a reconstruction, and not "raw" data.

## Keyword Arguments

  * `method = AverageLocalModel(2)` : Subtype of [`AbstractLocalModel`](@ref).
  * `ntype = FixedMassNeighborhood(2)` : Subtype of [`AbstractNeighborhood`](@ref).
  * `stepsize = 1` : Prediction step size.

## Description

Given a query point, the function finds its neighbors using neighborhood `ntype`. Then, the neighbors `xnn` and their images `ynn` are used to make a prediction for the future of the query point, using the provided `method`. The images `ynn` are the points `xnn` shifted by `stepsize` into the future.

The algorithm is applied iteratively until a prediction of length `p` has been created, starting with the query point to be the last point of the timeseries.

## References

[1] : D. Engster & U. Parlitz, *Handbook of Time Series Analysis* Ch. 1, VCH-Wiley (2006)

```julia
?AbstractLocalModel
```

    search: AbstractLocalModel

```
AbstractLocalModel
```

Supertype of methods for making a prediction of a query point `q` using local models, following the methods of [1]. Concrete subtypes are `AverageLocalModel` and `LinearLocalModel`.

All models weight neighbors with the following weight function

$$
\begin{aligned}
ω_i = \left[ 1- \left(\frac{d_i}{d_{max}}\right)^n\right]^n
\end{aligned}
$$

with $d_i = ||x_{nn,i} -q||_2$ and degree `n`, to ensure smoothness of interpolation.

### Average Local Model

```
AverageLocalModel(n::Int)
```

The prediction is simply the weighted average of the images $y_{nn, i}$ of the neighbors $x_{nn, i}$ of the query point `q`:

$$
\begin{aligned}
y_{pred} = \frac{\sum{\omega_i^2 y_{nn,i}}}{\sum{\omega_i^2}}
\end{aligned}
$$

### Linear Local Model

```
LinearLocalModel(n::Int, μ::Real)
LinearLocalModel(n::Int, s_min::Real, s_max::Real)
```

The prediction is a weighted linear regression over the neighbors $x_{nn, i}$ of the query and their images $y_{nn,i}$ as shown in [1].

Giving either `μ` or `s_min` and `s_max` determines which type of regularization is applied.

  * `μ` : Ridge Regression

    $$
    \begin{aligned}
    f(\sigma) = \frac{\sigma^2}{\mu^2 + \sigma^2}
    \end{aligned}
    $$
  * `s_min`, `s_max` : Soft Threshold

    $$
    \begin{aligned}
    f(\sigma) = \begin{cases} 0, & \sigma < s_{min}\\
    \left(1 - \left( \frac{s_{max}-\sigma}{s_{max}-s_{min}}\right)^2 \right)^2,
    &s_{min} \leq \sigma \leq s_{max} \\
    1, & \sigma > s_{max}\end{cases}
    \end{aligned}
    $$

## References

[1] : D. Engster & U. Parlitz, *Handbook of Time Series Analysis* Ch. 1, VCH-Wiley (2006)

```julia
?localmodel_stts
```

    search: localmodel_stts localmodel_tsp

```
localmodel_stts(U::AbstractVector{<:AbstractArray{T, Φ}}, D, τ, p, B, k; kwargs...)
```

Perform a spatio-temporal timeseries prediction for `p` iterations, using local weighted modeling [1]. The function always returns an object of the same type as `U`, with each entry being a predicted state. The returned data always contains the final state of `U` as starting point (total returned length is `p+1`).

`(D, τ, B, k)` are used to make a [`STReconstruction`](@ref) on `U`. In most cases `k=1` and `τ=1,2,3` give best results.

## Keyword Arguments

  * `boundary, weighting` : Passed directly to [`STReconstruction`](@ref).
  * `method = AverageLocalModel(2)` : Subtype of [`AbstractLocalModel`](@ref).
  * `ntype = FixedMassNeighborhood(3)` : Subtype of [`AbstractNeighborhood`](@ref).
  * `printprogress = true` : To print progress done.

## Description

This method works identically to [`localmodel_tsp`](@ref), by expanding the concept from vector-states to general array-states.

## Performance Notes

Be careful when choosing `B` as memory usage and computation time depend strongly on it.

## References

[1] : U. Parlitz & C. Merkwirth, [Phys. Rev. Lett. **84**, pp 1890 (2000)](https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.84.1890)

```julia
?STReconstruction
```

    search: STReconstruction

```
STReconstruction(U::AbstractVector{<:AbstractArray}, D, τ, B, k, boundary, weighting)
 <: AbstractDataset
```

Perform spatio-temporal(ST) delay-coordinates reconstruction from a ST-timeseries `s`.

## Description

An extension of [`Reconstruction`](@ref) to support inclusion of spatial neighbors into reconstructed vectors. `B` is the number of spatial neighbors along each direction to be included. The parameter `k` indicates the spatial sampling density as described in [1].

To better understand `B`, consider a system of 2 spatial dimensions, where the state is a `Matrix`, and choose a point of the matrix to reconstruct. Giving `B = 1` will choose the current point and 8 points around it, *not* 4! For `Φ` dimensions, the number of spatial points is then given by `(2B + 1)^Φ`. Temporal embedding via `D` and `τ` is treated analogous to [`Reconstruction`](@ref). Therefore the total embedding dimension is `D*(2B + 1)^Φ`.

## Other Parameters

  * `boundary = 20` : Constant boundary value used for reconstruction of states close to the border. Pass `false` for periodic boundary conditions.
  * `weighting = (a,b)` or `nothing` : If given numbers `(a, b)`, adds `Φ` additional entries to reconstructed states. These are a spatial weighting that may be useful for considering spatially inhomogenous dynamics. Each entry is a normalized spatial coordinate $-1\leq\tilde{x}\leq 1$:

    $$
    \begin{aligned}
    \omega(\tilde{x}) = a \tilde{x} ^ b.
    \end{aligned}
    $$

## References

[1] : U. Parlitz & C. Merkwirth, [Phys. Rev. Lett. **84**, pp 1890 (2000)](https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.84.1890)


# Random Number

```python
import numpy as np
import matplotlib.pyplot as plt
```

    c:\Users\leona\anaconda3\lib\site-packages\numpy\_distributor_init.py:30: UserWarning: loaded more than 1 DLL from .libs:
    c:\Users\leona\anaconda3\lib\site-packages\numpy\.libs\libopenblas.PYQHXLVVQ7VESDPUVUADXEVJOBGHJPAY.gfortran-win_amd64.dll
    c:\Users\leona\anaconda3\lib\site-packages\numpy\.libs\libopenblas.XWYDX2IKJW2NMTWSFYNGFUWKQU3LYTCZ.gfortran-win_amd64.dll
      warnings.warn("loaded more than 1 DLL from .libs:"

```python
np.random.rand(1)
```

    array([0.30236111])

```python
np.random.randint(low = 1, high = 10)
```

    9

```python
n = 1000000
x = np.random.rand(n)
y = np.random.rand(n)
r = np.sqrt(x**2+y**2)
inside = 0
outside = 0
for i in range(len(r)):
    if r[i] <= 1 :
        inside += 1
inside
```

    784948

```python
r
```

    array([0.60574234, 0.93723258, 0.9383093 , ..., 0.60240754, 0.39812025,
           0.96042183])

```python
inside*4/n
```

    3.139792

```python
inside*4/n
```

    3.139792

```python
np.pi
```

    3.141592653589793

# Contoh

\begin{equation}
g(x) := 4+3x
\end{equation}

```python
x = np.linspace(0,5, 10000)
gx = 4+3*x
x_ = np.random.uniform(low = 0, high = 2, size = 10000)
y = np.random.uniform(low=-0, high = 10, size = 10000)
plt.figure(figsize =(16,16))
plt.plot(x,gx, 'g')
plt.xlim(-3,3)
plt.ylim(-25,25)
plt.grid()
plt.scatter(x_,y, 0.5)
plt.show()
```

\begin{equation}
f(x) := \sin(4x) + \cos(\sin(2x^3))
\end{equation}

```python
x = np.linspace(-5,5, 10000)
fx = np.sin(4*x) + np.cos(np.sin(2*x**3))
x_ = np.random.uniform(low = -5, high = 5, size = 10000)
y = np.random.uniform(low=-0.5, high = 2, size = 10000)
plt.figure(figsize =(16,16))
plt.plot(x,fx, 'g')
plt.scatter(x_,y, 0.5)
plt.show()
```


# Vectors
Vectors, and vector spaces, are fundamental to *linear algebra*, and they're used in many machine learning models. Vectors describe spatial lines and planes, enabling you to perform calculations that explore relationships in multi-dimensional space.

## What is a Vector
At its simplest, a vector is a numeric element that has both *magnitude* and *direction*. The magnitude represents a distance (for example, "2 miles") and the direction indicates which way the vector is headed (for example, "East"). Vectors are defined by an n-dimensional coordinate that describe a point in space that can be connected by a line from an arbitrary origin.

That all seems a bit complicated, so let's start with a simple, two-dimensional example. In this case, we'll have a vector that is defined by a point in a two-dimensional plane: A two dimensional coordinate consists of an *x* and a *y* value, and in this case we'll use **2** for *x* and **1** for *y*.

Our vector can be written as **v**=(2,1), but more formally we would use the following notation, in which the dimensional coordinate values for the vector are shown as a matrix:
\begin{equation}\vec{v} = \begin{bmatrix}2 \\ 1 \end{bmatrix}\end{equation}

So what exactly does that mean? Well, the coordinate is two-dimensional, and describes the movements required to get to the end point (of *head*) of the vector - in this case, we need to move 2 units in the *x* dimension, and 1 unit in the *y* dimension. Note that we don't specify a starting point for the vector - we're simply describing a destination coordinate that encapsulate the magnitide and direction of the vector. Think about it as the directions you need to follow to get to *there* from *here*, without specifying where *here* actually is!

It can help to visualize the vector, and with a two-dimensional vector, that's pretty straightforward. We just define a two-dimensional plane, choose a starting point, and plot the coordinate described by the vector relative to the starting point.

Run the code in the following cell to visualize the vector **v** (which remember is described by the coordinate (2,1)).

```python
%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt

# We'll use a numpy array for our vector
v = np.array([2,1])

# and we'll use a quiver plot to visualize it.
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, *v, scale=10, color='r')
plt.show()
```

Note that we can use a numpy array to define the vector in Python; so to create our (2,1) vector, we simply create a numpy array with the elements [2,1]. We've then used a quiver plot to visualize the vector, using the point 0,0 as the starting point (or *origin*). Our vector of (2,1) is shown as an arrow that starts at 0,0 and moves 2 units along the *x* axis (to the right) and 1 unit along the *y* axis (up).

## Calculating Vector Magnitude and Direction
We tend to work with vectors by expressing their components as *cartesian coordinates*; that is, *x* and *y* (and other dimension) values that define the number of units travelled along each dimension. So the coordinates of our (2,1) vector indicate that we must travel 2 units along the *x* axis, and *1* unit along the *y* axis.

However, you can also work with verctors in terms of their *polar coordinates*; that is coordinates that describe the magnitude and direction of the vector. The magnitude is the overall distance of the vector from tail to head, and the direction is the angle at which the vector is oriented.

### Calculating Magnitude
Calculating the magnitude of the vector from its cartesian coordinates requires measuring the distance between the arbitrary starting point and the vector head point. For a two-dimensional vector, we're actually just calculating the length of the hypotenuse in a right-angled triangle - so we could simply invoke Pythagorean theorum and calculate the square root of the sum of the squares of it's components, like this:

\begin{equation}\|\vec{v}\| = \sqrt{v_{1}\;^{2} + v_{2}\;^{2}}\end{equation}

The notation for a vector's magnitude is to surround the vector name with vertical bars - you can use single bars (for example, |**v**|) or double bars (||**v**||). Double-bars are often used to avoid confusion with absolute values. Note that the components of the vector are indicated by subscript indices (v1, v2,...v*n*),

In this case, the vector **v** has two components with values **2** and **1**, so our magnitude calculation is:

\begin{equation}\|\vec{v}\| = \sqrt{2^{2} + 1^{2}}\end{equation}

Which is:

\begin{equation}\|\vec{v}\| = \sqrt{4 + 1}\end{equation}

So:

\begin{equation}\|\vec{v}\| = \sqrt{5} \approx 2.24\end{equation}

You can run the following Python code to get a more precise result (note that the elements of a numpy array are zero-based)

```python
import math

vMag = math.sqrt(v[0]**2 + v[1]**2)
print (vMag)
```

This calculation works for vectors of any dimensionality - you just take the square root of the sum of the squared components:

\begin{equation}\|\vec{v}\| = \sqrt{v_{1}\;^{2} + v_{2}\;^{2} ... + v_{n}\;^{2}}\end{equation}

In Python, *numpy* provides a linear algebra library named **linalg** that makes it easier to work with vectors - you can use the **norm** function in the following code to calculate the magnitude of a vector:

```python
import numpy as np

vMag = np.linalg.norm(v)
print (vMag)
```

### Calculating Direction
To calculate the direction, or *amplitude*, of a vector from its cartesian coordinates, you must employ a little trigonometry. We can get the angle of the vector by calculating the *inverse tangent*; sometimes known as the *arctan* (the *tangent*  calculates an angle as a ratio - the inverse tangent, or **tan-1**, expresses this in degrees).

In any right-angled triangle, the tangent is calculated as the *opposite* over the *adjacent*. In a two dimensional vector, this is the *y* value over the *x* value, so for our **v** vector (2,1):

\begin{equation}tan(\theta) = \frac{1}{2}\end{equation}

This produces the result ***0.5***, from which we can use a calculator to calculate the inverse tangent to get the angle in degrees:

\begin{equation}\theta = tan^{-1} (0.5) \approx 26.57^{o}\end{equation}

Note that the direction angle is indicated as ***θ***.

Run the following Python code to confirm this:

```python
import math
import numpy as np

v = np.array([2,1])
vTan = v[1] / v[0]
print ('tan = ' + str(vTan))
vAtan = math.atan(vTan)
# atan returns the angle in radians, so convert to degrees
print('inverse-tan = ' + str(math.degrees(vAtan)))
```

There is an added complication however, because if the value for *x* or *y* (or both) is negative, the orientation of the vector is not standard, and a calculator can give you the wrong tan-1 value. To ensure you get the correct direction for your vector, use the following rules:
- Both *x* and *y* are positive: Use the tan-1 value.
- *x* is negative, *y* is positive: Add 180 to the tan-1 value.
- Both *x* and *y* are negative: Add 180 to the tan-1 value.
- *x* is positive, *y* is negative: Add 360 to the tan-1 value.

To understand why we need to do this, think of it this way. A vector can be pointing in any direction through a 360 degree arc.  Let's break that circle into four quadrants with the x and y axis through the center. Angles can be measured from the x axis in both the positive (counter-clockwise) and negative (clockwise) directions. We'll number the quadrants in the positive (counter-clockwise) direction (which is how we measure the *positive* angle) like this:

    2 | 1
    - o -
    3 | 4

OK, let's look at 4 example vectors

 1. Vector [2,4] has positive values for both x and y. The line for this vector travels through the point 0,0 from quadrant 3 to quadrant 1. Tan-1 of 4/2 is around 63.4 degrees, which is the positive angle from the x axis to the vector line - so this is the direction of the vector.
 2. Vector [-2,4] has a negative x and positive y. The line for this vector travels through point 0,0 from quadrant 4 to quadrant 2. Tan-1 of 4/-2 is around -64.4 degrees, which is the *negative* angle from x to the vector line; but in the wrong direction (as if the vector was travelling from quadrant 2 towards quadrant 4). So we need the opposite direction, which we get by adding 180.
 3. Vector [-2,-4] has negative x and y. The line for the vector travels through 0,0 from quadrant 1 to quadrant 3. Tan-1 of -4/-2 is around 63.4 degrees, which is the angle between the x axis and the line, but again in the opposite direction, from quadrant 3 to quadrant 1; we need to go a further 180 degrees to reflect the correct direction.
 4. Vector [2,-4] has positive x and negative y. It travels through 0,0 from quadrant 2 to quadrant 4. Tan-1 of -4/2 is around -64.4 degrees, which is the *negative* angle from the x axis to the vector line. Technically it's correct, the line is travelleing down and to the right at an angle of -63.4 degrees; but we want to express the *positive* (counter-clockwise) angle, so we add 360.

In the previous Python code, we used the *math.**atan*** function to calculate the inverse tangent from a numeric tangent. The *numpy* library includes a similar ***arctan*** function. When working with numpy arrays, you can also use the *numpy.**arctan2*** function to return the inverse tangent of an array-based vector in *radians*, and you can use the *numpy.**degrees*** function to convert this to degrees. The ***arctan2*** function automatically makes the necessary adjustment for negative *x* and *y* values.

```python
import numpy as np

v = np.array([2,1])
print ('v: ' + str(np.degrees(np.arctan2(v[1], v[0]))))

s = np.array([-3,2])
print ('s: ' + str(np.degrees(np.arctan2(s[1], s[0]))))
```

## Vector Addition
So far, we've worked with one vector at a time. What happens when you need to add two vectors.

Let's take a look at an example, we already have a vector named **v**, as defined here:
\begin{equation}\vec{v} = \begin{bmatrix}2 \\ 1 \end{bmatrix}\end{equation}
Now let's create a second vector, and called **s** like this:
\begin{equation}\vec{s} = \begin{bmatrix}-3 \\ 2 \end{bmatrix}\end{equation}

Run the cell below to create **s** and plot it together with **v**:

```python
import math
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

v = np.array([2,1])
s = np.array([-3,2])
print (s)

# Plot v and s
vecs = np.array([v,s])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['r', 'b'], scale=10)
plt.show()
```

You can see in the plot that the two vectors have different directions and magnitudes. So what happens when we add them together?

Here's the formula:
\begin{equation}\vec{z} = \vec{v}+\vec{s}\end{equation}

In terms of our vector matrices, this looks like this:
\begin{equation}\vec{z} = \begin{bmatrix}2 \\ 1 \end{bmatrix} + \begin{bmatrix}-3 \\ 2 \end{bmatrix}\end{equation}

Which gives the following result:
\begin{equation}\vec{z} = \begin{bmatrix}2 \\ 1 \end{bmatrix} + \begin{bmatrix}-3 \\ 2 \end{bmatrix} = \begin{bmatrix}-1 \\ 3 \end{bmatrix}\end{equation}

Let's verify that Python gives the same result:

```python
z = v + s
print(z)
```

So what does that look like on our plot?

```python
vecs = np.array([v,s,z])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['r', 'b', 'g'], scale=10)
plt.show()
```

So what's going on here?
Well, we added the dimensions of **s** to the dimensions of **v** to describe a new vector **z**. Let's break that down:
- The dimensions of **v** are (2,1), so from our starting point we move 2 units in the *x* dimension (across to the right) and 1 unit in the *y* dimension (up). In the plot, if you start at the (0,0) position, this is shown as the red arrow.
- Then we're adding **s**, which has dimension values (-3, 2), so we move -3 units in the *x* dimension (across to the left, because it's a negative number) and then 2 units in the *y* dimension (up). On the plot, if you start at the head of the red arrow and make these moves, you'll end up at the head of the green arrow, which represents **z**.

The same is true if you perform the addition operation the other way around and add **v** to **s**, the steps to create **s** are described by the blue arrow, and if you use that as the starting point for **v**, you'll end up at the head of the green arrow, which represents **z**.

Note on the plot that if you simply moved the tail of the blue arrow so that it started at the head of red arrow, its head would end up in the same place as the head of the green arrow; and the same would be true if you moved tail of the red arrow to the head of the blue arrow.

Vector and Matrices by Hiren/03-01-Vectors.ipynb


# First Steps

- [DifferentialEquations.jl docs](https://diffeq.sciml.ai/dev/index.html)

## Radioactive decay

How to define your model, state variables, initial (and/or boundary) conditions, and parameters.

As a simple example, the concentration of a decaying nuclear isotope could be described as an exponential decay:

$$
\frac{d}{dt}C(t) = - \lambda C(t)
$$

**State variable(s)**
- $C(t)$: The concentration of a decaying nuclear isotope.

**Parameter(s)**
- $\lambda$: The rate constant of decay. The half-life $t_{\frac{1}{2}} = \frac{ln2}{\lambda}$

### Standard operating procedures

- Define a model function representing the right-hand-side (RHS) of the sysstem.
  - Out-of-place form: `f(u, p, t)` where `u` is the state variable(s), `p` is the parameter(s), and `t` is the independent variable (usually time). The output is the right hand side (RHS) of the differential equation system.
  - In-place form: `f!(du, u, p, t)`, where the output is saved to `du`. The rest is the same as the out of place form. The in-place form has potential performance benefits since it allocates less arrays than the out-of-place form.
- Initial conditions (`u0`) for the state variable(s).
- (Optional) parameter(s) `p`.
- Define a problem (e.g. `ODEProblem`) using the modeling function (`f`), initial conditions (`u0`), simulation time span (`tspan == (tstart, tend)`), and parameter(s) `p`.
- Solve the problem by calling `solve(prob)`.

```julia
using DifferentialEquations
using Plots

# The Exponential decay ODE model, out-of-place form
expdecay(u, p, t) = p * u

p = -1.0 # Parameter
u0 = 1.0 # Initial condition
tspan = (0.0, 2.0) # Simulation start and end time points
prob = ODEProblem(expdecay, u0, tspan, p) # Define the problem
sol = solve(prob) # Solve the problem
```

### Visualization

`DifferentialEquations.jl` has a plot recipe so that `plot(sol)` directly visualizes the time series of the state variable(s).

```julia
using Plots
plot(sol) # Visualize the solution
```

### Solution handling

Docs: https://diffeq.sciml.ai/stable/basics/solution/

`sol(t)`: solution at time `t` with interpolations.

```julia
sol(1.0)  #
```

`sol.t`: time points of the solution. Notice *t=1* may not in one of the time points.

```julia
sol.t
```

`sol.u`: The solution at time points `sol.t`

```julia
sol.u
```

## The SIR model

A more complicated example is the [SIR model](https://www.maa.org/press/periodicals/loci/joma/the-sir-model-for-spread-of-disease-the-differential-equation-model) describing infectious disease spreading. There are more state variables and parameters.

$$
\begin{align}
\frac{d}{dt}S(t) &= - \beta S(t)I(t)  \\
\frac{d}{dt}I(t) &= \beta S(t)I(t)  - \gamma I(t)  \\
\frac{d}{dt}R(t) &= \gamma I(t)
\end{align}
$$

**State variable(s)**

- $S(t)$ : the fraction of susceptible people
- $I(t)$ : the fraction of infectious people
- $R(t)$ : the fraction of recovered (or removed) people

**Parameter(s)**

- $\beta$ : the rate of infection when susceptible and infectious people meet
- $\gamma$ : the rate of recovery of infectious people

```julia
using DifferentialEquations
using Plots

# SIR model, in-place form
function sir!(du, u, p ,t)
	s, i, r = u
	β, γ = p
	v1 = β * s * i
	v2 = γ * i
    du[1] = -v1
    du[2] = v1 - v2
    du[3] = v2
	return nothing
end

# Parameters of the SIR model
p = (β = 1.0, γ = 0.3)
u0 = [0.99, 0.01, 0.00]  # s, i, r
tspan = (0.0, 20.0)

# Define a problem
prob = ODEProblem(sir!, u0, tspan, p)

# Solve the problem
sol = solve(prob)

# Visualize the solution
plot(sol, label=["S" "I" "R"], legend=:right)
```

### Solution handling

`sol[i, j]`: `i`th component at timestep `j`

```julia
sol[2]
```

```julia
sol[1, 2]
```

`sol[i, :]`: the timeseries for the `i`th component.

```julia
sol[1, :]
```

`sol(t,idxs=1)`: the 1st element in time point(s) `t` with interpolation. `t` can be a scalar (single point) or an vector-like sequence. (multiple time points)

```julia
sol(10, idxs=2)
```

```julia
sol(0.0:0.1:20.0, idxs=2)
```

## Lorenz system

The Lorenz system is a system of ordinary differential equations having chaotic solutions for certain parameter values and initial conditions. ([Wikipedia](https://en.wikipedia.org/wiki/Lorenz_system))

$$
\begin{align}
  \frac{dx}{dt} &=& \sigma(y-x) \\
  \frac{dy}{dt} &=& x(\rho - z) -y \\
  \frac{dz}{dt} &=& xy - \beta z
\end{align}
$$

In this example, we will use [LabelledArrays.jl](https://github.com/SciML/LabelledArrays.jl) to get DSL-like syntax.

```julia
using LabelledArrays
using DifferentialEquations
using Plots

function lorenz!(du,u,p,t)
    du.x = p.σ*(u.y-u.x)
    du.y = u.x*(p.ρ-u.z) - u.y
    du.z = u.x*u.y - p.β*u.z
end

u0 = LVector(x=1.0, y=0.0, z=0.0)
p = LVector(σ=10.0, ρ=28.0, β=8/3)
tspan = (0.0,100.0)
prob = ODEProblem(lorenz!,u0,tspan,p)
sol = solve(prob)
```

### Visualization

`vars=(1, 2, 3)`: make a phase plot with 1st, 2nd, and the 3rd state variable. With LabelledArrays, you can use symbols instead of numbers.

```julia
plot(sol, vars=(:x, :y, :z))
```


# Solving equations of motion for a particle in magnetic and electric field

In the following we discuss the solutions for the equation of motion for a particle in a magnetic and electric field

\begin{equation}
  m\mathbf{\ddot{r} } = q\mathbf{E} + \frac{q}{c} \mathbf{\dot{r}} \times \mathbf{B}
\end{equation}

governed by the Lorentz Force. $\mathbf{B} = (0, 0, B_z)$, $\mathbf{E} = (0, E_y, E_z)$ and the initial conditions $\mathbf{r} = (x_0, y_0, z_0)$, $\mathbf{v_0} = (v_{0x}, v_{0y}, v_{0z})$.

## Import libraries

```python
import numpy as np
import matplotlib.pyplot as plt
```

## Define the solutions of the problem

The following solution for the movement in the x-y plane can be derived by solving above differential equation:

\begin{split}
  x(t) &= x_0 + \frac{E_z c}{B_z}t + \left(v_{0x} - \frac{E_z c}{Bz}\right) \frac{cm}{qB_z}\sin{\frac{qB_z}{cm}t} \\
  y(t) &= y_0 + \left(v_{0x} - \frac{E_z c}{Bz}\right) \frac{cm}{qB_z} \left(\cos{\frac{qB_z}{cm}t} - 1\right) \\
  z(t) &= 0
\end{split}

```python
def x_t(t, x_0, v_0x, E_y, B_z, q, c, m):
    return x_0 + ((E_y * c) / B_z) * t + (v_0x - (E_y * c) / B_z) * ((c * m) / (q * B_z)) * np.sin(((q * B_z) / (c * m)) * t)
```

```python
def y_t(t, y_0, v_0x, E_y, B_z, q, c, m):
    return y_0 + ((v_0x - (E_y * c) / B_z) * ((c * m) / (q * B_z))) * (np.cos(((q * B_z) / (c * m)) * t) - 1)
```

### Define initial conditions

```python
x_0 = 0
y_0 = 0

v_0x = 0

E_y = 1
B_z = 1

c = 1# 299792458
m = 1# np.exp(-31)
q = 1#1.6 * np.exp(-19)

t = 0
```

### Solve and plot

```python
t_start = 0
t_end = 200
t = np.linspace(0, t_end, 1000)

x = x_t(t, x_0, v_0x, E_y, B_z, q, c, m)
y = y_t(t, y_0, v_0x, E_y, B_z, q, c, m)
```

```python
plt.figure(figsize=(15,10))

plt.subplot(211)

plt.title('Solutions of the x(t) and y(t) coordinates')

plt.ylabel('$x(t)$ in m')
plt.xlim(xmin=t_start, xmax=t_end)
plt.plot(t, x)

plt.subplot(212)

plt.ylabel('$y(t)$ in m')
plt.xlabel('$t$ in s')
plt.xlim(xmin=t_start, xmax=t_end)
plt.plot(t, y)
```