# Implementing Neural Networks with Numpy for Absolute Beginners - Part 2: Linear Regression

##### In this tutorial, you will learn to implement Linear Regression for prediction using Numpy in detail and also visualize how the algorithm learns epoch by epoch. In addition to this, you will explore two layer Neural Networks.

In the previous tutorial, you got a very brief overview of the field of AI and learnt about perceptrons. In this tutorial, you will dig very deep into implementing a Linear Perceptron and build upon those concepts to understand Linear Regression.

## Linear Activation Function

Lets assume that there is only one input and bias to the perceptron as shown below:

>>>>>>

The resulting linear output(i.e., the sum) will be $y = m.x + b$. This is the equation of a straight line, as shown in the below figure.

>>>>>>>>>

*It must be noted here that when no activation function is used, we can say that the acitvation function is linear.*

This is a **multivariate(multiple variables) linear equation.**

Let us see how this is utilized for predicting the actual output of $y$ in the next section i.e., ***Linear Regression***.

## Linear Regression

Fitting a linear equation on a given set of data in $n$-dimensional space is called **Linear Regression**. The image below shows an example of Linear Regression.

>>>>>>

In simple words, you try to find the best values of $m$ and $b$ that best fits the set of points as shown in the above figure. When we have obtained the best possible fit, we can predict the y values given x.

A very popular example is the *housing price prediction* problem. In this problem you are given a set of values like the area of the house and the number of rooms etc. as features and you must predict the price of the house given these values.

So the big question is... How does the prediction algorithm work? How does it learn to predict?

Let's learn this on the go!

Let's start importing the required packages.

```python
# Numpy for efficient Matrix and mathematical operations.
import numpy as np

# Pandas for table and other related operations
import pandas as pd

# Matplotlib for visualizing graphs
import matplotlib.pyplot as plt
from matplotlib.pylab import rcParams

# Sklearn for creating a dataset
from sklearn.datasets import make_regression

# train_test_split for splitting the data into training and testing data
from sklearn.model_selection import train_test_split

% matplotlib inline

# Set parameters for plotting
params = {'axes.titlesize': 'xx-large',               # Set title size
          'axes.labelsize': 'x-large',                # Set label size
          'figure.figsize': (8, 6)                    # Set a figure Size
}

rcParams.update(params)
```

You'll use the sklearn dataset generator for creating the dataset. You shall also the sklearn package for splitting the data into training and test data. If you are not aware of sklearn, it is a rich package with many machine learning algorithms. Although, you get prebuilt functions for performing linear regression, you are going to build it from scratch.

For creating the dataset, you must first set a list of hyperparameters $-$ while $m$ and $b$ are parameters, the number of samples, the number of input features, the number of neurons, the learning rate, the number of iterations/epochs for training etc. are called hyperparameters. You shall learn about these hyperparameters as you implement the algorithm.

For now, you shall set the number of training samples, the number of input features, the learning rate and epochs.You shall understand learning rate and epochs in a short while.

```python
# Sample size
M = 200

# No. of input features
n = 1

# Learning Rate - Define during explanation
l_r = 0.05

# Number of iterations for updates - Define during explanation
epochs = 51
```

Your first task would be to import or generate the data. In this tutorial, you'll generate the dataset using `sklearn`'s `make_regression` function.

For purpose of learning, we shall keep the number of features minimal so that it is easy to visualize.
Hence, you must choose only one feature.

```python
X, y = make_regression(n_samples=M, n_features=n, n_informative=n,
                             n_targets=1, random_state=42, noise=10)
```

Now, it's time to visualize what the data generator has cooked up!

```python
def plot_graph(X, y):

    # Plot the original set of datapoints
    _ = plt.scatter(X, y, alpha=0.8)

    _ = plt.title('Plot of Datapoints generated')
    _ = plt.xlabel('x')
    _ = plt.ylabel('y')

    plt.show()
```

```python
plot_graph(X, y)
```

Let's check the shape of the vectors for consistency.

```python
print('Shape of vector X:', X.shape)
print('Shape of vector y:', y.shape)
```

    Shape of vector X: (200, 1)
    Shape of vector y: (200,)

```python
# Function to reset the sizes
def reset_sizes(*args):

    return tuple(arg.reshape((arg.shape[0], 1)) for arg in args)
```

```python
# Reset the size from (200,) -> (200, 1)
X, y = reset_sizes(X, y)
X.shape
```

    (200, 1)

Next you will have to split the dataset into train and test sets, so that you can test the accuracy of the regression model using a part of the dataset once you have trained the model.

Now let's split the data into train set and test set. You shall also reset the sizes so there is no discrepency in doing matrix computations.

```python
# Split the data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
```

In our case, the training set is 80% and the test set is 20%.

Let's check the shape of the Train and Test datasets created.

```python
print(X_train.shape, y_train.shape)
print(X_test.shape, y_test.shape)
```

    (160, 1) (160, 1)
    (40, 1) (40, 1)

As you can see, 80% of the data i.e., 80% of 200 data points is 160 which is correct.

The next step is to randomly generate a line with a random slope and an intercept(bias). The goal is to achieve the best fit for the line.

```python
# Function to generate parameters of the linear regression model, m & b.
def init_params():

    m = np.random.normal(scale=10)
    b = np.random.normal(scale=10)

    return m, b
```

```python
# Call function to generate paramets
m, b = init_params()
```

Now, given $m$ & $b$, we can plot the line so generated.

Let's update the function ```plot_graph``` to show the predicted line too.

```python
def plot_graph(dataset, pred_line=None):

    X, y = dataset['X'], dataset['y']

    # Plot the set of datapoints
    _ = plt.scatter(X, y, alpha=0.8)

    if(pred_line != None):

        x_line, y_line = pred_line['x_line'], pred_line['y_line']

        _ = plt.plot(x_line, y_line, linewidth=2, markersize=12, color='red', alpha=0.8)      # Plot the randomly generated line

        _ = plt.title('Random Line on set of Datapoints')

    else:
        _ = plt.title('Plot of Datapoints')

    _ = plt.xlabel('x')
    _ = plt.ylabel('y')

    plt.show()
```

```python
# Function to plot predicted line
def plot_pred_line(X, y, m, b):

    # Generate a set of datapoints on x for creating a line.
    x_line = np.linspace(np.min(X), np.max(X), 10)

    # Calculate the corresponding y with random values of m & b
    y_line = m * x_line + b

    dataset = {'X': X, 'y': y}

    pred_line = {'x_line': x_line, 'y_line':y_line}

    plot_graph(dataset, pred_line)

    return
```

```python
plot_pred_line(X_train, y_train, m, b)
```

Since the line is now generated, you'll need to predict the values it is producing for a given value of $x$. From this value, all there is to do is to calculate their mean squared error.

Let us predict the values of $y(y_{pred})$ from the parameters $m$ & $b$ given the datapoints $X_{train}$ by defining a function ```forward_prop```

```python
def forward_prop(X, m, b):

    y_pred = m * X + b

    return y_pred
```

```python
y_pred = forward_prop(X_train, m, b)
```

## Cost/Loss Function

Now, that you have both the corresding values for $X_{train}$ and the predicted values for $y(y_{pred})$ you'll calculate the Cost/Error/Loss Function. We shall stick to the term Loss.

For the $Loss$, you must compute the $Mean \space Squared \space Error$ and minimize it.

The $Loss(Mean \space Squared \space Error)$ is:
\begin{equation}
MSE = (y'^{(i)} - y^{(i)})^2
\end{equation}

Summing over all $M$ examples, we obtain the $Cost/Loss \space fn.$ as below:

\begin{equation}
L = \frac{1}{2M}\sum_{i=0}^M(y'^{(i)} - y^{(i)})^2
\end{equation}

Hence, our goal would be to minimize the $Loss$ thereby fitting the curve.

Let us now codify this.

We shall also save each value of $loss$ that will be computed.

```python
def compute_loss(y, y_pred):
    loss = 1 / 2 * np.mean((y_pred - y)**2)

    return loss
```

```python
losses = []

compute_loss(y_train, y_pred)
```

    4005.265725705774

Lets modify the above ```plot_graph``` function defined above to plot the Loss too.

```python
def plot_graph(dataset, pred_line=None, losses=None):

    plots = 2 if losses!=None else 1

    fig = plt.figure(figsize=(8 * plots, 6))

    X, y = dataset['X'], dataset['y']

    ax1 = fig.add_subplot(1, plots, 1)
    ax1.scatter(X, y, alpha=0.8)                                # Plot the original set of datapoints

    if(pred_line != None):

        x_line, y_line = pred_line['x_line'], pred_line['y_line']

        ax1.plot(x_line, y_line, linewidth=2, markersize=12, color='red', alpha=0.8)      # Plot the randomly generated line

        ax1.set_title('Predicted Line on set of Datapoints')

    else:
        ax1.set_title('Plot of Datapoints generated')

    ax1.set_xlabel('x')
    ax1.set_ylabel('y')

    if(losses!=None):
        ax2 = fig.add_subplot(1, plots, 2)
        ax2.plot(np.arange(len(losses)), losses, marker='o')

        ax2.set_xlabel('Epoch')
        ax2.set_ylabel('Loss')
        ax2.set_title('Loss')

    plt.show()
```

```python
def plot_pred_line(X, y, m, b,losses=None):

    # Generate a set of datapoints on x for creating a line.
    # We shall consider the range of X_train for generating the line so that the line superposes the datapoints.
    x_line = np.linspace(np.min(X), np.max(X), 10)

    # Calculate the corresponding y with the parameter values of m & b
    y_line = m * x_line + b

    plot_graph(dataset={'X': X, 'y': y}, pred_line={'x_line': x_line, 'y_line':y_line}, losses=losses)

    return
```

You'll shall visualize the line created from the parameters $m$ and $b$.

```python
plot_pred_line(X_train, y_train, m, b,losses)
```

Now that you have computed the loss, let's minimize it.

## Gradient Descent for Linear Regression

Since, $Loss$ is the dependent variable and $m$ & $c$ are the independent variables, we'll have to update m & b so as to find the minimum Loss.

So the next question would be, How can I update the parameters $m$ and $b$?

Let us for instance consider just a single parameter $p$ as shown below and let $t(target)$ be the actual value that has to be predicted.. We see that as $cost$ converges to the minima, the parameter $p$ reaches an optimum value for the minimum $cost$. Let's say the optimum value of $p$ is $a$.

>>

Let's gain a bit of intuition as to what the graph is saying.

It is clear from the graph, that as $p$ moves towards $a$, the Cost decreases and as it moves away from it, the cost increase.

**Now, how can we make $p$ move towards $a$, whether it is on the left or to the right of $a$ as shown in figure?**

Let us consider the $slope$ of the curve. From calculus, we know that the $slope$ of a curve at a point is given by $\frac{\mathrm dy}{\mathrm dx}$(here it is $\frac{\mathrm dL}{\mathrm dp}$ where $L \rightarrow Loss$). From the fig., when $p$ is to the right of $a$, the $slope$ is obviously $-ve$ and when it's to the right, the $slope$ would be $+ve$. But we see that if $p$ is to the left of $a$, some value must be added to $p$. Likewise, some value must subtracted when $p$ is to the right of $a$.

This means that when $slope \rightarrow -ve \implies p = p + (some \space val.)$ and when $slope \rightarrow +ve \implies p = p - (some \space val.)$ to move towards $a$

$\therefore$ We subtract $slope$ itself to $p$. This way, slope is negated so that it could be appropriately added or subtracted.The resulting equation would be,

\begin{equation}
p = p - slope \\
= p - \dfrac{\mathrm dL}{\mathrm dp} \\
\implies p = p - \mathrm dp \qquad \qquad
\end{equation}

It must also be observed that if the cost is too high, the $slope$ will be too high. Hence, while subtracting the $slope$ from $p$, $p$ value might overshoot $a$. It implies that it is necessary to decrease the value of $slope$ so that $p$ does not overshoot $a$. Therefore, we introduce a dampening factor called $Learning \space Rate (\alpha)$ to the $slope$.

What we finally obtain would be,
\begin{equation}
p = p - \alpha .\mathrm dp \qquad \qquad
\end{equation}

A shown in the figure, the trajectory taken by $p$ against $Cost$ is that of a Bel curve.

This method is called the **Gradient Descent**.

In our case, we use two parameters $m$ and $b$. Therefore, the bel curve would be *3-dimensional* as shown in the below figure.
>>>>

We compute the partial derivative of the loss function w.r.t to the parameters $m$ & $b$ to obtain i.e.,

\begin{equation}
\frac{\partial L}{\partial m} = \partial{m} = \frac{1}{M}.\sum_{i=0}^M\Big(y'^{(i)} - y^{(i)}\Big).x^{(i)}\qquad--(1)
\end{equation}

\begin{equation}
\& \qquad \qquad \qquad
\end{equation}

\begin{equation}
\quad \frac{\partial L}{\partial b} = \partial{b} = \frac{1}{M}.\sum_{i=0}^M\Big(y'^{(i)} - y^{(i)}\Big)\qquad\qquad--(2)
\end{equation}

```python
def grad_desc(m, b, X_train, y_train, y_pred):
    dm = np.mean((y_pred - y_train) * X_train)
    db = np.mean(y_pred - y_train)

    return dm, db
```

### Updating the parameters

\begin{equation}
\qquad m = m - \alpha . \partial{m} \qquad\qquad\qquad\qquad--(3)\\
\qquad\space\space b = b - \alpha . \partial{b} \qquad\qquad\qquad\qquad\quad--(4)
\end{equation}

```python
def update_params(m, b, dm, db, l_r):

    m -= l_r * dm
    b -= l_r * db

    return m, b
```

Let us define a function ```back_prop```, which calls both ```grad_desc``` and ```update_params```.

```python
def back_prop(X_train, y_train, y_pred, m, b, l_r):

    dm, db = grad_desc(m, b, X_train, y_train, y_pred)

    m, b = update_params(m, b, dm, db, l_r)

    return m, b
```

We shall now combine and call all the functions at once to see how the algorithm works.

We shall again set and tune the parameters to improve the accuracy of our linear regression model.

```python
# Sample size
M = 200

# No. of input features
n = 1

# Learning Rate - Define during explanation
l_r = 0.05

# Number of iterations for updates - Define during explanation
epochs = 61
```

```python
X, y = make_regression(n_samples=M, n_features=n, n_informative=n,
                         n_targets=1, random_state=42, noise=10)

dataset = {'X': X, 'y': y}

plot_graph(dataset)

m, b = init_params()

X, y = reset_sizes(X, y)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

losses = []

for i in range(epochs):
    y_pred = forward_prop(X_train, m, b)

    loss = compute_loss(y_train, y_pred)
    losses.append(loss)

    m, b = back_prop(X_train, y_train, y_pred, m, b, l_r)

    if(i%10==0):
        print('Epoch: ', i)
        print('Loss = ', loss)
        plot_pred_line(X_train, y_train, m, b, losses)

del losses[:]
```

## Prediction

```python
# Prediction
print('Prediction: ')
y_pred = forward_prop(X_test, m, b)
loss = compute_loss(y_test, y_pred)
print('Loss = ', loss)
accuracy = np.mean(np.fabs((y_pred - y_test) / y_test)) * 100
print('Accuracy = {}%'.format(round(accuracy, 4)))
plot_pred_line(X_test, y_test, m, b)

print('Hence \nm = ', m)
print('b = ', b)
```

## Two Layer Neural Neutwork with Linear Activation Function

Now, let us consider scaling this up to a 2 layer network as shown in the below figure.
>>>>>

From the image, we observe that there are two inputs each to the two neurons in the first layer and an output neuron in the second layer.

We can represent them in vector form as:
\begin{equation}
z_1^{[1]} = x.w_1^{[1]} \\ =
\begin{bmatrix}
x_0 &
x_1 &
x_2
\end{bmatrix} .
\begin{bmatrix}
w_{10}^{[1]} \\
w_{11}^{[1]} \\
w_{12}^{[1]}
\end{bmatrix}\\
= w_{10}^{[1]} + w_{11}^{[1]}.x_1 + w_{12}^{[1]}.x_2, \\
\end{equation}

While doing matrix computations, we'll need to take care of the dimensions and multiply. Hence, we rearrange a bit to arrive at the required output.

The expansion of the equation is not required and hence let's stick to

\begin{equation}z_1^{[1]} = x.w_1^{[1]}\end{equation}

Similarly, the value of

\begin{equation}
z_2^{[1]} =  x.w_2^{[1]}
\end{equation}

\begin{equation}
\therefore z^{[1]}  =
\begin{bmatrix}
z_1^{[1]} &
z_2^{[1]}
\end{bmatrix}
\end{equation}

Now the ouput from the $2^{nd}$ layer will be:
\begin{equation}
z^{[2]} =  z^{[1]}.w^{[2]} = w_{0}^{[2]} + w_{1}^{[2]}.z_1^{[1]} + w_{2}^{[2]}.z_2^{[1]} \\
= w_{0}^{[2]} + w_{1}^{[2]}.(x.w_1^{[1]}) + w_{2}^{[2]}.(x.w_2^{[1]}) \\
\implies z^{[2]} = w_0' + w_1'.x \qquad \text{[where, } w_0' \space \& \space w_1' \text{are some values]}
\end{equation}

From the above set of equations, we see that a neural network with a linear activation function reduces to a *linear equation*. This defeats the purpose of the whole neural network which tries to approximate to complex curves in space, so as to find the optimal solution. **Hence, it should be strictly noted that a linear function cannot be used as an activation function for the neural network,** *although it can be used only in the last layer for regression problems*.

### Conclusion

In this tutorial, you learnt
1. The concept of  perceptron and also neural networks.

2. Linear Activation functions perform the tasks of regression i.e., learn to predict and forecast values. This method is extensively called *Linear Regression* everywhere.

2. An MLP(Multi-Layer Perceptron) with a linear activation function reduces to a normal Linear Regression task. Hence, linear activations must not be used in the hiiden layers of a network. However, it can be used in the last layer for regression/prediction tasks.

In the next tutorial, you'll learn about Sigmoid Activation Function and perform Logistic Regression.

NN with Numpy 2/Neural_Networks_for_Absolute_Beginners_Part_2_Linear_Regression.ipynb


# Матрицы

Достаточно осведомленные знают, что систему линейных уравнений можно представить в матричной форме.
Действительно,

$$
X = a \cos(\alpha) - b \sin(\alpha) + x \\
Y = a \sin(\alpha) + b \cos(\alpha) + y \\
$$

можно представить как

$$
\begin{bmatrix}
    X \\
    Y
\end{bmatrix} =
\begin{bmatrix}
    \cos(\alpha) & -\sin(\alpha) \\
    \sin(\alpha) & \cos(\alpha)
\end{bmatrix}
\begin{bmatrix}
    a \\
    b
\end{bmatrix} +
\begin{bmatrix}
    x \\
    y
\end{bmatrix}
$$

или как

$$
\begin{bmatrix}
    X \\
    Y \\
    1
\end{bmatrix} =
\begin{bmatrix}
    \cos(\alpha) & -\sin(\alpha) & x \\
    \sin(\alpha) & \cos(\alpha) & y \\
    0 & 0 & 1
\end{bmatrix}
\begin{bmatrix}
    a \\
    b \\
    1
\end{bmatrix}
$$

Последний вариант удобен тем, что _комбинирование_ перемещений (смещений и поворотов) достигается простым перемножением таких матриц.

```python
from sympy import *
t = symbols("t")
x, y, z = symbols("x, y, z")
```

Рассмотрим пример комбинирования при работе в плоскости:

```python
a, b = symbols("a, b")
alpha, beta = symbols("alpha, beta")
x1, x2, y1, y2 = symbols("x_1, x_2, y_1, y_2")
X0, Y0 = symbols("X_0, Y_0")
```

Подставим начальное положение $(X_0, Y_0)$ в систему уравнений:

```python
X1 = X0 * cos(alpha) - Y0 * sin(alpha) + x1
X1
```

```python
Y1 = X0 * sin(alpha) + Y0 * cos(alpha) + y1
Y1
```

Подставим результаты во второе подобное уравнение:

```python
X2 = X1 * cos(beta) - Y1 * sin(beta) + x2
X2.simplify()
```

```python
Y2 = X1 * sin(beta) + Y1 * cos(beta) + y2
Y2.simplify()
```

Теперь запишем эти преобразования в матричном виде:

```python
XY = simplify(Matrix([
    [cos(beta), -sin(beta), x2],
    [sin(beta), cos(beta), y2],
    [0, 0, 1]
]) * Matrix([
    [cos(alpha), -sin(alpha), x1],
    [sin(alpha), cos(alpha), y1],
    [0, 0, 1]
]) * Matrix([
    [X0],
    [Y0],
    [1]
]))
XY
```

## Работа в трехмерном пространстве

```python
def rz(a):
    return Matrix([
        [cos(a), -sin(a), 0, 0],
        [sin(a), cos(a), 0, 0],
        [0, 0, 1, 0],
        [0, 0, 0, 1]
    ])

def ry(a):
    return Matrix([
        [cos(a), 0, sin(a), 0],
        [0, 1, 0, 0],
        [-sin(a), 0, cos(a), 0],
        [0, 0, 0, 1]
    ])

def rx(a):
    return Matrix([
        [1, 0, 0, 0],
        [0, cos(a), -sin(a), 0],
        [0, sin(a), cos(a), 0],
        [0, 0, 0, 1]
    ])

def trs(x, y, z):
    return Matrix([
        [1, 0, 0, x],
        [0, 1, 0, y],
        [0, 0, 1, z],
        [0, 0, 0, 1]
    ])

def vec(x, y, z):
    return Matrix([
        [x],
        [y],
        [z],
        [1]
    ])
```

Рассмотрим манипулятор с шестью степенями подвижности:

```python
q0, q1, q2, q3, q4, q5 = symbols("q_0, q_1, q_2, q_3, q_4, q_5")
l0, l1, l2, l3 = symbols("l_0, l_1, l_2, l_3")
```

```python
base = Matrix([
    [1, 0, 0, 0],
    [0, 1, 0, 0],
    [0, 0, 1, 0],
    [0, 0, 0, 1]
])
m0 = base * trs(0, 0, l0) * rz(q0)
m1 = m0 * ry(q1)
m2 = m1 * trs(0, 0, l1)
m3 = m2 * ry(q2)
m4 = m3 * trs(0, 0, l2)
m5 = m4 * rz(q3)
m6 = m5 * ry(q4)
m7 = m6 * trs(0, 0, l3)
m8 = m7 * rz(q5)
position = m8 * vec(0, 0, 0)
simplify(position)
```


# Understanding the impact of timing on defaults
> How do delays in recognising defaults impact the apparent profitability of Afterpay?

- toc: true
- badges: true
- comments: true
- categories: [Sympy,Finance,Afterpay]

- image: images/2020-10-03-Afterpay-Customer-Defaults-Part-7/header.png

## The Context

The thesis of this post is actually pretty simple. There is a delay between when customers make a transaction, and when *Afterpay* realises that they have defaulted. Because of this delay, combined with the rapid growth in the total value of transactions, defaults as a percentage of transaction value may be artificially reduced.

> Important:  Obviously I need a disclaimer. If you use anything I say as the basis for any decision, financial or otherwise, you are an idiot.

## The Model

First off, let's load in a bunch of libraries.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

from io import StringIO
import pandas as pd

import scipy.optimize

plt.rcParams["figure.figsize"] = (10,10)
from warnings import filterwarnings
filterwarnings('ignore')

```

While reading through Afterpay's releases to the markets, I came across this chart, which appears on page 3 of [this](https://www.afterpaytouch.com/images/APT_ASX-Announcement_Q2-FY18_16-Jan-Final-2.pdf) release. Let's use this to build a simple quadratic model of the reported sales.

## Loading the data

```python
#Underlying sales
csv_data = StringIO('''anz_underlying_sales_value,date,month_count
0,FY15,0
37.3,FY16,12
561.2,FY17,24
2184.6,FY18,36
4314.1,FY19,48
6566.9,FY20,60''')

df = pd.read_csv(csv_data, sep=",")
```

## Fitting a curve

Let's first fit quadratic:

```python
def quadratic(t, a, b, c):
    y = a * t**2 + b * t + c
    return y

xdata = df.month_count.values
ydata = df.anz_underlying_sales_value.values

popt, pcov = scipy.optimize.curve_fit(quadratic, xdata, ydata)

print(popt)
```

    [  2.17012649 -17.61639881 -58.725     ]

```python
x = np.linspace(0,60, 61)
y = quadratic(x, *popt)

plt.plot(xdata, ydata, 'o', label='data')
plt.plot(x,y, label='fit')

plt.title('ANZ Sales by preceding Financial Year ($M AUD)')
plt.xlabel('Months after launch')
plt.ylabel('ANZ Sales ($M AUD)')
plt.legend(loc='best')
plt.show()
```

## Delays in reporting.

So we found that we could model the annual reported sales as: $$2.170 t^2 - 17.61t - 58.725$$

The instantanious rate of sales, is given by : $$0.1808t^2 + 0.7021t -9.36$$.

Don't wory how I arrived at this, I will show how in in the appendix of this post.

```python
t = np.linspace(0,60, 61)
sales = 0.1808*t**2 + 0.7021* t - 9.36

plt.plot(sales)

plt.title('ANZ Sales by month ($M AUD)')
plt.xlabel('Months after launch')
plt.ylabel('ANZ Sales by month ($M AUD)')
plt.show()
```

Now let's model a delay of 6 months between when the transaction happens, and when *Afterpay* finally realised there was a default.

From this we can see there is a potentially significant difference between the true rate at which losses are occurring, and the rate at which we observe them occurring, at any point in time.

```python
delay = 6 #months
true_loss_rate = 0.01

losses_true = true_loss_rate*(0.1808*t**2 + 0.7021* t - 9.36)
losses_observed = true_loss_rate*(0.1808*(t-delay)**2 + 0.7021* (t-delay) - 9.36)
plt.plot(losses_observed,label='Observed')
plt.plot(losses_true,label='True')

plt.legend()
plt.title('ANZ losses by month ($M AUD)')
plt.xlabel('Months after launch')
plt.ylabel('ANZ losses by month ($M AUD)')
plt.show()
```

Now let's integrate by financial year.

```python
def integrate_by_year(y):
    integrated = np.array([0,np.sum(y[0:12]),np.sum(y[12:24]),np.sum(y[24:36]),np.sum(y[36:48]),np.sum(y[48:60])])
    return(integrated)

observed_loss_rate = integrate_by_year(losses_true)/integrate_by_year(losses_observed)

plt.plot(observed_loss_rate)

plt.title('Ratio of true losses to observed losses')
plt.xlabel('Years after launch')
plt.ylabel('Ratio of true losses to observed losses')
plt.ylim(1,2.5)
plt.xlim(2,5)
plt.grid()
plt.show()
```

## Conclusion
In conclusion, we can clearly see the impact of a delay in recognising losses, in situations where there is rapid growth. Even after years of growth, with a 6 month delay in recognising losses, the true losses could be 30-40% higher than reported.

# Appendix

### Finding an integral

So we found that we could model the annual reported sales as: $$2.170 t^2 - 17.61t - 58.725$$

Let's call this function $f(t)$

We want to find the function $g(t)$, which is the underlying rate of sales, which I claimed was : $$0.1808t^2 + 0.7021t -9.36$$.

This function, when integrated over 12 months, will give us the annual reported sales.

To help us with the algebraic manipulation, we can use [Sympy](https://docs.sympy.org/latest/index.html). An alternative is to do the algebraic manipulation by hand, but this is probably faster and more scalable.

```python
import sympy as sym
sym.init_printing(use_unicode=True)

a,b,c,d,t = sym.symbols('a b c d t')
```

So we are looking for a quadratic function, the definite integral of which is equal to $$2.170 t^2 - 17.61t - 58.725$$. Let's start by forming the definite integral.

```python
expr = sym.simplify((a*t**3 + b*t**2 + c*t + d) - (a*(t-12)**3 + b*(t-12)**2 + c*(t-12) + d))
```

```python
print(sym.collect(expr,t))
```

    36*a*t**2 + 1728*a - 144*b + 12*c + t*(-432*a + 24*b)

```python
fitted_quadratic = t**2 * 2.17012649 + t*-17.61639881 -58.725
```

### Solving for the coefficients

Let's now form a set of simultaneous equations, and solve for each of the coefficients of $$t$$.

```python
equations = []
for i in [2,1,0]:
    eq = sym.collect(expr,t).coeff(t, i)
    coeff = sym.collect(fitted_quadratic,t).coeff(t, i)
    equations.append(sym.Eq(eq,coeff))

result = sym.solve(equations,(a,b,c))

print(result)

```

    {a: 0.0602812913888889, b: 0.351046627916667, c: -9.36169642500000}

### Finding the derivative

Now all that's left to do, is to find the derivative of the indefinite integral.

```python
expr = result[a] * t**3 + result[b]*t**2 + result[c]*t

print(sym.diff(expr, t))
```

    0.180843874166667*t**2 + 0.702093255833333*t - 9.361696425

Voila!


# Generalisation and overfitting

In this notebook we will explore the issue of overfitting and how we can measure how well the models we train generalise their predictions to unseen data. This will build upon the introduction to generalisation given in the [fourth lecture](http://www.inf.ed.ac.uk/teaching/courses/mlp/2016/mlp04-learn.pdf).

## Exercise: overfitting and model complexity in a 1D regression problem

As an exercise we will consider a regression problem. In particular we will attempt to use a multiple layer network model to learn to predict output values from inputs, given a fixed set of (noisy) observations of the underlying functional relationship between inputs and outputs. The aim of the exercise will be to visualise how increasing the complexity of the model we fit to the training data effects the ability of the model to make predictions across the input space.

### Function

To keep things simple we will consider a single input-output function defined by a fourth degree polynomial (quartic)

$$ f(x) = 10 x^4 - 17 x^3 + 8 x^2 - x $$

with the observed values being the function values plus zero-mean Gaussian noise

$$ y = f(x) + 0.01 \epsilon \qquad \epsilon \sim \mathcal{N}\left(\cdot;\,0,\,1\right) $$

The inputs will be drawn from the uniform distribution on $[0, 1]$.

First import the necessary modules and seed the random number generator by running the cell below.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('ggplot')
seed = 17102016
rng = np.random.RandomState(seed)
```

Write code in the cell below to calculate a polynomial function of one dimensional inputs.

If $\boldsymbol{c}$ is a length $P$ vector of coefficients corresponding to increasing powers in the polynomial (starting from the constant zero power term up to the $P-1^{\textrm{th}}$ power) the function should correspond to the following

\begin{equation}
  f_{\textrm{polynomial}}(x,\ \boldsymbol{c}) = \sum_{p=0}^{P-1} \left( c_p x^p \right)
\end{equation}

```python
def polynomial_function(inputs, coefficients):
    """Calculates polynomial with given coefficients of an array of inputs.

    Args:
        inputs: One-dimensional array of input values of shape (num_inputs,)
        coefficients: One-dimensional array of polynomial coefficient terms
           with `coefficients[0]` corresponding to the coefficient for the
           zero order term in the polynomial (constant) and `coefficients[-1]`
           corresponding to the highest order term.

    Returns:
        One dimensional array of output values of shape (num_inputs,)

    """
    raise NotImplementedError()
```

Run the cell below to test your implementation.

```python
test_coefficients = np.array([-1., 3., 4.])
test_inputs = np.array([0., 0.5, 1., 2.])
test_outputs = np.array([-1., 1.5, 6., 21.])
assert polynomial_function(test_inputs, test_coefficients).shape == (4,), (
    'Function gives wrong shape output.'
)
assert np.allclose(polynomial_function(test_inputs, test_coefficients), test_outputs), (
    'Function gives incorrect output values.'
)
```

We now need to use the random number generator to sample input values and calculate the corresponding target outputs using your polynomial implementation with the relevant coefficients for our function. Do this by running the cell below.

```python
coefficients = np.array([0, -1., 8., -17., 10.])
input_dim, output_dim = 1, 1
noise_std = 0.01
num_data = 80
inputs = rng.uniform(size=(num_data, input_dim))
epsilons = rng.normal(size=num_data)
targets = (polynomial_function(inputs[:, 0], coefficients) +
           epsilons * noise_std)[:, None]
```

We will split the generated data points in to equal sized training and validation data sets and use these to create data provider objects which we can use to train models in our framework. As the dataset is small here we will use a batch size equal to the size of the data set. Run the cell below to split the data and set up the data provider objects.

```python
from mlp.data_providers import DataProvider
num_train = num_data // 2
batch_size = num_train
inputs_train, targets_train = inputs[:num_train], targets[:num_train]
inputs_valid, targets_valid = inputs[num_train:], targets[num_train:]
train_data = DataProvider(inputs_train, targets_train, batch_size=batch_size, rng=rng)
valid_data = DataProvider(inputs_valid, targets_valid, batch_size=batch_size, rng=rng)
```

We can now visualise the data we will be modelling. Run the cell below to plot the target outputs against inputs for both the training and validation sets. Note the clear underlying smooth functional relationship evident in the noisy data.

```python
fig = plt.figure(figsize=(8, 4))
ax = fig.add_subplot(111)
ax.plot(inputs_train[:, 0], targets_train[:, 0], '.', label='training data')
ax.plot(inputs_valid[:, 0], targets_valid[:, 0], '.', label='validation data')
ax.set_xlabel('Inputs $x$', fontsize=14)
ax.set_ylabel('Ouputs $y$', fontsize=14)
ax.legend(loc='best')
fig.tight_layout()
```

### Model

We will fit models with a varying number of parameters to the training data. As multi-layer logistic sigmoid models do not tend to perform well in regressions tasks like this we will instead use a [radial basis function (RBF) network](https://en.wikipedia.org/wiki/Radial_basis_function_network).

This model predicts the output as the weighted sum of basis functions (here Gaussian like bumps) tiled across the input space. The cell below generates a random set of weights and bias for a RBF network and plots the modelled input-output function across inputs $[0, 1]$. Run the cell below for several different number of weight parameters (specified with `num_weights` variable) to get a feel for the sort of predictions the RBF network models produce.

```python
num_weights = 15
weights_scale = 1.
bias_scale = 1.

def basis_function(x, centre, scale):
    return np.exp(-(x - centre)**2 / scale**2)

weights = rng.normal(size=num_weights) * weights_scale
bias = rng.normal() * bias_scale

centres = np.linspace(0, 1, weights.shape[0])
scale = 1. / weights.shape[0]

xs = np.linspace(0, 1, 200)
ys = np.zeros(xs.shape[0])

fig = plt.figure(figsize=(12, 4))
ax = fig.add_subplot(1, 1, 1)
for weight, centre in zip(weights, centres):
    ys += weight * basis_function(xs, centre, scale)
ys += bias
ax.plot(xs, ys)
ax.set_xlabel('Input', fontsize=14)
ax.set_ylabel('Output', fontsize=14)
```

You do not need to read in to the details of how to implement this model. All of the additional code you need to fit RBF networks is provided in the `RadialBasisFunctionLayer` in the `mlp.layers` module. The `RadialBasisFunctionLayer` class has the same interface as the layer classes we encountered in the previous lab, defining both `fprop` and `bprop` methods, and we can therefore include it as a layer in a `MultipleLayerModel` as with any other layer.

Here we will use the `RadialBasisFunctionLayer` as the first layer in a two layer model. This first layer calculates the basis function terms which are then be weighted and summed together in an `AffineLayer`, the second and final layer. This illustrates the advantage of using a modular modelling framework - we can reuse the code we previously implemented to train a quite different model architecture just by defining a new layer class.

Run the cell below to run some necessary setup code.

```python
from mlp.models import MultipleLayerModel
from mlp.layers import AffineLayer, RadialBasisFunctionLayer
from mlp.errors import SumOfSquaredDiffsError
from mlp.initialisers import ConstantInit, UniformInit
from mlp.learning_rules import GradientDescentLearningRule
from mlp.optimisers import Optimiser

# Regression problem therefore use sum of squared differences error
error = SumOfSquaredDiffsError()
# Use basic gradient descent learning rule with fixed learning rate
learning_rule = GradientDescentLearningRule(0.1)
# Initialise weights from uniform distribution and zero bias
weights_init = UniformInit(-0.1, 0.1)
biases_init = ConstantInit(0.)
# Train all models for 2000 epochs
num_epoch = 2000
```

The cell below defines RBF network models with varying number of weight parameters (equal to the number of basis functions) and fits each to the training set, recording the final training and validation set errors for the fitted models. Run it now to fit the models and calculate the error values.

```python
num_weight_list = [2, 5, 10, 25, 50, 100]
models = []
train_errors = []
valid_errors = []
for num_weight in num_weight_list:
    model = MultipleLayerModel([
        RadialBasisFunctionLayer(num_weight),
        AffineLayer(input_dim * num_weight, output_dim,
                    weights_init, biases_init)
    ])
    optimiser = Optimiser(model, error, learning_rule,
                          train_data, valid_data)
    print('-' * 80)
    print('Training model with {0} weights'.format(num_weight))
    print('-' * 80)
    _ = optimiser.train(num_epoch, -1)
    outputs_train = model.fprop(inputs_train)[-1]
    outputs_valid = model.fprop(inputs_valid)[-1]
    models.append(model)
    train_errors.append(error(outputs_train, targets_train))
    valid_errors.append(error(outputs_valid, targets_valid))
    print('  Final training set error: {0:.1e}'.format(train_errors[-1]))
    print('  Final validation set error: {0:.1e}'.format(valid_errors[-1]))
```

In the cell below write code to [plot bar charts](http://matplotlib.org/examples/api/barchart_demo.html) of the training and validation set errors for the different fitted models.

Some questions to think about from the plots:

  * Do the models with more free parameters fit the training data better or worse?
  * What does the validation set error value tell us about the models?
  * Of the models fitted here which would you say seems like it is most likely to generalise well to unseen data?
  * Do any of the models seem to be overfitting?

```python

```

Now let's visualise what the fitted model's predictions look like across the whole input space compared to the 'true' function we were trying to fit.

In the cell below, for each of the fitted models stored in the `models` list above:
  * Compute output predictions for the model across a linearly spaced series of 500 input points between 0 and 1 in the input space.
  * Plot the computed predicted outputs and true function values at the corresponding inputs as line plots on the same axis (use a new axis for each model).
  * On the same axis plot the training data sets input-target pairs as points.

```python

```

You should be able to relate your answers to the questions above to what you see in these plots - ask a demonstrator if you are unsure what is going on. In particular for the models which appeared to be overfitting and generalising poorly you should now have an idea how this looks in terms of the model's predictions and how these relate to the training data points and true function values.


```python
from sympy import *
init_printing()
```

```python
alpha, a, d, theta = symbols('alpha a d theta')
```

```python
def sTx(v):
    R = Matrix([[1,0,0,0],[0,cos(v),-sin(v),0],[0, sin(v), cos(v),0],[0,0,0,1]])
    return R

def sTy(v):
    R = Matrix([[cos(v),0,sin(v),0],[0,1,0,0],[sin(v), 0, cos(v),0],[0,0,0,1]])
    return R

def sTz(v):
    R = Matrix([[cos(v),-sin(v),0,0],[sin(v),cos(v),0,0],[0, 0, 1,0],[0,0,0,1]])
    return R

def sD(v,op):
    if op.lower() == 'x':
        D = Matrix([[1,0,0,v],[0,1,0,0],[0,0,1,0],[0,0,0,1]])
    if op.lower() == 'y':
        D = Matrix([[1,0,0,0],[0,1,0,v],[0,0,1,0],[0,0,0,1]])
    if op.lower() == 'z':
        D = Matrix([[1,0,0,0],[0,1,0,0],[0,0,1,v],[0,0,0,1]])
    return D
```

```python
Tx = sTx(alpha)
Tx
```

$$\left[\begin{matrix}1 & 0 & 0 & 0\\0 & \cos{\left (\alpha \right )} & - \sin{\left (\alpha \right )} & 0\\0 & \sin{\left (\alpha \right )} & \cos{\left (\alpha \right )} & 0\\0 & 0 & 0 & 1\end{matrix}\right]$$

```python
Dx = sD(d,'x')
Dz = sD(d,'z')
Dz
```

$$\left[\begin{matrix}1 & 0 & 0 & 0\\0 & 1 & 0 & 0\\0 & 0 & 1 & d\\0 & 0 & 0 & 1\end{matrix}\right]$$

```python
def FrameT():
    return sTx(alpha)*sD(a,'x')*sTz(theta)*sD(d,'z')
```

```python
x = FrameT()
x
```

$$\left[\begin{matrix}\cos{\left (\theta \right )} & - \sin{\left (\theta \right )} & 0 & a\\\sin{\left (\theta \right )} \cos{\left (\alpha \right )} & \cos{\left (\alpha \right )} \cos{\left (\theta \right )} & - \sin{\left (\alpha \right )} & - d \sin{\left (\alpha \right )}\\\sin{\left (\alpha \right )} \sin{\left (\theta \right )} & \sin{\left (\alpha \right )} \cos{\left (\theta \right )} & \cos{\left (\alpha \right )} & d \cos{\left (\alpha \right )}\\0 & 0 & 0 & 1\end{matrix}\right]$$

```python

```


# Exercise 1 - Linear and Logistic Regression with Error Estimation

## a) Linear regression

### Program 1 - Implementation of Linear Regression (From scratch and using Sklearn)

#### AIM:

To implement linear regression in python from scratch and using scikit learn

#### FORMULA:

Equation of the regression line, $y = b_0 + b_1 x$

$$
\newcommand{\vect}{\mathbf}
\begin{align}
b_1 &= \frac{\displaystyle \sum_{i=1}^{m}(x_i-\bar{x})(y_i-\bar{y})}{\displaystyle \sum_{i=1}^{m}(x_i-\bar{x})^2}
= \frac{(\vect{x}-\bar{x})^T(\vect{y}-\bar{y})}{(\vect{x}-\bar{x})^T(\vect{x}-\bar{x})}
\\
b_0 &= \bar{y}-b_1\bar{x}
\\
\text{RMSE} &= \sqrt{\frac{1}{m}\sum_{i=1}^{m}(y_i-\hat{y_i})^2}
= \sqrt{\frac{(\vect{y}-\hat{\vect{y}})^T(\vect{y}-\hat{\vect{y}})}{m}}
\\
\text{SS}_{\text{tot}} &= \sum_{i=1}^{m}(y_i - \bar{y})^2 = (\vect{y}-\bar{y})^T(\vect{y}-\bar{y})
\\
\text{SS}_{\text{res}} &= \sum_{i=1}^{m}(y_i - \hat{y}_i)^2 = (\vect{y}-\hat{\vect{y}})^T(\vect{y}-\hat{\vect{y}})
\\
R^{2} \text{score} &= 1-\frac{\text{SS}_{\text{res}}}{\text{SS}_{\text{tot}}}
\end{align}
$$
where,
$\vect{x} = (x_1,x_2,...,x_n)$ - input vector
$\vect{y} = (y_1,y_2,...,y_n)$ - output vector
$\bar{x}$ - mean of $\vect{x}$
$\bar{y}$ - mean of $\vect{y}$
$\hat{\vect{y}} = (\hat{y}_1,\hat{y}_2,...,\hat{y}_n)$ - Predicted output vector

#### Part 1 - Importing modules and defining class for Linear Regression

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
```

```python
class LinearRegression:
    def fit(self,X,y):
        m = X.shape[0]
        X_mean, y_mean = np.mean(X), np.mean(y)
        X_mean_diff, y_mean_diff = X-X_mean, y-y_mean
        self.b1 = (X_mean_diff @ y_mean_diff) / (X_mean_diff @ X_mean_diff)
        self.b0 = y_mean - (self.b1 * X_mean)
        print(f"(b0,b1):({self.b0:.3f},{self.b1:.3f})")
        return self

    def predict(self,X):
        return self.b0 + X*self.b1

    def evaluate(self,X,y):
        y_pred = self.predict(X)
        y_diff,y_mean_diff  = y-y_pred , y-np.mean(y)
        rmse = np.sqrt(y_diff @ y_diff/X.shape[0])
        ss_tot = y_mean_diff @ y_mean_diff
        ss_res = y_diff @ y_diff
        r2 = 1 - ss_res/ss_tot
        print("Root mean squared Error:",rmse)
        print("R^2 value:",r2)
```

#### Part 2 - Ploting function for regression

```python
def regression_plot(X,y,model,title=""):
    plt.figure(figsize=(14,7))
    plt.title(title)
    plt.xlabel("Head Size(cm^3)")
    plt.ylabel("Brain Weights(grams)")

    x_line = np.array([np.min(X) - 100,np.max(X) + 100]).reshape(-1,1)
    y_line = model.predict(x_line)

    plt.scatter(X, y,c='orange', label='Original Data Points')
    plt.plot(x_line, y_line,linewidth=4, label='Regression Line')
    plt.legend()
```

#### Part 3 - Loading and processing the dataset

```python
data = pd.read_csv('./datasets/headbrain.csv')
print("size:",data.size,"; shape",data.shape)
data.head()
```

    size: 948 ; shape (237, 4)

      Gender
      Age Range
      Head Size(cm^3)
      Brain Weight(grams)

      0
      1
      1
      4512
      1530

      1
      1
      1
      3738
      1297

      2
      1
      1
      4261
      1335

      3
      1
      1
      3777
      1282

      4
      1
      1
      4177
      1590

#### Part 4 - Implementing Linear Regression from scratch

```python
X = data['Head Size(cm^3)'].values
y = data['Brain Weight(grams)'].values
```

```python
lin_reg_model= LinearRegression()
lin_reg_model.fit(X,y)
regression_plot(X,y,lin_reg_model,title="Regression Using Oridinary Least Squares Method")
lin_reg_model.evaluate(X,y)
```

#### Part 5 - Implementing Linear Regression using Scikit Learn

```python
from sklearn.linear_model import LinearRegression as SkLinearRegression
from sklearn.metrics import mean_squared_error
```

```python
# sklearn requires 2d array for X values
# thus we reshape X to X1 as follows
X1 = X.reshape(-1,1)
```

```python
sk_lin_reg_model = SkLinearRegression().fit(X1, y)

regression_plot(X1,y,sk_lin_reg_model,title="Linear Regression Using Scikit Learn")

y_hat = sk_lin_reg_model.predict(X1)
rmse = np.sqrt(mean_squared_error(y, y_hat))
r2_score = sk_lin_reg_model.score(X1, y)
print("Root Mean Squared Error:",rmse)
print("R^2 value:",r2_score)
```

## b) Logistic Regression

### Program 1 - Preprocessing and implementing Logistic Regression on titanic dataset using Scikit Learn

#### AIM:
To preprocess and implement Logistic Regression in titanic dataset using Scikit Learn

#### Part 1 - Importing modules and Loading the dataset

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

titanic_df = pd.read_csv('./datasets/titanic.csv')
titanic_df.head()
```

      PassengerId
      Survived
      Pclass
      Name
      Sex
      Age
      SibSp
      Parch
      Ticket
      Fare
      Cabin
      Embarked

      0
      1
      0
      3
      Braund, Mr. Owen Harris
      male
      22.0
      1
      0
      A/5 21171
      7.2500
      NaN
      S

      1
      2
      1
      1
      Cumings, Mrs. John Bradley (Florence Briggs Th...
      female
      38.0
      1
      0
      PC 17599
      71.2833
      C85
      C

      2
      3
      1
      3
      Heikkinen, Miss. Laina
      female
      26.0
      0
      0
      STON/O2. 3101282
      7.9250
      NaN
      S

      3
      4
      1
      1
      Futrelle, Mrs. Jacques Heath (Lily May Peel)
      female
      35.0
      1
      0
      113803
      53.1000
      C123
      S

      4
      5
      0
      3
      Allen, Mr. William Henry
      male
      35.0
      0
      0
      373450
      8.0500
      NaN
      S

#### Part 2 - Visualizing the dataset

```python
def plot(title,plot_func,*args,**kwargs):
    plt.title(title)
    plot_func(*args,**kwargs)
    plt.show()

plot('Visualize Missing Data',
   sns.heatmap,titanic_df.isnull(), cbar=False)
plot("Survival vs Sex",
    sns.countplot,x='Survived', hue='Sex', data=titanic_df)
plot("Survival vs Passenger Class",
    sns.countplot,x='Survived', hue='Pclass', data=titanic_df)
plot("Histogram of Passengers Age",
    plt.hist,titanic_df["Age"].dropna())
plot("Histogram of ship Fare",
    plt.hist,titanic_df['Fare'])
plot("Passenger class vs Age",
    sns.boxplot,x='Pclass', y='Age',data=titanic_df)
```

#### Part 3 - Dealing with missing, categorial and irrelevant  data

```python
mean_ages = {
    p_class:titanic_df[titanic_df["Pclass"]==p_class]["Age"].mean()
    for p_class in titanic_df["Pclass"].unique()
}

def impute_missing_age(columns):
    age , p_class = columns
    if pd.isnull(age):
        return mean_ages[p_class]
    return age

titanic_df['Age'] = titanic_df[['Age', 'Pclass']].apply(
    impute_missing_age,axis = 1
)
plot("Missing Passengers Age Data",
    sns.heatmap,titanic_df.isnull(), cbar=False)
```

```python
titanic_df.drop('Cabin', axis=1, inplace = True)
titanic_df.dropna(inplace = True)
sex_data = pd.get_dummies(titanic_df['Sex'], drop_first = True)
embarked_data = pd.get_dummies(titanic_df['Embarked'], drop_first = True)
titanic_df = pd.concat([titanic_df, sex_data, embarked_data], axis = 1)
titanic_df.drop(
    ['Name', 'PassengerId', 'Ticket', 'Sex', 'Embarked'],
    axis =1, inplace = True
)
titanic_df.head()
```

      Survived
      Pclass
      Age
      SibSp
      Parch
      Fare
      male
      Q
      S

      0
      0
      3
      22.0
      1
      0
      7.2500
      1
      0
      1

      1
      1
      1
      38.0
      1
      0
      71.2833
      0
      0
      0

      2
      1
      3
      26.0
      0
      0
      7.9250
      0
      0
      1

      3
      1
      1
      35.0
      1
      0
      53.1000
      0
      0
      1

      4
      0
      3
      35.0
      0
      0
      8.0500
      1
      0
      1

#### Part 4 - Implementing Logistic Regression using Scikit Learn

```python
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.preprocessing import minmax_scale

titanic_df[["Age","Fare"]] = minmax_scale(titanic_df[["Age","Fare"]])

X = titanic_df.drop('Survived', axis = 1)
y = titanic_df['Survived']
X_train, X_test, y_train, y_test =(
    train_test_split(X, y, test_size = 0.3)
)
model = LogisticRegression()
model.fit(X_train, y_train)
y_hat = model.predict(X_test)

def print_title(title): print(f"{title:^50}\n{'='*50}")

print_title("Classification Report")
print(classification_report(y_test, y_hat))
print_title("Confusion Matrix")
print(confusion_matrix(y_test, y_hat))
```

                  Classification Report
    ==================================================
                  precision    recall  f1-score   support

               0       0.84      0.87      0.85       178
               1       0.71      0.67      0.69        89

        accuracy                           0.80       267
       macro avg       0.78      0.77      0.77       267
    weighted avg       0.80      0.80      0.80       267

                     Confusion Matrix
    ==================================================
    [[154  24]
     [ 29  60]]

### Program 2 - Logistic regression for diabetes prediction( From scrath vs using Scikit Learn)

#### AIM:
To implement Logistic regression using Gradient descent from scratch and compare it with Logistic regression with Scikit Learn

#### FORMULA:

$$\text{Log-likelihood}, LL= \sum_{i=1}^{m}y_i log(p_i)+(1-y_i) log(1-p_i)$$
where, $$p_i = p( \vect{x}_i,\vect{w}) = \frac{1}{1+e^{-\vect{w}^T\vect{x}_i}}$$
$\vect{w} = (w_0,w_1,...,w_n)$ - weight vector
$\vect{x}_i = (1, x_1, x_2,...,x_n)$ - $i$th input vector

To find $\vect{w}$ such that,$$\max_{\vect{w}}{LL} = \min_{\vect{w}}(-LL)$$
gradient of $-LL$, $$\frac{d(-LL)}{d\vect{w}} = \sum_{i=1}^{m}(p_i-y_i) \vect{x}_i$$
weight update step is given by,
$$\vect{w} := \vect{w} - \alpha \frac{d(-LL)}{d\vect{w}}
= \vect{w} - \alpha \sum_{i=1}^{m}(p_i-y_i)\vect{x}_i
=\vect{w} - \alpha (\vect{p}-\vect{y})^T X$$
where $\alpha$ - learning rate

#### Part 1 - Defining Class for Logistic Regression

```python
import numpy as np
import pandas as pd
```

```python

class LogitRegression() :
    def __init__( self, learning_rate, iterations) :
        self.learning_rate = learning_rate
        self.iterations = iterations

    def p(self,X):
        return 1/(1+np.exp(-(X @ self.w)))

    def fit(self, X, y) :
        m,n = X.shape
        X = np.hstack([np.ones((m,1)),X])
        y = y.squeeze()
        self.w = np.zeros(n+1)

        for i in range(self.iterations) :
            self.w = self.w - self.learning_rate * ((self.p(X)-y) @ X)

    def predict(self, X) :
        m = X.shape[0]
        X = np.hstack([np.ones((m,1)),X])
        y_hat = np.where( self.p(X) > 0.5, 1, 0 )
        return y_hat
```

#### Part 2 - Loading and Processing Dataset

```python
from sklearn.preprocessing import minmax_scale
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
```

```python

diabetes_df = pd.read_csv( "./datasets/diabetes.csv" )
X = minmax_scale(diabetes_df.iloc[:,:-1].values)
y = diabetes_df.iloc[:,-1:].values.reshape(-1)
X_train, X_test, y_train, y_test =train_test_split(
    X, y, test_size = 1/3, random_state =6
)
```

#### Part 3 - Comparing Models

```python

models = [
    LogitRegression(learning_rate = .1, iterations = 1000),
    LogisticRegression()
]
for model in models:
    model.fit(X_train,y_train)

def compute_accuracy(model,X_test,y_test):
    y_hat = model.predict(X_test)
    return (y_hat==y_test).mean() * 100

print("Accuracy on test set by our implementation of Logistic Reg model :",
    compute_accuracy(models[0],X_test,y_test)
)
print("Accuracy on test set by sklearn model :",
    compute_accuracy(models[1],X_test,y_test)
)
```

    Accuracy on test set by our implementation of Logistic Reg model : 64.0625
    Accuracy on test set by sklearn model : 78.515625


# B-Splines FEM algorithms

Let $\Omega \subset \mathbb{R}^d$ be a computational domain that is the image of a logical domain $\mathcal{P}$, *i.e.* a unit line (in *1d*), square (in *2d*) or a cube (in *3d*) with a **mapping** function

$$
\Omega = \mathcal{G} (\mathcal{P})
$$

We consider the Poisson problem with Homogeneous Dirichlet boundary conditions:

$$
  \mbox{Find} ~  u \in H^1_0(\Omega), ~ \mbox{such that}
$$
$$
  - \nabla^2 u = f, ~~ \Omega
$$

Using a Galerkin-Rietz method, we introduce a discrete finite elements space $\mathcal{V}_h = \mathbf{span}\{ \phi_j, 1 \leq j \leq n_V \}$ for trial and test functions. Here the index $j$ is a multi-index. For example, in $2D$, we have $j = (j_1, j_2)$

Let $u_h \in \mathcal{V}_h$ such that $u_h = \sum_{j=1}^{n_V} u_j \phi_j$. Then the weak formulation for the Poisson equation writes

$$
  \sum_{j=1}^{n} u_j \int_{\Omega} \nabla \phi_j \cdot \nabla \phi_i = \int_{\Omega} f \phi_i, \quad \forall  1 \leq i \leq n
$$

which can be written in a matrix form

$$
  M U = F
$$

where $U$ denotes the coefficients $(u_j, ~ 1 \leq j \leq n)$ and $F$ is a vector consisting of the terms $\int_{\Omega} f \phi_i$ for $1 \leq i \leq n$. Finally, the matrix $M$ is called the **stiffness** matrix and its entries are defined as

$$
  M_{ij} = \int_{\Omega} \nabla \phi_j \cdot \nabla \phi_i
$$

We will denote our basis function by $b_i$ and $b_j$ rather than $\phi_i$ and $\phi_j$. In this case, in order to solve the Poisson problem, one needs to

1. Assemble the matrix $M$ of entries
$$
  M_{ij} = \int_{\Omega} \nabla b_j \cdot \nabla b_i
$$
2. Assemble the right hand side $F$ of entries
$$
  F_{i} = \int_{\Omega} f b_i
$$
3. Solve the linear system
$$
  M U = F
$$

Since at the end, we need to compute the $L^2$-norm of the error $u-u_h$, we need also to evaluate (discretize) the following (scalar) expression:

$$
\| e_h \|_{L^2}^2 := \| u-u_h \|_{L^2}^2 = \int_{\Omega} (u-u_h)^2
$$

## Discretization of the Poisson problem

Now let's go back to a matrix entry $M_{ij}$, we have
$$
M_{ij} = \sum_e \int_{Q_e} \nabla b_i \cdot \nabla b_j
$$
We know that every basis function $b_i$ is a polynomial over the element $Q_i$. We can then use one of the Gauss quadrature formulae.

**Note**:
In the case of *B-splines* we will avoid having to evaluate on the boundary of our element (B-Splines are only defined on the right, if we don't do so, evaluating derivatives may lead to wrong results). Hence we will use the Gauss-Legendre formula.

**Note**:
In general, the quadrature formulae are given on the interval $[-1, 1]$. We therefor need to remap these points for every element, in out partition.

Let's assume now that our quadrature points have been remapped, and let's call them $\xi_k$, $\omega_k$ will denote the associated weight to $\xi_k$.
In this case $M_{ij}$ can be written as

$$
M_{ij} = \sum_e \sum_k \omega_k \nabla b_i(\xi_k) \cdot \nabla b_j( \xi_k)
$$

Now let's go back to a rhs entry $F_{i}$, we have
$$
F_{i} = \sum_e \int_{Q_e} f b_i
$$
We know that every basis function $b_i$ is a polynomial over the element $Q_i$. We can then use one of the Gauss quadrature formulae.

Since our quadrature points have been remapped ( points : $\xi_k$, weights : $\omega_k$ ).

In this case $F_{i}$ can be written as

$$
F_{i} = \sum_e \sum_k \omega_k b_i(\xi_k) f( \xi_k)
$$

Now let's go back to a norm $e_h$, we have
$$
\| e_h \|_{L^2}^2 = \sum_e \int_{Q_e} (u-u_h)^2
$$

which leads to

$$
\| e_h \|_{L^2}^2 = \sum_e \sum_k \omega_k (u(\xi_k) - u_h( \xi_k))^2
$$

The missing part here, is how to evaluate the discrete field $u_h$. This will be covered in the next part.

## Expressions

When using a Finite Elements method, one needs to compute the following kind of expressions:

* **bilinear form**. Example : $\int_{\Omega} \phi_i \phi_j ~ d\Omega$
* **linear form**. Example : $\int_{\Omega} f \phi_i ~ d\Omega$
* **functional**, i.e. integral of a scalar (vector) expression. Example : $\int_{\Omega} (u-u_h)^2$

A general formal definition of a variational form $a$ is given by Eq. \ref{eq:var-form-def}, as real-valued \textit{multi-linear} map on the product of a sequence of function spaces  $\{V_j\}_{j=1}^\rho$ :

\begin{align}
  a =
  \begin{cases}
  V_\rho \times \ldots \times V_2 \times V_1 \times \mathcal{P} \rightarrow \mathbb{R}
  \\
  a(v_\rho, \dots, v_1; w_1, \dots, w_\nu, c_1, \dots, c_\eta)
  \end{cases}
  \label{eq:var-form-def}
\end{align}

where $\rho$ is the arity of the variational form, and the special cases $\rho=0,1,2$ yield to the notions of \emph{functional}, \emph{linear form} and \emph{bilinear form}, respectively.
Each of the spaces $V_j$ may be a scalar/vector function space, or a product of other function spaces.
$\mathcal{P}$ is the space of all parameters (which can be empty), containing either functions or constants (with respect to differential operators).
A function parameter is referred to as a \emph{field} and is an element of a function space, therefore it may be represented by its coefficients over a given basis.
The general definition of the space of parameters is given as $\mathcal{P} := \left( W_1 \times \ldots \times W_\nu \right) \times \mathbb{R}^\eta$ where $\left( W_i \right)_{1 \le i \le \nu}$ can be scalar or vector function spaces and $\eta$ is the number of (real) constants.
We shall refer to the parameters as \textit{free variables}, as they will not appear in the signature of the variational form, and are defined as global variables (or local to the current scope).

In the sequel, we shall see how the different expressions can be evaluated. We will developp the different needed algorithms, that we will use in the next chapters.

Since, we will end up by computing an integral over the domain, we will need the following procedures

* Assembly over the domain: this will loop over the elements of the domain. In each element, we shall call a kernel procedure that performs the computation over one single element.
* depending on the multi-linear map (expression) to evaluate, we shall have three kinds of kernels:
  - a kernel for computing expressions with arity of 2
  - a kernel for computing expressions with arity of 1
  - a kernel for computing expressions with arity of 0
* we shall not discuss for the moment the data structure used to store the results of the kernels

## Notations

In the sequel, we shall use the following arguments

* n_elements_d (d=1,2,3) number of elements of the grid in each direction
* an element will always refer to a grid element, which considered as the a tuple
* k_d (d=1,2,3) will denote the number of quadrature points in each direction, with respect to current element
* The operator $\texttt{coeff}$ will denote the local coefficients of a field over the current element
* The operator $\texttt{value}$ will denote the values of a field over the current element

## Assembly procedures

The generic form of the assembly procedure is described next. In general, it is splitted in 3 parts:

* Pre-process: select the needed data, related to the current element.
* Process: call the kernel for computations on the given element
* Post-process:

### Pre-process

For each function space (both $V_i$ and $W_i$) involved in the expression, we need to get the basis functions over the current element, as well as their spans.

The following examples are not general, since we consider only one function space.

## Evaluation of a discrete field on a given element

When a free-variable is used in the expression, one needs to evaluate it on the quadrature points. In what follows, we describe the evaluation of a discrete field, with respect to a given element

## Kernel for an expression with arity = 1

#### Computing contributions of test basis functions (1D)

#### Computing contributions of test basis functions (2D)

## Kernel for an expression with arity = 2

#### Computing contributions between test and trial basis functions (1D)

#### Computing contributions between test and trial basis functions (2D)


# Testing proposed analysis of structure factor $S(q)$

This notebook provides a proof of concept and tutorial of how the oscillatory behavior of the static structure factor post-quench is analyzed without the need for non-linear fitting.

```python
import numpy as np
import matplotlib.pyplot as plt
from numpy import pi as π

%matplotlib inline
%config InlineBackend.figure_format = 'retina'

from collections import defaultdict
import pickle

import dgutils.colors as colortools

plt.style.use('../include/notebook.mplstyle')
colors = plt.rcParams['axes.prop_cycle'].by_key()['color']

π = np.pi
```

## We assume the functional form:

\begin{equation}
f(t) = \alpha + \beta \cos(\gamma t) + \delta t
\end{equation}

and we aim to extract all parameters $\alpha,\beta,\gamma,\delta$ from a simple analysis of the function (i.e. no fitting).

### 1. Define the function and plot

```python
def f(t,α,β,γ,δ):
    return α + β*np.cos(γ*t) + δ*t
```

```python
from scipy.interpolate import interp1d
α,β,γ,δ = 1+1E-2, 0.5*(1/1.2-1.2),2.07,1E-4

t_raw = np.arange(0,10,0.1)
_f_raw = f(t_raw,α,β,γ,δ)

f_spline = interp1d(t_raw, _f_raw, kind='cubic')
t_spline = np.arange(0,t_raw[-1],0.0001)

_f_spline = f_spline(t_spline)

plt.plot(t_spline,_f_spline,'-', color=colors[-4], lw=1)
plt.plot(t_raw,_f_raw,'o', mfc='None')
plt.xlabel(r'$t$')
plt.ylabel(r'$f(t)$')
```

### Decide whether we analyze the raw data or the spline fit

```python
spline = True
if spline:
    t = t_spline
    _f = _f_spline
else:
    t = t_raw
    _f = _f_raw
```

## 2. Extract the period by looking at extrema

We also check that this works by plotting

```python
from scipy.signal import argrelextrema
max_idx = argrelextrema(_f_spline[10:], np.greater)[0]
min_idx = argrelextrema(_f_spline[10:], np.less)[0]
t_idx = np.zeros(1,dtype='int')
t_idx = np.append(t_idx,max_idx)
t_idx = np.append(t_idx,min_idx)
t_idx = np.sort(t_idx)

plt.plot(t_spline,_f_spline,'o', mfc='None')
plt.plot(t_spline[t_idx],_f_spline[t_idx], 'x', color=colors[-4])

plt.xlabel(r'$t$')
plt.ylabel(r'$f(t)$')
```

## 3. Extract the period

We extract using the first and second extrema
Compare with the actual one.

```python
γ_ext = [π/t[t_idx[1]],2*π/t[t_idx[2]]]

γ̄_ext = np.average(γ_ext)
print(f'γ_ext = {γ_ext}\n')
print(f'γ̄_ext = {γ̄_ext:.3f},\n γ_gt = {γ:.3f}')
```

    γ_ext = [2.0711976882844096, 2.0707881178497085]

    γ̄_ext = 2.071,
     γ_gt = 2.070

### 4. Integrate up to 1/2, 1 and 3/2 periods

\begin{equation}
I_{n} = \frac{\gamma}{2n\pi} \int_0^{2n\pi/\gamma} f(t).
\end{equation}

```python
from scipy import integrate
I = []
for j in range(1,4):
    Δt = (j/2)*2*π/γ̄_ext
    Δt = t[t_idx[j]]
    I.append(integrate.simps(_f[:t_idx[j]+1],t[:t_idx[j]+1])/Δt)

print(f'I = {I}')
```

    I = [1.009969444033466, 1.010221346531876, 1.0101939605338615]

## 5. Compute the drift $\delta$ from these values

\begin{align}
\delta &= \frac{2\gamma}{\pi}(I_1 - I_{1/2}) \\
&= \frac{2\gamma}{\pi}(I_{3/2}-I_1) \\
&= \frac{\gamma}{\pi}(I_{3/2}-I_{1/2})\,.
\end{align}

```python
δ_ext = [None]*3
δ_ext[0] = (2*γ̄_ext/π)*(I[1]-I[0])
δ_ext[1] = (2*γ̄_ext/π)*(I[2]-I[1])
δ_ext[2] = (γ̄_ext/π)*(I[2]-I[0])
print(f'δ_ext = {δ_ext}\n')

δ̄_ext = np.average(δ_ext)
print(f'δ̄_ext = {δ̄_ext}\n δ_gt = {δ}')
```

    δ_ext = [0.0003321170781806277, -3.610665912813717e-05, 0.00014800520952624527]

    δ̄_ext = 0.00014800520952624527
     δ_gt = 0.0001

## 6. Compute the average $\alpha$ from these values

\begin{align}
\alpha &= 2I_{1/2} - I_1 \\
&= \frac{1}{2}(3I_{1/2} - I_{3/2}) \\
&= 3I_1 - 2I_{3/2}.
\end{align}

```python
α_ext = [None]*3
α_ext[0] = 2*I[0] - I[1]
α_ext[1] = 0.5*(3*I[0] - I[2])
α_ext[2] = 3*I[1]-2*I[2]
print(f'α_ext = {α_ext}\n')

ᾱ_ext = np.average(α_ext)
print(f'ᾱ_ext = {ᾱ_ext}\n α_gt = {α}')
```

    α_ext = [1.0097175415350559, 1.009857185783268, 1.010276118527905]

    ᾱ_ext = 1.0099502819487431
     α_gt = 1.01

```python

```


```python
from sympy import *
import math
init_printing()
```

```python
x=IndexedBase('x')
y=IndexedBase('y')
p=IndexedBase('p')
q=IndexedBase('q')
alpha,omega,lam=symbols('α,ω,λ')
pr,pz,r,z=symbols('p_r,p_z,r,z')
a,b,c=symbols('a,b,c')
```

```python
H=pr*pr/4+omega*omega/(4*r*r)+(alpha+2)*pz*pz/(4*alpha)-1/(2*alpha*r)-2/sqrt(r*r+z*z);H
```

```python
H.subs(r,a*q[1]+b).subs(pr,p[1]/a).subs(z,c*q[2]).subs(pz,p[2]/c)
```

```python
H_taylor=series(H,z).removeO();H_taylor
```

```python
series(H_taylor,r,1/b,5)
```

```python
hoge=1/sqrt(r*r+z*z)
hoge
```

```python
series(series(hoge,r).removeO(),z)
```

```python
def mtaylor(funexpr,x,mu,order=1):

    nvars = len(x)
    hlist = ['__h' + str(i+1) for i in range(nvars)]
    command=''
    command="symbols('"+'  '.join(hlist) +"')"
    hvar = eval(command)
    #mtaylor is utaylor for specificly defined function
    t = symbols('t')
    #substitution
    loc_funexpr = funexpr
    for i in range(nvars):
        locvar = x[i]
        locsubs = mu[i]+t*hvar[i]
        loc_funexpr = loc_funexpr.subs(locvar,locsubs)
    #calculate taylorseries
    g = 0
    for i in range(order+1):
        g+=loc_funexpr.diff(t,i).subs(t,0)*t**i/math.factorial(i)

    #resubstitute
    for i in range(nvars):
        g = g.subs(hlist[i],x[i]-mu[i])

    g = g.subs(t,1)
    return g
```

```python
x1,x2,x3,x4,x5 = symbols('x1 x2 x3 x4 x5')
funexpr=1/(x1*x1+x2*x2)
x=[x1,x2,x3,x4,x5]
mu=[1,0,0,0,0]
mygee = mtaylor(funexpr,x,mu,order=4)
```

```python
mygee
```

```python
a=omega**(3/2)*alpha/(4*alpha+1)
b=omega**2*alpha/(4*alpha+1)
c=omega**1.5*alpha**0.5*(alpha+2)**0.25/(sqrt(2)*(4*alpha+1)**0.75)
```


# A notebook to carry out calculations

This notebook accompanies the work described in the article **An evolutionary game theory model for devaluing rhinos**.
It is used for generating plots and analysing data and verify some of the algebra, all described in the paper.

```python
import sympy as sym
import scipy as sp
import numpy as np
import pandas as pd
sym.init_printing()

import imp
tools = imp.load_source('tools', '../tools.py')

assert sym.__version__ == "1.0"
assert sp.__version__ == "0.19.0"
assert np.__version__ == "1.12.1"
```

```python
#import matplotlib
import matplotlib.pyplot as plt
import matplotlib

%matplotlib inline

font = {'size'   : 15,
        'weight': 'bold'}
matplotlib.rc('font', **font)
```

```python
from scipy.optimize import brentq
from scipy.integrate import odeint
```

Defining all our variables (with assumption that they are positive).

```python
r, s, x, theta_r, alpha, H, beta, Gamma, F, tau, N_r  = sym.symbols("r, s, x, theta_r, alpha, \
                                                                   H, beta, Gamma, F, tau, N_r", positive=True)
```

The model
----------

```python
# the theta function
tools.theta(r, s)
```

```python
# gain
tools.gain(1), tools.gain(0)
```

```python
rs = np.linspace(0, 0.999, 100)
```

```python
gain = sym.lambdify((H, r, theta_r, x, alpha),
                    H * tools.theta(r=r, s=x, theta_r=theta_r) ** -alpha)
```

```python
fig =  plt.figure();
colors = iter(matplotlib.cm.plasma(np.linspace(0, 1, 6)))
linestyles = ['-', ':', '--', '-.', '-']
for i, alphas in enumerate(np.linspace(1, 0, 5)):
    ys = [gain(10, rval, 0.3, 0.2, alphas) for rval in rs]
    c = next(colors)

    plt.plot(rs, ys, label=r'$\alpha={}$'.format(alphas), linestyle=linestyles[i],
             linewidth=2, color=c)
plt.ylabel(r'$H\theta(r, x)^{-\alpha}$')
plt.xlabel(r'$r$')

plt.legend()
plt.savefig('../images/gain_curve.pdf', bbox_inches='tight')
```

# Cost

Here we simulate time spent in the park to offer an empirical verification of the obtained formula.

```python
import random
import math

def simulate_count_of_devalued_rhinos(r_val):
    count = 0
    while random.random() < r_val:
        count += 1
    return count

def simulate_selective_time(r_val, tau_val, repetitions=500):
    times = []
    for seed in range(repetitions):
        random.seed(seed)
        devalued_rhinos_count = simulate_count_of_devalued_rhinos(r_val)
        times.append(devalued_rhinos_count + tau_val)
    return times

def simulate_indiscriminate_time(r_val, tau_val, repetitions=500):
    times = []
    N_r = math.ceil(1 / theta_r_val)
    for seed in range(repetitions):
        random.seed(seed)
        devalued_rhinos_count = simulate_count_of_devalued_rhinos(r_val)
        times.append(min(N_r, devalued_rhinos_count + 1) * tau_val)
    return times
```

```python
r_val=.2
tau_val= 1.5
repetitions = 10 ** 6

simulated_values = simulate_selective_time(r_val=r_val, tau_val=tau_val, repetitions=repetitions)
simulated_mean = np.mean(simulated_values)
theoretic_mean = tools.selective_time(r=r_val, tau=tau_val)
plt.hist(simulated_values, bins=20)
plt.axvline(simulated_mean, color="black")
plt.axvline(theoretic_mean, color="red", linestyle="dashed")
plt.title("Simulated mean = {:.02f}, theoretic mean = {:.02f}".format(simulated_mean, theoretic_mean))
```

```python
theta_r_val = .2
N_r_val = math.ceil(1 / theta_r_val)

repetitions = 10 ** 6
simulated_values = simulate_indiscriminate_time(r_val=r_val,
                                                tau_val=tau_val,
                                                repetitions=repetitions)
simulated_mean = np.mean(simulated_values)
theoretic_mean = tools.indiscriminate_time(r=r_val, tau=tau_val, N_r=N_r_val)
plt.hist(simulated_values, bins=20)
plt.axvline(simulated_mean, color="black")
plt.axvline(theoretic_mean, color="red", linestyle="dashed")
plt.title("Simulated mean = {:.02f}, theoretic mean = {:.02f}".format(simulated_mean, theoretic_mean))
```

```python
taus = np.linspace(.5, 1.5, 100)
theta_r_val = .15

fig, axarr = plt.subplots(1, 3, figsize=(15, 7))

for i, r_val in enumerate((.4, .6, .8)):
    N_r_val = math.ceil(1 / theta_r_val)

    ax = axarr[i]
    ax.plot(taus, [tools.indiscriminate_time(r=r_val, tau=tau, N_r=N_r_val) for tau in taus], label="Indiscriminate time")
    ax.plot(taus, [tools.selective_time(r=r_val, tau=tau) for tau in taus], label="Selective time")
    ax.set_xlabel("$\\tau$")
    ax.set_ylabel("Time in park")
    ax.set_title("$r={}$".format(r_val))

ax.legend(bbox_to_anchor=(-1.7, -.2, 1., .102), loc="lower left",
          ncol=2)

fig.tight_layout()
fig.savefig("../images/indiscriminate_vs_selective_time.pdf", bbox_inches="tight")
```

```python
F_val = 5
r_val = 0.4
theta_r_val = 0.15
```

```python
betas = (.95, .99, 1.05)
fig, axarr =  plt.subplots(len(betas), 1, figsize=(12,10))
for plot_index, (beta_val, s_val) in enumerate(((.95, 1), (1.05, 1), (.95, 0))):
    tpe = "Selective" if s_val == 1 else "Indisriminate"
    colors = iter(matplotlib.cm.plasma(np.linspace(0, 1, 6)))
    linestyles = ['-', ':', '--', '-.', '-']
    for i, tau_val in enumerate(np.linspace(1, 2, 5)):
        ys = [tools.cost(s=s_val,
                         r=rval,
                         tau=tau_val,
                         theta_r=theta_r_val,
                         F=F_val,
                         beta=beta_val)
              for rval in rs]
        c = next(colors)

        ax = axarr[plot_index]
        ax.plot(rs, ys, label='$\\tau={}$'.format(tau_val), linestyle=linestyles[i],
                            linewidth=2, color=c)
        ax.set_xlabel(r'$r$')
        ax.set_title("{} cost, $\\beta={}$".format(tpe, beta_val))

ax.legend(bbox_to_anchor=(0.08, -.6, 1., .102), loc="lower left",
          ncol=5)
fig.tight_layout()
fig.savefig('../images/betas_curve.pdf', bbox_inches='tight')
```

Utility
-------

One the cost and the gain of a poacher have been defined we bring all together in the utility function.

```python
tools.utility()
```

Here we verify that how we write the utility in the paper is in line with our code.

```python
expr = H * (theta_r * r * (1 - s) - r + 1) * tools.theta(r, x) ** (-alpha) - \
       F * (s * r + s * tau * (1 - r) + (1 - s) * tau * (1 - r ** (N_r))) * (1 - r) ** (beta - 1) - Gamma * (1 - s)
expr
```

```python
(tools.utility() - expr).subs({N_r: sym.ceiling(1 / theta_r)}).simplify()
```

# Observing the system

Solve the differential equations numerically and observe the outcomes:

```python
def dx(x_1, t, F_v, H_v, r_v, alpha_v, beta_v, tau_v, theta_r_v, Gamma=0):
    """
    Define the derivate of x
    """

    x_1 = x_1[0]
    f_1 = tools.utility(s=1, x=x_1, F=F_v, H=H_v, r=r_v,
                        alpha=alpha_v, beta=beta_v, tau=tau_v, theta_r=theta_r_v)
    f_2 = tools.utility(s=0, x=x_1, F=F_v, H=H_v, r=r_v,
                        alpha=alpha_v, beta=beta_v, tau=tau_v, theta_r=theta_r_v, Gamma=Gamma)

    return x_1 * (1 - x_1) * (f_1 - f_2)
```

```python
fig, axarr =  plt.subplots(nrows=3, ncols=2, figsize=(20, 10))
tss = [np.linspace(0, 100, 1000),
       np.linspace(0, 20, 1000),
       np.linspace(0, 20, 1000),
       np.linspace(0, 40, 1000),
       np.linspace(0, 20, 1000),
       np.linspace(0, 20000, 1000),
      ]
linestyles = ['-', ':', '--', '-.', '-', ':']
labels = ["0.01", "0.1", "0.5", "1-10^{-2}", "1-10^{-3}"][::-1]
starting_populations = [.01, .1, .5, .99, .999][::-1]
scenarios = [{"F_v": 5, "H_v": 50, "r": 0.54, "alpha_v": 2, "beta_v": .99, "tau": 1.5, "theta_r": 0.01},
             {"F_v": 5, "H_v": 50, "r": 0.54, "alpha_v": 2, "beta_v": .99, "tau": 1.1, "theta_r": 0.01},
             {"F_v": 5, "H_v": 50, "r": 0.6, "alpha_v": 2, "beta_v": .99, "tau": 1.8, "theta_r": 0.01},
             {"F_v": 5, "H_v": 50, "r": 0.6, "alpha_v": 2, "beta_v": .99, "tau": 1.5, "theta_r": 0.01},
             {"F_v": 5, "H_v": 192, "r":.2, "alpha_v":1.2,  "beta_v": .99, "tau":2, "theta_r": .01},
             {"F_v": 5, "H_v": 192, "r":.2, "alpha_v":1.2,  "beta_v": .99, "tau":1.5, "theta_r": .01},
            ]

for i, scenario, ts in zip(range(len(scenarios)), scenarios, tss):
    colors = iter(matplotlib.cm.plasma(np.linspace(0, 1, 6)))
    for linestyle, starting_population, label in zip(linestyles,
                                                 starting_populations,
                                                 labels):
        args = tuple(scenario[var] for var in ("F_v",  "H_v", "r", "alpha_v", "beta_v", "tau", "theta_r"))
        xs = odeint(func=dx, y0=[starting_population], t=ts, args=args)
        c = next(colors)

        ax = axarr[int(i / 2), i % 2]
        ax.plot(ts, xs, linestyle=linestyle, linewidth=2, color=c, label=r'$x(0)={}$'.format(label))
        ax.set_title(r'$r={}$ $\tau={}$'.format(scenario["r"], scenario["tau"]))
        ax.set_ylabel(r'$x$')
        ax.set_xlabel(r'$time$')

ax.legend(bbox_to_anchor=(0.6, -.4), ncol=5)
plt.tight_layout()
plt.savefig('../images/evolution_of_system.pdf', bbox_inches='tight')
```

# Calculating the Jacobian

```python
jacobian = sym.diff(x * (1 - x) * (tools.utility(s=1, x=x) - tools.utility(s=0, x=x)), x).expand().simplify()
jacobian.subs({x: sym.Symbol("a")})
```

# All poachers are selective

The condition is:

$$J(1) > 0$$

```python
jacobian.subs({x: 1}).expand().simplify()
```

```python
lower_bound = sym.solveset(jacobian.subs({x: 1}).expand().simplify(), tau).args[0]
lower_bound
```

```python
rs = np.linspace(.1, .9, 1000)
for g in (0, 5, 10):
    bounds = [lower_bound.subs(
        {F: 5, H: 50, r: r_val, alpha: 2, beta: .99, tau: 1.5, theta_r: 0.01, Gamma: g}
                            ) for r_val in rs]
    plt.plot(rs, bounds, label="$\Gamma={}$".format(g))
plt.xlabel("$r$")
plt.legend();
```

```python
sym.limit(lower_bound, r, 1, dir="-")
```

```python
sym.limit(lower_bound.subs({Gamma: 0}), r, 0)
```

# All poachers are indiscriminate

The condition is:

$$J(0) > 0$$

```python
jacobian.subs({x: 0}).expand().simplify()
```

```python
upper_bound = sym.solveset(jacobian.subs({x: 0}).expand().simplify(), tau).args[0]
upper_bound
```

```python
sym.limit(upper_bound, r, 1, dir="-")
```

# Possibility of a mixed population being stable

We first need to find \\(x^*\\) such that \\(u_1(x^*) = u_2(x^*)\\):

```python
# Not using solveset
x_mixed = sym.solve(sym.fraction((tools.utility(1, x) - tools.utility(0, x)).expand().simplify())[0], x)[0]
x_mixed
```

```python
# Need to check condition for which 0<x^*<1
```

```python
x_mixed.subs({theta_r: .01, H: 192, F: 5, beta: .99, alpha:1.2, tau:1.5, r:.2, Gamma:0})
```

# Stable population

```python
def is_converged(values, percent=10 ** -1, tol=10 ** -5):
    N = len(values)
    last = int(N * percent)
    return max(values[-last:]) - min(values[-last:]) < tol

def get_ts(exponent, number_of_points_scale=2):
    return np.linspace(0, 10 ** exponent, number_of_points_scale * 10 ** exponent)

def solve_till_convergence(scenario, starting_population=1 / 2, percent=10 ** -1, tol=10 ** -5):
    args = tuple(scenario[var] for var in ("F_v",  "H_v", "r", "alpha_v", "beta_v", "tau", "theta_r", "Gamma"))

    exponent = 1

    while exponent == 1 or (not is_converged(xs)):
        ts = get_ts(exponent)
        xs = odeint(func=dx, y0=[starting_population], t=ts, args=args)
        exponent += 1

    return ts, xs
```

```python
data = []
starting_population = 1 / 2
scenarios = [{"F_v": 5, "H_v": 50, "r": .45, "alpha_v": 2, "beta_v": .99, "tau": 2, "theta_r": .05, "Gamma": 0},
             {"F_v": 5, "H_v": 50, "r": .4, "alpha_v": 2.5, "beta_v": .99, "tau": 1.8, "theta_r": .05, "Gamma": 0},
             {"F_v": 5, "H_v": 25, "r": .45, "alpha_v": 2, "beta_v": .99, "tau": 2, "theta_r": .05, "Gamma": 0},
             {"F_v": 5, "H_v": 25, "r": .4, "alpha_v": 2.5, "beta_v": .99, "tau": 1.8, "theta_r": .05, "Gamma": 0},
             {"F_v": 5, "H_v": 25, "r": .99, "alpha_v": 2, "beta_v": .99, "tau": 2, "theta_r": .05, "Gamma": 4},
             {"F_v": 5, "H_v": 25, "r": .99, "alpha_v": 2.5, "beta_v": .99, "tau": 1.8, "theta_r": .05, "Gamma": 4},
            ]
```

```python
for scenario in scenarios:
    print(lower_bound.subs(scenario))
```

    (-r + 1)**(-alpha - beta)*(F*r*(-r + 1)**(alpha + beta) + Gamma*r*(-r + 1)**alpha - Gamma*(-r + 1)**alpha - H*r**2*theta_r + H*r*theta_r)/(F*(r - r**ceiling(1/theta_r)))
    (-r + 1)**(-alpha - beta)*(F*r*(-r + 1)**(alpha + beta) + Gamma*r*(-r + 1)**alpha - Gamma*(-r + 1)**alpha - H*r**2*theta_r + H*r*theta_r)/(F*(r - r**ceiling(1/theta_r)))
    (-r + 1)**(-alpha - beta)*(F*r*(-r + 1)**(alpha + beta) + Gamma*r*(-r + 1)**alpha - Gamma*(-r + 1)**alpha - H*r**2*theta_r + H*r*theta_r)/(F*(r - r**ceiling(1/theta_r)))
    (-r + 1)**(-alpha - beta)*(F*r*(-r + 1)**(alpha + beta) + Gamma*r*(-r + 1)**alpha - Gamma*(-r + 1)**alpha - H*r**2*theta_r + H*r*theta_r)/(F*(r - r**ceiling(1/theta_r)))
    (-r + 1)**(-alpha - beta)*(F*r*(-r + 1)**(alpha + beta) + Gamma*r*(-r + 1)**alpha - Gamma*(-r + 1)**alpha - H*r**2*theta_r + H*r*theta_r)/(F*(r - r**ceiling(1/theta_r)))
    (-r + 1)**(-alpha - beta)*(F*r*(-r + 1)**(alpha + beta) + Gamma*r*(-r + 1)**alpha - Gamma*(-r + 1)**alpha - H*r**2*theta_r + H*r*theta_r)/(F*(r - r**ceiling(1/theta_r)))

```python
def numeric_lower_bound(scenario, lower_bound=lower_bound):
    return lower_bound.subs({
        alpha: scenario["alpha_v"],
        r: scenario["r"],
        beta: scenario["beta_v"],
        F: scenario["F_v"],
        Gamma: scenario["Gamma"],
        theta_r: scenario["theta_r"],
        H: scenario["H_v"],
                            })
```

```python
numeric_lower_bound(scenario), scenario["tau"]
```

```python
number_of_points = 1000

variables = ("tau", "Gamma", "r")
linspaces = (np.linspace(.5, 2.5, number_of_points),
             np.linspace(0, 8, number_of_points),
             np.linspace(0.01, .99, number_of_points),
             )
x_labels = (r"$\tau$", "$\Gamma$", "$r$",)
stable_population_data_path = "../data/main.csv"
```

```python
for base_scenario in scenarios:
    for i, (variable, linspace, x_label) in enumerate(zip(variables, linspaces, x_labels)):
        for value in linspace:
            scenario = {key: value for key, value in base_scenario.items()}
            scenario[variable] = value

            ts, xs = solve_till_convergence(scenario=scenario)
            scenario["x"] = xs[-1][0]
            scenario["lower_bound"] = numeric_lower_bound(scenario=scenario)
            data.append(scenario)

df = pd.DataFrame(data)
df.head()
df.to_csv(stable_population_data_path)
```

Test that the theoretic condition is true:

```python
df = pd.read_csv(stable_population_data_path)
~(np.isclose(df["x"], 1, rtol=10 ** -4))
```

    array([ True,  True,  True, ...,  True,  True,  True], dtype=bool)

```python
all(
    (
        (
            (np.isclose(df["x"], 1, rtol=10 ** -4)) &  # and
            (df["tau"] > df["lower_bound"])
        ) |  # or
        (
            (~(np.isclose(df["x"], 1, rtol=10 ** -4))) &  # and
            (df["tau"] <= df["lower_bound"])
        )
    )
)
```

    True

```python
with open("../list_of_scenarios.tex", "w") as f:
    for i, scenario in enumerate(scenarios):
        f.write("\item Scenario {i}: \(H={H_v}\)  \(r={r}\) \(\\alpha={alpha_v}\), \(\\tau={tau}\), \(\\Gamma={Gamma}\)\n".format(i=i +1, **scenario))
```

```python
linestyles = ['-', ':', '--', '-.', '-', ':']
colors = list(matplotlib.cm.plasma(np.linspace(0, 1, len(scenarios) + 1)))

fig, axarr =  plt.subplots(nrows=3, ncols=1, figsize=(10, 12))

for i, (variable, x_label) in enumerate(zip(variables, x_labels)):
    ax = axarr[i]
    for j, (scenario, color, linestyle) in enumerate(zip(scenarios, colors, linestyles)):
        index = np.all(np.array([df[key] == value for key, value in scenario.items() if key != variable]), axis=0)
        sorted_df = df[index].sort_values(variable)
        ax.plot(sorted_df[variable], sorted_df["x"], color=color, linestyle=linestyle, label="Scenario: {}".format(j + 1))
        ax.set_xlabel(x_label)
        ax.set_ylabel("Evolutionary stable $x$")

ax.legend(bbox_to_anchor=(1.3, 2.1))
plt.tight_layout()
plt.savefig('../images/steady_populations.pdf', bbox_inches='tight')
```