# Linear Regression

```{note}
Linear Regression = Linear Model + Mean Square Loss<br/>
We can use either gradient descent or analytic solution to solve linear regression.<br/>
Linear Regression has nice geometric and probabilistic Interpretations.
```

## Model

Suppose $x \in \mathbb{R}^{d}$, $y \in \mathbb{R}$. Linear model is:

$$h(x) = w^{T}x + b$$

For simplicity, let:

$$x := [x,1] \in \mathbb{R}^{d + 1}$$

$$\theta := [w, b] \in \mathbb{R}^{d + 1}$$

Then linear model could be write as:

$$h(x) = \theta^{T}x$$

## Loss

Loss Function is mean square loss:

$$J(\theta) = \frac{1}{2}\sum_{i=1}^{n}(h(x^{(i)}) - y^{(i)})^{2}$$

## Update Rule

Gradient Descent:

$$\theta \to \theta - \alpha\nabla{J(\theta)}$$

Gradient of Linear Regression:

$$
\begin{equation}
\begin{split}
\frac{\partial }{\partial \theta_{j}}J(\theta) &=  \frac{\partial }{\partial \theta}_{j}\frac{1}{2}\sum_{i=1}^{n}(h(x^{(i)}) - y^{(i)})^2  \\
&=\sum_{i=1}^{n}(h(x^{(i)}) - y^{(i)})\cdot\frac{\partial }{\partial \theta_{j}}(h(x^{(i)}) - y^{(i)})\\
& =\sum_{i=1}^{n}(h(x^{(i)}) - y^{(i)})\cdot{x_{j}}^{(i)}
\end{split}
\end{equation}
$$

Combine all dimensions:

$$\theta \to \theta - \alpha\sum_{i=1}^{n}(h(x^{(i)}) - y^{(i)})\cdot{x^{(i)}} $$

Write in matrix form:

$$\theta \to \theta - \alpha{X^{T}}(X\theta-y) $$

where $X \in \mathbb{R}^{n\times{d}}, y \in \mathbb{R}^{n}$.

## Analytic Solution

From above, we have:

$$\nabla{J(\theta)} = X^{T}X\theta - X^{T}y$$

If $X^{T}X$ is invertible:

$$\theta = (X^{T}X)^{-1}X^{T}y$$

Else the equation also have solution:

$$X^{T}X\theta=0 \Rightarrow \theta^{T}X^{T}X\theta = (X\theta)^{T}X\theta=0 \Rightarrow X\theta = 0$$

$$\mbox{null}(X^{T}X) = \mbox{null}(X) \Rightarrow \mbox{range}(X^{T}X) = \mbox{range}(X^{T})$$

## Examples

```python
from sklearn.datasets import load_boston

X, y = load_boston(return_X_y=True)
X.shape, y.shape
```

    ((506, 13), (506,))

```python
from sklearn.linear_model import LinearRegression

reg = LinearRegression()
reg.fit(X, y)
```

    LinearRegression()

```python
from sklearn.metrics import mean_squared_error
mean_squared_error(y, reg.predict(X))
```

    21.894831181729202

## Geometric Interpretation

Denote linear space $S = \mbox{span}\left \{\mbox{columns of } X \right \}$, linear combination of $S$ should be written as $X\theta$.

$X\theta$ is the projection of $y$ on $S \Leftrightarrow$ $X\theta - y$ orthogonal with $S \Leftrightarrow$ orthogonal with columns of $X \Leftrightarrow X^{T}(X\theta - y)=0$

Linear regression $\Leftrightarrow$ Finding the projection of $y$ on $S$.

## Probabilistic Interpretation

Assume targets and inputs are related via:

$$y^{(i)} = \theta^{T}x^{(i)} + \epsilon^{(i)}$$

where $\epsilon^{(i)}$ is the error term and distributed IID according to Gaussian with mean 0 and variance $\sigma^{2}$:

$$p(\epsilon^{(i)}) = \frac{1}{\sqrt{2\pi}\sigma}exp\left (-\frac{(\epsilon^{(i)})^{2}}{2\sigma^{2}}\right )$$

This is equivalent to say(we should denote that $\theta$ is not a random variable here):

$$p(y^{(i)}|x^{(i)}; \theta) = \frac{1}{\sqrt{2\pi}\sigma}exp\left ( -\frac{(y^{(i)} - \theta^{T}x^{(i)})^{2}}{2\sigma^{2}}\right)$$

The likelihood function:

$$
\begin{equation}
\begin{split}
L(\theta) &= \prod_{i=1}^{n}p(y^{(i)}|x^{(i)}; \theta) \\
&= \prod_{i=1}^{n}\frac{1}{\sqrt{2\pi}\sigma}exp\left ( -\frac{(y^{(i)} - \theta^{T}x^{(i)})^{2}}{2\sigma^{2}}\right)
\end{split}
\end{equation}
$$

Maximize the log likelihood:

$$
\begin{equation}
\begin{split}
\log(L(\theta)) &= \log\prod_{i=1}^{n}\frac{1}{\sqrt{2\pi}\sigma}\exp\left ( -\frac{(y^{(i)} - \theta^{T}x^{(i)})^{2}}{2\sigma^{2}}\right) \\
&= \sum_{i=1}^{n}\log\frac{1}{\sqrt{2\pi}\sigma}\exp\left ( -\frac{(y^{(i)} - \theta^{T}x^{(i)})^{2}}{2\sigma^{2}}\right) \\
&= n\log\frac{1}{\sqrt{2\pi}\sigma} - \frac{1}{2\sigma^{2}}\sum_{i=1}^{n}(y^{(i)} - \theta^{T}x^{(i)})^{2}
\end{split}
\end{equation}
$$

hence, maximizing the log likelihood gives the same answer as minimizing:

$$\frac{1}{2}\sum_{i=1}^{n}(y^{(i)} - \theta^{T}x^{(i)})^{2} = J(\theta)$$

Linear regression $\Leftrightarrow $ Maximum Likelihood Estimate given Gaussian error.


```python
import numpy as np
import matplotlib.pyplot as plt

%matplotlib notebook
```

# 6.1 Example: Learning $XOR$

```python
X = np.array([[0, 0],
              [0, 1],
              [1, 0],
              [1, 1]])

y = np.array([[0],
              [1],
              [1],
              [0]])

bias = np.ones((4,1), dtype=int)
```

## Linear approach

Attempt to derive a model for $f(x; \, \theta)$ using the _normal equations_ : $(X^{\intercal} X)^{-1} \, X^{\intercal} y$

```python
Xb = np.hstack((X, bias))

XbTXb = Xb.T.dot(Xb)

Xb_y = Xb.T.dot(y)

sol = np.linalg.inv(XbTXb).dot(Xb_y)
print(sol)
```

    [[1.66533454e-16]
     [1.11022302e-16]
     [5.00000000e-01]]

The solution turns out to be

\begin{align}
  \text{linear solution} &= \begin{bmatrix}
    0 \\
    0 \\
    \frac{1}{2}
  \end{bmatrix}
\end{align}

which ignores the inputs and simply yields $\frac{1}{2}$

```python
def _relu(x):
    """ Return max{0, x} """
    return x * (x > 0)
```

```python
W = np.array([[1, 1],
              [1, 1]])

c = np.array([[0, -1]])
```

```python
colors = ['#7570b3', '#d95f02', '#d95f02', '#7570b3']
markers = list('oxxo')

fig, (ax1, ax2) = plt.subplots(1, 2, gridspec_kw={'width_ratios': [1, 3]})

ax1.table(cellText=X, colLabels=[r'$x_1$', r'$x_2$'], loc='center')
ax1.axis('off')

for (x,y), col, m in zip(X, colors, markers):
    ax2.scatter(x, y, color=col, marker=m)
ax2.set_xlabel(r'$x_1$')
ax2.set_ylabel(r'$x_2$')
fig.subplots_adjust(wspace=0.5, left=-.5, right=0.5)
```

```python
XW = X.dot(W)

fig, (ax1, ax2) = plt.subplots(1, 2, gridspec_kw={'width_ratios': [1, 3]})

ax1.table(cellText=XW, colLabels=[r'$x_1$', r'$x_2$'], loc='center')
ax1.axis('off')

for (x,y), col, m in zip(XW, colors, markers):
    ax2.scatter(x, y, color=col, marker=m)
ax2.set_xlabel(r'$x_1$')
ax2.set_ylabel(r'$x_2$')
fig.subplots_adjust(wspace=0.5, hspace=1.2, left=-.5, right=1.)
```

```python
XW_c = XW + c

fig, (ax1, ax2) = plt.subplots(1, 2, gridspec_kw={'width_ratios': [1, 3]})

ax1.table(cellText=XW_c, colLabels=[r'$x_1$', r'$x_2$'], loc='center')
ax1.axis('off')

for (x,y), col, m in zip(XW_c, colors, markers):
    ax2.scatter(x, y, color=col, marker=m)
ax2.set_xlabel(r'$x_1$')
ax2.set_ylabel(r'$x_2$')
fig.subplots_adjust(wspace=0.5, left=-.5, right=1.)
```

```python
h = _relu(XW_c)

fig, (ax1, ax2) = plt.subplots(1, 2, gridspec_kw={'width_ratios': [1, 3]})

ax1.table(cellText=h, colLabels=[r'$x_1$', r'$x_2$'], loc='center')
ax1.axis('off')

for (x,y), col, m in zip(h, colors, markers):
    ax2.scatter(x, y, color=col, marker=m)
ax2.set_xlabel(r'$x_1$')
ax2.set_ylabel(r'$x_2$')
fig.subplots_adjust(wspace=0.5, left=-.5, right=1.)
```

```python
w = np.array([[1],
              [-2]])

b = 0

y = h.dot(w) + b
print(y)
```

    [[0]
     [1]
     [1]
     [0]]

```python

```


## Least square fit

```python
from jupyterthemes import jtplot
jtplot.style(theme='onedork', context='notebook', ticks=True, grid=False)
```

```python
import sympy as sp
import pandas as pd
import numpy as np
```

```python
c_0,c_1 = sp.symbols('c_0 c_1')
epsilon = sp.IndexedBase("epsilon")
y = sp.IndexedBase("y")
x = sp.IndexedBase("x")
i, N = sp.var("i,N", integer=True)

linear_equation = sp.Eq(y[i], c_0+c_1*x[i]+epsilon[i])
```

```python
linear_equation
```

$\displaystyle {y}_{i} = c_{0} + c_{1} {x}_{i} + {\epsilon}_{i}$

```python
epsilon_equation = sp.Eq(epsilon[i],sp.solve(linear_equation, epsilon[i])[0])
epsilon_equation
```

$\displaystyle {\epsilon}_{i} = - c_{0} - c_{1} {x}_{i} + {y}_{i}$

## Criteria 1
Mean error should be 0.

$mean(\epsilon_i)=0 $

```python
criteria_1_equation = sp.Eq(1/N*sp.Sum(epsilon[i], (i, 1, N)),0)
criteria_1_equation
```

$\displaystyle \frac{\sum_{i=1}^{N} {\epsilon}_{i}}{N} = 0$

```python
criteria_1_equation = sp.Eq(sp.Sum(sp.solve(epsilon_equation,epsilon[i])[0], (i, 1, N)),0)
criteria_1_equation
```

$\displaystyle \sum_{i=1}^{N} \left(- c_{0} - c_{1} {x}_{i} + {y}_{i}\right) = 0$

```python
criteria_1_equation = sp.Eq(sp.expand(criteria_1_equation.lhs).doit(),0)
criteria_1_equation
```

$\displaystyle - N c_{0} + \sum_{i=1}^{N} - c_{1} {x}_{i} + \sum_{i=1}^{N} {y}_{i} = 0$

```python
criteria_1_equation = sp.Eq(c_0,sp.solve(criteria_1_equation,c_0)[0])
criteria_1_equation
```

$\displaystyle c_{0} = \frac{\sum_{i=1}^{N} \left(- c_{1} {x}_{i} + {y}_{i}\right)}{N}$

```python
criteria_1_equation = criteria_1_equation.expand().doit()
criteria_1_equation
```

$\displaystyle c_{0} = \frac{\sum_{i=1}^{N} - c_{1} {x}_{i}}{N} + \frac{\sum_{i=1}^{N} {y}_{i}}{N}$

```python
S_y,S_x,x_i,y_i = sp.symbols('S_y S_x x_i y_i')
```

```python
criteria_1_equation = criteria_1_equation.subs([(y[i],y_i),
                         (x[i],x_i),
                         ])
```

```python
criteria_1_equation
```

$\displaystyle c_{0} = \frac{\sum_{i=1}^{N} - c_{1} {x}_{i}}{N} + \frac{\sum_{i=1}^{N} {y}_{i}}{N}$

```python
S_y,S_x = sp.symbols('S_y S_x')
S_y_equation = sp.Eq(S_y, sp.Sum(y_i,(i, 1, N)))
S_y_equation
```

$\displaystyle S_{y} = \sum_{i=1}^{N} y_{i}$

```python
S_x_equation = sp.Eq(S_x, sp.Sum(x_i,(i, 1, N)))
S_x_equation
```

$\displaystyle S_{x} = \sum_{i=1}^{N} x_{i}$

```python
eqs=(S_y_equation,
     S_x_equation,
     criteria_1_equation
)
sp.solve(eqs, c_0, S_y,S_x,)
```

    {S_x: x_i*Sum(1, (i, 1, N)),
     S_y: y_i*Sum(1, (i, 1, N)),
     c_0: Sum(-c_1*x[i] + y[i], (i, 1, N))/N}

```python
criteria_1_equation.subs(S_y_equation.rhs,S_y_equation.lhs)
```

$\displaystyle c_{0} = \frac{\sum_{i=1}^{N} - c_{1} {x}_{i}}{N} + \frac{\sum_{i=1}^{N} {y}_{i}}{N}$

## Square error should be minimized

```python
square_error_equation = sp.Eq(epsilon[i]**2,
                              sp.Sum((sp.solve(epsilon_equation,epsilon[i])[0])**2, (i, 1, N)))
square_error_equation
```

$\displaystyle {\epsilon}_{i}^{2} = \sum_{i=1}^{N} \left(- c_{0} - c_{1} {x}_{i} + {y}_{i}\right)^{2}$

```python
square_error_zero_derivative_equation = sp.Eq(square_error_equation.rhs.diff(c_1),0)
square_error_zero_derivative_equation
```

$\displaystyle \sum_{i=1}^{N} - 2 \left(- c_{0} - c_{1} {x}_{i} + {y}_{i}\right) {x}_{i} = 0$

```python
square_error_zero_derivative_equation = square_error_zero_derivative_equation.expand().doit()
square_error_zero_derivative_equation
```

$\displaystyle \sum_{i=1}^{N} 2 c_{0} {x}_{i} + \sum_{i=1}^{N} 2 c_{1} {x}_{i}^{2} + \sum_{i=1}^{N} - 2 {x}_{i} {y}_{i} = 0$

```python
square_error_zero_derivative_equation = square_error_zero_derivative_equation.subs(c_0,
                                                 sp.solve(criteria_1_equation, c_0)[0]).doit().expand()
square_error_zero_derivative_equation
```

$\displaystyle \sum_{i=1}^{N} 2 c_{1} {x}_{i}^{2} + \sum_{i=1}^{N} - 2 {x}_{i} {y}_{i} + \sum_{i=1}^{N} \frac{2 {x}_{i} \sum_{i=1}^{N} - c_{1} {x}_{i}}{N} + \sum_{i=1}^{N} \frac{2 {x}_{i} \sum_{i=1}^{N} {y}_{i}}{N} = 0$

```python
sp.solve(square_error_zero_derivative_equation,c_1)
```

    []

```python
solution = sp.solve(eqs,c_0,c_1)[0]
```

```python
c0_equation = sp.Eq(c_0,solution[0].simplify())
c0_equation
```

$\displaystyle c_{0} = \frac{\sum_{i=1}^{N} \left(- c_{1} {x}_{i} + {y}_{i}\right)}{N}$

```python
c1_equation = sp.Eq(c_1,solution[1].simplify())
c1_equation
```

$\displaystyle \text{True}$

```python
solution[1]
```

$\displaystyle - \frac{\sum_{i=1}^{N} \frac{\left(\sum_{i=1}^{N} - 2 c_{1} {y}_{i} + \sum_{i=1}^{N} 2 c_{1}^{2} {x}_{i}\right) {y}_{i}}{\sum_{i=1}^{N} \left(- c_{1} {x}_{i} + {y}_{i}\right)} + \sum_{i=1}^{N} \frac{\left(\left(\sum_{i=1}^{N} - 2 c_{1} {y}_{i} + \sum_{i=1}^{N} 2 c_{1}^{2} {x}_{i}\right)^{2}\right) {x}_{i}}{2 \left(\sum_{i=1}^{N} \left(- c_{1} {x}_{i} + {y}_{i}\right)\right)^{2}}}{2 \sum_{i=1}^{N} \left(\frac{\left(\sum_{i=1}^{N} - 2 c_{1} {y}_{i} + \sum_{i=1}^{N} 2 c_{1}^{2} {x}_{i}\right) {x}_{i}}{2 \sum_{i=1}^{N} \left(- c_{1} {x}_{i} + {y}_{i}\right)} + {y}_{i}\right)}$

```python
s = sp.Sum(c_0+c_1*x[i], (i, 1, N))

```

```python
sp.solve(epsilon_equation,epsilon[i])[0]
```

$\displaystyle - c_{0} - c_{1} {x}_{i} + {y}_{i}$

```python

```

```python
sp.Vector()
```

```python
linear_equation
```

$\displaystyle y = c_{0} + c_{1} x + \epsilon$

```python
from sympy import IndexedBase
from sympy.tensor.array import Array

a,b,c,d,e,f = sp.symbols("a b c d e f")

X = Array([[a, b, c], [d, e, f]])
X
```

$\displaystyle \left[\begin{matrix}a & b & c\\d & e & f\end{matrix}\right]$

```python
Array([[1]])
```

$\displaystyle \left[\left[\begin{matrix}1\end{matrix}\right]\right]$

```python
X = IndexedBase("X")

W = IndexedBase("W")

i, j, M, K = sp.var("i,j,M,K", integer=True)
s = sp.Sum(X[i, j]*W[j], (i, 1, M), (j, 1, K))
```

```python
s
```

$\displaystyle \sum_{\substack{1 \leq i \leq M\\1 \leq j \leq K}} {W}_{j} {X}_{i,j}$

```python

```

```python
s
```

$\displaystyle \sum_{i=1}^{N} \left(c_{0} + c_{1} {x}_{i}\right)$

```python

```


In this notebook, we will go over several Python examples for detecting anomalies (outliers) from data. Anomaly detection is the task of identifying instances whose characteristics differ significantly from the rest of the data. In this tutorial, we will provide examples of applying different anomaly detection techniques using Python and its library packages.

This approach assumes that the majority of the data instances are governed by some well-known probability distribution, e.g., Binomial or Gaussian distribution. Anomalies can then detected by seeking for observations that do not fit the overall distribution of the data.

In this example, our goal is to detect anomalous changes in the daily closing prices of various stocks. The input data *stocks.csv* contains the historical closing prices of stocks for 3 large corporations (Microsoft, Ford Motor Company, and Bank of America).

```python
import pandas as pd

stocks = pd.read_csv('data/stocks.csv', header='infer' )
stocks.index = stocks['Date']
stocks = stocks.drop(['Date'],axis=1)
stocks.head()
```

We can compute the percentage of changes in the daily closing price of each stock as follows:
\begin{equation}
\Delta(t) = 100 \times \frac{x_t - x_{t-1}}{x_{t-1}}
\end{equation}

where $x_t$ denotes the price of a stock on day $t$ and $x_{t-1}$ denotes the price on its previous day, $t-1$.

```python
import numpy as np

N,d = stocks.shape
delta = pd.DataFrame(100*np.divide(stocks.iloc[1:,:].values-stocks.iloc[:N-1,:].values, stocks.iloc[:N-1,:].values),
                    columns=stocks.columns, index=stocks.iloc[1:].index)
delta.head()
```

We can plot the distribution of the percentage daily changes in stock price.

```python
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
%matplotlib inline

fig = plt.figure(figsize=(8,5)).gca(projection='3d')
fig.scatter(delta.MSFT,delta.F,delta.BAC)
fig.set_xlabel('Microsoft')
fig.set_ylabel('Ford')
fig.set_zlabel('Bank of America')
plt.show()
```

Assuming the data follows a multivariate Gaussian distribution, we can compute the mean and covariance matrix of the 3-dimensional data as follows

```python
meanValue = delta.mean()
covValue = delta.cov()
print(meanValue)
print(covValue)
```

To determine the anomalous trading days, we can compute the Mahalanobis distance between the percentage of price change on each day against the mean percentage of price change.
\begin{equation}
\textrm{Mahalanobis}(x) = (x - \bar{x}) \Sigma^{-1}(x - \bar{x})^T
\end{equation}
where $x$ is assumed to be a row vector.

See Equation 9.4 in Section 9.3.1 for more information about using Mahalanobis distance for detecting anomalies in multivariate Gaussian distribution.

```python
from numpy.linalg import inv

X = delta.as_matrix()
S = covValue.as_matrix()
for i in range(3):
    X[:,i] = X[:,i] - meanValue[i]

def mahalanobis(row):
    return np.matmul(row,S).dot(row)

anomaly_score = np.apply_along_axis( mahalanobis, axis=1, arr=X)

fig = plt.figure(figsize=(10,6))
ax = fig.add_subplot(111, projection='3d')
p = ax.scatter(delta.MSFT,delta.F,delta.BAC,c=anomaly_score,cmap='jet')
ax.set_xlabel('Microsoft')
ax.set_ylabel('Ford')
ax.set_zlabel('Bank of America')
fig.colorbar(p)
plt.show()
```

The top-2 anomalies are shown as a brown point in the figure above. The highest anomaly corresponds to the day in which the prices for all 3 stocks increase significantly whereas the second highest anomaly corresponds  to the day in which all 3 stocks suffer a large percentage drop in their closing prices. We can examine the dates associated with the top-2 highest anomaly scores as follows.

```python
anom = pd.DataFrame(anomaly_score, index=delta.index, columns=['Anomaly score'])
result = pd.concat((delta,anom), axis=1)
result.nlargest(2,'Anomaly score')
```

Note that the sharp drop in the stock prices on October 7, 2008 coincide with the beginning of the global financial crisis (https://en.wikipedia.org/wiki/Global_financial_crisis_in_October_2008) while the increase in the stock prices on April 9, 2009.

```python
fig, (ax1,ax2) = plt.subplots(nrows=1, ncols=2, figsize=(15,6))

ts = delta[440:447]
ts.plot.line(ax=ax1)
ax1.set_xticks(range(7))
ax1.set_xticklabels(ts.index)
ax1.set_ylabel('Percent Change')

ts = delta[568:575]
ts.plot.line(ax=ax2)
ax2.set_xticks(range(7))
ax2.set_xticklabels(ts.index)
ax2.set_ylabel('Percent Change')
```

The distance-based approach is a model-free anomaly detection approach as it does not require constructing an explicit model of the normal class to determine the anomaly score of data instances. The example code shown below employs the k-nearest neighbor approach to calculate anomaly score. Specifically, a normal instance is expected to have a small distance to its k-th nearest neighbor whereas an anomaly is likely to have a large distance to its k-th nearest neighbor. In the example below, we apply the distance-based approach with k=4 to identify the anomalous trading days from the stock market data described in the previous section.

```python
from sklearn.neighbors import NearestNeighbors
import numpy as np
from scipy.spatial import distance

knn = 4
nbrs = NearestNeighbors(n_neighbors=knn, metric=distance.euclidean).fit(delta.as_matrix())
distances, indices = nbrs.kneighbors(delta.as_matrix())

anomaly_score = distances[:,knn-1]

fig = plt.figure(figsize=(10,6))
ax = fig.add_subplot(111, projection='3d')
p = ax.scatter(delta.MSFT,delta.F,delta.BAC,c=anomaly_score,cmap='jet')
ax.set_xlabel('Microsoft')
ax.set_ylabel('Ford')
ax.set_zlabel('Bank of America')
fig.colorbar(p)
plt.show()
```

The results are slightly different than the one shown in Section 9.1 since we have used Euclidean distance (instead of Mahalanobis distance) to detect the anomalies. We can examine the dates associated with the top-5 highest anomaly scores as follows.

```python
anom = pd.DataFrame(anomaly_score, index=delta.index, columns=['Anomaly score'])
result = pd.concat((delta,anom), axis=1)
result.nlargest(5,'Anomaly score')
```

```python
fig = plt.figure(figsize=(10,4))

ax = fig.add_subplot(111)
ts = delta[445:452]
ts.plot.line(ax=ax)
ax.set_xticks(range(7))
ax.set_xticklabels(ts.index)
ax.set_ylabel('Percent Change')
```


```python
import sympy as sm
import numpy as np
from scipy import optimize
import ModelProject as fct
import matplotlib.pyplot as plt
%load_ext autoreload
%autoreload 2
from __future__ import print_function
from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets
from IPython.display import display
```

# 1. Introduction

We make use of an OLG model where people live in two periods $(1t)$ and $(2t+1)$. We have homogenous firms that maximise their profits in the following way:
$$
max_{K_{t},\space L_{t}}\space AK_{t}^{\alpha}L_{t}^{1-\alpha}-r_{t}K_{t}-w_{t}L_{t}
$$
where $\alpha \in (0,1)$, and the total factory productivty $A$ is strictly positive. Also, $r_{t}$ denotes the rental rate of capital while $w_{t}$ refers to wage rate.

To continue, we define the following utility of young people born in period $1t$:
$$
U_{t}=ln(c_{1t})+\frac{1}{1+\rho}ln(c_{2t+1})
$$
where $\rho>-1$ and $c_{1t}$ refers to the consumption of the young generation while $c_{2t+1}$ refers to the consumption of the old generation.

We also assume that a government provides a 'pay-as-you-go' scenario where the young generation contributes an amount $\tau$ of their wage where $\tau \in (0,1)$ that is paid in the same period to the old. As such, the budget constraints are:

$$
c_{1t}= (1-\tau)w_{t}-s_{t}
$$

$$
c_{2t+1}=R_{t+1}s_{t}+\tau w_{t+1}
$$
where $R_{t+1}=1+r_{t+1}-\delta$.

Finally, we assume that population growth is constant and that the old generation does not work. As such, the old generation depends financially on income from their savings as well as on the contributions given by the young generation.

# 2. Analytical analysis

```python
A=sm.symbols('A')
K=sm.symbols('K_t')
L=sm.symbols('L_t')
alpha=sm.symbols('alpha')
rho=sm.symbols('rho')
c1 = sm.symbols('c_1t')
c2=sm.symbols('c_2t+1')
u=sm.symbols('u_t')
U=sm.symbols('U_t')
uc1=sm.ln(c1)
uc2=sm.ln(c2)
s=sm.symbols('s_t')
tau=sm.symbols('tau')
r=sm.symbols('r_t')
rt=sm.symbols('R_t+1')
w=sm.symbols('w_t')
wt=sm.symbols('w_t+1')
k = sm.symbols('k_t')
kt=sm.symbols('k_t+1')
lamb=sm.symbols('lambda_t')
tau=sm.symbols('tau')
kstar=sm.symbols('k_{ss}^{*}')
sstar=sm.symbols('s_{t}^{*}')
```

**2.1. Firm maximisation problem**

We begin by maximising the firm profit problem with respect to $K_t$ and $L_t$ in order to find the rental rate of capital and the wage rate: $$
max_{K_{t}, \space {L_t}} \space AK_{t}^{\alpha}L_{t}^{1-\alpha}-r_{t}K_{t}-w_{t}L_{t}
$$

```python
#Define the firm maximisation problem
firmmax=A*K**(alpha)*L**(1-alpha)-(r*K)-(w*L)
```

FOC w.r.t $K_{t}$ and $L_{t}$ respectively:

```python
#We compute the first order conditions of our firm maximisation problem w.r.t to K_{t} and L_{t}
FOCK = sm.Eq(0,sm.diff(firmmax, K))
FOCL = sm.Eq(0,sm.diff(firmmax, L))
display(FOCK)
display(FOCL)
```

$\displaystyle 0 = \frac{A K_{t}^{\alpha} L_{t}^{1 - \alpha} \alpha}{K_{t}} - r_{t}$

$\displaystyle 0 = \frac{A K_{t}^{\alpha} L_{t}^{1 - \alpha} \left(1 - \alpha\right)}{L_{t}} - w_{t}$

Using the fact that $k_{t}=\frac{K_t}{L_t}$, our FOCs above become:

```python
rentalrateK =sm.Eq(r,A*alpha* k**(alpha-1))
wagerate= sm.Eq(w,A*k**alpha * (1-alpha))
display(rentalrateK)
display(wagerate)
```

$\displaystyle r_{t} = A \alpha k_{t}^{\alpha - 1}$

$\displaystyle w_{t} = A k_{t}^{\alpha} \left(1 - \alpha\right)$

**2.2. Household maximisation problem**

We maxmimise the household problem of intertemporal allocation of resources with respect to the budget constraints mentioned above:
$$
max_{c_{1t}, \space c_{2t+1}}\space U_{t}=ln(c_{1t})+\frac{1}{1+\rho}ln(c_{2t+1})+\lambda_{t}((1-\tau)w_{t}-c_{1t}-(\frac{c_{2t+1}-\tau w_{t+1}}{R_{t+1}})
$$

```python
#We define the household maximisation problem
maxU =uc1 + (1/(1+rho))*uc2 + lamb*((1-tau)*w-c1 -((c2-tau*wt)/rt))
display(sm.Eq(U,maxU))
```

$\displaystyle U_{t} = \lambda_{t} \left(- c_{1t} + w_{t} \left(1 - \tau\right) - \frac{c_{2t+1} - \tau w_{t+1}}{R_{t+1}}\right) + \log{\left(c_{1t} \right)} + \frac{\log{\left(c_{2t+1} \right)}}{\rho + 1}$

FOC of $U_{t}$ w.r.t $c_{1t}$ and $c_{2t+1}$ are respectively:

```python
FOCc1=sm.Eq(0,sm.diff(maxU,c1))
FOCc2=sm.Eq(0,sm.diff(maxU,c2))
display(FOCc1)
display(FOCc2)
```

$\displaystyle 0 = - \lambda_{t} + \frac{1}{c_{1t}}$

$\displaystyle 0 = \frac{1}{c_{2t+1} \left(\rho + 1\right)} - \frac{\lambda_{t}}{R_{t+1}}$

Isolating $\lambda_{t}$ in the first FOC and plugging it into the second FOC yields the Euler equation:

```python
#We isolate lambda_{t} in the first FOC:
lambisol=sm.solve(FOCc1, lamb)
#Then we plug in the isolated lambda_{t} into the second FOC:
Eulereq=FOCc2.subs(lamb, lambisol[0])
display(Eulereq)
```

$\displaystyle 0 = \frac{1}{c_{2t+1} \left(\rho + 1\right)} - \frac{1}{R_{t+1} c_{1t}}$

Then we isolate for $c_1$ so that our Euler equation is:

```python
#We isolate for c1:
Eulereqiso=sm.solve(Eulereq,c1)
display(sm.Eq(c1,Eulereqiso[0]))
```

$\displaystyle c_{1t} = \frac{c_{2t+1} \left(\rho + 1\right)}{R_{t+1}}$

The Euler equation gives us households' optimal consumption path. Therefore, the marginal utility of consumption today is equal to the future value of discounted marginal
utility of consumption tomorrow, discounted by $\rho$, when the above condition holds.

**2.3. Optimal savings $s_{t}^{*}$**

To continue, we plug in the budget constraints in our maximisation problem to find the optimal individual savings behaviour $s_t$:
$$
max_{s_t}\space U_{t}=ln((1-\tau)w_{t}-s_{t})+\frac{1}{1+\rho}ln((R_{t+1})s_{t}+\tau (w_{t+1}))+\lambda_{t}((1-\tau)w_{t}-c_{1t}-(\frac{c_{2t+1}-\tau w_{t+1}}{R_{t+1}})
$$

```python
#We define our budget contraints
bc1=(1-tau)*w-s
bc2=(rt*s)+tau*(wt)
#We define the logs of our budget contraints
lnbc1=sm.ln(bc1)
lnbc2=sm.ln(bc2)
```

```python
#We plug in our budget constraints into our household max problem:
maxUsavings=lnbc1+(1/(1+rho))*lnbc2+lamb*((1-tau)*w-c1 -((c2-tau*wt)/rt))
display(sm.Eq(U,maxUsavings))
```

$\displaystyle U_{t} = \lambda_{t} \left(- c_{1t} + w_{t} \left(1 - \tau\right) - \frac{c_{2t+1} - \tau w_{t+1}}{R_{t+1}}\right) + \log{\left(- s_{t} + w_{t} \left(1 - \tau\right) \right)} + \frac{\log{\left(R_{t+1} s_{t} + \tau w_{t+1} \right)}}{\rho + 1}$

We then find the FOC of our household maximisation problem w.r.t $s_t$:

```python
FOCsavings=sm.Eq(0,sm.diff(maxUsavings,s))
display(FOCsavings)
```

$\displaystyle 0 = \frac{R_{t+1}}{\left(\rho + 1\right) \left(R_{t+1} s_{t} + \tau w_{t+1}\right)} - \frac{1}{- s_{t} + w_{t} \left(1 - \tau\right)}$

We isolate for $s_t$ so that our optimal savings is:

```python
#isolate for savings:
savings=(sm.solve(FOCsavings,s))
optsavings=sm.Eq(sstar,savings[0])
display(optsavings)
```

$\displaystyle s_{t}^{*} = \frac{- R_{t+1} \tau w_{t} + R_{t+1} w_{t} - \rho \tau w_{t+1} - \tau w_{t+1}}{R_{t+1} \left(\rho + 2\right)}$

```python
#Defining optimal savings into a python function:
optsavingsfct=sm.lambdify((rt,tau,w,wt,rho),savings)
```

```python
optsavingsfct(0.8,0.1,0.6,0.9,0.1)
print('Analytical optimal savings =', optsavingsfct(0.8,0.1,0.6,0.9,0.1))
```

    Analytical optimal savings = [0.19821428571428568]

We can see that the higher the gross return on savings $R_{t+1}$ is, the higher the optimal savings $s_{t}^{*}$ is. Also, we notice that the higher the contribution $\tau$ from young to old is, the lower the optimal savings $s_{t}^{*}$ are. Finally, we notice that the wage rate in period 1 ($w_t)$ is positively correlated to $s_{t}^{*}$ while the wage rate in period 2 $(w_{t+1})$ is negatively correlated to $s_{t}^{*}$. As a result, the optimal savings $s_{t}^{*}$ decrease the more $w_{t+1}$ increases.

**2.4. Capital accumulation**

In order to find the the capital accumulation equation, we use the optimal savings equation $s_t^{*}$ derived above and replace $s_t^{*}$ with $k_{t+1}$, and we plug in the values for $w_t$, $w_{t+1}$, $r_t$ and $r_{t+1}$. Also, we now assume that capital full depreciates so that $\delta=1$. Therefore, $R_{t+1}=r_{t+1}$ since $R_{t+1}=1+r_{t+1}-1$.

Note that our sympy commands provide a different capital accumulation and steady-state. As such, we perform parameter substitution ourselves in order to compute capital accumulation and the steady state. Our capital accumulation equation $k_{t+1}=s_{t}^{*}$ looks as followed:

```python
#We define $w_{t+1}$ and $r_{t+1}$:
rentalrateK1=rentalrateK.subs(k,kt)
wagerate1=wagerate.subs(k,kt)
display(rentalrateK1)
display(wagerate1)
```

$\displaystyle r_{t} = A \alpha k_{t+1}^{\alpha - 1}$

$\displaystyle w_{t} = A k_{t+1}^{\alpha} \left(1 - \alpha\right)$

```python
#We set $s_t^{*}=k_{t+1}$ and plug in the values for $w_t$, $w_{t+1}$, $r_t$ and $r_{t+1}$:
capitalacc=(((1-alpha)*(1-tau)*alpha)/((2+rho)*alpha+(1+rho)*(1-alpha)*tau))*A*(k**alpha)
capitalacc=sm.Eq(kt,capitalacc)
display(capitalacc)
```

$\displaystyle k_{t+1} = \frac{A \alpha k_{t}^{\alpha} \left(1 - \alpha\right) \left(1 - \tau\right)}{\alpha \left(\rho + 2\right) + \tau \left(1 - \alpha\right) \left(\rho + 1\right)}$

**2.5. Steady state**

To find our steady state, we assume that $k_{ss}^{*}=k_{t+1}=k_{t}$. Thus, we plug in $k_{ss}^{*}=k_{t+1}=k_{t}$ in our equation avove so that:

```python
#We substitute $k_{ss}^{*}=k_{t+1}=k_{t}$ into our capital accumulation equation:
sstate=(((1-alpha)*(1-tau)*alpha*A)/((2+rho)*alpha+(1+rho)*(1-alpha)*tau))**(1/(1-alpha))
SS=sm.Eq(kstar,sstate)
display(SS)
```

$\displaystyle k_{ss}^{*} = \left(\frac{A \alpha \left(1 - \alpha\right) \left(1 - \tau\right)}{\alpha \left(\rho + 2\right) + \tau \left(1 - \alpha\right) \left(\rho + 1\right)}\right)^{\frac{1}{1 - \alpha}}$

```python
#Defining steady state into a python function:
optSSfct=sm.lambdify((A,alpha,tau,rho),sstate)
```

```python
optSSfct(100,0.1,0.003,0.1)
print('Analytical steady state =', optSSfct(100,0.1,0.003,0.1))
```

    Analytical steady state = 63.84582238491508

We can see that $\alpha$, $\tau$ and $\rho$ are all negatively correlated to the steady state $k_{ss}^{*}$ as an increase in either parameter decreases the steady state. In contrast, the total factor productivity $A$ is positively correlated, meaning that an increase in $A$ increases the steady state.

# 3. Numerical analysis

We will now use functions that have been created in the `ModelProject.py` file in order to solde this model numericaly.

We first set the model parameters :

```python
alpha=0.7
rho=0.1
tau=0.05
A=20
w=10
s=2
r=0.3
L=50
delta=1
K=100
```

```python
alpha1=0.33
rho1=0.05
tau1=0.2
A1=2
w1=10
s1=2
r1=0.3
L1=50
delta1=1
K1=100
```

**3.1. Firm maximization problem**

We have the following results:

```python
#profit maximisation:
MaxProfit = fct.profit(K,L,w,r,alpha,A)
print('Firm profits =', MaxProfit)
#Optimal wage:
optw=fct.optiWage(A,K,alpha)
print('Optimal wage =',optw)
#Optimal rental rate of capital:
optr=fct.optirr(A,K,alpha)
print('Optimal rental rate of capital =',optr)
#Optimal level of capital:
optK=fct.MaxProf(K,L,w,r,alpha,A,Kopt=True)
print('Optimal level of capital =',optK)
#Optimal level of labour:
optL=fct.MaxProf(K,L,w,r,alpha,A,Kopt=False)
print('Optimal level of labour =',optL)
```

    Firm profits = 1094.5047927124708
    Optimal wage = 150.7131858905748
    Optimal rental rate of capital = 3.5166410041134113
    Optimal level of capital = 9999.99970046681
    Optimal level of labour = 48.202904223331565

**3.2. Household maximization problem**

We have the following results:

```python
#Young generation consumption:
c1=fct.consoY(s,tau,w)
print('Young generation consumption =',c1)
#Old generation consumption:we first calculate the gross return on savings and then old generation consumption:
R=fct.GrossRet(r,delta)
c2=fct.consoO(R,s,tau,w)
c2
print('Old generation consumption =',R)
#Euler equation:
c1E=fct.eulerC1(c2,rho,R)
print('Euler equation =',c1E)
#Utility of young generation:
u=fct.utilityY(c1,c2,rho)
print('Utility =',u)
#savings:
s=fct.savings(w,rho)
opts=fct.optiS(tau,rho,w,R)
print('Savings = ',s)
print('Optimal savings = ',opts)
```

    Young generation consumption = 7.5
    Old generation consumption = 0.30000000000000004
    Euler equation = 4.033333333333333
    Utility = 2.1015486385461966
    Savings =  4.761904761904762
    Optimal savings =  3.6507936507936507

**3.3. Capital steady state**

```python
#We define the parameters:
n=50
k=100
l=[k]
```

We create a list that contains the different levels of capital accross a period of time *n*:

```python
for t in range(n):
    k=((1-alpha)*(1-tau)*alpha)/((2+rho)*alpha+(1+rho)*(1-alpha)*tau)*A*k**alpha
    l.append(k)
```

Numerical steady state:

```python
K_ss=(((1-alpha)*(1-tau)*alpha)/((2+rho)*alpha+(1+rho)*(1-alpha)*tau)*A)**(1/(1-alpha))
print('Numerical steady state =',K_ss)
```

    Numerical steady state = 26.875709644766083

We now plot the list created earlier and the numerical steady state:

```python
fig = plt.figure(figsize=(6,6/1.5))
ax = fig.add_subplot(1,1,1)
ax.plot(l,label=r'$K_{t+1}$')
ax.axhline(K_ss,ls='--',color='red',label='Analytical steady state')
ax.legend(frameon=True)
fig.tight_layout()
plt.grid()
```

We now make our results interactive :

```python
A_inter=interactive(fct.f,x=20)
print('Choose A:')
display(A_inter)
```

    Choose A:

    interactive(children=(IntSlider(value=20, description='x', max=60, min=-20), Output()), _dom_classes=('widget-…

```python
alpha_inter=interactive(fct.f,x=0.7)
print('Choose alpha:')
display(alpha_inter)
```

    Choose alpha:

    interactive(children=(FloatSlider(value=0.7, description='x', max=2.0999999999999996, min=-0.7), Output()), _d…

```python
rho_inter=interactive(fct.f,x=0.1)
print('Choose rho:')
display(rho_inter)
```

    Choose rho:

    interactive(children=(FloatSlider(value=0.1, description='x', max=0.30000000000000004, min=-0.1), Output()), _…

```python
tau_inter=interactive(fct.f,x=0.05)
print('Choose tau:')
display(tau_inter)
```

    Choose tau:

    interactive(children=(FloatSlider(value=0.05, description='x', max=0.15000000000000002, min=-0.05), Output()),…

```python
#We create a list of the level of capital from t to t+n:
n=50
k=100
l_inter=[k]
```

```python
def plotfct(A,alpha,rho,tau,k=100):
    l_inter=[]
    for t in range(50):
        k=((1-alpha)*(1-tau)*alpha)/((2+rho)*alpha+(1+rho)*(1-alpha)*tau)*(A)*k**alpha
        l_inter.append(k)

    K_ss_inter=(((1-alpha)*(1-tau)*alpha)/((2+rho)*alpha+(1+rho)*(1-alpha)*tau)*(A))**(1/(1-alpha))

    #We plot the analytical steady state and the level of capital generated by the list above:
    fig = plt.figure(figsize=(6,6/1.5))
    ax = fig.add_subplot(1,1,1)
    ax.plot(l_inter,label=r'$K_{t+1}$')
    ax.axhline(K_ss_inter,ls='--',color='red',label='Analytical steady state')
    ax.legend(frameon=True)
    fig.tight_layout()
    plt.grid()
```

```python
import ipywidgets as widgets
```

```python
widgets.interact(plotfct,
                 A=widgets.FloatSlider(description="A",min=1,max=100,step=5,value=20),
                 alpha=widgets.FloatSlider(description="alpha",min=0.01,max=0.99,step=0.05,value=0.5),
                 rho=widgets.FloatSlider(description="rho",min=0.01,max=0.99,step=0.05,value=0.1),
                 tau=widgets.FloatSlider(description="tau",min=0.01,max=0.99,step=0.05,value=0.2),
                 k=widgets.fixed(100))
```

    interactive(children=(FloatSlider(value=20.0, description='A', min=1.0, step=5.0), FloatSlider(value=0.5, desc…

    <function __main__.plotfct(A, alpha, rho, tau, k=100)>

```python
for t in range(n):
    k=((1-alpha_inter.result)*(1-tau_inter.result)*alpha_inter.result)/((2+rho_inter.result)*alpha_inter.result+(1+rho_inter.result)*(1-alpha_inter.result)*tau_inter.result)*(A_inter.result)*k**alpha_inter.result
    l_inter.append(k)
```

```python
K_ss_inter=(((1-alpha_inter.result)*(1-tau_inter.result)*alpha_inter.result)/((2+rho_inter.result)*alpha_inter.result+(1+rho_inter.result)*(1-alpha_inter.result)*tau_inter.result)*(A_inter.result))**(1/(1-alpha_inter.result))
print('The analytical steady state is :',K_ss_inter)
```

    The analytical steady state is : 31.622116722161557

```python
#We plot the analytical steady state and the level of capital generated by the list above:
fig = plt.figure(figsize=(6,6/1.5))
ax = fig.add_subplot(1,1,1)
ax.plot(l_inter,label=r'$K_{t+1}$')
ax.axhline(K_ss_inter,ls='--',color='red',label='Analytical steady state')
ax.legend(frameon=True)
fig.tight_layout()
plt.grid()
```

# 4. Extension: Dynamics of capital accumulation

**4.1. We first use the basic level of savings $s_{t}$**

```python
# Define parameters
capital_1,capital_2 = fct.transition_curve(A,alpha,rho)

# Plot
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.plot(capital_1,capital_2)
ax.plot(capital_1,capital_1, '--', color='grey')
plt.text(45,32,'$k_t+1$')
plt.text(35,45,'$k_t=k_t+1$')
plt.text(32,30,'$k^*$')
ax.set_xlabel('$k_t$')
ax.set_ylabel('$k_t+1$')
ax.set_title('Transition curve')
ax.set_xlim([-0.001,50])
ax.set_ylim([0,50]);
plt.grid()
```

We can observe that there is an intersection between transition curve (in blue) and the 45° line (in grey) at around $k_{t+1}=k_{t}=20$. This intersection represent the capital steady state $k_{ss}^{*}$.

**4.2. we use the optimal level of savings $s_{t}^{*}$:**

```python
# Define parameters
capital_1,capital_2 = fct.transition_curve2(A,alpha,tau,rho,R)

# Plot
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.plot(capital_1,capital_2)
ax.plot(capital_1,capital_1, '--', color='grey')
plt.text(27,17,'$k_t+1$')
plt.text(22,27,'$k_t=k_t+1$')
plt.text(13,10,'$k^*$')
ax.set_xlabel('$k_t$')
ax.set_ylabel('$k_t+1$')
ax.set_title('Transition curve')
ax.set_xlim([-0.001,30])
ax.set_ylim([0,30]);
plt.grid()
```

We observe that by using the optimal savings $s_{t}^{*}$, we obtain a smaller capital steady state level $k_{ss}^{*}$.

```python
kvec=np.linspace(1e-8,35,500)
for i in range(499):
    kt1=fct.transition3(kvec[i+1],kvec[i],A,alpha,tau,rho,delta)
    kvec[i]=kt1
print(kvec)
```

    [ 0.42250596  0.68556991  0.91017317  1.11289324  1.30074739  1.47754239
      1.64563879  1.80662929  1.96165339  2.11156233  2.25701345  2.39852792
      2.53652794  2.67136161  2.80332032  2.93265115  3.05956589  3.18424788
      3.30685723  3.42753475  3.54640518  3.66357968  3.77915785  3.89322938
      4.00587544  4.11716972  4.22717949  4.33596625  4.44358652  4.55009234
      4.65553177  4.75994935  4.86338639  4.96588137  5.06747016  5.16818627
      5.2680611   5.3671241   5.46540293  5.56292365  5.65971079  5.75578755
      5.85117583  5.94589638  6.03996887  6.13341195  6.22624337  6.31847998
      6.41013785  6.50123229  6.59177791  6.68178865  6.77127785  6.86025827
      6.94874213  7.03674112  7.12426648  7.21132899  7.29793899  7.38410642
      7.46984086  7.5551515   7.64004722  7.72453656  7.80862775  7.89232873
      7.97564719  8.05859051  8.14116587  8.22338017  8.30524011  8.38675217
      8.46792261  8.5487575   8.62926273  8.709444    8.78930683  8.86885661
      8.94809853  9.02703765  9.10567889  9.18402701  9.26208666  9.33986233
      9.41735843  9.49457922  9.57152886  9.64821138  9.72463074  9.80079076
      9.87669518  9.95234766 10.02775174 10.1029109  10.17782851 10.25250788
     10.32695223 10.40116469 10.47514835 10.5489062  10.62244118 10.69575615
     10.7688539  10.84173718 10.91440867 10.98687098 11.05912667 11.13117825
     11.20302817 11.27467885 11.34613262 11.41739179 11.48845862 11.55933531
     11.63002404 11.70052692 11.77084603 11.84098342 11.91094108 11.98072098
     12.05032503 12.11975513 12.18901312 12.25810083 12.32702003 12.39577248
     12.46435989 12.53278396 12.60104633 12.66914864 12.73709249 12.80487944
     12.87251104 12.93998881 13.00731424 13.07448879 13.1415139  13.20839099
     13.27512145 13.34170666 13.40814796 13.47444667 13.5406041  13.60662154
     13.67250024 13.73824144 13.80384638 13.86931625 13.93465224 13.99985552
     14.06492724 14.12986852 14.19468048 14.25936422 14.32392082 14.38835135
     14.45265686 14.51683837 14.58089692 14.6448335  14.7086491  14.7723447
     14.83592126 14.89937973 14.96272105 15.02594614 15.0890559  15.15205124
     15.21493303 15.27770216 15.34035948 15.40290584 15.46534208 15.52766903
     15.5898875  15.65199829 15.71400221 15.77590004 15.83769254 15.89938049
     15.96096464 16.02244573 16.0838245  16.14510168 16.20627798 16.26735411
     16.32833077 16.38920866 16.44998844 16.51067081 16.57125642 16.63174594
     16.69214    16.75243927 16.81264436 16.87275591 16.93277454 16.99270085
     17.05253547 17.11227897 17.17193196 17.23149503 17.29096874 17.35035368
     17.4096504  17.46885946 17.52798143 17.58701684 17.64596623 17.70483014
     17.76360911 17.82230364 17.88091427 17.93944149 17.99788582 18.05624777
     18.11452781 18.17272645 18.23084417 18.28888144 18.34683875 18.40471657
     18.46251535 18.52023557 18.57787766 18.6354421  18.69292932 18.75033976
     18.80767387 18.86493207 18.9221148  18.97922247 19.03625551 19.09321434
     19.15009937 19.206911   19.26364964 19.32031569 19.37690954 19.43343159
     19.48988223 19.54626184 19.60257081 19.6588095  19.7149783  19.77107758
     19.82710769 19.88306902 19.93896191 19.99478672 20.05054382 20.10623354
     20.16185623 20.21741225 20.27290192 20.3283256  20.38368361 20.43897629
     20.49420396 20.54936695 20.60446558 20.65950018 20.71447106 20.76937854
     20.82422293 20.87900453 20.93372366 20.98838062 21.04297571 21.09750923
     21.15198147 21.20639274 21.26074331 21.31503348 21.36926354 21.42343377
     21.47754445 21.53159586 21.58558828 21.63952198 21.69339723 21.74721431
     21.80097349 21.85467502 21.90831917 21.96190621 22.0154364  22.06890998
     22.12232722 22.17568838 22.22899369 22.28224341 22.33543779 22.38857708
     22.44166151 22.49469133 22.54766678 22.60058809 22.6534555  22.70626926
     22.75902958 22.8117367  22.86439085 22.91699225 22.96954114 23.02203772
     23.07448224 23.1268749  23.17921593 23.23150554 23.28374395 23.33593138
     23.38806802 23.44015411 23.49218984 23.54417542 23.59611105 23.64799696
     23.69983333 23.75162036 23.80335827 23.85504724 23.90668748 23.95827918
     24.00982254 24.06131774 24.11276499 24.16416446 24.21551636 24.26682087
     24.31807817 24.36928845 24.4204519  24.4715687  24.52263902 24.57366305
     24.62464097 24.67557296 24.72645919 24.77729984 24.82809508 24.87884508
     24.92955003 24.98021008 25.0308254  25.08139618 25.13192256 25.18240473
     25.23284284 25.28323706 25.33358755 25.38389447 25.43415799 25.48437826
     25.53455545 25.5846897  25.63478119 25.68483005 25.73483646 25.78480055
     25.83472249 25.88460242 25.9344405  25.98423688 26.0339917  26.08370512
     26.13337727 26.18300831 26.23259839 26.28214764 26.3316562  26.38112423
     26.43055186 26.47993924 26.5292865  26.57859379 26.62786123 26.67708897
     26.72627715 26.7754259  26.82453535 26.87360564 26.92263691 26.97162928
     27.02058288 27.06949786 27.11837433 27.16721243 27.21601228 27.26477401
     27.31349776 27.36218364 27.41083179 27.45944232 27.50801536 27.55655104
     27.60504947 27.65351078 27.70193509 27.75032252 27.79867319 27.84698722
     27.89526473 27.94350583 27.99171065 28.03987929 28.08801188 28.13610853
     28.18416936 28.23219447 28.28018398 28.32813801 28.37605666 28.42394005
     28.47178829 28.51960148 28.56737975 28.61512318 28.66283191 28.71050602
     28.75814564 28.80575086 28.85332179 28.90085854 28.94836121 28.99582991
     29.04326474 29.0906658  29.13803319 29.18536702 29.23266739 29.2799344
     29.32716815 29.37436874 29.42153627 29.46867084 29.51577254 29.56284148
     29.60987774 29.65688144 29.70385266 29.75079149 29.79769805 29.84457241
     29.89141468 29.93822494 29.98500329 30.03174983 30.07846465 30.12514783
     30.17179947 30.21841966 30.26500849 30.31156605 30.35809243 30.40458772
     30.451052   30.49748537 30.54388792 30.59025972 30.63660088 30.68291146
     30.72919157 30.77544128 30.82166069 30.86784986 30.9140089  30.96013788
     31.00623689 31.052306   31.09834531 31.1443549  31.19033484 31.23628522
     31.28220612 31.32809761 31.37395979 31.41979273 31.46559651 31.5113712
     31.5571169  31.60283367 31.64852159 31.69418074 31.7398112  31.78541305
     31.83098636 31.87653121 31.92204767 31.96753582 32.01299573 32.05842749
     32.10383115 32.14920681 32.19455452 32.23987438 32.28516643 32.33043077
     32.37566746 35.        ]

```python

```

```python
obj2 = lambda k: k - fct.transition3(k,k1,A,alpha,tau,rho,delta)
result = optimize.root_scalar(obj2,bracket=[1e-8,1],method='brentq')
result.root
```

```python
# Define parameters
capital_5,capital_6 = fct.transition_curve3(A1,alpha1,tau1,rho1,delta1)

# Plot
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.plot(capital_5,capital_6)
ax.plot(capital_5,capital_5, '--', color='grey')
#plt.text(27,17,'$k_t+1$')
#plt.text(22,27,'$k_t=k_t+1$')
#plt.text(13,10,'$k^*$')
ax.set_xlabel('$k_t$')
ax.set_ylabel('$k_t+1$')
ax.set_title('Transition curve')
ax.set_xlim([-0.001,1])
ax.set_ylim([0,1]);
plt.grid()
```

```python
K_ss_1=(((1-alpha1)*(1-tau1)*alpha1)/((2+rho1)*alpha1+(1+rho1)*(1-alpha1)*tau1)*(A1))**(1/(1-alpha1))
K_ss_1
```

    0.28660524259061815

**4.3. Finally, we study the capital accumulation $k_{t+1}$ for different levels of the discount parameter $\rho$:**

```python
# Make a plot
fig = plt.figure()
ax = fig.add_subplot(1,1,1)

#transition curves
for rho in [0.01,0.05,0.1,0.2,0.4]:
    capital_3,capital_4 = fct.transition_curve2(A,alpha,tau,rho,R)
    ax.plot(capital_3,capital_4,label=f'rho={rho}')

# Plot transition curves
ax.legend(loc='lower right',facecolor='white',frameon=True)
ax.plot(capital_3,capital_3, '--', color='grey')
ax.set_xlabel('$k_t$')
ax.set_ylabel('$k_t+1$')
ax.set_title('Transition curve')
ax.set_xlim([-0.001,20])
ax.set_ylim([0,20]);
fig.set_figwidth(9)
fig.set_figheight(5)
plt.grid()
```

We can conclude from this graph that the higher the discount parameter $\rho$ is, the lower the capital steady state $k_{ss}^{*}$ is. Thus, $\rho$ is negatively correlated to the capital steady state $k_{ss}^{*}$.

# 5. Conclusion

We solved a two-period overlapping generations model (OLG) in a 'pay-as-you-go' setting in order to find the optimal savings, capital accumulation and steady state of our model. The model was solved analytically with a solver. Then, the numerical part was solved with help of `Model.Project.py`, and different dynamics of capital accumulation were shown.

Overall, we found that our analytical and numerical steady converge to the same value in the long-run. Also, we found that the contribution that the young generation provide to the old, i.e. $\tau$, and the discouting parameter $\rho$ and optimal savings $s_{t}^{*}$ are all negatively correlated to capital accumulation.


#T-SNE and Kullback-Leibler divergence

This notebook contains the code for a recent blog post I wrote about the use of t-distributed stochastic neighbor embedding and its connection to the Kullback-Leibler divergence. T-SNE can be used to visualize and cluster high-dimensional datasets to make them accessible and gain intuition about the sparisity and distribution

# Kullback-Leibler divergence

The [Kullback-Leibler divergence](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence) is a well known quantity in information studiesa nd statistics. It is used to measure the differences between two probability distributions $P$ and $Q$, where $P$ is called the "true" distribution and $Q$ its model or approximation. The KL divergence is defined as

$$
\begin{align}
\text{KL}(P||Q) = & \sum_i P_i \log \left( \frac{P_i}{Q_i} \right) \\
 = & \sum_i P_i \log \left( P_i \right) - \sum_i P_i \log \left( Q_i \right) \\
 = & - H(P) + H(P, Q)
\end{align}
$$

where $H(P)$ and $H(P,Q)$ are the self-entropy and mutual information measures of the probability distributions.

Some properties:
- $\text{KL}(P||Q) \geq 0$: Easy to see when using first order Taylor expansion $\ln x \leq x-1$
- If $P = P_1 \cdot P_2$ with two independent distributions $P_1$, $P_2$, then $\text{KL}(P||Q) = \text{KL}(P_1||Q_1) + \text{KL}(P_2||Q_2)$, where $Q_1$, $Q_2$ are the respective models of $P_1$ and $P_2$
- $\text{KL}(P||Q)$ is **not** a metric, as it violates the triangle inequality $||x + y || \leq ||x|| + ||y||$ and is not symmeric, i.e. $\text{KL}(P||Q) \neq \text{KL}(Q||P)$

##Simulation of the Kullback-Leibler divergence

Let's have a look at our most favorite theorem - the central limit theorem. An example of it is given by $n$-identically distributed Bernoulli variables. If $Y_i \sim \text{Bern} (p)$, $i\in \{1,..., n\}$ then their sum approaches a Normal distribution

$$
\begin{equation}
Y = \sum_{i=1}^n Y_i \rightarrow \mathcal{N} \big(np, np(1-p)\big)
\end{equation}
$$

We can now use the KL divergence to quantify how far away from a normal distribution we are. In this case $\mathcal{N} \big(np, np(1-p)\big)$ is our model and the distribution of $Y$ is generated numerically

###Preparations and code

```python
import pymc
import seaborn as sns
import scipy.stats as stats
import numpy as np
import matplotlib.pyplot as plt

%matplotlib inline
```

```python
def calculate_single_kl_value(p, q):
    return p * (np.log(p) - np.log(q))

def single_bernoulli_draw(p, n_bern):
    bernoulli = pymc.Bernoulli('bern', p, size = n_bern)
    return float(bernoulli.value.sum())

def perform_bernoulli_experiment(p, n_bern, n_samples):
    bernoulli_successes = []
    for idx in range(n_samples):
        bernoulli_successes.append(single_bernoulli_draw(p, n_bern))
    return bernoulli_successes

def calculate_full_kullback_leibler(p, n_samples, n_bern):
    bernoulli_successes = perform_bernoulli_experiment(p, n_bern, n_samples)

    bern_hist, bern_loc = np.histogram(bernoulli_successes, bins=100, density=True)

    mu = p * n_bern
    sigma = mu * (1 - p)

    kl = 0
    null_cnt = 0
    for idx in range(len(bern_loc)-1):
        x = bern_loc[idx]
        p = bern_hist[idx]
        q = stats.norm.pdf(x, loc=mu, scale=sigma)
        if (p > 0.0) & (q > 0.0):
            kl += calculate_single_kl_value(p, q)
        else:
            null_cnt += 1

    return kl / (n_samples - null_cnt)
```

###Histogram of the Bernoulli variable

... looks pretty normal, I would say

```python

```

    0.00884955752212 0.0107956438107

```python
from scipy.stats import norm

fig = plt.figure(figsize=(10, 6.5))
data = perform_bernoulli_experiment(0.02, 10000, 10000)
hist = plt.hist(data , bins=99, normed= True, label = 'n-fold Bernoulli distribution');
plt.xlabel('number of successes', fontsize = 20);
plt.ylabel('number of occurences', fontsize = 20);
plt.tick_params(axis='both', which='major', labelsize=16);
plt.tick_params(axis='both', which='minor', labelsize=12);

mu, std = norm.fit(data)

x = np.linspace(min(data), max(data), 100)
p = norm.pdf(x, mu, std)
plt.plot(x, p, '--k', linewidth=2, label='Gaussian fit')

plt.xlim([min(data), max(data)])
plt.legend(loc='best', fontsize=16)
```

###KL curve with uncertainty and mean

```python
p = 0.02
n_bern = 10000
sample_sizes = [100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000, 50000]

lst_kl_values = []
for idx in range(10):
    kl_values = []
    for n_samples in sample_sizes:
        kl_values.append(calculate_full_kullback_leibler(p, n_samples, n_bern))
    lst_kl_values.append(kl_values)
kl_array = np.asarray(lst_kl_values)
```

```python
fig = plt.figure(figsize=(10, 6.5))
p1 = plt.loglog(sample_sizes, kl_array.mean(axis=0), '--o', markersize=10, linewidth = 3, label = 'mean');

p2 = plt.fill_between(sample_sizes,
                 kl_array.mean(axis=0) + 3* kl_array.std(axis=0),
                 kl_array.mean(axis=0) - 3* kl_array.std(axis=0),
                 color='green',
                 alpha='0.35',
                 label= 'sigma range')
plt.xlabel('number of realizations', fontsize = 20);
plt.ylabel('Kullback-Leibler divergence', fontsize = 20);
plt.xlim([min(sample_sizes), max(sample_sizes)])
plt.tick_params(axis='both', which='major', labelsize=16);
plt.tick_params(axis='both', which='minor', labelsize=12);
```

The Kullback-Leibler divergence quickly approaches zero and with as few as 1000 samples we're already well below $10^{-2}$ in difference to a Normal distribution. Long live the central limit theorem!

# Stochastic neighbor embedding

We can leverage the property of the KL divergency to do effective visualization of high dimensional datasets. The key trick is to realize that we can convert distances into probabilities in the high-dimensional embedding space and the low-dimensional mapping space. Let

$$
\begin{equation}
p_{ij} = \frac{\text{exp}\big(-||x_i-x_j||/2\sigma^2\big)} {\sum_{k\neq l}\text{exp}\big(-||x_k-x_l||/2\sigma^2\big)}
\end{equation}
$$

and

$$
\begin{equation}
q_{ij} = \frac{\big(1+||y_i-y_j||^2\big)^{-1}} {\sum_{k\neq l}\big(1+||y_k-y_l||^2\big)^{-1}}.
\end{equation}
$$

The vectors $x_i$ and $y_j$ refer to the embedded space and mapping space respectively. The distribution in the mapping space is a t-distribution with a single degree of freedom (Cauchy distribution), hence the name *t-SNE*: t-distributed Stochastic Neighbor Embedding.

To find a good embedding we want to minimize the Kullback-Leibler divergence between $p$ and $q$. In this way we enforce that the distances in the embedded space and mapping space lead to similar probabilities and hence the structure of the data in the embedded space is preserved during the mapping.

## Stochastic gradient descent

The cost function we want to minimize is

$$
\begin{equation}
C = \text{KL}(P||Q) = \sum_{i,j} p_{ij} \log \left( \frac{p_{ij}} {q_{ij}} \right)
\end{equation}
$$

and is best done using a stochastic gradient descent. The gradient is given by

$$
\begin{equation}
\frac{\partial C}{\partial y_i} = 4\sum_i (p_{ij} - q_{ij})(y_i - y_j) \big(1+||y_i-y_j||^2\big)^{-1}
\end{equation}
$$

and has a quite nice interpretation:
- the difference between the joint probability distributions acts as a "spring" force between the point $i$ and $j$.
- depending on the position and the spring constant the force either repels the points (the points are too close in the mapping space, such that $q_{ij} > p_{ij}$) or attracts points towards each other (due to large distances).
- the "inverse square law" prevents the so-called crowding problem, i.e. to many small attractive forces lead to a crowding of mapped points (e.g. in Sammon mapping). It also introduces a scale invariance at large distances, making clusters of points interact in the same way as single points - just like in gravity!!

##Numerical experiment with artifically clustered data

We explore t-SNE using artifically clustered data that lie at the corners of a 3-dim cube. Training a good t-SNE representation will allow us to learn an effective 2-dim distribution that reflects the true 3-dim distribution.

```python
import numpy as np
import numpy.random as rd
import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline
```

    :0: FutureWarning: IPython widgets are experimental and may change in the future.

```python
## This method can be used to produce clustered data in arbitraty dimensions.
## However, we will limit ourselves here to 3-dimensions and explicitly define the center points of the clusters

dimensions = 10
number_clusters = 18
initial_pts = [tuple(2 * rd.binomial(1, .5, size=dimensions) - 1) for x in range(number_clusters)]

def generate_random_points(center, number_of_points):
    return np.array([rd.normal(loc=loc, scale=0.05, size=number_of_points) for loc in center]).transpose()
```

```python
# initial_pts = [
#     (-1, -1, -1),
#     (-1, -1, 1),
#     (-1, 1, -1),
#     (1, -1, -1),
#     (-1, 1, 1),
#     (1, -1, 1),
#     (1, 1, -1),
#     (1, 1, 1)
# ]
```

```python
size_cluster = 100
lst = []
label = []

for idx, pt in enumerate(initial_pts):
    lst += [item for item in generate_random_points(pt, size_cluster).tolist()]
    label.append((np.ones(size_cluster) * idx).tolist())
```

```python
import sklearn.manifold as mf
```

```python
tsne =mf.TSNE(n_components=2, n_iter=175000, perplexity=87.5, random_state=13, method='exact')

Y = tsne.fit_transform(lst)
```

```python
fig = plt.figure(figsize=(10, 6.5))
plt.scatter(Y[:, 0], Y[:, 1], c=label, cmap=plt.cm.jet, s=50, alpha=1)
ax = plt.gca();
ax.axes.get_xaxis().set_ticklabels([]);
ax.axes.get_yaxis().set_ticklabels([]);
plt.xlabel("dimension 1 of t-SNE", fontsize=20);
plt.ylabel("dimension 2 of t-SNE", fontsize=20);
```

```python
from scipy.spatial.distance import pdist
```

```python
dst = pdist(lst, 'sqeuclidean')
fig = plt.figure(figsize=(10, 6.5));
plt.hist(dst, bins=100, normed=True);
plt.xlabel('square distances in {} dim'.format(dimensions), fontsize=20);

plt.tick_params(axis='both', which='major', labelsize=18)
plt.tick_params(axis='both', which='minor', labelsize=14)
```

```python
sigma = 30
prob_dist = np.exp(-1*dst/ sigma) / np.exp(-1*dst/ sigma).sum()

fig = plt.figure(figsize=(10, 6.5))
plt.hist(prob_dist, bins=1000);
plt.ylim([0, 12000]);
plt.gca().get_xaxis().set_ticks([])
plt.gca().get_yaxis().set_ticks([])
plt.xlabel('probability', fontsize=18)
plt.ylabel('occurence', fontsize=18)

```

```python
dst_2 = pdist(Y, 'sqeuclidean')
fig = plt.figure(figsize=(10, 6.5));
plt.hist(dst_2, bins=100, normed=True);

plt.xlabel('t-SNE square distances in 2 dim', fontsize=16);
plt.tick_params(axis='both', which='major', labelsize=14)
plt.tick_params(axis='both', which='minor', labelsize=12)
```

```python
sigma = 300
prob_dist = np.power(1+ dst_2 / sigma, -1) / np.power(1+ dst_2 / sigma, -1).sum()

fig = plt.figure(figsize=(10, 6.5))
plt.hist(prob_dist, bins=1000);
plt.ylim([0, 7000]);
plt.gca().get_xaxis().set_ticks([])
plt.gca().get_yaxis().set_ticks([])
plt.xlabel('probability', fontsize=18)
plt.ylabel('occurence', fontsize=18)
```

# Average distance between uniformly distributed points in $d$ dimensions

```python
import numpy.random as rd
from scipy.spatial.distance import pdist

mean_dist = []
for dim in range(1, 101):
    X = rd.uniform(low=-1.0, high=1.0, size=(1000, dim))
    mean_dist.append(pdist(X).mean())
```

```python
fig = plt.figure(figsize=(10,6.5));
plt.plot(mean_dist, linewidth=4);
d = np.arange(len(mean_dist))
y =np.sqrt( 0.6666 *  (d+1))
plt.plot(d, y);
plt.xlabel('dimension', fontsize=18);
plt.ylabel('avg. distance', fontsize=18);
plt.xticks(fontsize=14);
plt.yticks(fontsize=14);
```

```python

```

Numerical-Experimentation/t-SNE and the KL Divergence.ipynb


# Assignment-2

```python
import numpy as np
from matplotlib import pyplot as plt
from sympy import *
```

```python
def cov_inv(w):
    w_cov=np.cov(w.transpose())
    return w_cov,np.linalg.inv(w_cov)
def get_mean(w):
    return np.mean(w,axis=0)
def get_X():
    x,y=symbols('x y')
    X=np.array([x,y]).transpose()
    return x,y,X
def get_equation(X,w_mean,w_cov,w_inv,pw):
    c1=-0.5*(np.dot(np.subtract(X,w_mean).transpose(),np.dot(w_inv,np.subtract(X,w_mean))))
    c2=-(w_cov.shape[0]/2)*np.log(2*np.pi)
    c3=-0.5*np.log(np.linalg.det(w_cov))
    c4=np.log(pw)
    return c1+c2+c3+c4

def move_sympyplot_to_axes(p, ax):
    backend = p.backend(p)
    backend.ax = ax
    backend._process_series(backend.parent._series, ax, backend.parent)
    backend.ax.spines['right'].set_color('none')
    backend.ax.spines['bottom'].set_position('zero')
    backend.ax.spines['top'].set_color('none')
    plt.close(backend.fig)

def plot(w1,w2,g,x,y):
    p1 = plot_implicit(g, (x, -20, 20), (y, -20, 20), show=False)
    fig, ax = plt.subplots()
    move_sympyplot_to_axes(p1, ax)
    plt.setp(ax.yaxis.get_label(), 'rotation', 0)
    plt.scatter([x[0] for x in w1 ],[x[1] for x in w1 ],label="Class W1")
    plt.scatter([x[0] for x in w2 ],[x[1] for x in w2 ],label="Class W2")
    plt.title(label='Decision Boundary', fontsize='20', pad='20')

    plt.legend()
    plt.show()

```

```python
def get_result(w1,w2,pw1,pw2):
    w1_cov,w1_inv=cov_inv(w1)
    w2_cov,w2_inv=cov_inv(w2)
    w1_mean=get_mean(w1)
    w2_mean=get_mean(w2)
    x,y,X=get_X()
    g1=get_equation(X,w1_mean,w1_cov,w1_inv,pw1)
    g2=get_equation(X,w2_mean,w2_cov,w2_inv,pw2)
    g=g1-g2
    if((w1_cov==w2_cov).all() and (w1_cov[0][1]==0 and w1_cov[1][0]==0)):
        print("Following are the covariance matrix of the given classes:\n\nClass W1 Covariance Matrix:\n",w1_cov,"\n\nClass W2 Covariance Matrix:\n",w2_cov,"\n\nCase_1 of Baye's Decision Boundary(Since Covariance Matrices are equal and of form 1):\n")
    elif((w1_cov==w2_cov).all()):
        print("Following are the covariance matrix of the given classes:\n\nClass W1 Covariance Matrix:\n",w1_cov,"\n\nClass W2 Covariance Matrix:\n",w2_cov,"\n\nCase_2 of Baye's Decision Boundary(Since Covariance Matrices are equal but not of form 1):\n")
    else:
        print("Following are the covariance matrix of the given classes:\n\nClass W1 Covariance Matrix:\n",w1_cov,"\n\nClass W2 Covariance Matrix:\n",w2_cov,"\n\nCase_3 of Baye's Decision Boundary(Since Covariance Matrices are not equal):\n")

    plot(w1,w2,g,x,y)

```

# Question 1
Find and plot the decision boundary between class ω1 and ω2. Assume P(ω1) = P(ω2)

ω1 = [1,6; 3,4; 3,8; 5,6]

ω2 = [3,0; 1,-2; 3,-4; 5,-2]

```python
w1=np.array([[1,6],[3,4],[3,8],[5,6]])
w2=np.array([[3,0],[1,-2],[3,-4],[5,-2]])
pw1=0.5
pw2=0.5
get_result(w1,w2,pw1,pw2)
```

# Question 2
Find and plot the decision boundary between class ω1 and ω2. Assume P(ω1) = P(ω2)

ω1 = [1,-1; 2,-5; 3,-6; 4,-10; 5,-12; 6,-15]

ω2 = [-1,1; -2,5; -3,6; -4,10, -5,12; -6, 15]

```python
w1=np.array([[1,-1], [2,-5], [3,-6], [4,-10], [5,-12], [6,-15]])
w2=np.array([[-1,1], [-2,5], [-3,6], [-4,10], [-5,12], [-6,15]])
pw1=0.3
pw2=0.7
get_result(w1,w2,pw1,pw2)
```

# Question 3
Find and plot the decision boundary between class ω1 and ω2. Assume P(ω1) = P(ω2)

ω1 = [2,6; 3,4; 3,8; 4,6]

ω2 = [3,0; 1,-2; 3,-4; 5,-2]

```python
w1=np.array([[2,6], [3,4], [3,8], [4,6]])
w2=np.array([[3,0], [1,-2], [3,-4], [5,-2]])
pw1=0.5
pw2=0.5
get_result(w1,w2,pw1,pw2)
```