```julia
using CSV
using DataFrames
using PyPlot
using ScikitLearn # machine learning package
using StatsBase
using Random
using LaTeXStrings # for L"$x$" to work instead of needing to do "\$x\$"
using Printf

# (optional)change settings for all plots at once, e.g. font size
rcParams = PyPlot.PyDict(PyPlot.matplotlib."rcParams")
rcParams["font.size"] = 16

# (optional) change the style. see styles here: https://matplotlib.org/3.1.1/gallery/style_sheets/style_sheets_reference.html
PyPlot.matplotlib.style.use("seaborn-white")
```

## classifying breast tumors as malignant or benign

source: [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic))

> Features are computed from a digitized image of a fine needle aspirate (FNA) of a breast mass. They describe characteristics of the cell nuclei present in the image.

The mean radius and smoothness of the cell nuclei (the two features) and the outcome (M = malignant, B = benign) of the tumor are in the `breast_cancer_data.csv`.

```julia
df = CSV.read("breast_cancer_data.csv")
df[!, :class] = map(row -> row == "B" ? 0 : 1, df[:, :outcome])
first(df, 5)
```

	mean_radius	mean_smoothness	outcome	class

	Float64	Float64	String	Int64

5 rows × 4 columns

1	13.85	1.495	B	0

2	9.668	2.275	B	0

3	9.295	2.388	B	0

4	19.69	4.585	M	1

5	9.755	1.243	B	0

## visualize the two classes distributed in feature space

```julia
markers = Dict("M" => "x", "B" => "o")

figure()
xlabel("mean radius")
ylabel("mean smoothness")
for df_c in groupby(df, :outcome)
    outcome = df_c[1, :outcome]
    scatter(df_c[:, :mean_radius], df_c[:, :mean_smoothness], label="$outcome",
        marker=markers[outcome])
end
legend()
axis("equal")
```

## get data ready for classifiation in scikitlearn

scikitlearn takes as input:
* a feature matrix `X`, which must be `n_samples` by `n_features`
* a target vector `y`, which must be `n_samples` long (of course)

```julia
n_tumors = nrow(df)

X = zeros(n_tumors, 2)
y = zeros(n_tumors)
for (i, tumor) in enumerate(eachrow(df))
    X[i, 1] = tumor[:mean_radius]
    X[i, 2] = tumor[:mean_smoothness]
    y[i] = tumor[:class]
end
X # look at y too!
```

    300×2 Array{Float64,2}:
     13.85   1.495
      9.668  2.275
      9.295  2.388
     19.69   4.585
      9.755  1.243
     16.11   4.533
     14.78   2.45
     15.78   3.598
     15.71   1.972
     14.68   3.195
     13.71   3.856
     21.09   4.414
     11.31   1.831
      ⋮
     11.08   1.719
     18.94   5.486
     15.32   4.061
     14.25   5.373
     20.6    5.772
      8.671  1.435
     11.64   2.155
     12.06   1.171
     13.88   1.709
     14.9    3.466
     19.59   2.916
     14.81   1.677

## logistic regression

let $\mathbf{x} \in \mathbb{R}^2$ be the feature vector describing a tumor. let $T$ be the random variable that denotes whether the tumor is benign (0) or malignant (1). the logistic model is a probabilistic model for the probability that a tumor is malignant given its feature vector:

\begin{equation}
    \log \frac{Pr(T=1 | \mathbf{x})}{1-Pr(T=1 | \mathbf{x})} = \beta_0 + \boldsymbol \beta^\intercal \mathbf{x}
\end{equation}
where $\beta_0$ is the intercept and $\boldsymbol \beta \in \mathbb{R}$ are the weights for the features.

we will use scikitlearn to learn the  $\beta_0$ and $\boldsymbol \beta$ that maximize the likelihood.

```julia
@sk_import linear_model : LogisticRegression
```

    WARNING: redefining constant LogisticRegression

    PyObject <class 'sklearn.linear_model.logistic.LogisticRegression'>

```julia
lr = LogisticRegression(penalty="none", solver="newton-cg")

lr.fit(X, y)

println("β = ", lr.coef_)
println("β₀ = ", lr.intercept_)
```

    β = [1.1686607782175527 0.9420681231447378]
    β₀ = [-19.387890643955814]

prediction of the probability that a new tumor is 0 (benign) or 1 (malignant)

```julia
lr.predict_proba([10.0 2.5])
```

    1×2 Array{Float64,2}:
     0.995256  0.00474403

## visualize the learned model $Pr(T=1|\mathbf{x})$

```julia
radius = 5:0.25:30
smoothness = 0.0:0.25:20.0

lr_prediction = zeros(length(smoothness), length(radius))
for i = 1:length(radius)
    for j = 1:length(smoothness)
        x = [radius[i] smoothness[j]]
        lr_prediction[j, i] = lr.predict_proba(x)[2] # proba for probability
    end
end
```

```julia
figure()
pcolor(radius, smoothness, lr_prediction, cmap="viridis",
    vmin=0.0, vmax=1.0) # vmin,vmax enforce colorbar scale
colorbar()

for df_c in groupby(df, :outcome)
    outcome = df_c[1, :outcome]
    scatter(df_c[:, :mean_radius], df_c[:, :mean_smoothness], label="$outcome",
        marker=markers[outcome])
end
legend()
```

## making decisions: the ROC curve

this depends on the cost of a false positive versus false negative. (here, "positive" is defined as testing positive for "malignant")

> "I equally value minimizing (1) false positives and (2) false negatives."

$\implies$ choose $Pr(T=1|\mathbf{x})=0.5$ as the decision boundary.

> "I'd rather predict that a benign tumor is malignant (false positive) than predict that a malignant tumor is benign (false negative)."

$\implies$ choose $Pr(T=1|\mathbf{x})=0.2$ as the decision boundary. Even if there is a relatively small chance that the tumor is malignant, we still take action and classify it as malignant...

the receiver operator characteristic (ROC) curve is a way we can evaluate a classification algorithm without imposing our values and specifying where the decision boundary should be.

```julia
@sk_import metrics : roc_curve
@sk_import metrics : auc
```

    WARNING: redefining constant roc_curve

    PyObject <function auc at 0x7fe80b1fb7b8>

```julia
scores = lr.predict_proba(X)[:, 2]
```

    300-element Array{Float64,1}:
     0.14263839860751615
     0.0026092670076477047
     0.0018782588829001088
     0.9996447815854005
     0.0010942250321098578
     0.9760986635528334
     0.5480964640686227
     0.9200581620768566
     0.6962552287733829
     0.6852396841339906
     0.5663718098629624
     0.9999187138695917
     0.011596238415532208
     ⋮
     0.008004504524685206
     0.9996348109797668
     0.9122747253539892
     0.9111094723703275
     0.9999599017831867
     0.0003696570017181287
     0.022876063860596256
     0.014910308332017176
     0.17409413839661506
     0.7842086355766936
     0.9980794947953938
     0.37749924416866815

```julia
fpr, tpr, thresholds = roc_curve(y, scores)
```

    ([0.0, 0.0, 0.0, 0.005555555555555556, 0.005555555555555556, 0.011111111111111112, 0.011111111111111112, 0.027777777777777776, 0.027777777777777776, 0.03333333333333333  …  0.37222222222222223, 0.43333333333333335, 0.43333333333333335, 0.4722222222222222, 0.4722222222222222, 0.5055555555555555, 0.5055555555555555, 0.7666666666666667, 0.7666666666666667, 1.0], [0.0, 0.008333333333333333, 0.7416666666666667, 0.7416666666666667, 0.7583333333333333, 0.7583333333333333, 0.7666666666666667, 0.7666666666666667, 0.8083333333333333, 0.8083333333333333  …  0.9583333333333334, 0.9583333333333334, 0.975, 0.975, 0.9833333333333333, 0.9833333333333333, 0.9916666666666667, 0.9916666666666667, 1.0, 1.0], [1.9999999999999254, 0.9999999999999254, 0.7959831048160733, 0.7882403199356263, 0.7732403837747389, 0.7622568750185913, 0.7614526612308641, 0.6962552287733829, 0.6282401802701606, 0.5888691371793174  …  0.09889839902698462, 0.065245657335008, 0.0601149551828247, 0.05261077303643403, 0.0518823253258252, 0.043527327155129115, 0.04345926758794682, 0.008416240229296301, 0.008004504524685206, 0.00011874278075119236])

```julia
figure()
plot([0, 1], [0, 1], color="k")
plot(fpr, tpr, color="darkorange",
         label=@sprintf("ROC curve (area = %0.2f)", auc(fpr, tpr))
    )
legend()
xlabel("false positive rate")
ylabel("true positive rate")
```

tradeoff:
* threshold too small: classify all of the tumors as malignant, false positive rate very high
* threshold too large: classify all of the tumors as benign, false negative rate very high

somewhere in the middle (but still depending on the cost of a false positive versus false negative) is where we should operate.

the `auc`, area under the curve, has a probabilistic interpretation:
>  the area under the curve (often referred to as simply the AUC) is equal to the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative one (assuming 'positive' ranks higher than 'negative') -[Wikipedia](https://en.wikipedia.org/wiki/Receiver_operating_characteristic)

**warning**: always split your data into test or train or do cross-validation to assess model performance. we trained on all data here to see the mechanics of fitting a logistic regression model to data, visualizing the model, and creating an ROC curve.


# Initialize Enviroment

```python
from IPython.display import display, Math
from sympy import *
init_printing()

from helper import comparator_factory, comparator_eval_factory, comparator_method_factory

x,y,z = symbols('x y z')

comparator = comparator_factory('Before applying {}:','After:')
method_comparator = comparator_method_factory('Before calling {}:','After:')
eval_comparator = comparator_eval_factory('Before evaluation:','After evaluation:')
```

# Calculus

In this section, we will cover how to perform baisc calculus tasks such as derivatives, integrals,litmis and series expansion.

## Derivatives
To take derivatives, use diff()

### basic derivatives
Pass in an expresison and one symbol with respect to which the derivative is applied.

```python
expr = sin(x)

expr_diff = diff(expr,x)

comparator(expr, diff, x)
```

diff() can also be called as a method

```python
expr = sin(x)

method_comparator(expr, 'diff', x)
```

To create an unevaluated derivative, use Derivative class and initiate it with the same syntax with diff()

```python
expr = sin(x)

diff_expr = Derivative(expr,x)

print('Before evaluation:')
display(diff_expr)
```

And call ```doit()``` method to evaluate it.

```python
print('After evaluation:')
display(diff_expr.doit())
```

To reduce code duplication, we user ```eval_comparator``` to handle the comparasion in later notes

### higher order derivative
To take n order derivatives, pass the symbol n times or pass the symbol once and then follows it with n.

```python
expr = Derivative(x**4,x,x,x)

eval_comparator(expr)
```

You can achive the same calculation with the second syntax.

```python
expr = Derivative(x**4,x,3)

eval_comparator(expr)
```

### higher order partial derivatives.

Just pass the symbols in order, with the same syntax with single variable derivative.

```python
expr = Derivative(exp(x*y*z),x, y, y, z, z, z, z)

eval_comparator(expr)
```

Or user number to control the derivatives order for each symbol

```python
expr = Derivative(exp(x*y*z),x, y, 2, z, 4)

eval_comparator(expr)
```

# Integrals

## indefinte integrals
Similiar with derivatives, integrals can be performed through integral() by function or method. To construct an unevaluated expression, initialize an Integral class and call doit() to evaludate the calculation.

```python
expr = Integral(cos(x),x)

eval_comparator(expr)
```

### definite integrals

To perform a definite integral, pass in a tuple of variable symbol, lower bound and upper bound.

```python
expr = Integral(exp(-x),(x,0,oo))

eval_comparator(expr)
```

Note, $\infty$ in Sympy is oo (double lower case 'O')

### multiple integrals
To perform a multiple integral, pass multiple tuples of symbol variable and integral bounds.

```python
expr = Integral(exp(-x**2 - y**2), (x, -oo, oo), (y, -oo, oo))

eval_comparator(expr)
```

If Sympy cannot calculate an integral on an expression, it returns an unevaluated integral expression.

```python
expr = Integral(x**x)

eval_comparator(expr)
```

# Limits

Similiar with derivatives, limits can be performed through limit() by function or method. To construct an unevaluated expression, initialize an Limit class and call doit() to evaludate the calculation.

By default the limit is calculated from right to left with default setting ```dir = '+'```.

```python
expr = Limit(sin(x)/x, x, 0)

eval_comparator(expr)
```

To caluculate a limit from left to right. Pass ```dir='-'```

```python
expr = Limit(sin(x)/x, x, 0, dir = '-')

eval_comparator(expr)
```

# Series Expansion

Sympy can calculate an asymptotic series expansion of a funciton around a point $x_0$ with respect to  $O(x-x_0)^n$ by calling ```series``` method of an expression.

```python
expr = sin(x)

method_comparator(expr, 'series', x,0,6)
```

To remove the order term, call ```removeO()``` method

```python
print('Expanded expression without order term:')
expr.series(x,0,6).removeO()
```

# Reference
[Sympy Documentation](http://docs.sympy.org/latest/index.html)

# Related Articles
* [Sympy Notes I]({filename}0026_sympy_intro_1_en.ipynb)
* [Sympy Notes II]({filename}0027_sympy_intro_2_en.ipynb)
* [Sympy Notes III]({filename}0028_sympy_intro_3_en.ipynb)
* [Sympy Notes IV]({filename}0029_sympy_intro_4_en.ipynb)


```python
import numpy as np

import numpy as np
import scipy as sp
from scipy import linalg
from scipy import optimize
from scipy import interpolate
import sympy as sm

%matplotlib inline
import matplotlib.pyplot as plt
plt.style.use('seaborn-whitegrid')
from matplotlib import cm
from mpl_toolkits.mplot3d import Axes3D
```

# Gradient descent

Let $\boldsymbol{x} = \left[\begin{array}{c}
x_1 \\
x_2\\
\end{array}\right]$ be a two-dimensional vector. Consider the following algorithm:

**Algorithm:** `gradient_descent()`

**Goal:** Minimize the function $f(\boldsymbol{x})$.

1. Choose a tolerance $\epsilon>0$, a scale factor $ \Theta > 0$, and a small number $\Delta > 0$
2. Guess on $\boldsymbol{x}_0$ and set $n=1$
3. Compute a numerical approximation of the jacobian for $f$ by

    $$
    \nabla f(\boldsymbol{x}_{n-1}) \approx \frac{1}{\Delta}\left[\begin{array}{c}
    f\left(\boldsymbol{x}_{n-1}+\left[\begin{array}{c}
    \Delta\\
    0
    \end{array}\right]\right)-f(\boldsymbol{x}_{n-1})\\
    f\left(\boldsymbol{x}_{n-1}+\left[\begin{array}{c}
    0\\
    \Delta
    \end{array}\right]\right)-f(\boldsymbol{x}_{n-1})
    \end{array}\right]
    $$

4. Stop if the maximum element in $|\nabla f(\boldsymbol{x}_{n-1})|$ is less than $\epsilon$
5. Set $\theta = \Theta$
6. Compute $f^{\theta}_{n} = f(\boldsymbol{x}_{n-1} - \theta \nabla f(\boldsymbol{x}_{n-1}))$
7. If $f^{\theta}_{n} < f(\boldsymbol{x}_{n-1})$ continue to step 9
8. Set $\theta = \frac{\theta}{2}$ and return to step 6
9. Set $x_{n} = x_{n-1} - \theta \nabla f(\boldsymbol{x}_{n-1})$
10. Set $n = n + 1$ and return to step 3

**Question:** Implement the algorithm above such that the code below can run.

Define the rosenbrock function

```python
def _rosen(x1,x2):
    f = (1.0-x1)**2 + 2*(x2-x1**2)**2

x1 = sm.symbols('x_1')
x2 = sm.symbols('x_2')
f = (1.0-x1)**2 + 2*(x2-x1**2)**2

```

```python

def gradient_descent(f,x0,epsilon=1e-6,Theta=0.1,Delta=1e-8,max_iter=10_000):

    # step 1: initialize
    x = x0
    fx = f(x0)
    n = 1

    # step 2-6: iteration
    while n < max_iter:

        x1_variable=[x[0]+Delta,x[1]]
        x2_variable=[x[0],x[1]+Delta]
        # step 2: function and derivatives
        def rosen_jac(x):
            jac = np.zeros(2)
            jac[0] = f(x1_variable)-f(x)
            jac[1] = f(x2_variable)-f(x)
            return jac*(1/Delta)
        # step 3: check convergence
        if np.max(np.array(abs(rosen_jac(x)))) < epsilon:
            break

        x_prev = x
        fx_prev = fx

        # step 2: evaluate gradient
        jacx = rosen_jac(x)

        # step 3: find good step size (line search)

        fx_ast = np.inf
        theta_ast = Theta
        theta = Theta / 2
        if fx < fx_ast:
            fx_ast = fx
            theta_ast = theta

        # step 4: update guess
        x = x_prev - theta_ast * jacx

        # step 5: check convergence
        fx = f(x)
        if fx > fx_prev:
            break

        # d. update i
        n += 1

    return x,n
```

**Test case:**

```python
def rosen(x):
    return (1.0-x[0])**2+2*(x[1]-x[0]**2)**2

x0 = np.array([1.1,1.1])
try:
    x,it = gradient_descent(rosen,x0)
    print(f'minimum found at ({x[0]:.4f},{x[1]:.4f}) after {it} iterations')
    assert np.allclose(x,[1,1])
except:
    print('not implemented yet')
```

    minimum found at (1.0000,1.0000) after 578 iterations

```python
gradient_descent(rosen,x0)
```

    (array([1.00000114, 1.00000253]), 578)

Can use Nelder-Mead without analytical hessian

Newton if we have the analytical hessian

BFGS is the best without analytical

Maybe potential step sizes are the theta function with theta/2

##


Neccesary packages to run the project:

```python
import matplotlib.pyplot as plt    # A Python 2D plotting library to produce figures
import numpy as np                 # Package for scientific computing with Python
from scipy import linalg           # Used to do linear algebra functions
from scipy import optimize         # For maximizing and minimizing functions
import sympy as sm                 # Used to do symbolic mathematics
from sympy import *
import ipywidgets as widgets       # Package for making interactive figures
sm.init_printing(use_unicode=True) # This will make all expressions pretty print with unicode characters.
```

# Analysis of the Solow

This project will try to analyse a wide range of Solow models. More specificly the project will try to analyse the models of chapter 3, 5, 6 and 7 from the book "Introducing Advanced Macroeconomics - Growth and business cycles".

## The basic Solow model (Chapter 3)

### Introducing the model

The basic Solow model consists of 4 essential equations which is stated below.
1. $Y_t = F(K_t^d, L_t^d) = B K_t^\alpha L_t^{1-\alpha}$
2. $S_t = sY_t$
3. $L_{t+1} = (1+n)L_t$
4. $K_{t+1} - K_t = S_t - \delta K_t$

The model consists of the following variables and parameters:
1. $Y_t$ is GDP which is given as a Cobb-Douglas function.
2. $S_t$ is the amount of saving in the economy and is given as a constant fraction, $0<s<1$, of GDP.
3. $L_t$ is the amount of labour.
4. $K_t$ is the amount of capital.
5. $B$ is total factor productivity (TFP), $B>0$.
6. $\alpha$ is the share of income/output spent on capital, $0<\alpha<1$

### Analysing the model

In the first part of the analysis of the basic Solow model we find the steady state values for $k_t$ and $y_t$ respectively. In order to do that we use the sympy package which is a great tool for doing symbolic mathematics. We start by defing the symbols of model:

```python
Y = sm.symbols('Y_t')
K = sm.symbols('K_t')
L = sm.symbols('L_t')
y = sm.symbols('y_t')
k = sm.symbols('k_t')
B = sm.symbols('B')
s = sm.symbols('s')
n = sm.symbols('n')
alpha = sm.symbols('alpha')
delta = sm.symbols('delta')
```

### Finding steady state analytical

To find the steady state value of capital per worker we'll use the transition equation:

$k_{t+1} = \frac{1}{1+n} sBk_t^\alpha+(1-\delta)k_t$

```python
Transtion_equation = sm.Eq(k,(s*B*k**alpha+(1-delta)*k)/(1+n))
kss = sm.solve(Transtion_equation ,k)
print('The steady state value of capital per worker is given as')
kss
```

To find the steady state value of $y_t$ we firstly define the capital per capita function and then insert our steady state expression for $k^*$ into the output per capita production function:

```python
y=B*k**alpha
yss =y.subs({k:kss[0]})
print('The steady state value of income per worker is given as')
yss
```

We now got an expresison for both capital per worker and income per worker in steady state. In the next part of the analysis we'll find an numerical solution the the steady state values.

### Numerical solution to the model

We can find a numerical solution to the model as well, which we only do for income per worker. There are several methods to do this, but we will only demonstrate the Brent, Newton and Bisect method.

**Brent method:**

```python
def brent_basic_solow(a,b, B=1, alpha=1/3, delta=0.01, n=0.02, s=0.5):

    return optimize.brentq(lambda k: s*B*k**alpha - (n+delta)*k, a,b)

print(f'The steady state value with the Brent method is: {brent_basic_solow(1,100):.20f}')
```

    The steady state value with the Brent method is: 68.04138174397705540741

**Newton method:**

```python
def newton_basic_solow(x, B=1, alpha=1/3, delta=0.01, n=0.02, s=0.5):

    return optimize.newton(lambda k: s*B*k**alpha - (n+delta)*k, x)

print(f'The steady state value with the Newton method is: {newton_basic_solow(100):.20f}')
```

    The steady state value with the Newton method is: 68.04138174397716909425

**Bisect method:**

```python
def bisect_basic_solow(a,b, B=1, alpha=1/3, delta=0.01, n=0.02, s=0.5):

    return optimize.bisect(lambda k: s*B*k**alpha - (n+delta)*k, a,b)
print(f'The steady state with the Bisect method is: {bisect_basic_solow(1,100):.20f}')
```

    The steady state with the Bisect method is: 68.04138174397708382912

We see that steady value of income per worker is roughly the same for all three methods although you might notice they change slightly as we get to 12. digit. From this it is safe to say that the steady value is around 68.04.

### Visualisation of Solow diagram

We'll now try to visualise how capital converges towards it's steady state value.

```python
# We define the variables of the Solow diagram
def solowdiagram_basic_solow(k,B,s,n,delta,alpha,T):

# We make two empty lists. One for k and the 45-degree line
    k_next_period = [k]
    Linear_line = [0]

# And we create our functions which growths as time goes.
    for t in range(0,T):
        line = (n+delta)*t
        Linear_line.append(line)
        k_plus1 = s*B*t**alpha
        k_next_period.append(k_plus1)

# Plotting figure, limits and labels.
    plt.figure(dpi=120)
    plt.plot(k_next_period[:T], label='Growth, $sBk_t^a$', color='b')
    plt.plot(Linear_line[:T], label = 'Depression,$(n+\delta)*k_t$', color='r')
    plt.xlim(0,T)
    plt.ylim(0,Linear_line[-1])
    plt.xlabel('$k_t$')
    plt.ylabel('')
    plt.grid()
    plt.legend()
    plt.show()

# Showing figure. Change yourself to obtain a higher or lower steady state.
solowdiagram_basic_solow(0,1,0.5,0.02,0.01,1/3,200)
```

The blue line is the growth in capital. On the positive side we have that the higher the savings rate the more capital is accumulated, the higher total factor producitvity the more workers produce and the higher an alpha the higher capitals share is. The red line represents the depreciation. We have that population growth decreases capital per worker. Furthermore, the higher $\delta$ is the more depreciation there is on capital which also decreses capital per worker. You can change the values yourself to move see how the steady state value of capital per worker changes.

As you probably notice the steady state value is the same as we found in the numerical analysis of the model.

The presence of technology in the model allows there to be long-run growth in output per capita. This will conclude the analysis of the basic Solow model. In the next part of the analysis we'll expand the the model by including technologycal growth from the general Solow model.

## The general Solow model (chapter 5)
The next part of the project is going to analyze the Solow model with technology from chapter 5. The model looks a lot like the model from chapter 3 except that the total factor productivity is now being changed with an exogenious technological growth. For a better overview we'll introduce the full model although a lot of the equations is the same as in chapter 3.

### Introducing the model

1. $Y_t = K_t^\alpha (A_t L_t)^{1-\alpha}$
2. $S_t = sY_t$
3. $K_{t+1} - K_t = S_t - \delta K_t$
4. $L_{t+1} = (1+n)L_t$
5. $A_{t+1} = (1+g)A_t$

where $A_{t+1}$ is the growth in technology which is given exogenously as $(1+g)A_t$ and $\tilde{k}_t = \frac{K_t}{A_t L_t}$ and the model has the parameters $\alpha$, $n$, $g$ and $\delta$. The rest of the parameters have been described previously.

### Analysing the model

In the first part of the analysis of the general Solow model we find the marginal product with respect to capital and labour. Following that we'll once again find the steady state values for $k_t$ and $y_t$ respectively. We start by defing the symbols of model.

```python
Y = sm.symbols('Y_t')
y = sm.symbols('y_t')
K = sm.symbols('K_t')
k = sm.symbols('k_t')
A = sm.symbols('A')
s = sm.symbols('s')
n = sm.symbols('n')
g = sm.symbols('g')
L = sm.symbols('L_t')
alpha = sm.symbols('alpha')
delta = sm.symbols('delta')
kstar = sm.symbols('k_t^*')
ystar = sm.symbols('y_t^*')
ktilde = sm.symbols('ktilde_t')
ytilde = sm.symbols('ytilde_t')
ytildestar = sm.symbols('ytilde_t^*')
```

### Finding the marginal product with respect to capital and labour

To find the marginal product of capital we take the derivative of the production function with respect to K. Following this we take the derivative of the production function with respect to L to find the the marginal product of labour.

```python
Y = K**alpha*A**(1-alpha)*L**(1-alpha)

r = diff(Y, K)
print('The marginal product of capital which equals the real rental rate is given as')
simplify(r)
```

This can also be written as:

$r_t = \alpha \left(\frac{K_t}{A_tL_t} \right)^{\alpha-1}$

We find the marginal product with respect to L.

```python
w = diff(Y, L)
print('The marginal product of labour which equals the real wage is given as')
simplify(w)
```

Which is an expression for the real wage. This can also be written as:

$w_t = (1-\alpha) \left(\frac{K_t}{A_tL_t} \right)^\alpha A_t$

### Finding steady state analytical

This time we'll take our starting point in the Solow equation which is stated below:

$\tilde{k}_{t+1}-\tilde{k}_t = \frac{1}{(1+n)(1+g)} (s*\tilde{k}_t^\alpha-(n+g+delta+ng)*\tilde{k}_t)$

We know that in steady state $\tilde{k}_{t+1} = \tilde{k}_t = \tilde{k}^*$ which we use to find steady state:

```python
eq = sm.Eq((s*ktilde**alpha-(n+g+delta+n*g)*ktilde)/((1+n)*(1+g)))
eq
```

We then isolate k which gives us the result $\tilde{k}^*$:

```python
k_tilde_star = sm.solve(eq,ktilde)[0]
k_tilde_star
```

What we're really interested in is the capital per worker and not technology-adjusted capital per worker. Since $\tilde{k}_t^* = Ak_t^*$ we get the following result:

```python
k_star = sm.Eq(kstar, A*k_tilde_star)
k_star
```

We now got the steady state value for capital per worker. It's quite easy to find $\tilde{y}_t^*$ now since we know that $\tilde{y}_t^* = \tilde{k}_t^\alpha$

```python
y_tilde_star = sm.Eq(ytildestar, k_tilde_star**alpha)
y_tilde_star
```

Output per worker is then given as:

```python
y_star = sm.Eq(ystar, A*k_tilde_star**alpha)
y_star
```

### Numerical solution to the model

We will also find a numerical solution to the general Solow model where we once again will use the Brent, Newton and Bisect method.

```python
def brent_general_solow(a,b, alpha=1/3, delta=0.01, n=0.02, s=0.5, g=0.01):

    return optimize.brentq(lambda k: s*k**alpha - (n+delta+g+n*g)*k, a,b)

print(f'The steady state value with the Brent method is: {brent_general_solow(1,100):.20f}')
```

    The steady state value with the Brent method is: 43.86477710563418241918

```python
def newton_general_solow(x, alpha=1/3, delta=0.01, n=0.02, s=0.5, g=0.01):

    return optimize.newton(lambda k: s*k**alpha - (n+delta+g+n*g)*k, x)

print(f'The steady state value with the Newton method is: {newton_general_solow(100):.20f}')
```

    The steady state value with the Newton method is: 43.86477710563421794632

**Bisect method:**

```python
def bisect_general_solow(a,b, alpha=1/3, delta=0.01, n=0.02, s=0.5, g=0.01):

    return optimize.bisect(lambda k: s*k**alpha - (n+delta+g+n*g)*k, a,b)
print(f'The steady state with the Bisect method is: {bisect_general_solow(1,100):.20f}')
```

    The steady state with the Bisect method is: 43.86477710563546850153

Once again we see that the steady state value of income per worker is roughly the same until the 12. digit.

### Visualization of the general Solow diagram

We visualise the Solow diagram of the model.

```python
# We define the variabels of the Solow diagram
def solowdiagram_general_solow(k,alpha,delta,s,n,g,T):

# We make a list for k and the 45-degree line.
    k_next_period = [k]
    Linear_line = [0]

# And we create our functions which goes from 1 to infinity which we call T(= time).
    for t in range(1,T):
        line = (n+delta+g+n*g)*t
        Linear_line.append(line)
        k_plus = s*t**alpha
        k_next_period.append(k_plus)

# Plotting figure, limits and labels.
    plt.figure(dpi=120)
    plt.plot(k_next_period[:T], label='Growth, $sk_t^a$', color='b')
    plt.plot(Linear_line[:T], label = 'Depression, $(n+\delta+g+ng)k_t$', color='r')
    plt.xlim(0,T)
    plt.ylim(0,Linear_line[-1])
    plt.xlabel('$k_t$')
    plt.ylabel('')
    plt.grid()
    plt.legend()
    plt.show()

# Showing figure. Change yourself to obtain a higher or lower steady state.
solowdiagram_general_solow(0,1/3,0.01,0.5,0.02,0.01,200)
```

We've set the parameters to the same values as before and set $g=0.01$. As before you can change the parameters yourself. Although it might be hard to see, the steady state value of $k_t$ has decreased compared with the model in chapter 3. This is because we now look at the technology-adjusted capital per worker.

Adding technology to the model ensures that there is growth in steady, but also brings it to a lower level. This makes sense as countries that are considered to be in steady state, still experience growth in GDP and GDP per capita. You may also notice that the intial steady state value is the same as we found in the numerical solution. This will conclude the analysis of the general Solow model.

In the next part of the project we will be focusing on the Solow model from chapter 6 which includes human capital.

## The Solow model with human capital (Chapter 6)

The third part of our project will focus on the Solow model with human capital. This model addresses the two discrepancies between the empiri and the models predictions.

### The model is given as following,

1. $Y_t = K_t^\alpha  H_t^\varphi (A_t L_t)^{1-\alpha-\varphi}$
2. $L_{t+1} = (1+n)L_t$
3. $A_{t+1} = (1+g)A_t$
4. $H_{t+1} = s_H Y_t + (1+\delta)H_t$
5. $K_{t+1} = s_K Y_t +(1+\delta)K_t$

$r_t = \alpha \left(\frac{K_t}{A_t L_t}\right)^{\alpha-1}\left(\frac{H_t}{A_t L_t}\right)^\varphi A_t$

$w_t = (1-\alpha) \left(\frac{K_t}{A_t L_t}\right)^\alpha \left(\frac{H_t}{A_t L_t}\right)^\varphi A_t$

The model consists of the following variables and parameters:
1. $Y_t$ is GDP, which is given as a Cobb-Douglas function.
2. $L_t$ is the amount of labour. The labour force has a growth rate of 'n'
3. $A_t$ is the technological level and it has a  growth rate of 'g'
4. $K_t$ is the amount of capital.
7. $s_H$ and $s_K$ is the amount of saving in the economy used on respectivly human capital and physical capital. They're both given as a constant fraction of GDP
5. $\sigma$ Physical capitals contribution to GDP growth
6. $\alpha$ Physical capitals contribution to GDP growth

```python
# Defining symbols for sympy

# period t
K = sm.symbols("K_t")
H = sm.symbols("H_t")
A = sm.symbols("A_t")
L = sm.symbols("L_t")

ktilde = sm.symbols("ktilde_t")
htilde = sm.symbols("htilde_t")
ytilde = sm.symbols("ytilde_t")

# Steady state
kstar = sm.symbols("ktilde^*")
hstar = sm.symbols("htilde^*")
ystar = sm.symbols("ytilde^*")

# Parameters
phi = sm.symbols("varphi")
sK = sm.symbols("s_K")
sH = sm.symbols("s_H")
alpha = sm.symbols('alpha')
delta = sm.symbols('delta')
varphi = sm.symbols('varphi')
g = sm.symbols('g')
n = sm.symbols('n')
```

```python
# Defining the variables in sympy

# GDP
Y = K**alpha*H**phi*(A*L)**(1-alpha-phi)

# Physical capital
Kt1 = sK*Y +(1-delta)*K

# Human capital
Ht1 = sH*Y +(1-delta)*H

# Technology level and labor force
At1 = (1+g)*A
Lt1 = (1+n)*L
```

```python
# By differentiation Yt (GDP) respectively for K and L you get the interest rate and the wages in the economy.
r = sm.simplify(sm.diff(Y,K))
w = sm.simplify(sm.diff(Y,L))
```

```python
# interest rate
r
```

The intresrest rate is equivalent to the one shown in the model presentation

$ H_t^\varphi K_t^{\alpha-1}\alpha(A_t L_t)^{-\alpha-\varphi+1} =
\alpha \left(\frac{K_t}{A-t L_t}\right)^{\alpha-1} \left(\frac{H_t}{A_t L_t}\right)^\varphi$

```python
# real Wages
w
```

The real wages is equivalent to the one shown in the model presentation
$ \frac{H_t^\varphi K_t^{\alpha}(A_t L_t)^{-\alpha-\varphi+1} (-\alpha-\varphi+1)}{L_t} =
(1-\alpha) \left(\frac{K_t}{A-t L_t}\right)^{\alpha} \left(\frac{H_t}{A_t L_t}\right)^\varphi A_t$

### Analysing the model (The Law of motion)
The Transition equation is a bit more complicated than the one in chapter 5 since this model includes two first order differential equations.
Even so the models are very a like, by setting $\varphi =  1$ and dropping equation 4 the model from chapter 5 appears.

The technology adjusted variables per effective worker are defined as following:

$ \tilde{yt}  \equiv \frac{y_t}{A_t}   \equiv \frac{Y_t}{A_tL_t} \hspace 1 cm \tilde{k_t}  \equiv \frac{k_t}{A_t}   \equiv \frac{K_t}{A_tL_t} \hspace 1 cm  \tilde{h_t}  \equiv \frac{h_t}{A_t}   \equiv \frac{H_t}{A_tL_t} $

We derive the technology adjusted variables by dividing both sides of the equations (1), (4) and (5)  by $A_{t+1}*L_{t+1}$

```python
# Calculation of the technology adjusted GDP per effective worker - Equation (5) / [(2)(3)]
ytilde = sm.simplify((Y/(A*L)))
ytilde
```

Per definition the technology adjusted GDP per effective worker can also be expressed as:

$ \tilde y_t = \tilde k_t^\alpha \tilde h_t^\phi$

```python
# Calculation of the technology adjusted physical capital per effective worker - Equation (5) / [(2)(3)]
ktildet1 = sm.simplify(Kt1/(At1*Lt1))
ktildet1
```

```python
# Calculation of the technology adjusted human capital per effective worker - Equation (4) / [(2)(3)]
htildet1 = sm.simplify(Ht1/(At1*Lt1))
htildet1
```

By using the definition of $\tilde k_t \hspace 0.3 cm \text and \hspace 0.3 cm  \tilde h_t$ ,  substituting Y into the techonology adjusted human capital and physical capital, it can be reduced to the following functions:

$\text 6. \hspace 1 cm \tilde k_t = \frac{1}{(1+n)(1+g)} (s_K \tilde k_t^\alpha \tilde h_t^\varphi +(1-\delta)\tilde k_t)\\
\text 7. \hspace 1 cm \tilde h_t = \frac{1}{(1+n)(1+g)} (s_h \tilde k_t^\alpha \tilde h_t^\varphi +(1-\delta)\tilde h_t)$

### Finding steady state values for key endogenous variable

**Step 1:** Subtracting $\tilde k_t$ on both sides of equation 6 and subtracting $\tilde h_t$ on both sides of equation 7 to rewrite them to the solow equations

```python
# Defining/calculting the solow equation for physical capital
ksolow = sm.simplify(ktildet1-K/(A*L))
ksolow
```

```python
# Defining/calculting the solow equation for human capital
hsolow = sm.simplify(htildet1 - H/(A*L))
hsolow
```

**Step 2:** In steady state $ \tilde k_{t+1} = \tilde k_t$ and $\tilde h_{t+1} = \tilde h_t,$ so it makes perfect sence to set the equation equal to 0.

```python
ksolowss= sm.Eq(ksolow)
ksolowss
```

```python
hsolowss = sm.Eq(hsolow)
hsolowss
```

```python
# By using the definitons listed through out this chapter of the project the two equations above can be reduced to:

ksolowss1 = sm.Eq(sK*ktilde**alpha*htilde**varphi - (n + g + delta + n*g)*ktilde)
hsolowss1= sm.Eq(sH*ktilde**alpha*htilde**varphi - (n + g + delta + n*g)*htilde)
```

```python
# The solow equation for physical capital in steady state
ksolowss1
```

```python
# The solow equation for physical capital in steady state
hsolowss1
```

**Step 3:** isolating $H_t$ and $ K_t$

```python
# Finding steady state for physical capital
kss = sm.solve(ksolowss1, ktilde)[0]
kSS = sm.Eq(kstar, sm.solve(ksolowss1, ktilde)[0])
kSS
```

substituting  $\tilde h^{-\varphi}$ as a fuction of respectivly $\tilde k_t$

```python
kss1 = kss.subs(htilde, (ktilde*sH)/sK)
kss2 = sm.Eq(kstar, kss.subs(htilde, (ktilde*sH)/sK))
kss2
```

```python
# finding steady state for human capital
hss = sm.solve(hsolowss1, htilde)[0]
hSS = sm.Eq(hstar, sm.solve(hsolowss1, htilde)[0])
hSS
```

substituting  $\tilde k^{-\alpha}$ as a fuction of $\tilde h_t$

```python
hss1 = hss.subs(ktilde, (htilde*sH)/sK)
hss2 = sm.Eq(hstar, hss.subs(ktilde, (htilde*sH)/sK))
hss2
```

```python
# substituting steady state variables into technology adjusted GDP per efficient worker

ys = sm.Eq(ystar, kss1**alpha * hss1 **varphi)
ys
```

### Numerical solution to the model

Once again we find a numerical solution with the Brent, Newton and Bisect method.

```python
def human_brentq(a=1, b=100, alpha=0.3, phi=0.3, delta=0.01, sK=0.2, sH=0.2, n=0.02, g=0.01):

    return print(f'The steady state value for k with the Brent method is: {optimize.brentq(lambda k: sK*k**alpha*((sH/sK)*k)**phi-(n+g+delta+n*g)*k, a, b):.20f} \nThe steady state value for h with the Brent method is: {optimize.brentq(lambda h: sH*((sK/sH)*h)**alpha*h**phi-(n+g+delta+n*g)*h, a, b):.20f}')
human_brentq()
```

    The steady state value for k with the Brent method is: 55.20899689926860531841
    The steady state value for h with the Brent method is: 55.20899689926860531841

```python
def human_newton(x=1000, alpha=0.3, phi=0.3, delta=0.01, sK=0.2, sH=0.2, n=0.02, g=0.01):

    return print(f'The steady state value for k with the Newton method is: {optimize.newton(lambda k: sK*k**alpha*((sH/sK)*k)**phi-(n+g+delta+n*g)*k, x):.20f} \nThe steady state value for h with the Newton method is: {optimize.newton(lambda h: sH*((sK/sH)*h)**alpha*h**phi-(n+g+delta+n*g)*h, x):.20f}')
human_newton()
```

    The steady state value for k with the Newton method is: 55.20899689926865505640
    The steady state value for h with the Newton method is: 55.20899689926865505640

```python
def human_bisect(a=1, b=100, alpha=0.3, phi=0.3, delta=0.01, sK=0.2, sH=0.2, n=0.01, g=0.02):

    return print(f'The steady state value for k with the Bisect method is: {optimize.bisect(lambda k: sK*k**alpha*((sH/sK)*k)**phi-(n+g+delta+n*g)*k, a, b):.20f} \nThe steady state value for h with the Bisect method is: {optimize.bisect(lambda h: sH*((sK/sH)*h)**alpha*h**phi-(n+g+delta+n*g)*h, a, b):.20f}')
human_bisect()
```

    The steady state value for k with the Bisect method is: 55.20899689926760345315
    The steady state value for h with the Bisect method is: 55.20899689926760345315

And we can conclude that the steady steady value is roughly the same howsoever the method.

### Simulation and visiual presentation of results -  solow with human capital

We now present the model by visualising the Solow diagram.

```python
def simulate_human_capital(sK, sH, g, n, T, alpha, phi, delta, htilde, ktilde):

# Defining arays to contain values of the functions
    ktilde_movement = [ktilde]
    htilde_movement = [htilde]

# Defing functions to add to our arays
    for t in range(1,T):
        ktilde = t
        ktilde_np = ((ktilde**(-alpha+1)*(delta+n*g+n+g))/sK)**(1/(phi))
        ktilde_movement.append(ktilde_np)
        htilde = t
        htilde_np = ((htilde**(-alpha)*(delta+n*g+n+g))/sH)**(1/(phi-1))
        htilde_movement.append(htilde_np)

# Setting up the data plot
    plt.figure(dpi=120)
    plt.plot(htilde_movement[:T], label='$\Delta \~{h} = 0$', color='b')
    plt.plot(ktilde_movement[:T], label='$\Delta \~{k} = 0$', color='r')
    plt.xlim(0,T)
    plt.ylim(0,T)
    plt.xlabel('$\~{k}$')
    plt.ylabel('$\~{h}$')
    plt.grid()
    plt.legend()
    plt.show()

widgets.interact(simulate_human_capital,
                 htilde = widgets.fixed(0),
                 ktilde = widgets.fixed(0),
                 alpha = widgets.FloatSlider(description='$\u03B1$', min=0, max=0.5, step=0.05, value=0.3),
                 phi = widgets.FloatSlider(description='$\u03C6$', min=0, max=0.5, step=0.05, value=0.3),
                 delta = widgets.FloatSlider(description='$\u03B4$', min=0.01, max=0.1, step=0.01, value=0.01),
                 sK = widgets.FloatSlider(description='$s_K$', min=0.1, max=0.4, step=0.01, value=0.2),
                 sH = widgets.FloatSlider(description='$s_H$', min=0.1, max=0.4, step=0.01, value=0.2),
                 n = widgets.FloatSlider(description='$n$', min=0.01, max=0.1, step=0.005, value=0.02),
                 g = widgets.FloatSlider(description='$g$', min=0.01, max=0.1, step=0.005, value=0.01),
                 T = widgets.IntSlider(description='$T$', min=1, max=1000, step=10, value=100))
```

    interactive(children=(FloatSlider(value=0.2, description='$s_K$', max=0.4, min=0.1, step=0.01), FloatSlider(va…

    <function __main__.simulate_human_capital(sK, sH, g, n, T, alpha, phi, delta, htilde, ktilde)>

As shown in the visual representation by adding an extra parameter, human capital, the convergence towards steady state happens at a slower speed. This means that for an initial amount of human and physical capital, it is going to take a longer amount of time for a country to get to their steady state compared to the previous chapters. Once again we see that the steady state value equals the one we found in the numerical analysis.

## Solow Model with land (Chapter 7)

```python
import matplotlib.pyplot as plt
from sympy.plotting import plot
import numpy as np                 # Package for scientific computing with Python
from scipy import linalg           # Used to do linear algebra functions
from scipy import optimize         # For maximizing and minimizing functions
import sympy as sm                 # For solving mathematics with symbols
from sympy import *
from sympy import latex            # Used to print results as LaTeX code
import ipywidgets as widgets
sm.init_printing(use_unicode=True) # This will make all expressions pretty print with unicode characters.
```

### The model

The model with land consists of the following equations:

1. $Y_t=K^\alpha_t(A_tL_t)^\beta X^\kappa$,$\hspace{0.5cm}$ $\alpha>0,$ $\hspace{0.5cm}$ $\beta>0$,$\hspace{0.5cm}$$\kappa>0$,$\hspace{0.5cm}$ $\alpha+\beta+\kappa=1$
2. $K_{t+1}=sY_t+(1-\delta)K_t$,$\hspace{0.5cm}$    $0<s<1$,$\hspace{0.5cm}$  $0<\delta<1$.
3. $L_{t+1}=(1+n)L_t$, $\hspace{0.5cm}$  $n \geq 0$.
4. $A_{t+1}=(1+g)A_t$, $\hspace{0.5cm}$ $g \geq 0$.

Where $Y_t$ is output, $K_t$ is capital, $A_t$ is technology, $L_t$ is the population, and $X$ is land (which does not evolve over time). Using the above equations one will arrive at a per-capita output that can be written as:

$y_t=k_t^\alpha A_t^\beta x_t^\kappa$

Using that $z_t=\frac{k_t}{y_t}$ one gets the following expression:

$z_t=k_t^{1-\alpha}A_t^{-\beta}x_t^{-\kappa}$

We wish to find the steady state capital-output ratio and this is done by firstly defining the variables which ables us to use the sympy function to do calculations with the variables.

```python
#defining variables:
y = sm.symbols('y_t')
Y = sm.symbols('Y_t')
K = sm.symbols('K_t')
k = sm.symbols('k_t')
X = sm.symbols('X')
x = sm.symbols('x_t')
A = sm.symbols('A_t')
L = sm.symbols('L_t')
z = sm.symbols('z_t')
z1 = sm.symbols('z_t+1')
A1 = sm.symbols('A_t+1')
L1 = sm.symbols('L_t+1')
x1 = sm.symbols('x_t+1')
k1 = sm.symbols('k_t+1')
K1 = sm.symbols('K_t+1')
n = sm.symbols('n')
s = sm.symbols('s')
g = sm.symbols('g')
S = sm.symbols('z^*')
kappa = sm.symbols('kappa',positive = True)
alpha = sm.symbols('alpha', positive = True)
beta = sm.symbols('beta',positive = True)
delta = sm.symbols('delta',positive = True)
```

```python
cap_out = sm.Eq(z,(k**(1-alpha)*A**(-beta)*x**(-kappa)))
cap_out
```

```python
#Redifining the equation to t+1 by substituting
cap_out = cap_out.subs(z,z1)
cap_out = cap_out.subs(A,A1)
cap_out = cap_out.subs(k,k1)
cap_out = cap_out.subs(x,x1)
cap_out
```

Inserting that $k_{t+1}=\left(\frac{K_{t+1}}{L_{t+1}}\right)$  and $x_{t+1}=\left(\frac{X}{L_{t+1}}\right)$ and also inserting equations 2, 3 and 4 from the model one gets:

$z_{t+1}=\left( \frac{sY_t+(1-\delta)K_t}{(1+n)L_t}  \right)^{1-\alpha}(1+g)^{-\beta}A_t^{-\beta}\left(   \frac{X}{(1+n)L_t} \right)^{-\kappa}$

$=\left(\frac{1}{(1+n)(1+g)}\right)^\beta(sy_t+(1-\delta)k_t)^{1-\alpha}A_t^{-\beta}x_t^{-\kappa}$

$=\left(\frac{1}{(1+n)(1+g)}\right)^\beta \left( s\frac{y_t}{k_t}+(1-\delta)\right)^{1-\alpha}k_t^{1-\alpha} A_t^{-\beta}x_t^{-\kappa}$

Lastly inserting $z_t=k_t^{1-\alpha}A_t^{-\beta}x_t^{-\kappa}$ one arrives at:

$=\left(\frac{1}{(1+n)(1+g)}\right)^\beta \left(\frac{s}{z_t}+(1-\delta)\right)^{1-\alpha}z_t$

This can finally be written as:

```python
#Creating a sympy equation
Transition = sm.Eq(z1, (1/((1+n)*(1+g)))**beta*(s+(1-delta)*z)**(1-alpha)*z**alpha)
Transition
```

The sympy solve for function gives an error when trying to isolate $z_t$ which is known to be a common error when solving equations with powers. Therefore the isolating is done by brute-force setting $z_{t+1}=z_t=z^*$ :

$z_t^{1-\alpha}=\left(\frac{1}{(g+1)(n+1)}\right)^\beta (s+z_t(-\delta+1))^{1-\alpha}$

$z_t=\left(\frac{1}{(g+1)(n+1)}\right)^\frac{\beta}{1-\alpha}s+z_t(-\delta+1)\left(\frac{1}{(g+1)(n+1)}\right)^\frac{\beta}{1-\alpha}$

$z_t-z_t(-\delta+1)\left(\frac{1}{(g+1)(n+1)}\right)^\frac{\beta}{1-\alpha}=\left(\frac{1}{(g+1)(n+1)}\right)^\frac{\beta}{1-\alpha}s$

$z_t\left(1-(-\delta+1)\left(\frac{1}{(g+1)(n+1)}\right)^\frac{\beta}{1-\alpha}\right)=\left(\frac{1}{(g+1)(n+1)}\right)^\frac{\beta}{1-\alpha}s$

$z_t=\frac{\left(\left(\frac{1}{(g+1)(n+1)}\right)^\frac{\beta}{1-\alpha}s\right)}{\left(1-(-\delta+1)\left(\frac{1}{(g+1)(n+1)}\right)^\frac{\beta}{1-\alpha}\right)}$

$z_t^{-1}=\frac{\left(1-(-\delta+1)\left(\frac{1}{(g+1)(n+1)}\right)^\frac{\beta}{1-\alpha}\right)}{\left(\left(\frac{1}{(g+1)(n+1)}\right)^\frac{\beta}{1-\alpha}s\right)}$

$z_t^{-1}=\frac{1}{\left(\left(\frac{1}{(g+1)(n+1)}\right)^\frac{\beta}{1-\alpha}s\right)}-\frac{(-\delta+1)\left(\frac{1}{(g+1)(n+1)}\right)^\frac{\beta}{1-\alpha}}{\left(\left(\frac{1}{(g+1)(n+1)}\right)^\frac{\beta}{1-\alpha}s\right)}$

$z_t^{-1}=\frac{((1+n)(1+g))^{\frac{\beta}{1-\alpha}}}{s}-\frac{(-\delta+1)}{s}$

$z_t=\frac{s}{((1+n)(1+g))^{\frac{\beta}{1-\alpha}}-(-\delta+1)}$

This can then be denoted as $z^*$ for steady state:

$z^*=\frac{s}{((1+n)(1+g))^{\frac{\beta}{1-\alpha}}-(-\delta+1)}$

Furthermore by using that $1-\alpha=\beta+\kappa$ we can substitute this into the steady state function:

```python
ss = sm.Eq(S, (((s)/((((1+n)*(1+g))**(beta/(1-alpha)))-(-delta+1)))))
ss = ss.subs(1-alpha, beta+kappa) #inserting beta+kappa for 1-alpha
ssz = sm.solve(ss, S) #solving the equation for steady state capital-output ratio
ssz
```

```python
ssz_func = sm.lambdify((s,delta,g,n,beta,kappa),ssz) #Turning the variables into a python function
ssz_func(0.25, 0.05, 0.02, 0.01, 0.6, 0.2)
```

So the model (with the reasonable parameter values) suggests that an economy in steady state has a capital-output ratio of around 3.445.

### Visualization of convergence to steady state

Using the transition equation for the capital-output ratio from earlier, a diagram showing the convergence to the steady state value of 3.445 will be illustrated in the following section:

```python
# Here the intervals for the different parameters are defined
s = np.arange(0,1)
z = np.arange(100)
beta = np.arange(0,1)
delta = np.arange(0,0.1)
n = np.arange(0,0.2)
g = np.arange(0,0.04)
alpha = np.arange(0,1)

def interactive_transition1(z,s,beta,delta,n,g,alpha):
    #Defines the capital-output ratio and a 45 degree line
    z0 = (((1)/((1+n)*(1+g)))**beta*(s+((1-delta)*z))**(1-alpha)*z**alpha)
    m0 = z

    fig = plt.figure(dpi=120, figsize=[4,4])
    ax = fig.add_subplot(1,1,1)
    ax.plot(z0, label = 'Capital/output')         #plots the capital-output function and the 45degree line
    ax.plot(m0, label = '45 degrees line')
    ax.set_xlim([0,100]) # fixed x range          #sets limits for the axis
    ax.set_ylim([0,100]) # fixed y range
    plt.xlabel('z in period t')                   #Creates x- and y-axis
    plt.ylabel('z in period t+1')
    plt.title('Transition diagram')               #Creates title for the diagram
    ax.grid()                                 #Makes a grid on the graph
    ax.legend(loc='upper left')

    fig = plt.figure(dpi=120, figsize=[4,4])
    ax = fig.add_subplot(1,1,1)
    ax.plot(z0, label = 'Capital/output')
    ax.plot(m0, label = '45 degrees line')
    ax.set_xlim([3,4]) # fixed x range            #sets limits for the axis
    ax.set_ylim([3,4]) # fixed y range
    plt.xlabel('z in period t')                   #Creates x- and y-axis
    plt.ylabel('z in period t+1')
    plt.title('Transition diagram (zoomed in)')   #Creates title for the diagram
    ax.grid()                                 #Makes a grid on the graph
    ax.legend(loc='upper left')

    #Defining the intervals and step sizes for the sliders
widgets.interact(interactive_transition1,
    s=widgets.FloatSlider(description="$s$", min=0, max=1, step=0.005, value=0.25),
    beta=widgets.FloatSlider(description="$beta$", min=0, max=1, step=0.005, value=0.6),
    delta=widgets.FloatSlider(description="$delta$", min=0, max=0.1, step=0.005, value=0.05),
    n=widgets.FloatSlider(description="$n$", min=0, max=0.2, step=0.01, value=0.01),
    g=widgets.FloatSlider(description="$g$", min=0, max=0.04, step=0.001, value=0.02),
    alpha=widgets.FloatSlider(description="$alpha$", min=0, max=1, step=0.005, value=0.2),
    z=widgets.fixed(z));
```

    interactive(children=(FloatSlider(value=0.25, description='$s$', max=1.0, step=0.005), FloatSlider(value=0.6, …

From the above visual representation of the convergence to the capital-output steady state, one can see that the intersection point lies at around 3.5, which was also what we saw when calculating the steady state value based on given parameters. In comparison to the US, they had a capital-output ratio of 3.04 in 2015$^{(1)}$. This is relatively close to that of the model, where parameters that arguably could fit an OECD country were picked. A parameter like the savings rate however is arguable set a bit high for that of the US, where a lower savings rate, would make the model result come even closer the the capital-output ratio of the US.

$^{(1)}$ "Financial Times, Andrew Smithers - February 25, 2015 - https://www.ft.com/content/97337405-66fb-3391-b92c-4c244db5f8c0"

### Steady state output growth path and the effect of land

The steady state output growth path can be found using the steady state capital-output ratio and the per capita output function form earlier:

$y_t=k_t^\alpha A_t^\beta x_t^\kappa$

$y_t^{1-\alpha}=\frac{k_t^\alpha A_t^\beta x_t^\kappa}{y_t^\alpha}$

$y_t^{1-\alpha}=\frac{k_t^\alpha}{y_t^\alpha} A_t^\beta x_t^\kappa$

${y_t^*}^{1-\alpha}= (z^*)^\alpha A_t^\beta x_t^\kappa$

$y_t^*= (z^*)^{\frac{\alpha}{1-\alpha}} A_t^{\frac{\beta}{1-\alpha}} x_t^{\frac{\kappa}{1-\alpha}}$

$y_t^*= (z^*)^{\frac{\alpha}{1-\alpha}} A_t^{\frac{\beta}{1-\alpha}} \left(\frac{X}{L_t}\right)^{\frac{\kappa}{1-\alpha}}$

$y_t^*= (z^*)^{\frac{\alpha}{1-\alpha}} A_0^{\frac{\beta}{1-\alpha}} \left(\frac{X}{L_0}\right)^{\frac{\kappa}{1-\alpha}}(1+g)^{\frac{\beta}{1-\alpha}}(1+n)^{\frac{-\kappa}{1-\alpha}}$

Where one sees that the model suggests that land contributes positively to the steady state per capita output. Comparing to the solow model where technology was introduced (chapter 5) one will see that the technological growth has a smaller contribution to the output per capita in steady state. However setting $\kappa=0$ (rather unrealistic when assuming that land does have and effect) will give the steady state output per capita, that was found for chapter 5. So the contribution that land brings to the steady state per capita output in relation to the chapter 5 model, resides from technology, making the direct effect from development in technology lesser.

## Conclusion

The goal of this project was to see how the basic solow model compares to extensions of the model with technology, human capital and land. For the model with technology one saw that adding technology ensured that there could be growth in output per capita in steady state. This makes sense as countries that are considered to be in steady state, still experience growth in GDP and GDP per capita. In the next part of the project the model with human capital was analysed. Here one saw that the convergence towards steady state happens at a slower pace than for the general solow model. In the final part of the project the model with land was looked at and here one saw that land has a positive effect on output per capita. However this positive effect resides from the contribution from technological growth making that lesser.


##  Linear Regression using Gradient Descent

Video Explanation :  https://youtu.be/CzW0w7guvFo

```python
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
```

```python
X = [3,4,6,12,9,15,10,1,8,13] ## Experience
y = [16,29,43,65,51,89,57,9,53,68]  ## Salary
```

```python
#X = [3,4,6]
#y = [16,29,43]
```

```python
plt.scatter(X,y)
plt.xlabel('Experience')
plt.ylabel('Salary')
plt.grid(alpha=0.25)
plt.show()
```

```python
# Lets lienar regession line is y = mx + c
# lets
# y = mx +c
c=0
# # y_pred = mx
# calcaulating the y_out we will mutiply the original X values by the slope to produce y_out values for each X
def multiply_matrix(x,m,c):
    y_out = []
    for i in range(len(x)):
        y_out.append(x[i]*m + c)
    return y_out
```

```python
# Lets
c = 0
m = 1
y_m1=multiply_matrix(X,m,c)
y_m1
```

    [3, 4, 6, 12, 9, 15, 10, 1, 8, 13]

```python
# line at m =1
plt.scatter(X, y,label='Scatter Plot')
plt.plot([min(X), max(X)], [min(y_m1), max(y_m1)], color='red',label='Regression best fit Line at m=1')
plt.xlabel('Experience')
plt.ylabel('Salary')
plt.grid(alpha=0.25)
plt.legend()
plt.show()
```

```python
# calculate cost function 1/n*sum(y-y_out)^2 by looping each sample
# calculate mse 1/n*sum(y-y_out)^2 by looping each sample
def mse(actual, predicted):
    sum_error = 0.0
    # loop over all values
    for i in range(len(actual)):
        # the error is the sum of (actual - prediction)^2
        SSE =  (actual[i] - predicted[i])**2
        sum_error =  sum_error + SSE
    mean_error = sum_error / float(len(actual))
    return (mean_error)
```

```python
mse_m1 = mse(y,y_m1)
mse_m1
```

    1953.5

```python
# calculate mse for different m values ,lets take m = range(0,9)
mse_values = []  # store the MSE for each m where m = range(0,9)
m = range(0,9)
c = 0
for i in m:
    y_out_values = multiply_matrix(X, m[i],c)
    mse_values.append(mse(y, y_out_values))
    print("mse for m :",m[i], "and c :",c ," is ", mse(y, y_out_values))
    plt.figure(figsize=(3,2))
    plt.scatter(X, y,label='Scatter Plot')
    plt.plot([min(X), max(X)], [min(y_out_values), max(y_out_values)], color='red',label='regression line')
    plt.xlabel('Experience')
    plt.ylabel('Salary')
    plt.grid(alpha=0.25)
    #plt.legend()
    plt.show()
```

```python
plt.plot(m,mse_values,label = "cost function line")
plt.xlabel('slope')
plt.ylabel('Cost Function')
plt.grid(alpha=0.25)
plt.legend()
```

```python
np.min(mse_values)  # at this MSE we get the best bit line when m = 5 and c =0
# but we have to iterate the both m & c to get the best bit line taht we can get with the help
# of gradient descent(taking derivative of MSE wrt m and c)
```

    28.0

Cost Function

\begin{equation}
Linear Equation ==> y = mx + c
\end{equation}

\begin{equation}
J(x) = 1/n \sum_{i=1}^{n} (y^{(i)} - y^{(i)}predicted)^2
\end{equation}

\begin{equation}
y^{(i)}predicted = (mx^{(i)} + c)
\end{equation}

\begin{equation}
J(x) = 1/n \sum_{i=1}^{n} (y^{(i)} - (mx^{(i)} + c))^2
\end{equation}

Gradient

**Derivative of cost function with respect to   m   ==>  slope**
\begin{equation}
\frac{\partial J(x^{(i)})}{\partial m} = -2/n\sum_{i=1}^{n}(x^{(i)}).(y^{(i)} - (mx^{(i)} + c))
\end{equation}

**Derivative of cost function with respect to   c   ==>  intercept**
\begin{equation}
\frac{\partial J(x^{(i)})}{\partial c} = -2/n\sum_{i=1}^{n}(y^{(i)} - (mx^{(i)} + c))
\end{equation}

**Why do we take derivative?**

To get the direction of where is global minimum. If we are on left side of minimum value of parabola then the derivate is negative and negative sign before derivate will make the new value of X go right and thus more closer to minimum value. Vice versa for right side.

```python
import numpy as np
def Simple_LinearRegression_Gradient_Descent(x,y,epochs,learning_rate,n):
    m_curr = c_curr = 0
    for i in range(epochs):
        y_pred = m_curr * x + c_curr
        ## MSE equation == cost_function
        cost = (1/n) * sum([val**2 for val in (y-y_pred)])
        md = -(2/n)*sum(x*(y-y_pred))
        cd = -(2/n)*sum(y-y_pred)
        m_curr = m_curr - learning_rate * md  # step_size = learning_rate * md
        c_curr = c_curr - learning_rate * cd  # # step_size = learning_rate * cd
        print('>epoch=%d, m=%.3f,c=%.3f, md=%.3f,cd=%.3f,step_size_m=%.3f,step_size_c=%.3f,cost=%.3f' %
              (i,m_curr,c_curr, md,cd,md*learning_rate,cd*learning_rate,(cost)))
```

```python
x = np.array([3,4,6,12,9,15,10,1,8,13])
y = np.array([16,29,43,65,51,89,57,9,53,68])
epochs = 30
n = len(x)
learning_rate = 0.001
Simple_LinearRegression_Gradient_Descent(x,y,epochs,learning_rate,n)
```

    >epoch=0, m=0.977,c=0.096, md=-976.600,cd=-96.000,step_size_m=-0.977,step_size_c=-0.096,cost=2845.600
    >epoch=1, m=1.787,c=0.176, md=-809.999,cd=-79.987,step_size_m=-0.810,step_size_c=-0.080,cost=1964.756
    >epoch=2, m=2.458,c=0.243, md=-671.814,cd=-66.705,step_size_m=-0.672,step_size_c=-0.067,cost=1358.756
    >epoch=3, m=3.016,c=0.298, md=-557.197,cd=-55.688,step_size_m=-0.557,step_size_c=-0.056,cost=941.840
    >epoch=4, m=3.478,c=0.345, md=-462.128,cd=-46.550,step_size_m=-0.462,step_size_c=-0.047,cost=655.012
    >epoch=5, m=3.861,c=0.384, md=-383.274,cd=-38.971,step_size_m=-0.383,step_size_c=-0.039,cost=457.679
    >epoch=6, m=4.179,c=0.417, md=-317.870,cd=-32.684,step_size_m=-0.318,step_size_c=-0.033,cost=321.917
    >epoch=7, m=4.443,c=0.444, md=-263.620,cd=-27.469,step_size_m=-0.264,step_size_c=-0.027,cost=228.515
    >epoch=8, m=4.661,c=0.467, md=-218.623,cd=-23.143,step_size_m=-0.219,step_size_c=-0.023,cost=164.256
    >epoch=9, m=4.842,c=0.487, md=-181.301,cd=-19.555,step_size_m=-0.181,step_size_c=-0.020,cost=120.045
    >epoch=10, m=4.993,c=0.503, md=-150.344,cd=-16.579,step_size_m=-0.150,step_size_c=-0.017,cost=89.628
    >epoch=11, m=5.117,c=0.517, md=-124.668,cd=-14.110,step_size_m=-0.125,step_size_c=-0.014,cost=68.700
    >epoch=12, m=5.221,c=0.530, md=-103.370,cd=-12.063,step_size_m=-0.103,step_size_c=-0.012,cost=54.301
    >epoch=13, m=5.307,c=0.540, md=-85.705,cd=-10.364,step_size_m=-0.086,step_size_c=-0.010,cost=44.393
    >epoch=14, m=5.378,c=0.549, md=-71.053,cd=-8.955,step_size_m=-0.071,step_size_c=-0.009,cost=37.576
    >epoch=15, m=5.436,c=0.557, md=-58.900,cd=-7.786,step_size_m=-0.059,step_size_c=-0.008,cost=32.884
    >epoch=16, m=5.485,c=0.563, md=-48.820,cd=-6.816,step_size_m=-0.049,step_size_c=-0.007,cost=29.655
    >epoch=17, m=5.526,c=0.569, md=-40.459,cd=-6.011,step_size_m=-0.040,step_size_c=-0.006,cost=27.432
    >epoch=18, m=5.559,c=0.575, md=-33.524,cd=-5.344,step_size_m=-0.034,step_size_c=-0.005,cost=25.901
    >epoch=19, m=5.587,c=0.580, md=-27.772,cd=-4.790,step_size_m=-0.028,step_size_c=-0.005,cost=24.846
    >epoch=20, m=5.610,c=0.584, md=-23.001,cd=-4.331,step_size_m=-0.023,step_size_c=-0.004,cost=24.120
    >epoch=21, m=5.629,c=0.588, md=-19.043,cd=-3.949,step_size_m=-0.019,step_size_c=-0.004,cost=23.618
    >epoch=22, m=5.645,c=0.591, md=-15.761,cd=-3.633,step_size_m=-0.016,step_size_c=-0.004,cost=23.272
    >epoch=23, m=5.658,c=0.595, md=-13.039,cd=-3.370,step_size_m=-0.013,step_size_c=-0.003,cost=23.032
    >epoch=24, m=5.669,c=0.598, md=-10.781,cd=-3.153,step_size_m=-0.011,step_size_c=-0.003,cost=22.866
    >epoch=25, m=5.678,c=0.601, md=-8.908,cd=-2.972,step_size_m=-0.009,step_size_c=-0.003,cost=22.750
    >epoch=26, m=5.685,c=0.604, md=-7.354,cd=-2.821,step_size_m=-0.007,step_size_c=-0.003,cost=22.669
    >epoch=27, m=5.691,c=0.606, md=-6.065,cd=-2.697,step_size_m=-0.006,step_size_c=-0.003,cost=22.612
    >epoch=28, m=5.696,c=0.609, md=-4.997,cd=-2.593,step_size_m=-0.005,step_size_c=-0.003,cost=22.571
    >epoch=29, m=5.700,c=0.612, md=-4.110,cd=-2.507,step_size_m=-0.004,step_size_c=-0.003,cost=22.542

```python
## Plot the cost with each iteration
import matplotlib.pyplot as plt
def Simple_LinearRegression_Gradient_Descent_cost_plot(x,y,epochs,learning_rate,n):
    m_curr = 0
    c_curr = 0
    iter = epochs
    cost_val = [] # list to store the cost in each iteration
    intercept = []
    coefficient = []
    for i in range(iter):
        y_pred = (m_curr * x) + c_curr
        cost = 1/n * sum([data**2 for data in (y - y_pred)])
        cost_val.append(cost) ## append the cost_val
        md = -(2/n) * sum(x * (y - y_pred))
        cd = -(2/n) * sum(y - y_pred)
        m_curr = m_curr - (learning_rate * md)  #
        coefficient.append(m_curr)
        c_curr = c_curr - (learning_rate * cd)
        intercept.append(c_curr)
    return intercept,coefficient,cost_val
```

```python
x = np.array([3,4,6,12,9,15,10,1,8,13])
y = np.array([16,29,43,65,51,89,57,9,53,68])
epochs = 100
n = len(x)
learning_rate = 0.001
intercept,coefficient,cost_val = Simple_LinearRegression_Gradient_Descent_cost_plot(x,y,epochs,learning_rate,n)

plt.figure(figsize=(5,5))
plt.plot(coefficient, cost_val, '.r',color='blue',label="cost function curve")
plt.xlabel('coefficient')
plt.ylabel('cost_val')
plt.grid(alpha=0.25)
plt.legend()
```

```python
## Now plot grapg to show the regression best fit line
import matplotlib.pyplot as plt
%matplotlib inline

def Simple_LinearRegression_Gradient_Descent_best_fit_line(x,y,epochs,learning_rate,n):
    m_curr = 0
    c_curr = 0
    iter = epochs
    plt.scatter(x,y,color = "red",label='Scatter Plot')
    for i in range(iter):
        ## y = mx + c
        y_pred = m_curr * x + c_curr
        ## MSE equation == cost_function
        cost = 1/n*sum([val**2 for val in (y - y_pred)])
        ## lets take derivative
        md = -2/n * sum(x*(y - y_pred))
        cd = -2/n * sum(y - y_pred)
        plt.plot(x,y_pred,color = "blue",linewidth='.1')
        plt.xlabel('Experience')
        plt.ylabel('Salary')
        #plt.style.use('fivethirtyeight')
        #plt.legend()
        m_curr = m_curr - learning_rate * md
        c_curr = c_curr - learning_rate * cd
```

```python
#x = np.array([3,4,6,12,9,15,10,1,8,13])
#y = np.array([16,29,43,65,51,89,57,9,53,68])
epochs = 100
n = len(x)
learning_rate = 0.001
Simple_LinearRegression_Gradient_Descent_best_fit_line(x,y,epochs,learning_rate,n)
```

```python
lr = LinearRegression()
lr.fit(x.reshape(-1,1),y)
```

    LinearRegression()

```python
lr.intercept_,lr.coef_
```

    (5.334568554790884, array([5.26733722]))

```python
y_predict = lr.predict(x.reshape(-1,1))
y_predict
```

    array([21.1365802 , 26.40391742, 36.93859185, 68.54261514, 52.74060349,
           84.34462679, 58.00794071, 10.60190577, 47.47326628, 73.80995236])

```python
mean_squared_error(y,y_predict)
```

    17.499947061937544

```python
from sklearn.linear_model import ElasticNet,Ridge,Lasso
```

```python
lre = ElasticNet()
lre.fit(x.reshape(-1,1),y)
```

    ElasticNet()

```python
lre.intercept_,lre.coef_
```

    (6.643630737493552, array([5.1057246]))

```python
y_predict_e = lre.predict(x.reshape(-1,1))
y_predict_e
```

    array([21.96080454, 27.06652914, 37.27797834, 67.91232594, 52.59515214,
           83.22949974, 57.70087674, 11.74935534, 47.48942754, 73.01805054])

```python
mean_squared_error(y,y_predict_e)
```

    17.993328121953727


---
author: Nathan Carter (ncarter@bentley.edu)
---

This answer assumes you have imported SymPy as follows.

```python
from sympy import *                   # load all math functions
init_printing( use_latex='mathjax' )  # use pretty math output
```

SymPy has support for piecewise functions built in, using `Piecewise`.
The function above would be written as follows.

```python
var( 'x' )
formula = Piecewise( (x**2, x>2), (1+x, x<=2) )
formula
```

$\displaystyle \begin{cases} x^{2} & \text{for}\: x > 2 \\x + 1 & \text{otherwise} \end{cases}$

We can test to be sure the function works correctly by plugging in a few
$x$ values and ensuring the correct $y$ values result.
Here we're using the method from how to substitute a value for a symbolic variable.

```python
formula.subs(x,1), formula.subs(x,2), formula.subs(x,3)
```

$\displaystyle \left( 2, \  3, \  9\right)$

For $x=1$ we got $1+1=2$.  For $x=2$ we got $2+1=3$.  For $x=3$, we got $3^2=9$.

database/tasks/How to write a piecewise-defined function/Python, using SymPy.ipynb


```python
## This cell just imports necessary modules
%pylab notebook
from sympy import sin, cos, Function, Symbol, diff, integrate, exp
```

```python
###### FIRST PARTIAL DERIVATIVES ######
###### Lecture 2, slide 10 ######

# Define the independent variables using Symbol
r = Symbol('r')
h = Symbol('h')
# Define the function V(r,h)
V = pi*(r**2)*h

# The first partial derivative of V w.r.t h (i.e. r is kept constant)
print("The first partial derivative of V w.r.t. h is: ", diff(V, h))
# The first partial derivative of V w.r.t r (i.e. h is kept constant)
print("The first partial derivative of V w.r.t. r is: ", diff(V, r))

```

```python
###### SECOND PARTIAL DERIVATIVES ######
###### Lecture 2, slide 12 ######

x = Symbol('x')
y = Symbol('y')
f = (x**2)*sin(y)

f_x = diff(f, x)
f_y = diff(f, y)

print("The first partial derivatives of f = (x**2)*sin(y) are: ")
print("f_x = ", f_x)
print("f_y = ", f_y, "\n")

f_xx = diff(f_x, x)
f_xy = diff(f_x, y)
f_yx = diff(f_y, x)
f_yy = diff(f_y, y)

print("The second partial derivatives of f = (x**2)*sin(y) are: ")
print("f_xx = ", f_xx)
print("f_xy = ", f_xy)
print("f_yy = ", f_yy)
print("f_yx = ", f_yx)
```

```python
###### CHAIN RULE ######
###### Lecture 2, slide 19 ######
a = Symbol('a')
b = Symbol('b')
t = Symbol('t')

x = a*t
y = b*t

T = 3*y*sin(x)

# SymPy automatically applies the chain rule here:
print("Differentiating T = 3*y*sin(x) wrt t using the chain rule: \n")
print(diff(T, t))
```

```python
###### DEFINITE AND INDEFINITE INTEGRALS ######
###### Lecture 2, slide 21 ######
x = Symbol('x')
y = Symbol('y')

# Remember: Indefinite integrals result in a constant 'c'. SymPy sets this to zero.
# f is the function we want to integrate.
f = cos(x)

# The second argument is the variable we want to integrate with respect to.
# (in this example, it is 'x').
print("Integrating cos(x) yields:")
print(integrate(f, x))

# Using integrate(2*x, (x, a, b)) evaluates a DEFINITE integral between x=a and x=b
a = 0
b = 2
print("Integrating 2*x between x=0 and x=2 yields:")
print(integrate(2*x, (x, a, b)))
```

```python
###### DOUBLE INTEGRALS ######
###### Lecture 2, slide 26 ######

# The function we want to integrate.
f = 2*(x**2)*y

# First integrate f wrt y between y=0 and y=2.
inner_integral = integrate(f, (y, 0, 2))
print("The inner integral is: ", inner_integral)

# Then integrate the inner_integral wrt x.
outer_integral = integrate(inner_integral, (x, 0, 2))
print("The outer integral is: ", outer_integral)
```


# Etivity 1.1.1: Modelling an infected cohort

In this etivity, you will code your first simple model and use it to answer a research question. To get used to the format for models in R, we'll start by giving you some parts of the code, and you need to fill in the missing numbers or code blocks by replacing the *#YOUR CODE#* placeholders. At the end of each etivity, we'll provide a solution file that contains the full code, annotated with comments. Make sure you understand each line, and soon you'll be able to write your own model from scratch!

Your task is to find out how long it takes for
a cohort of infected people to recover. As you saw in the video,
to answer this question you need to keep track of 2 populations:
those that are infected (compartment I), and those that have recovered (compartment R).
Infected people recover at a rate *gamma*. The differential equations describing this are:

\begin{align}
\frac{dI}{dt} & = -\gamma I \\
\frac{dR}{dt} & = \gamma I
\end{align}

Load the packages which you need for this etivity, by running the following cell:

```
library(deSolve)   # package to solve the model
library(reshape2)  # package to change the shape of the model output
library(ggplot2)   # package for plotting
```

To start, it is useful to code what we know about the situation we want to model.
We are looking at a cohort of 10$^6$ currently infected people, and no one has recovered so far. The average duration of infection is 10 days. The question we want to answer is how many people will recover from the infection over a 4-week period.

Given this data, fill in the correct values to the following variables, and run the cell:

```
initial_number_infected <- 1000000  # the initial infected population size
initial_number_recovered <- 0#YOUR CODE#  # the initial number of people in the recovered state
recovery_rate <- 0.1            # the rate of recovery gamma, in units of days^-1
follow_up_duration <- 4*7 #YOUR CODE#        # the duration to run the model for, in units of days

# Hint: the units of the recovery rate and the follow-up duration should be consistent.
```

Now, we combine this data into objects that are recognised by the deSolve package as **model input**. To do this, again run the code below.

```
# The initial state values are stored as a vector and each value is assigned a name.
initial_state_values <- c(I = initial_number_infected,
                          R = initial_number_recovered)

# Parameters are also stored as a vector with assigned names and values.
parameters <- c(gamma = recovery_rate)
# In this case we only have one parameter, gamma.
```

Think about: what kind of information is stored in the *initial_state_values* and *parameters* vectors?

Additionally, we need to specify the time we want the model to run for. This depends on the question we want to answer. In the cell below, the duration you specified earlier is automatically filled in when you run it.

```
# The times vector creates a sequence of timepoints at which we want to calculate the number of people
# in the I and R compartment.
times <- seq(from = 0, to = follow_up_duration, by = 1)
```

Think about: what kind of information is stored in the *times* vector?

Check your answers by having a look at each of these vectors to familiarise yourself with the structure: in the following code cell we have typed the object names, so you just need to press "Run" to see what each of them contains.

```
initial_state_values
parameters
times
```

I

1e+06

R

0

gamma: 0.1

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

The next step is specifying the model. Using the example code in the introductory document, complete the following model function with the differential equations above.

```
cohort_model <- function(time, state, parameters) {

    with(as.list(c(state, parameters)), {

      dI <- -gamma*I
      dR <- gamma*I

      return(list(c(dI, dR)))
    })

}
```

Now all there's left to do is solving this set of equations using the deSolve package. Fill in the following command, which calculates and stores the number of infected and recovered people at each timestep in the "output" dataframe. Don't forget to run it!

```
# Hint: if you can't remember what those arguments correspond to, just look up the ode help file:
??ode
```

```
output <- as.data.frame(ode(y = initial_state_values,
                            times = times,
                            func = cohort_model,
                            parms = parameters))
```

Printing the **model output** returns a dataframe with columns time (containing our times vector), I (containing the number of infected people at each timestep) and R (containing the number of recovered people at each timestep):

```
output
```

time	I	R

 0          	1.000000e+06	      0.0

 1          	3.678794e+05	 632120.6

 2          	1.353351e+05	 864664.9

 3          	4.978697e+04	 950213.0

 4          	1.831559e+04	 981684.4

 5          	6.737923e+03	 993262.1

 6          	2.478741e+03	 997521.3

 7          	9.118772e+02	 999088.1

 8          	3.354606e+02	 999664.5

 9          	1.234090e+02	 999876.6

10          	4.539958e+01	 999954.6

11          	1.670156e+01	 999983.3

12          	6.144155e+00	 999993.9

13          	2.260306e+00	 999997.7

14          	8.315196e-01	 999999.2

15          	3.058986e-01	 999999.7

16          	1.125336e-01	 999999.9

17          	4.139872e-02	1000000.0

18          	1.522972e-02	1000000.0

19          	5.602676e-03	1000000.0

20          	2.061173e-03	1000000.0

21          	7.582818e-04	1000000.0

22          	2.789371e-04	1000000.0

23          	1.026131e-04	1000000.0

24          	3.773270e-05	1000000.0

25          	1.391230e-05	1000000.0

26          	5.151106e-06	1000000.0

27          	1.851387e-06	1000000.0

28          	6.704086e-07	1000000.0

### Question: Based on the output, how many people have recovered after 4 weeks? What proportion of the total population does this correspond to?

Now, plot your model output in the following cell, with time on the x axis and the number of infected and recovered people on the y axis.  You can use the introductory document for help with this.

```
# First turn the output dataset into a long format, so that the number in each compartment at each timestep
# are all in the same column
output_long <- melt(as.data.frame(output), id = "time")

# Plot the number of people in each compartment over time
ggplot(data = output_long,                                               # specify object containing data to plot
        aes(x = time, y = value, colour = variable, group = variable))+
       # in the long-format output dataset, the number in each compartment is in the "value" column and the
       # compartment the number relates to is in the "variable" column. We are telling ggplot to assign a different
       # colour to each compartment/group, which automatically generates a legend
  geom_line() +
       # we want to represent the data over time as lines. This command automatically looks to the specifications saved
       # in the ggplot command above to know which data to plot
  xlab("Time (days)")+                                                   # add label for x axis
  ylab("Number of people") +                                             # add label for y axis
  labs(title = paste("Number infected and recovered over time when gamma =",parameters["gamma"],"days^-1")) # add title
# Using the paste() command, we can combine sentences with the values stored in variables (here gamma)
```

### Question: Based on the plot, at what timepoint were infected and recovered individuals equal in number?

For the last part of the etivity, try varying *gamma* to see how it affects the output. For example, in the cell below change *gamma* to correspond to an average infectious period of 2 days and 20 days. What is the recovery rate in these 2 cases?

7 days

```
parameters <- c(gamma = 1.0)
output <- as.data.frame(ode(y = initial_state_values,
                            times = times,
                            func = cohort_model,
                            parms = parameters))

output_long <- melt(as.data.frame(output), id = "time")                  # turn output dataset into long format

ggplot(data = output_long,                                               # specify object containing data to plot
       aes(x = time, y = value, colour = variable, group = variable)) +  # assign columns to axes and groups
  geom_line() +                                                          # represent data as lines
  xlab("Time (days)")+                                                   # add label for x axis
  ylab("Number of people") +                                             # add label for y axis
  labs(title = paste("Number infected and recovered over time when gamma =",parameters["gamma"],"days^-1")) # add title
# Since the initial number in each compartment and the timesteps of interest haven't changed,
# these are the only parts of the code we need to rerun.

# Now, copy-paste your plot code from above here to visualise the output.
```

### Question: What changes do you observe in the transition to the recovered compartment if *gamma* is higher or lower? For example, how long does it take for everyone to recover in both cases?

higher gamma, faster recovery time

## Well done on writing your first model code. Now, check the solutions!
Once you are done with an etivity, you can open the "Solutions to etivities" folder in your Jupyter workspace. In this folder, you will find feedback and model code for all the coding etivities throughout the course. After completing an exercise, you should always carefully compare your answers and code with the solutions! It is especially important to get the coding right from the start, because we are building on the same modelling framework throughout the course. Additionally, the solution files sometimes contain further information that will help you deepen your understanding of the lesson.
To make the most of the learning experience, we always recommend trying to complete the whole etivity before checking the solutions - but if you are stuck at any point, they can also help you to move on to the next part.

```

```

course-1/utf-8''IDM1.1.07 - Modelling an infected cohort.ipynb


# SymPy Tutorial

*Arthur Ryman* <br/>
*Last Updated: 2020-04-14*

This notebook contains the examples from the
[SymPy Tutorial](https://docs.sympy.org/latest/tutorial/index.html).

## Calculus

```python
from sympy import *
x, y, z = symbols('x y z')
init_printing(use_unicode=True)
```

### Derivatives

```python
diff(cos(x), x)
```

```python
diff(exp(x**2), x)
```

```python
diff(x**4, x, x, x)
```

```python
diff(x**4, x, 3)
```

```python
expr = exp(x*y*z)
diff(expr, x, y, y, z, z, z, z)
```

```python
diff(expr, x, y, 2, z, 4)
```

```python
diff(expr, x, y, y, z, 4)
```

```python
expr.diff(x, y, y, z, 4)
```

```python
deriv = Derivative(expr, x, y, y, z, 4)
deriv
```

```python
deriv.doit()
```

```python
m, n, a, b = symbols('m n a b')
expr = (a*x + b)**m
expr.diff((x,n))
```

### Integrals

```python
integrate(cos(x), x)
```

```python
integrate(exp(-x), (x, 0, oo))
```

```python
integrate(exp(-x**2 - y**2), (x, -oo, oo), (y, -oo, oo))
```

```python
expr = integrate(x**x, x)
print(expr)
expr
```

```python
expr = Integral(log(x)**2, x)
expr
```

```python
expr.doit()
```

```python
integ = Integral((x**4 + x**2*exp(x) - x**2 - 2*x*exp(x) - 2*x -
                 exp(x))*exp(x)/((x - 1)**2*(x + 1)**2*(exp(x) + 1)), x)
integ
```

```python
integ.doit()
```

```python
integ = Integral(sin(x**2), x)
integ
```

```python
integ.doit()
```

```python
integ = Integral(x**y*exp(-x), (x, 0, oo))
integ
```

```python
integ.doit()
```

### Limits

```python
limit(sin(x)/x, x, 0)
```

```python
expr = x**2/exp(x)
expr.subs(x, oo)
```

```python
limit(expr, x, oo)
```

```python
expr = Limit((cos(x) - 1)/x, x, 0)
expr
```

```python
expr.doit()
```

```python
limit(1/x, x, 0, '+')
```

```python
limit(1/x, x, 0, '-')
```

### Series Expansion

```python
expr = exp(sin(x))
expr.series(x, 0, 4)
```

```python
x + x**3 + x**6 + O(x**4)
```

```python
x*O(1)
```

```python
expr.series(x, 0, 4).removeO()
```

```python
exp(x - 6).series(x, x0=6)
```

### Finite Differences

```python
f, g = symbols('f g', cls=Function)
differentiate_finite(f(x)*g(x))
```

```python
differentiate_finite(f(x)*g(x), evaluate=True)
```

```python
f = Function('f')
dfdx = f(x).diff(x)
dfdx.as_finite_difference()
```

```python
f = Function('f')
d2fdx2 = f(x).diff(x, 2)
h = Symbol('h')
d2fdx2.as_finite_difference([-3*h,-h,2*h])
```

```python
finite_diff_weights(2, [-3, -1, 2], 0)[-1][-1]
```

```python
x_list = [-3, 1, 2]
y_list = symbols('a b c')
apply_finite_diff(1, x_list, y_list, 0)
```


# Objective: Semi-supervised classification of MNIST Dataset

Less than 1.5 % labelling in each class

```python
# load the required  libs
from tqdm import tqdm
import numpy as np
import mnist
import torch
import faiss
from matplotlib import pyplot as plt
from torch_pdegraph.utilities import *
from torch_pdegraph.operators import MeanCurv, GradPlusInfNorm, GradMinusInfNorm
```

```python
# create a smaller dataset
# We will take all the 70000 images and label only 100 in each class later
images_tr = mnist.train_images()
labels_tr = mnist.train_labels()
images_te = mnist.test_images()
labels_te = mnist.test_labels()
images = np.concatenate((images_tr,images_te),axis=0)
labels = np.concatenate((labels_tr, labels_te),axis=0)
images_flatten = np.reshape(images, (images.shape[0], images.shape[1]*images.shape[2])).astype(np.float32) * 1000
```

# Graph construction:

- In order to simply show the effectiveness of PDEs on graph,  I am only creating a simple K-NN based graphs. This may or maynot be the best graph for a given problem at hand.

- One can create graph using whatsoever apt approach or one can even use third-party network datasets and run a PDE on that graph.  PDEs are extensible to any given graph/network at hand as long as that graph has edges and weights( edge_index and edge_attr).

Although torch_cluster comes with a knn-graph method. I found it to be limited and  slow when the node-features have high dimensions. We shall be using facebook's faiss library which is blazingly fast for a KNN-graph construction.

```python
# Create the intial front(level-sets) for each class with 100 labels(seeds) in each class
"""
Initial front creation process in literature it is
also known as intial seed or level-set creation process.
"""
Front = genInitialSeeds(**dict(labels = labels, num_seeds = 100))

# Create the Knn graph of the flattened image features and
# assign weights to the edges
res = faiss.StandardGpuResources()
index = faiss.IndexFlatL2(images_flatten.shape[1])
gpu_index_flat = faiss.index_cpu_to_gpu(res,0,index)
gpu_index_flat.add(images_flatten/1000)
k = 30
D, I = gpu_index_flat.search(images_flatten/1000,k+1)

#Graph
edge_index = np.vstack((I[:,1:].flatten(), np.repeat(I[:,0].flatten(),k)))
edge_attr = np.exp(-(D[:,1:].flatten()/505000))

edge_index = torch.tensor(edge_index, dtype=torch.long).to('cuda:0')
edge_attr = torch.tensor(edge_attr, dtype=torch.float32).to('cuda:0')
edge_attr = edge_attr.view(-1,1)
graph = Graph(edge_index, edge_attr)
```

    Success!

# Run a manually defined PDE:

\begin{equation}
\mathbf{x}^{n+1}_{i} = \mathbf{x}^{n}_{i} + \Delta t \kappa_w(i,\mathbf{x}^{n}) \|\nabla^{+}_w\mathbf{x}^{n}_{i}\|_{\infty}, \quad if\quad \kappa_w(i,\mathbf{x}^{n}) > 0
\end{equation}

\begin{equation}
\mathbf{x}^{n+1}_{i} = \mathbf{x}^{n}_{i} + \Delta t \kappa_w(i,\mathbf{x}^{n}) \|\nabla^{-}_w\mathbf{x}^{n}_{i}\|_{\infty}, \quad if\quad \kappa_w(i,\mathbf{x}^{n}) < 0
\end{equation}

- $\mathbf{x}_{i}$ is the node feature/signal at the $i^{th}$ node
- $\nabla^{-}_{w}$, $\nabla^{+}_{w}$ are the negative and positive directional gradients on weighted graphs respectively.
- $\kappa(i,\mathbf{x})$ is the mean curvatrue operator.

**Example:**

```python
from torch_pdegraph.operators import GradMinusInfNorm, GradPlusInfNorm, MeanCurv
# Instantiate the operators
ope_normM = GradMinusInfNorm.OPE(graph)
ope_normP = GradPlusInfNorm.OPE(graph)
ope_curv = MeanCurv.OPE(graph)

# Run the above explicit PDE on intial front(level-set) on graph
Ip = (ope_curv(fr) > 0.0)
Im = (ope_curv(fr) < 0.0)
fr = fr + dt * ope_curv(fr) * (ope_normP(fr) * Ip.type(torch.int) + ope_normM(fr) * Im.type(torch.int)
```

To know more ref [DEL11](https://ieeexplore.ieee.org/document/6116433), PDE level-set method in section 6.3 of [M. Toutain's thesis](https://tel.archives-ouvertes.fr/tel-01258738)

```python
# Params
itr = 500
dt = 0.05

# Instantiate the operators
ope_curv = MeanCurv.OPE(graph)
ope_normP = GradPlusInfNorm.OPE(graph)
ope_normM = GradMinusInfNorm.OPE(graph)

#Evolved front
newFront = []

#Run the custome PDE on each initial front (initial level-set)
for fr in tqdm(Front):
    fr = torch.tensor(fr, dtype=torch.float32).to('cuda:0')
    for i in range(itr):
        Ip = (ope_curv(fr) > 0.0)
        Im = (ope_curv(fr) < 0.0)
        fr = fr + dt * ope_curv(fr) * (ope_normP(fr) * Ip.type(torch.int) + ope_normM(fr) * Im.type(torch.int))
    newfr = fr.to('cpu')
    newFront.append(newfr.numpy())
```

    100%|██████████| 10/10 [00:24<00:00,  2.42s/it]

```python
# Now see the results
from sklearn.preprocessing import MinMaxScaler
scalar = MinMaxScaler()
norm_fr = []

for fr  in newFront:
    scalar.fit(fr)
    norm_fr.append(scalar.transform(fr))

m = np.argmax(norm_fr, axis=0)
m = m[:,0]

import collections
from sklearn.metrics import confusion_matrix
print(f"The original number of elements in each class is : \n {collections.Counter(labels)}")
print(f"The predicted number of elements are: \n {collections.Counter(m)}")
print(f"The matrix of confusion: \n {confusion_matrix(labels,m)}")

nmask = (m == labels)
nmask = nmask.astype("int")
print("The accuracy of the classification is: \t {acc}".format(acc=np.sum(nmask)*100/len(labels)))
```

    The original number of elements in each class is :
     Counter({1: 7877, 7: 7293, 3: 7141, 2: 6990, 9: 6958, 0: 6903, 6: 6876, 8: 6825, 4: 6824, 5: 6313})
    The predicted number of elements are:
     Counter({1: 8618, 7: 7521, 9: 7515, 0: 7207, 6: 7076, 3: 7016, 2: 6375, 4: 6353, 5: 6329, 8: 5990})
    The matrix of confusion:
     [[6839    7    4    1    1    9   34    4    1    3]
     [   0 7825   15    1    3    2    6    9    1   15]
     [ 114  226 6290   21   25   12   34  223   30   15]
     [  23   60   28 6717    6  136    6   78   41   46]
     [   6   88    1    0 6135    0   34   10    1  549]
     [  45   44    0   75   16 5945  111    7    6   64]
     [  44   24    0    0    8   20 6779    0    1    0]
     [   4  141   11    2   25    2    0 6997    1  110]
     [  97  167   21  117   67  187   66   59 5898  146]
     [  35   36    5   82   67   16    6  134   10 6567]]
    The accuracy of the classification is: 	 94.27428571428571

### NOTE: Here I have used a simple knn-graph construction method but with a more sophisticated method of graph creation like [two-sided tangent distance](http://www.keysers.net/daniel/files/ICPR2000.pdf) one can achieve  more than 98% of classification accuracy with 1% labeling using this PDE level-set method, as it was shown in the section 6.4.5 in the thesis of [M. Toutain](https://tel.archives-ouvertes.fr/tel-01258738)


# Minimum Square Error method

```python
import numpy as np
from matplotlib import pyplot as plt
import sympy as sy
from scipy import linalg as sla
sy.init_printing()
%matplotlib inline
```

```python
xs=np.array([0.0,0.2,0.4,0.6,0.8,1.2,1.4,1.6,1.8,2.0])
ys=np.array([2.0,2.1,1.6,2.6,1.5,2.7,0.67,3.5,0.94,2.0])
size=len(xs)
x=sy.symbols('x')
```

```python
n=10
base=[x**k for k in range(n)]
fs=[np.vectorize(sy.Lambda(x,base[k].subs({x:x}))) for k in range(n)]
```

```python
phi=sy.Matrix([[np.dot(fs[i](xs), fs[j](xs)) for i in range(n)] for j in range(n)])
```

```python
b=np.array([np.dot(ys,fs[i](xs))  for i in range(n)])
b
```

    array([19.6100000000000, 19.289999999999999, 27.4428000000000,
           43.7709600000000, 73.9736160000000, 129.494736000000,
           232.319902080000, 424.628390016000, 787.705159449600,
           1478.97815892480], dtype=object)

```python
a=phi.solve(b)
a
```

```python
poly=sum([a[k] * base[k] for k in range(n)])
poly=sy.Lambda(x,poly.subs({x:x}))
```

```python
xplt=np.linspace(0,2,100)
yplt=np.vectorize(poly)(xplt)
plt.plot(xplt,yplt)
plt.plot(xs,ys,'gx')
```

```python

```

```python
import numpy as np
from matplotlib import pyplot as plt
import sympy as sy
sy.init_printing()
%matplotlib inline
```

```python
xs=np.array([0.0,0.2,0.4,0.6,0.8,1.2,1.4,1.6,1.8,2.0])
ys=np.array([2.0,2.1,1.6,2.6,1.5,2.7,0.67,3.5,0.94,2.0])
x=sy.symbols('x')
def approx(n):
    base=[x**k for k in range(n)]
    fs=[np.vectorize(sy.Lambda(x,base[k].subs({x:x}))) for k in range(n)]
    phi=sy.Matrix([[np.dot(fs[i](xs), fs[j](xs)) for i in range(n)] for j in range(n)])
    b=np.array([np.dot(ys,fs[i](xs))  for i in range(n)])
    a=phi.solve(b)
    poly=sum([a[k] * base[k] for k in range(n)])
    poly=sy.Lambda(x,poly.subs({x:x}))
    return poly

for n in range(8,11):
    poly=approx(n)
    xplt=np.linspace(0,2,100)
    yplt=np.vectorize(poly)(xplt)
    plt.plot(xplt,yplt)
    plt.ylim(-8.0,8.0)
    plt.plot(xs,ys,'bx')
```

```python

```

```python

```


# Question 1

## The Multivariate Gaussian distribution

The multivariate Gaussian distribution has the form

\begin{align}
\mathcal{N}(x; \mu, \Sigma) &= |2\pi \Sigma|^{-1/2} \exp\left( -\frac{1}{2} (x-\mu)^\top \Sigma^{-1} (x-\mu) \right) \\
& = \exp\left(-\frac{1}{2} x^\top \Sigma^{-1} x + \mu^\top \Sigma^{-1} x   - \frac{1}{2} \mu^\top \Sigma^{-1} \mu   -\frac{1}{2}\log \det(2\pi \Sigma) \right) \\
\end{align}

## Gaussian Processes Regression

In Bayesian machine learning, a frequent problem encountered is the regression problem where we are given a pairs of inputs $x_i \in \mathbb{R}^N$ and associated noisy observations $y_i \in \mathbb{R}$. We assume the following model

\begin{eqnarray*}
y_i &\sim&  {\cal N}(y_i; f(x_i), R)
\end{eqnarray*}

The interesting thing about a Gaussian process is that the function $f$ is not specified in close form, but we assume that the function values
\begin{eqnarray*}
f_i & = & f(x_i)
\end{eqnarray*}
are jointly Gaussian distributed as
\begin{eqnarray*}
\left(
  \begin{array}{c}
    f_1 \\
    \vdots \\
    f_L \\
  \end{array}
\right) & = & f_{1:L} \sim  {\cal N}(f_{1:L}; 0, \Sigma(x_{1:L}))
\end{eqnarray*}
Here, we define the entries of the covariance matrix $\Sigma(x_{1:L})$ as
\begin{eqnarray*}
\Sigma_{i,j} & = & K(x_i, x_j)
\end{eqnarray*}
for $i,j \in \{1, \dots, L\}$. Here, $K$ is a given covariance function. Now, if we wish to predict the value of $f$ for a new $x$, we simply form the following joint distribution:
\begin{eqnarray*}
\left(
  \begin{array}{c}
    f_1 \\
    f_2 \\
    \vdots \\
    f_L \\
    f \\
  \end{array}
\right) & \sim &  {\cal N}\left( \left(\begin{array}{c}
    0 \\
    0 \\
    \vdots \\
    0 \\
    0 \\
  \end{array}\right)
  , \left(\begin{array}{cccccc}
    K(x_1,x_1) & K(x_1,x_2) & \dots & K(x_1, x_L) &  K(x_1, x) \\
    K(x_2,x_1) & K(x_2,x_2) & \dots & K(x_2, x_L) & K(x_2, x) \\
    \vdots &\\
    K(x_L,x_1) & K(x_L,x_2) & \dots & K(x_L, x_L) &  K(x_L, x)   \\
    K(x,x_1) & K(x,x_2) & \dots & K(x, x_L) &  K(x, x)   \\
  \end{array}\right) \right) \\
\left(
\begin{array}{c}
    f_{1:L} \\
    f
  \end{array}
\right) & \sim &  {\cal N}\left( \left(\begin{array}{c}
    \mathbf{0} \\
    0 \\
  \end{array}\right)
  , \left(\begin{array}{cc}
    \Sigma(x_{1:L}) &  k(x_{1:L}, x) \\
    k(x_{1:L}, x)^\top &  K(x, x)   \\
  \end{array}\right) \right) \\
\end{eqnarray*}

Here,  $k(x_{1:L}, x)$ is a $L \times 1$ vector with entries $k_i$ where

\begin{eqnarray*}
k_i = K(x_i, x)
\end{eqnarray*}

Popular choices of covariance functions to generate smooth regression functions include a Bell shaped one
\begin{eqnarray*}
K_1(x_i, x_j) & = & \exp\left(-\frac{1}2 \| x_i - x_j \|^2 \right)
\end{eqnarray*}
and a Laplacian
\begin{eqnarray*}
K_2(x_i, x_j) & = & \exp\left(-\frac{1}2 \| x_i - x_j \| \right)
\end{eqnarray*}

where $\| x \| = \sqrt{x^\top x}$ is the Euclidian norm.

## Part 1
From your notes, derive the expressions to compute the predictive density
\begin{eqnarray*}
p(\hat{y}| y_{1:L}, x_{1:L}, \hat{x})
\end{eqnarray*}

\begin{eqnarray*}
p(y | y_{1:L}, x_{1:L}, x) &=& {\cal N}(y; m, S) \\
m & = & \\
S & = &
\end{eqnarray*}

## Part 2
Write a program to compute the mean and covariance of $p(\hat{y}| y_{1:L}, x_{1:L}, \hat{x})$ to generate a for the following data:

     x = [-2 -1 0 3.5 4]
     y = [4.1 0.9 2 12.3 15.8]

Try different covariance functions $K_1$ and $K_2$ and observation noise covariances $R$ and comment on the nature of the approximation.

## Part 3
Suppose we are using a covariance function parameterised by
\begin{eqnarray*}
K_\beta(x_i, x_j) & = & \exp\left(-\frac{1}\beta  \| x_i - x_j \|^2 \right)
\end{eqnarray*}
Find the optimum regularisation parameter $\beta^*(R)$ as a function of observation noise variance via maximisation of the marginal likelihood, i.e.
\begin{eqnarray*}
\beta^* & = & \arg\max_{\beta} p(y_{1:N}| x_{1:N}, \beta, R)
\end{eqnarray*}
Generate a plot of $b^*(R)$ for $R = 0.01, 0.02, \dots, 1$ for the dataset given in 2.

Here, remember that
\begin{eqnarray*}
p(y_{1:N}| x_{1:N}, \beta, R) = \mathcal{N}(y_{1:N}; 0, K(x_{1:N})+R)
\end{eqnarray*}

For your reference, we give a basic implementation below:

```python
def cov_fun_bell(x1,x2,delta=1):
    return np.exp(-0.5*np.abs(x1-x2)**2/delta)

def cov_fun_exp(x1,x2):
    return np.exp(-0.5*np.abs(x1-x2))

def cov_fun(x1,x2):
    return cov_fun_bell(x1,x2,delta=1)

R = 0.05

x = np.array([-2, -1, 0, 3.5, 4]);
y = np.array([4.1, 0.9, 2, 12.3, 15.8]);

Sig = cov_fun(x.reshape((len(x),1)),x.reshape((1,len(x)))) + R*np.eye(len(x))
SigI = np.linalg.inv(Sig)

xx = np.linspace(-5,5,100)
yy = np.zeros_like(xx)
ss = np.zeros_like(xx)
for i in range(len(xx)):
    z = np.r_[x,xx[i]]
    CrossSig = cov_fun(x,xx[i])
    PriorSig = cov_fun(xx[i],xx[i]) + R

    yy[i] = np.dot(np.dot(CrossSig, SigI),y)
    ss[i] = PriorSig - np.dot(np.dot(CrossSig, SigI),CrossSig)

plt.plot(x,y,'or')
plt.plot(xx,yy,'b.')
plt.plot(xx,yy+3*np.sqrt(ss),'b:')
plt.plot(xx,yy-3*np.sqrt(ss),'b:')
plt.show()
```

# Question 2

Read http://mbmlbook.com/TrueSkill.html
and
http://mbmlbook.com/TrueSkill_Modelling_the_outcome_of_games.html

* Write a program that produces 10,000 samples from a Gaussian with zero mean and a standard deviation of 1. Compute the percentage of these samples which lie between -1 and 1, between -2 and 2 and between -3 and 3. You should find that these percentages are close to those given in the caption of Figure 3.4.

* Construct a histogram of the samples created in the previous exercise (like the ones in Figure 3.7) and verify that it resembles a bell-shaped curve.

* Compute the mean, standard deviation and variance of your samples, referring to Panel 3.1. The mean should be close to zero and the standard deviation and variance should both be close to 1 (since 1^2=1).

* Produce a second set of 10,000 samples from a Gaussian with mean 1 and standard deviation 1. Plot a scatter plot like Figure 3.8 where the X co-ordinate of each point is a sample from the first set and the Y co-ordinate is the corresponding sample from the second set (pairing the first sample from each set, the second sample from each set and so on). Compute the fraction of samples which lie above the diagonal line where X=Y.

* Create double variables X and Y with priors of Gaussian(0,1) and Gaussian(1,1) respectively. Define a third random variable Ywins that equals to Y>X. Compute the posterior distribution numerically over Ywins and verify that it is close to the fraction of samples above the diagonal in the previous exercise.


# Build a Circuit to Emulate the Scope's Filter

Main reference: [Passive Low Pass Filter](https://www.electronics-tutorials.ws/filter/filter_2.html)

## Relavant Equiations

### RC Potential Divider Equation
$$ V_{out}=V_{in}\frac{X_C}{Z} \; (V)$$
where
\begin{align}
    X_C & = \frac{1}{2\pi fC} \; (\Omega)\\
    Z & = \sqrt{R^2+X_C^2} \; (\Omega)
\end{align}

Voltage gain can be written as
\begin{align}
    \frac{V_{out}}{V_{in}} &= \frac{1}{\sqrt{1+\left(\frac{R}{X_C}\right)^2}} \\
    &= \frac{1}{\sqrt{1+ \left( 2\pi fRC \right)^2 }}
\label{eq:voltage_gain} \tag{1}
\end{align}

### Power Gain Level

The [power gain level](https://en.wikipedia.org/wiki/Decibel#Root-power_(field)_quantities) is defined as
\begin{align}
    L_G &= 10\log\left( \frac{V_{out}}{V_{in}}\right)^2 \\
    &= 20\log\left( \frac{V_{out}}{V_{in}}\right) \; \textrm{(dB)}
\label{eq:power_gain_level} \tag{2}
\end{align}

We can make the famous [**Bode plot**](https://en.wikipedia.org/wiki/Bode_plot) with $RC=1$. (maybe not now...)

### Cutoff Frequency
The cutoff frequency is defined as the frequency at which the power gain drops to $50\%$. In decibel unit, this value is $-3$ dB, known as the [$3$ dB point](https://en.wikipedia.org/wiki/Cutoff_frequency#Electronics).

Here is an ambiguity. In literature about Butterworth filter, frequency actually means **angular frequency**. However, by glancing through the scope's manual, I cannot figure out whether the frequency is the **usual frequency** or the **angular frequency**.

For now, I will assume the frequency used by the scope is also **angular frequency**.

## The RC Values Leading to a 20 MHz Cutoff

With a power gain $\alpha$, we have
\begin{equation}
    \left( \frac{V_{out}}{V_{in}} \right)^2=\alpha
\label{eq:cutoff_power} \tag{3}
\end{equation}

Substitute eq.$~\eqref{eq:voltage_gain}$ into eq.$~\eqref{eq:cutoff_power}$ and rearrange, we have
\begin{equation}
    RC=\frac{1}{\omega}\sqrt{\frac{1}{\alpha}-1} \;\; (\omega=2\pi f)
\end{equation}

With $\alpha=0.5$, $\omega =20$ MHz, we obtain
\begin{equation}
    \boxed{RC=5\times 10^{-8}}
\end{equation}

```python
from math import *
RC=1/20e6*sqrt(1/.5-1)
print(RC)
```

    5e-08

## Scope Measurement

I inject sine waves with fixed frequency from the pulse generator. The pulse generator shows that the frequency it means is **normal frequency**.

The pulse amplitude I set up on the function generator is 1.65 V. However, I am measuring peak-to-peak voltage with the scope, which is enlarged by noise.

```python
import matplotlib.pyplot as plt
import math
import numpy as np
import scipy.optimize
x = [0.01e6, 0.1e6, 1e6, 5e6, 10e6, 15e6, 20e6, 25e6, 30e6, 35e6, 40e6, 45e6, 50e6, 60e6, 70e6, 80e6, 100e6]
y = [1.88, 1.88, 1.88, 1.82, 1.72, 1.56, 1.32, 1.08, 0.84, 0.6, 0.4, 0.32, 0.266, 0.207, 0.184, 0.173, 0.162]
```

```python
def divider_eq(x, vin, RC, voffset):
    return vin/np.sqrt(1+(2*math.pi*x*RC)**2) + voffset
# initial parameters
p_init = [1.88, 5e-8, 0.162]
fit_par, fit_err = scipy.optimize.curve_fit(divider_eq, x, y, p_init)
print(fit_par[0], fit_par[1], fit_par[2])

x_fit = np.linspace(.5e6, 51e6, 100)
y_fit = divider_eq(x_fit, fit_par[0], fit_par[1], fit_par[2])
plot_fit = plt.scatter(x=x_fit, y=y_fit, c='r', s=4)

plot_raw_meas = plt.scatter(x=x, y=y)
cache = plt.xlabel('frequency (not angular) (Hz)')
cache = plt.ylabel('amplitude (V)')
```

```python
vout = 1.32-0.162
vin = 1.88-0.162
p_gain = (vout/vin)**2
print('power gain: {:.2f}'.format(p_gain))
print('RC: {:.2e}'.format(1/2/math.pi/20e6*sqrt(1/p_gain-1)))
print('2πRC: {:.2e}'.format(1/20e6*sqrt(1/p_gain-1)))
```

    power gain: 0.45
    RC: 8.72e-09
    2πRC: 5.48e-08