This notebook is created by  Amir Mardan. For any feedback or suggestion, please contact me via email, (mardan.amir.h@gmail.com).

# Fully Connected Neural Networks

In this notebook, we learn about fully connected neural networks (FCNNs).
In FCNNs, every node in one layer is connected to all nodes in the previous and the next layer.

Figure 1: An example of fully connected neural network.

FCNNs are “structure agnostic” and there are no special assumptions needed to be made about the input.

Mathematically speaking, a fully connected layer is a function from $\mathbb{R}^m$ to $\mathbb{R}^n$.

    Imagine we have the input of fully connected layer as $x \in \mathbb{R}^m$.
Then, considering the $i$-th output of this layer as $y_i \in \mathbb{R}$,

$
y_i = \sigma (w_1 x_1 + \dotsb + w_m x_m),
$
where $\sigma$ is an [activation function](https://en.wikipedia.org/wiki/Activation_function) and $w_i$ are learnable parameters in the network.
Now, the output of this layer is

\begin{equation}
y = \left[
    \begin{array}{c}
    \sigma (w_{1,1} x_1 + \dotsb + w_{1,m} x_m)\\
    \vdots\\
    \sigma (w_{n,1} x_1 + \dotsb + w_{n,m} x_m)
    \end{array}
\right]
\end{equation}

This is how the connection between two layers works.
A network with multiple fully connected layers is called a ***deep*** network.
However, in this notebook, we create shallow networks from scratch and TensorFlow to learn the basic concept of fully connected networks.

This notebook will cover the following topics:

- [1. One Input - One Output](#1in_1out)
    - [1.1 What does training mean?](#meaning_training)
    - [1.2 How to update model parameters](#updating_parameters)
    - [1.3 Activation Function](#activation_function)
    - [1.4 Pseudo-code For Artificial Neural Network](#Pseudo_code)
    - [1.5 Training From Scratch](#training_1)
    - [1.6 Training With TensorFlow](#training_1_tf)
- [2. Multiple Input - One Output](#2in_1out)
    - [2.1 Training From Scratch](#training_2)
    - [2.2 Training With TensorFlow](#training_2_tf)

```python
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
import copy

print("TensorFlow version:", tf.__version__)
```

    TensorFlow version: 2.8.0

I'm currently using `TensorFlow version: 2.8.0`.

## 1. One Input - One Output

Let's talk about the easiest version of NN.

Imagine we have the input of a fully connected layer as $x \in \mathbb{R}^2$.
Then, considering the output of this layer as $y \in \mathbb{R}$,

\begin{equation}
y = \sigma (b + w_1 x_1),
\end{equation}

where $\sigma$ is an [activation function](https://en.wikipedia.org/wiki/Activation_function) and $w$ and $b$ are learnable parameters in the network.

Let's imagine our input has `100` samples while `m = 2`.
So, input should have 1 (m - 1) column (`input_shape=1`).
We also have only one output, `units = 1`.

```python
N = 100
m = 2
units = 1
input_shape = m - 1

X = np.random.randn(input_shape, N)
b_true = 2
w_true = np.array([5]).reshape(input_shape, units)

noise_scale = 0.5
noise = np.random.normal(scale=noise_scale, size=(1, N))
y = w_true.T @ X + b_true + noise

print(f"Shape of X: {X.shape}")
print(f'Shape of y: {y.shape}')
print(f'Shape of W: {w_true.shape}')
print(f'Shape of b: {np.shape(b_true)}')
```

    Shape of X: (1, 100)
    Shape of y: (1, 100)
    Shape of W: (1, 1)
    Shape of b: ()

```python
x_test = np.arange(-3, 3, 0.01).reshape(1, -1)
y_pred = w_true.T @ x_test + b_true

plt.scatter(X, y)
plt.plot(x_test.T, y_pred.T, 'r')
plt.xlabel('X')
plt.ylabel('y')
plt.grid()

```

### 1.1 What does training mean?

As you know, we train a network to predict the correct label for our data samples.
As is shown in previous cell (`y_pred = x_test @ w_true + b_true`) the predicted label depends on not only data samples, but also ***weight*** (`w_true`) and ***bias*** (`b_true`) of the model.
So, the goal of the neural network is to find the correct weight and bias for a model which can be called model parameters.
This is happening by using random values for these two parameters and comparing the predicted value with the correct one.
The difference between the predicted and observed value is called ***loss*** or ***cost***.
Hence, training is the problem of minimizing the loss by updating the model parameters.
So, the general procedure of training can be written as

1. Initialize model parameters ($W$ and $b$)
2. Calculate the loss
3. While loss > desired loss do:

    4. update the model parameters
    5. Calculate the loss

### 1.2 How to update model parameters?

The loss function we use for a regression problem can be mean squared error as

\begin{equation}
\ell = \frac{1}{N}\sum_{i=1}^{N}(y_{observed} - y_{predicted})^2,
\end{equation}

where $N$ is number of data points.

```python
def loss_function(y_observed, y_predicted):
    n = y_observed.size

    res = y_observed - y_predicted
    return 1/n * np.sum(res * res)
```

We know that `w_true = 5`. Let's plot the loss for `w` in a range of [-40 to 40].

```python
loss = []
y_obs = w_true.T @ x_test + b_true

ws = np.arange(-40, 40, 2)
for w in ws:
    w = w.reshape(-1, 1)
    y_pred = w.T @ x_test + b_true

    loss.append(loss_function(y_obs, y_pred))

plt.plot(ws, loss, label='Loss')
plt.plot(w_true, [0], 'or', label='Minimum loss')
plt.xlabel("w")
plt.ylabel('Loss')
plt.title('Loss function')
plt.legend()
plt.grid()
```

As we can we, the true weight can be found where loss function has an extreme point.
The extreme point of a function is where the gradient of the function is zero.
As you can see, if we choose a `w` the higher than the true one, gradient (slope of loss) is positive, so we need to update `w` with **negative** of gradient.
If we pick a small `w`, the gradient is negative and we need to update the model parameter again with **negative** of the gradient.
Hence, we can say that the gradient of the loss with respect to model parameters gives us the ***learning direction***.

You might have guessed from its name that the learning direction just specifies the direction of optimization.
The speed of training is defined by ***learning rate*** (`lr`).

By having learning direction and learning rate, we can now, update the model parameters

\begin{equation}
    \begin{aligned}
        &w := w - lr * \frac{d\ell}{dw}\\
        &b := b - lr * \frac{d\ell}{db}\\
    \end{aligned}
\end{equation}
where $\frac{d\ell}{dw}$ and $\frac{d\ell}{db}$ are the gradient of the loss with respect to $w$ and $b$, respectively.

### 1.3 Activation Function

The nodes in artificial neural networks perform two operations,
1. Calculate the weighted sum of inputs,
2. Pass the result of step 1 into an [activation function](https://en.wikipedia.org/wiki/Activation_function).

For linear regression this activation function is linear.
$
a = \sigma(z) = z.
$
where

$
z = W^T X  + b.
$
Thereby, we can even ignore it.
However as we're looking for the general structure of artificial neural networks, it's better to be considered in our computations.

```python
def sigma(z):
    return z
```

### 1.4 Pseudo-code For Artificial Neural Network

The following steps are showing the general procedure of training a fully connected network.

1. Choose $lr$
2. Initialize $W$ and $b$

    3. Calculate $z =  W^Tx + b$
    4. Activate $a = \sigma(z)$
    5. Calculate the cost function ($\ell$)
    6. Calculate $\frac{d\ell}{da}$
    7. Calculate $\frac{d\ell}{dz}$
    8. Calcculate $\frac{d\ell}{dw}$ and $\frac{d\ell}{db}$
    9. Update $W$ and $b$ as
        \begin{equation}
            \begin{aligned}
                W := W - \alpha * \frac{d\ell}{dw}\\
                b := b - \alpha * \frac{d\ell}{db}\\
            \end{aligned}
        \end{equation}
    10. Return to step 3.

Steps 6 to 8 are based on the [chain rule](https://www.khanacademy.org/math/ap-calculus-ab/ab-differentiation-2-new/ab-3-1a/a/chain-rule-review) where we compute the gradient of the cost function with respect to model parameters using [automatic differentiation](https://en.wikipedia.org/wiki/Automatic_differentiation).

Based on the chain rule,

\begin{equation}
    \begin{aligned}
        &\frac{d\ell}{dW} = \frac{d\ell}{da} \frac{da}{dz} \frac{dz}{dW},\\
        &\frac{d\ell}{db} = \frac{d\ell}{da} \frac{da}{dz} \frac{dz}{db}.
    \end{aligned}
\end{equation}

We have

\begin{equation}
    \begin{aligned}
        &\frac{d\ell}{da} = \frac{2}{m} (a - y),\\
    \end{aligned}
\end{equation}
and as we are using linear activation function,

\begin{equation}
    \begin{aligned}
         &\frac{da}{dz} = 1,  %(1 - a)
    \end{aligned}
\end{equation}

and
\begin{equation}
    \begin{aligned}
        &\frac{dz}{dW} = X^T,\\
        &\frac{dz}{db} = 1.\\
    \end{aligned}
\end{equation}

Finally

\begin{equation}
    \begin{aligned}
        &\frac{d\ell}{dW} = (a - y) X^T,\\
        &\frac{d\ell}{db} = (a - y).
    \end{aligned}
\end{equation}

```python
# Step 1. Choose learning rate

lr = 0.001
```

```python
# Step 2. Initialize W and b

def initialize_weights(units, n_input):
    params = {
        'w': np.random.random((n_input, units)),
        'b': np.random.random((1, 1))
    }
    return params
```

```python
# Step 3. Calculate z = xw + b

def fun(x, lerning_params):
    W = lerning_params['w']
    b = lerning_params['b']

    return  W.T @ x + b
```

```python
# Step 4. Activate function

def sigma(z):
    return z

```

```python
# Step 5. Calculate the cost function

def loss_function(z, y_true):
    m = y_true.size

    res = z - y_true
    return 1/m * np.sum(res * res)
```

```python
# Step 6. Calculate dl/da

def dl_da(a, y):
    return a - y
```

```python
# Step 7. Calculate dl/dz

def dl_dz(a, da):
    da_dz = 1

    dz = da * da_dz
    return dz
```

```python
# Step 8. Calculate dl/dw and dl/db

def dparams(X, dz):
    grad = {

        'dw': X @ dz.T,
        'db': np.sum(dz)
    }
    return grad
```

```python
# Step 9. Update W and b

def update(params, grad, lr):
    params['w'] -= lr * grad['dw']
    params['b'] -= lr * grad['db']
    return params
```

```python
def plot_test(lerning_params, title=None):

    x_test = np.arange(-3, 3, 0.01).reshape(1,-1)
    y_pred = fun(x_test, lerning_params)

    plt.scatter(X, y)
    plt.plot(x_test.T, y_pred.T, 'r')
    plt.xlabel('X')
    plt.ylabel('y')
    plt.grid()
    plt.title(title)
```

### 1.5 Training From Scratch

Now, we perform the training by executing the mentioned steps.

```python
# Training
epochs = 20  # Number of iteration
hist = []   # Placeholder to keep the loss in each step

lr = 0.002                                              # Step 1: Choose learning rate
lerning_params = initialize_weights(1, 1)               # Step 2. Initialize W and b
initial_weights = copy.deepcopy(lerning_params)

for i in range(epochs):
    z = fun(X, lerning_params)                          # Step 3. Calculate z = xw + b
    a = sigma(z)                                        # Step 4. Activate function
    loss = loss_function(a, y)                          # Step 5. Calculate the cost function
    hist.append(loss)     # keep track of losses
    da = dl_da(a, y)                                    # Step 6. Calculate dl/da
    dz = dl_dz(a, da)                                   # Step 7. Calculate dl/dz
    grad = dparams(X, dz)                               # Step 8. Calculate dl/dw and dl/db
    lerning_params = update(lerning_params, grad, lr)   # Step 9. Update W and b

    if i%20 ==0:
        print(f"Epoch: {i}, loss: {loss}")

plt.figure()
plt.plot(hist)
plt.grid()
```

Let's plot the result of untrained and trained models.

```python
fig = plt.figure(figsize=(9, 4))
ax = fig.add_subplot(121)
plot_test(initial_weights, 'Before training')

ax = fig.add_subplot(122)
plot_test(lerning_params, 'After training')

```

### 1.6 Training With TensorFlow

The same problem can be solved easily with TensorFlow.

We create a network with one layer which gets one input (`input_shape=[1]`) and gives one output (`units=1`). This layer uses linear activation (`activation='linear'`).

```python
from tensorflow import keras

class LinearRegression(keras.Model):
    def __init__(self, units, input_shape):
        super(LinearRegression, self).__init__()
        self.dense = keras.layers.Dense(units=units, input_shape=input_shape, activation='linear')

    def call(self, inputs):
        x = self.dense(inputs)
        return x

model = keras.Sequential([
    LinearRegression(units=1, input_shape=[1])
    ])

```

    2022-04-24 19:29:02.321553: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
    To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

After creating the model, we compile it.

```python
model.compile(optimizer='sgd',      # Gradient descent optimizer
              loss='mse'            # Mean squared error as loss function
)
```

And now, we train the model

```python
history = model.fit(X.T,              # Training samples
                    y.T,              # Training labels
                    epochs=epochs,  # Number of iteration
                    verbose=0       # Status of prining information
                    )
```

```python
model.summary()
```

    Model: "sequential"
    _________________________________________________________________
     Layer (type)                Output Shape              Param #
    =================================================================
     linear_regression (LinearRe  (None, 1)                2
     gression)

    =================================================================
    Total params: 2
    Trainable params: 2
    Non-trainable params: 0
    _________________________________________________________________

Let's compare the result of our linear regression and TensorFlow's.

```python
fig = plt.figure(figsize=(8, 4))
ax = fig.add_subplot(121)
ax.plot(history.history['loss'])
ax.set_title('TensorFlow')
ax.grid()
ax = fig.add_subplot(122)
ax.plot(hist)
ax.set_title('From scratch')
ax.grid()
```

We can get the model parameters using `model.weights`.

```python
tf_params = {'w': model.weights[0].numpy(),
             'b': model.weights[1].numpy()
}
```

```python
fig = plt.figure(figsize=(9, 4))
ax = fig.add_subplot(121)
plot_test(tf_params, 'TensorFlow')

ax = fig.add_subplot(122)
plot_test(lerning_params, 'From scratch')
```

Before going further, I organize our training code as follows

```python
class LinearRegression_scratch():
    def __init__(self, units, n_input, lr=0.002, verbose=0):
        self. n_input = n_input
        self.lr = lr
        self.verbose = verbose
        self.units = units

        self.weights = initialize_weights(units=units, n_input=n_input)

    def fit(self, X, y, epochs):
        history = {'loss': []}

        for i in range(epochs):
            z = fun(X, self.weights)
            a = sigma(z)
            loss = loss_function(a, y)
            history['loss'].append(loss)
            da = dl_da(a, y)
            dz = dl_dz(a, da)
            grad = dparams(X, dz)

            update(self.weights, grad, self.lr)

            if self.verbose == 1:
                print(f"Epoch: {i}, loss: {loss}")

        return history
```

## 2. Multiple Input - One Output

Now let's make a model with multi-dimensional input.

```python
N = 100
m = 3
units = 1
input_shape = m - 1

X = np.random.randn(input_shape, N)
b_true = 2
w_true = np.array([5, 2]).reshape(input_shape, units)

noise_scale = 0.5
noise = np.random.normal(scale=noise_scale, size=(1, N))
y = w_true.T @ X + b_true + noise

print(f"Shape of X: {X.shape}")
print(f'Shape of W: {w_true.shape}')
print(f'Shape of y: {y.shape}')
print(f'Shape of b: {np.shape(b_true)}')
```

    Shape of X: (2, 100)
    Shape of W: (2, 1)
    Shape of y: (1, 100)
    Shape of b: ()

```python
def plot_test_2in(lerning_params, title=None):

    x_test = np.vstack((np.arange(-3, 3, 0.01).reshape(1, -1),
                    np.arange(-3, 3, 0.01).reshape(1, -1)))

    y_pred = fun(x_test, lerning_params)

    fig = plt.figure(figsize=(10, 4))
    ax = fig.add_subplot(121)
    ax.scatter(X[0, :], y)
    plt.plot(x_test[0, :].T, y_pred.T, 'r')
    ax.set_xlabel('$X_1$')
    ax.set_ylabel('y')
    plt.grid()

    ax = fig.add_subplot(122)
    ax.scatter(X[1, :], y)
    plt.plot(x_test[1, :].T, y_pred.T, 'r')
    ax.set_xlabel('$X_2$')
    ax.set_ylabel('y')
    plt.grid()
    plt.suptitle(title)

plot_test_2in({'w':w_true, 'b':b_true}, title=None)

```

Thereby, the number of inputs in this part is 2.

```python
n_input = 2
```

### 2.1 Training From Scratch

```python
# Training
epochs = 10  # Number of iteration

model_scratch = LinearRegression_scratch(units=units, n_input=n_input)
hist =  model_scratch.fit(X, y, epochs)

plt.figure()
plt.plot(hist['loss'])
plt.grid()
```

Finally the result is

```python
plot_test_2in(model_scratch.weights, title='Trained Model')
```

### 2.2 Training With TensorFlow

Now, let's train the model using TensorFlow.

```python
model = keras.Sequential([
    LinearRegression(units=1, input_shape=[n_input])
    ])

model.compile(optimizer='sgd',
              loss='mse'
              )

history = model.fit(X.T, y.T, epochs=epochs, verbose=0)

model.summary()

```

    Model: "sequential_1"
    _________________________________________________________________
     Layer (type)                Output Shape              Param #
    =================================================================
     linear_regression_1 (Linear  (None, 1)                3
     Regression)

    =================================================================
    Total params: 3
    Trainable params: 3
    Non-trainable params: 0
    _________________________________________________________________

As is shown, we have 3 parameters to train ($w_1$, $w_2$, and $b$).

```python
fig = plt.figure(figsize=(8, 4))
ax = fig.add_subplot(121)
ax.plot(history.history['loss'])
ax.set_title('TensorFlow')
ax.grid()
ax = fig.add_subplot(122)
ax.plot(hist['loss'])
ax.set_title('From scratch')
ax.grid()
```

```python
tf_params = {'w': model.weights[0].numpy(),
             'b': model.weights[1].numpy()
}
```

```python
plot_test_2in(tf_params, title='TensorFlow')
```

```python
print(f"True value of weights: {w_true.reshape(-1)}")
print(f"Weights from our code: {model_scratch.weights['w'].reshape(-1)}")
print(f"Weights from TensorFlow: {tf_params['w'].reshape(-1)}")

```

    True value of weights: [5 2]
    Weights from our code: [4.46124921 1.88815475]
    Weights from our TensorFlow: [2.6159163 0.6902651]

```python
print(f"True value of bias: {b_true}")
print(f"Bias from our code: {model_scratch.weights['b'].reshape(-1)}")
print(f"Bias from TensorFlow: {tf_params['b'].reshape(-1)}")
```

    True value of bias: 2
    Bias from our code: [1.78891717]
    Bias from our TensorFlow: [1.0209153]

### [TOP ☝](#top)


# Trading Lists and Trading Trajectories

### Introduction

[Almgren and Chriss](https://cims.nyu.edu/~almgren/papers/optliq.pdf) provided a solution to the optimal liquidation problem by assuming the that stock prices follow a discrete arithmetic random walk, and that the permanent and temporary market impact functions are linear functions of the trading rate.

Almgren and Chriss showed that for each value of risk aversion there is a unique optimal execution strategy. This optimal execution strategy is determined by a trading trajectory and its associated trading list. The optimal trading trajectory is given by:

\begin{equation}
x_j = \frac{\sinh \left( \kappa \left( T-t_j\right)\right)}{ \sinh (\kappa T)}X, \hspace{1cm}\text{ for } j=0,...,N
\end{equation}

and the associated trading list is given by:

\begin{equation}
n_j = \frac{2 \sinh \left(\frac{1}{2} \kappa \tau \right)}{ \sinh \left(\kappa T\right) } \cosh \left(\kappa \left(T - t_{j-\frac{1}{2}}\right)\right) X, \hspace{1cm}\text{ for } j=1,...,N
\end{equation}

where $t_{j-1/2} = (j-\frac{1}{2}) \tau$.

Given some initial parameters, such as the number of shares, the liquidation time, the trader's risk aversion, etc..., the trading list will tell us how many shares we should sell at each trade to minimize our transaction costs.

In this notebook, we will see how the trading list varies according to some initial trading parameters.

## Visualizing Trading Lists and Trading Trajectories

Let's assume we have 1,000,000 shares that we wish to liquidate. In the code below, we will plot the optimal trading trajectory and its associated trading list for different trading parameters, such as trader's risk aversion, number of trades, and liquidation time.

```python
%matplotlib inline

import matplotlib.pyplot as plt

import utils

# We set the default figure size
plt.rcParams['figure.figsize'] = [17.0, 7.0]

# Set the number of days to sell all shares (i.e. the liquidation time)
l_time = 60

# Set the number of trades
n_trades = 60

# Set the trader's risk aversion
t_risk = 1e-6

# Plot the trading list and trading trajectory. If show_trl = True, the data frame containing the values of the
# trading list and trading trajectory is printed
utils.plot_trade_list(lq_time = l_time, nm_trades = n_trades, tr_risk = t_risk, show_trl = True)
```

# Implementing a Trading List

Once we have the trading list for a given set of initial parameters, we can actually implement it. That is, we can sell our shares in the stock market according to the trading list and see how much money we made or lost. To do this, we are going to simulate the stock market with a simple trading environment. This simulated trading environment uses the same price dynamics and market impact functions as the Almgren and Chriss model. That is, stock price movements evolve according to a discrete arithmetic random walk and the permanent and temporary market impact functions are linear functions of the trading rate. We are going to use the same environment to train our Deep Reinforcement Learning algorithm later on.

We will describe the details of the trading environment in another notebook, for now we will just take a look at its default parameters. We will distinguish between financial parameters, such the annual volatility in stock price, and the parameters needed to calculate the trade list using the Almgren and Criss model, such as the trader's risk aversion.

```python
import utils

# Get the default financial and AC Model parameters
financial_params, ac_params = utils.get_env_param()
```

### Default Financial Parameters

```python
financial_params
```

Financial Parameters

  Annual Volatility:	  12%	   Bid-Ask Spread:    	     0.125

  Daily Volatility:	  0.8%	   Daily Trading Volume:	 5,000,000

### Parameters for the Almgren and Chriss Model

```python
ac_params
```

Almgren and Chriss Model Parameters

  Total Number of Shares to Sell:	                  1,000,000	   Fixed Cost of Selling per Share:	 $0.062

  Starting Price per Share:	                         $50.00	     Trader's Risk Aversion:	           1e-06

  Price Impact for Each 1% of Daily Volume Traded:	 $2.5e-06	    Permanent Impact Constant:	       2.5e-07

  Number of Days to Sell All the Shares:	              60	       Single Step Variance:	             0.144

  Number of Trades:	                                   60	       Time Interval between trades:	      1.0

The code below implements the trading list resulting from different trading parameters, such as trader's risk aversion, number of trades, and liquidation time. All other parameters, such as total number shares to sell, are taken from the simulated trading environment (see above).

```python
%matplotlib inline

import matplotlib.pyplot as plt

import utils

# We set the default figure size
plt.rcParams['figure.figsize'] = [17.0, 7.0]

# Set the random seed
sd = 1

# Set the number of days to sell all shares (i.e. the liquidation time)
l_time = 60

# Set the number of trades
n_trades = 60

# Set the trader's risk aversion
t_risk = 1e-7

# Implement the trading list for the given parameters
utils.implement_trade_list(seed = sd, lq_time = l_time, nm_trades = n_trades, tr_risk = t_risk)
```


## PS1-5 Locally weighted linear regression

#### (a)

##### i.

Recall the cost function

\begin{align*}
J(\theta) & = \frac{1}{2} \sum_{i = 1}^{m} w^{(i)} \big( \theta^T x^{(i)} - y^{(i)} \big)^2 \\
          & = (X \theta - y)^T W (X \theta - y)
\end{align*}

Notice that there are only $m$ $w^{(i)}$'s, and $w^{(i)}$ only takes effect on the $i$-th entry of $(X \theta - y)^T$ and $(X \theta - y)$.
Hence, $W$ is a diagonal matrix, and the above equation holds by picking:

\begin{equation}
X = \begin{bmatrix}
     - (x^{(1)})^T - \\
     \vdots \\
     - (x^{(m)})^T - \\
     \end{bmatrix}
     \in \mathbb{R}^{m \times n}
\qquad
y = \begin{bmatrix}
     y^{(1)} \\
     \vdots \\
     y^{(m)} \\
     \end{bmatrix}
\qquad
W = \frac{1}{2} \mathrm{diag} (w^{(i)}, \dots , w^{(m)})
\end{equation}

##### ii.

To simplify the calculation, let $J(\theta) = \frac{1}{2} (X \theta - y)^T W (X \theta - y)$. To minimize $J$, take derivative w.r.t $\theta$ and set to 0:

\begin{align*}
\nabla_\theta J(\theta) & = \nabla_\theta \big( \frac{1}{2} (X \theta - y)^T W (X \theta - y) \big) \\
                        & = \frac{1}{2} \nabla_\theta (\theta^T X^T W X \theta - 2 y^T W X \theta + y^T W y) \\
                        & = \frac{1}{2} \nabla_\theta \big( \theta^T (X^T W X) \theta - 2 (X^T W^T y)^T \theta \big) \\
                        & = \frac{1}{2} (2 X^T W X \theta - 2 X^T W^T y) \\
                        & = X^T W X \theta - X^T W^T y \\
                        & = X^T W X \theta - X^T W y \\
                        & = 0
\end{align*}

By solving the equation, we obtain:

$$\theta = (X^T W X)^{-1} X^T W y$$

##### iii.

\begin{align*}
\ell (\theta) & = \sum_{i = 1}^{m} \log \frac{1}{\sqrt{2 \pi} \sigma^{(i)}} \exp \big( - \frac{(y^{(i)} - \theta^T x^{(i)})^2}{2 (\sigma^{(i)})^2} \big) \\
              & = - m \log \sqrt{2 \pi} \sigma^{(i)} - \sum_{i = 1}^{m} \frac{(y^{(i)} - \theta^T x^{(i)})^2}{2 (\sigma^{(i)})^2} \\
              & = - \frac{1}{2} \sum_{i = 1}^{m} \frac{1}{(\sigma^{(i)})^2} (\theta^T x^{(i)} - y^{(i)})^2 - m \log \sqrt{2 \pi} \sigma^{(i)}
\end{align*}

Thus, maximizing $\ell (\theta)$ is equivalent to minimizing

$$\frac{1}{2} \sum_{i = 1}^{m} \frac{1}{(\sigma^{(i)})^2} (\theta^T x^{(i)} - y^{(i)})^2$$

By setting $w^{(i)} = 1 / (\sigma^{(i)})^2$, finding the maximum likelihood estimate of $\theta$ reduces to minimizing $J(\theta)$.

#### (b)

```python
import math
import matplotlib.pyplot as plt
import numpy as np
import problem_set_1.src.util as util

from problem_set_1.src.linear_model import LinearModel
```

```python
x_train, y_train = util.load_dataset('data/ds5_train.csv', add_intercept=True)
x_valid, y_valid = util.load_dataset('data/ds5_valid.csv', add_intercept=True)
x_test, y_test = util.load_dataset('data/ds5_test.csv', add_intercept=True)
```

Take a look at the training set:

```python
plt.xlabel('x_1')
plt.ylabel('y')
plt.plot(x_train[:,-1], y_train, 'bx', linewidth=2)
```

Implement locally weighted linear regression:

```python
class LocallyWeightedLinearRegression(LinearModel):
    """Locally Weighted Regression (LWR).

    Example usage:
        > clf = LocallyWeightedLinearRegression(tau)
        > clf.fit(x_train, y_train)
        > clf.predict(x_eval)
    """

    def __init__(self, tau):
        super(LocallyWeightedLinearRegression, self).__init__()
        self.tau = tau
        self.x = None
        self.y = None

    def fit(self, x, y):
        """Fit LWR by saving the training set.

        :param x: Training example inputs. Shape (m, n).
        :param y: Training example labels. Shape (m,).
        """

        self.x = x
        self.y = y

    def predict(self, x):
        """Make predictions given inputs x.

        :param x: Inputs of shape (l, n).
        :return:  Outputs of shape (l,).
        """

        l, n = x.shape

        # Reshape the input x by adding an additional dimension so that it can broadcast
        w_vector = np.exp(- np.linalg.norm(self.x - np.reshape(x, (l, -1, n)), ord=2, axis=2)**2 / (2 * self.tau**2))

        # Turn the weights into diagonal matrices, each corresponds to a single input. Shape (l, m, m)
        w = np.apply_along_axis(np.diag, axis=1, arr=w_vector)

        # Compute theta for each input x^(i). Shape (l, n)
        theta = np.linalg.inv(self.x.T @ w @ self.x) @ self.x.T @ w @ self.y

        return np.einsum('ij,ij->i', x, theta)
```

Train the model:

```python
clf = LocallyWeightedLinearRegression(tau=0.5)
clf.fit(x_train, y_train)
```

Plot predictions on training set and validation set:

```python
def plot(x, y_label, y_pred, title):
    plt.figure()
    plt.plot(x[:,-1], y_label, 'bx', label='label')
    plt.plot(x[:,-1], y_pred, 'ro', label='prediction')
    plt.suptitle(title, fontsize=12)
    plt.legend(loc='upper left')
```

```python
y_train_pred = clf.predict(x_train)
plot(x_train, y_train, y_train_pred, 'Training Set')

y_valid_pred = clf.predict(x_valid)
plot(x_valid, y_valid, y_valid_pred, 'Validation Set')
```

From the figures, we can see the model is underfitting.

#### (c)

```python
taus = [3e-2, 5e-2, 1e-1, 5e-1, 1e0, 1e1]

lowest_mse = math.inf
best_tau = taus[0]

for tau in taus:
    clf = LocallyWeightedLinearRegression(tau)
    clf.fit(x_train, y_train)
    y_valid_pred = clf.predict(x_valid)

    mse = np.mean((y_valid_pred - y_valid)**2)
    if mse < lowest_mse:
        lowest_mse = mse
        best_tau = tau

    plot(x_valid, y_valid, y_valid_pred, f'Validation Set (Tau = {tau}, MSE = {mse})')

print(f'Tau = {best_tau} achieves the lowest MSE on the validation set.')
```

Use $\tau = 0.05$ to make predictions on test set:

```python
clf = LocallyWeightedLinearRegression(tau=0.05)
clf.fit(x_train, y_train)
y_test_pred = clf.predict(x_test)

plot(x_test, y_test, y_test_pred, f'Test Set (MSE = {np.mean((y_test_pred - y_test)**2)})')
```

problem_set_1/PS1-5 Locally weighted linear regression.ipynb


```python
import numpy as np
import scipy as sp
import scipy.stats
import pandas as pd
import matplotlib.pyplot as plt
import statsmodels.formula.api as smf
plt.style.use('fivethirtyeight')
```

    E:\Anaconda\lib\site-packages\statsmodels\compat\pandas.py:65: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
      from pandas import Int64Index as NumericIndex

#  Simple Linear Regression

This is a simple linear regression model as in every textbooks
$$
Y_i=\beta_1+\beta_2X_i+u_i\qquad i = 1, 2, ..., n
$$
where $Y$ is **dependent variable**, $X$ is **independent variable** and $u$ is **disturbance term**. $\beta_1$ and $\beta_2$ are unknown parameters that we are aiming to estimate by feeding the data in the model.

Following Gauss-Markov assumptions,
$$
E(Y|X_i) = {\beta}_1 + {\beta}_2X_i\\
Y_{i} \sim \mathrm{N}\left({\beta_{1}}+{\beta_{2}}X_{i}, \sigma^{2}\right)
$$
Similarly, the precision is defined as $\tau=\frac{1}{\sigma^2}$.

##  The Priors

Each of these three parameters, i.e. $\beta_1$, $\beta_2$ and $\tau$ will be estimated, therefore each of them have their own **prior elicitation** which means define a prior with all proper knowledge available. For demonstration purpose, we define $\beta$ priors as normal distributions.

$$
P(\beta_1)=(2 \pi)^{-\frac{1}{2}} \tau_1^{\frac{1}{2}} \exp{\left(-\frac{1}{2} \tau_1 \left(\beta_1-\mu_1\right)^{2}\right)}\\
P(\beta_2)=(2 \pi)^{-\frac{1}{2}} \tau_2^{\frac{1}{2}} \exp{\left(-\frac{1}{2} \tau_2 \left(\beta_1-\mu_1\right)^{2}\right)}\\
$$

However the precision $\tau$ can't be negative, therefore a Gamma distribution would be appropriate. The posterior of $\tau$ will follow a Gamma distribution too due to **Gamma-Normal conjugate**.
$$
P(\tau)=\frac{\theta_{0}^{\alpha_{0}} \tau^{\alpha_{0}-1} e^{-\theta_{0} \tau}}{\Gamma\left(\alpha_{0}\right)} \quad 0 \leq \tau \leq \infty
$$

##  Likelihood Function

With normality assumption, likelihood of $Y$ is
$$
\mathcal{L}\left(Y | \mu_{i}, \sigma^{2}\right)=\prod_{i=1}^n\frac{1}{\sigma \sqrt{2 \pi}} \exp{\left(-\frac{1}{2}\left(\frac{Y_i-\mu_i}{\sigma}\right)^{2}\right)}
$$

Equivalently, replace $\sigma$ by $\tau$ and rewrite the function into summation
$$
\mathcal{L}\left(Y | \mu_{i}, \tau\right)=(2 \pi)^{-\frac{n}{2}} \tau^{\frac{n}{2}} \exp{\left(-\frac{1}{2} \tau \sum_{i=1}^n\left(Y_{i}-\mu_{i}\right)^{2}\right)}
$$

Replace $\mu_i$ by ${\beta}_1 + {\beta}_2X_i$, note how we have change the notation of parameters
$$
\mathcal{L}\left(Y | \beta_1, \beta_2, \tau\right)=(2 \pi)^{-\frac{n}{2}} \tau^{\frac{n}{2}} \exp{\left(-\frac{1}{2} \tau \sum_{i=1}^n\left(Y_{i}-({\beta}_1 + {\beta}_2X_i)\right)^{2}\right)}
$$

The likelihood function are the same for $\beta_1$, $\beta_2$ and $\tau$.
$$
\mathcal{L}\left(Y | \beta_1\right)=\mathcal{L}\left(Y | \beta_2\right)=\mathcal{L}\left(Y | \tau\right)=(2 \pi)^{-\frac{n}{2}} \tau^{\frac{n}{2}} \exp{\left(-\frac{1}{2} \tau \sum_{i=1}^n\left(Y_{i}-({\beta}_1 + {\beta}_2X_i)\right)^{2}\right)}
$$

##  The Posteriors

Be careful about the notation, $\tau_1$ is the precision of $\beta_1$ which is a variable rather than a constant viewed in frequentists.

To start from $\beta_1$
$$
P\left(\beta_{1}\right) \times P\left(Y\mid \beta_{1}\right)=(2 \pi)^{-\frac{1}{2}} \tau_{1}^{\frac{1}{2}} \exp\left({-\frac{\tau_{1}}{2}\left(\beta_{1}-\mu_{1}\right)^{2}}\right) \times (2 \pi)^{-\frac{n}{2}} \tau^{\frac{n}{2}} \exp{\left(-\frac{1}{2} \tau \sum_{i=1}^n\left(Y_{i}-({\beta}_1 + {\beta}_2X_i)\right)^{2}\right)}
$$

Join terms
$$
P\left(\beta_{1}\right) \times P\left(Y\mid \beta_{1}\right)=(2 \pi)^{-\frac{n+1}{2}} \tau_{1}^{\frac{1}{2}} \tau^{\frac{n}{2}} \exp{\left(-\frac{1}{2} \tau_{1}\left(\beta_{1}-\mu_{1}\right)^{2}-\frac{1}{2} \tau \sum_{i=1}^n\left(Y_{i}-\beta_{1}-\beta_{2} X_{i}\right)^{2}\right)}
$$

Let's focus on exponential term for now, expand the brackets
\begin{align}
&\exp{\left(-\frac{1}{2} \tau_{1}\left(\beta_{1}-\mu_{1}\right)^{2}-\frac{1}{2} \tau \sum_{i=1}^n\left(Y_{i}-\beta_{1}-\beta_{2} X_{i}\right)^{2}\right)}\\
&= \exp{\left[-\frac{1}{2} \tau_{1}\left(\beta_{1}^{2}-2 \beta_{1} \mu_{1}+\mu_{1}^{2}\right)-\frac{1}{2} \tau\left(n \beta_{1}^{2}-2 \beta_{1} \sum_{i=1}^n\left(Y_{i}-\beta_{2} X_{i}\right)+\sum_{i=1}^n\left(Y_{i}-\beta_{2} X_{i}\right)^{2}\right)\right]}\\
&=\exp{\left[-\frac{1}{2}\left(\tau_{1}+n \tau\right) \beta_{1}^{2}-\frac{1}{2}\left[-2 \tau_{1} \mu_{1}-2 \tau \sum_{i=1}^n\left(Y_{i}-\beta_{2} X_{i}\right)\right] \beta_{1}-\frac{1}{2} \tau_{1} \mu_{1}^{2}-\frac{1}{2} \tau \sum_{i=1}^n\left(Y_{i}-\beta_{2} X_{i}\right)^{2}\right]}\\
&=\exp{\left[-\frac{1}{2}\left(\tau_{1}+n \tau\right) \beta_{1}^{2}+\left[\tau_{1} \mu_{1}+\tau \sum_{i=1}^n\left(Y_{i}-\beta_{2} X_{i}\right)\right] \beta_{1}+\frac{-\frac{1}{2}\left[\tau_{1} \mu_{1}+\tau \sum_{i=1}^n\left(Y_{i}-\beta_{2} X_{i}\right)\right]^{2}}{\tau_{1}+n \tau}-\frac{-\frac{1}{2}\left[\tau_{1} \mu_{1}+\tau \sum_{i=1}^n\left(Y_{i}-\beta_{2} X_{i}\right)\right]^{2}}{\tau_{1}+n \tau}-\frac{1}{2} \tau_{1} \mu_{1}^{2}-\frac{1}{2} \tau \sum_{i=1}^n\left(Y_{i}-\beta_{2} X_{i}\right)^{2}\right]}\\
\end{align}
The first three terms can be combined and the rest are free of $\beta_1$, which will be denoted as $C$.

$$
-\frac{1}{2} \left(\tau_{1}+n \tau\right)\left[\beta_{1}-\frac{\left(\tau_{1} \mu_{1}+\tau \sum_{i=1}^n\left(Y_{i}-\beta_{2} X_{i}\right)\right)}{\tau_{1}+n \tau}\right]^{2}
$$

Back to Bayes' Theorem
$$
P\left(\beta_{1} | Y\right)=\frac{(2 \pi)^{-\frac{n+1}{2}} \tau_{1}^{\frac{1}{2}} \tau^{\frac{n}{2}} \exp{\left[-\frac{1}{2}\left(\tau_{1}+n \tau\right)\left[\beta_{1}-\frac{\left(\tau_{1} \mu_{1}+\tau \sum_{i=1}^n\left(Y_{i}-\beta_{2} X_{i}\right)\right)}{\tau_{1}+n \tau}\qquad\right]^{2}+C\right]}}{\int_{-\infty}^{\infty}(2 \pi)^{-\frac{n+1}{2}} \tau_{1}^{\frac{1}{2}} \tau^{\frac{n}{2}} \exp{\left[-\frac{1}{2}\left(\tau_{1}+n \tau\right)\left[u-\frac{\left(\tau_{1} \mu_{1}+\tau \sum_{i=1}^n\left(Y_{i}-\beta_{2} X_{i}\right)\right)}{\tau_{1}+n \tau}\qquad\right]^{2}+C\right]} d u}
$$

Cancel out terms, we obtain
$$
P\left(\beta_{1} | Y\right)=\frac{\exp{\left[-\frac{1}{2}\left(\tau_{1}+n \tau\right)\left[\beta_{1}-\frac{\left(\tau_{1} \mu_{1}+\tau \sum_{i=1}^n\left(Y_{i}-\beta_{2} X_{i}\right)\right)}{\tau_{1}+n \tau}\qquad\right]^{2}\right]}}{\int_{-\infty}^{\infty} \exp{\left[-\frac{1}{2}\left(\tau_{1}+n \tau\right)\left[u-\frac{\left(\tau_{1} \mu_{1}+\tau \sum_{i=1}^n\left(Y_{i}-\beta_{2} X_{i}\right)\right)}{\tau_{1}+n \tau}\qquad\right]^{2}\right]} d u}
$$

Now multiply both numerator and denominator by
$$
\frac{\sqrt{\tau_{1}+n \tau}}{\sqrt{2 \pi}}
$$
which yields

$$
P\left(\beta_{1} | Y\right)=\frac{\frac{\sqrt{\tau_{1}+n \tau}}{\sqrt{2 \pi}}\exp{\left[-\frac{1}{2}\left(\tau_{1}+n \tau\right)\left[\beta_{1}-\frac{\left(\tau_{1} \mu_{1}+\tau \sum_{i=1}^n\left(Y_{i}-\beta_{2} X_{i}\right)\right)}{\tau_{1}+n \tau}\qquad\right]^{2}\right]}}{\int_{-\infty}^{\infty}\frac{\sqrt{\tau_{1}+n \tau}}{\sqrt{2 \pi}} \exp{\left[-\frac{1}{2}\left(\tau_{1}+n \tau\right)\left[u-\frac{\left(\tau_{1} \mu_{1}+\tau \sum_{i=1}^n\left(Y_{i}-\beta_{2} X_{i}\right)\right)}{\tau_{1}+n \tau}\qquad\right]^{2}\right]} d u}
$$

The denominator is equal to $1$, which leaves us
$$
P\left(\beta_{1} | Y\right)=(2\pi)^{-\frac{1}{2}}(\tau_1+nr)^{\frac{1}{2}}\exp{\left[-\frac{1}{2}\left(\tau_{1}+n \tau\right)\left[\beta_{1}-\frac{\left(\tau_{1} \mu_{1}+\tau \sum_{i=1}^n\left(Y_{i}-\beta_{2} X_{i}\right)\right)}{\tau_{1}+n \tau}\qquad\right]^{2}\right]}
$$
Compare with the prior
$$
P(\beta_1)=(2 \pi)^{-\frac{1}{2}} \tau_1^{\frac{1}{2}} \exp{\left(-\frac{1}{2} \tau_1 \left(\beta_1-\mu_1\right)^{2}\right)}
$$

The posterior hyperparameters of $\beta_1$ are
$$
\begin{gathered}
\mu_{\text {1posterior }}=\frac{\tau_{1} \mu_{1}+\tau \sum_{i=1}^n\left(Y_{i}-\beta_{2} X_{i}\right)}{\tau_{1}+n \tau} \\
\tau_{\text {1posterior }}=\tau_{1}+n \tau
\end{gathered}
$$

```python
def mu_tau_1_post(tau_1_prior, mu_1_prior, mu_2_prior, tau_lh, Y, X, n):
    mu_1_post = (tau_1_prior*mu_1_prior + tau_lh*(np.sum(Y - mu_2_prior*X)))/(tau_1_prior + n*tau_lh)
    tau_1_post = tau_1_prior + n*tau_lh
    sigma_1_post = 1/np.sqrt(tau_1_post)
    return mu_1_post, tau_1_post, sigma_1_post
```

The derivation of posterior for $\beta_2$ are very, will not be reproduced here
$$
\begin{gathered}
\mu_{2 \text { posterior }}=\frac{\tau_{2} \mu_{2}+\tau \sum_{i=1}^n X_{i}\left(Y_{i}-\beta_{1}\right)}{\tau_{2}+\tau \sum_{i=1}^n X_{i}^{2}} \\
\tau_{2 \text { posterior }}=\tau_{2}+\tau \sum_{i=1}^n X_{i}^{2}
\end{gathered}
$$

```python
def mu_tau_2_post(tau_2_prior, mu_2_prior, mu_1_prior, tau_lh, Y, X, n):
    mu_2_post = (tau_2_prior*mu_2_prior + tau_lh*np.sum(X*(Y-mu_1_prior)))/(tau_2_prior + tau_lh*np.sum(X**2))
    tau_2_post = tau_2_prior + tau_lh*np.sum(X**2)
    sigma_2_post = 1/np.sqrt(tau_2_post)
    return mu_2_post, tau_2_post, sigma_2_post
```

We will derive the posterior of $\tau$, prior multiply by likelihood equals
$$
P(\tau)  P(Y | \tau)=\frac{\beta_{0}^{\alpha_{0}} \tau^{\alpha_{0}-1} e^{-\beta_{0} \tau}}{\Gamma\left(\alpha_{0}\right)} *(2 \pi)^{-\frac{n}{2}} \tau^{\frac{n}{2}} \exp{\left[-\frac{1}{2} \tau \Sigma\left(y_{i}-\left(b_{0}-b_{1} x_{i}\right)\right)^{2}\right]}
$$

##  A Simulated Example

```python
beta_1, beta_2 = 8, 15
n = 30
X = np.arange(30)
u = np.random.rand(30)
Y = beta_1 + beta_2*X + u
```

```python
mu_1_prior, sigma_1_prior = 2, 3
mu_2_prior, sigma_2_prior = 20, 6
sigma_lh = np.std(Y)
tau_lh = 1/sigma_lh**2
tau_1_prior, tau_2_prior = 1/sigma_1_prior**2, 1/sigma_2_prior**2
```

```python
mu_1_post, tau_1_post, sigma_1_post = mu_tau_1_post(tau_1_prior, mu_1_prior,
                                                    mu_2_prior, tau_lh, Y, X, n)
mu_2_post, tau_2_post, sigma_2_post = mu_tau_2_post(tau_2_prior, mu_2_prior,
                                                    mu_1_prior, tau_lh, Y, X, n)
```

```python
betax_prior_1 = np.linspace(-15, 10, 100)
betax_prior_2 = np.linspace(0, 30, 100)

betay_prior_1 = sp.stats.norm.pdf(betax_prior_1, loc=mu_1_prior, scale=sigma_1_prior)
betay_prior_2 = sp.stats.norm.pdf(betax_prior_2, loc=mu_2_prior, scale=sigma_2_prior)

betax_post_1 = np.linspace(mu_1_post-3*sigma_1_post, mu_1_post+3*sigma_1_post, 100)
betax_post_2 = np.linspace(mu_2_post-sigma_2_post, mu_2_post+sigma_2_post, 100)

betay_post_1 = sp.stats.norm.pdf(betax_prior_1, loc=mu_1_post, scale=sigma_1_post)
betay_post_2 = sp.stats.norm.pdf(betax_prior_2, loc=mu_2_post, scale=sigma_2_post)

fig, ax = plt.subplots(figsize=(12, 6), nrows=1, ncols=2)
ax[0].plot(betax_prior_1, betay_prior_1, lw = 3, label='Prior')
ax[0].plot(betax_post_1, betay_post_1, lw=3, label='Posterior')
ax[0].set_title(r'$\beta_1$')
ax[1].plot(betax_prior_2, betay_prior_2, lw = 3, label='Prior')
ax[1].plot(betax_prior_2, betay_post_2, lw=3, label='Posterior')
ax[1].set_title(r'$\beta_2$')
ax[0].legend()
ax[1].legend()

print('mu_1_post: {:.4f}'.format(mu_1_post))
print('tau_1_post: {:.10f}'.format(tau_1_post))
print('sigma_1_post: {:.4f}'.format(sigma_1_post))
print('')
print('mu_2_post: {:.4f}'.format(mu_2_post))
print('tau_2_post: {:.10f}'.format(tau_2_post))
print('sigma_2_post: {:.4f}'.format(sigma_2_post))
plt.show()
```

##  An Example of Apartment Price

```python
df = pd.read_excel('Basic_Econometrics_practice_data.xlsx', 'CN_Cities_house_price')
```

```python
df.head()
```

      cities
      house_price
      salary

      0
      Shenzhen
      87957
      64878

      1
      Beijing
      64721
      69434

      2
      Shanghai
      59072
      72232

      3
      Xiamen
      49803
      58140

      4
      Guangzhou
      39851
      68304

The model to fit the data ill be exactly as we derived above, i.e.
$$
Y_i=\beta_1+\beta_2X_i+u_i\qquad i = 1, 2, ..., n
$$

This simple model aims to explain the influence of salary on apartment price per square meter. The $\beta_1$ doesn't have much meaning in the model, literally it means the apartment price when salary equals $0$. However $\beta_2$ has a concrete meaning that how much apartment price can raise based on every one yuan increase in average salary.

We can choose to elicit priors' hyperparameters for $\beta_1$ and $\beta_2$ as
$$
\mu_1 = 1000\\
\sigma_1 = 200 \qquad \tau_1 = \frac{1}{\sigma_1^2}\\
\mu_2 = 1.5\\
\sigma_2 = .3 \qquad \tau_2 = \frac{1}{\sigma_2^2}\\
$$

```python
Y, X = df['house_price'], df['salary']

mu_1_prior, sigma_1_prior = 1000, 200
mu_2_prior, sigma_2_prior = 1, .1
sigma_lh = 3000
tau_lh = 1/sigma_lh**2
tau_1_prior, tau_2_prior = 1/sigma_1_prior**2, 1/sigma_2_prior**2

betax_prior_1 = np.linspace(400, 1600, 100)
betax_prior_2 = np.linspace(.5, 2.5, 100)

betay_prior_1 = sp.stats.norm.pdf(betax_prior_1, loc=mu_1_prior, scale=sigma_1_prior)
betay_prior_2 = sp.stats.norm.pdf(betax_prior_2, loc=mu_2_prior, scale=sigma_2_prior)
n = len(X)

# use functions to calculate posterior hyperparameters
mu_1_post, tau_1_post, sigma_1_post = mu_tau_1_post(tau_1_prior, mu_1_prior, mu_2_prior, tau_lh, Y, X, n)
mu_2_post, tau_2_post, sigma_2_post = mu_tau_2_post(tau_2_prior, mu_2_prior, mu_1_prior, tau_lh, Y, X, n)

betay_post_1 = sp.stats.norm.pdf(betax_prior_1, loc=mu_1_post, scale=sigma_1_post)
betay_post_2 = sp.stats.norm.pdf(betax_prior_2, loc=mu_2_post, scale=sigma_2_post)

fig, ax = plt.subplots(figsize=(17, 6), nrows=1, ncols=2)

betax_post_1 = np.linspace(mu_1_post-3*sigma_1_post, mu_1_post+3*sigma_1_post, 100)
#betax_post_2 = np.linspace(mu_2_post-sigma_2_post, mu_2_post+sigma_2_post, 100)

ax[0].plot(betax_prior_1, betay_prior_1, lw = 3)
ax[0].plot(betax_post_1, betay_post_1, lw=3)
ax[0].set_title(r'$\beta_1$')
ax[1].plot(betax_prior_2, betay_prior_2, lw = 3)
ax[1].plot(betax_prior_2, betay_post_2, lw=3)
ax[1].set_title(r'$\beta_2$')

plt.show()

print('mu_1_post: {:.4f}'.format(mu_1_post))
print('tau_1_post: {:.10f}'.format(tau_1_post))
print('sigma_1_post: {:.4f}'.format(sigma_1_post))
print('')
print('mu_2_post: {:.4f}'.format(mu_2_post))
print('tau_2_post: {:.10f}'.format(tau_2_post))
print('sigma_2_post: {:.4f}'.format(sigma_2_post))
```

```python
model = smf.ols(formula='house_price ~ salary', data=df)
results = model.fit()
print(results.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:            house_price   R-squared:                       0.365
    Model:                            OLS   Adj. R-squared:                  0.337
    Method:                 Least Squares   F-statistic:                     13.21
    Date:                Sat, 02 Apr 2022   Prob (F-statistic):            0.00139
    Time:                        12:17:05   Log-Likelihood:                -274.08
    No. Observations:                  25   AIC:                             552.2
    Df Residuals:                      23   BIC:                             554.6
    Df Model:                           1
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    Intercept  -2.918e+04   1.66e+04     -1.758      0.092   -6.35e+04    5156.201
    salary         1.1010      0.303      3.635      0.001       0.474       1.728
    ==============================================================================
    Omnibus:                       14.403   Durbin-Watson:                   0.926
    Prob(Omnibus):                  0.001   Jarque-Bera (JB):               14.263
    Skew:                           1.457   Prob(JB):                     0.000800
    Kurtosis:                       5.280   Cond. No.                     3.12e+05
    ==============================================================================

    Notes:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
    [2] The condition number is large, 3.12e+05. This might indicate that there are
    strong multicollinearity or other numerical problems.

```python
ols_coeff = pd.DataFrame(results.params, columns =['OLS estimates'])
ols_coeff
```

      OLS estimates

      Intercept
      -29181.169815

      salary
      1.100985

```python

```

Chapter 3 - Bayesian Simple Linear Regression.ipynb


# LAPLACE TRANSFORMS FOR REACTION NETWORKS

# Preliminaries

```python
!pip -q install controlSBML
import controlSBML as ctl
import control
from controlSBML.util import makeSimulationTimes
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import sympy as sy
import tellurium as te
import sympy
print("controlSBML version: " + ctl.__version__)
```

    controlSBML version: 0.2.8

```python
s, S1, S2, S3, k1, k2, k3, k4 = sympy.symbols("s, S1, S2, S3, k1, k2, k3, k4 ")
```

# Sequential Pathway

$S_1 \xrightarrow[\text{}]{k_1 S_1} S_2$,
$S_2 \xrightarrow[\text{}]{k_2 S_2} S_3$

Find the transfer function for $\frac{S_3}{S_1}$

## Steps
1. Specify the input and output for the transfer function.
1. Write the system equations.
1. Convert to Laplace transforms.
1. Solve for the output LT in terms of the input LT and form the ratio.

### Specify the input and output for the transfer function

This is specified in the problem. The input is $S_1$ and the output is $S_3$.

### Write the system equations

The reactions are:

    S1 -> S2; k1*S1
    S2 -> S3; k2*S2

From inspections, the system equations are:

\begin{eqnarray}
\dot{S}_1 & = & - k_1 S_1 \\
\dot{S}_2 & = & k_1 S_1 - k_2 S_2
\end{eqnarray}

### Convert the system equations to Laplace transforms

This is a mechanical process based on the fact that $L[\dot{g}(t)] = s G(s)$,
where $L[g(t)] = G(s)$.

\begin{eqnarray}
s S_1(s) & = & - k_1 S_1(s) \\
sS_2 (s) & = & k_1 S_1(s) - k_2 S_2(s)
\end{eqnarray}

### Solve for LT of the output in terms of the input and form the ratio

## Simulate to check

## Analysis

### Poles

### Response to initial conditions

### DC Gain (Step Response)

# Branching

$S_1 \xrightarrow[\text{}]{k_1 S_1} S_2$,
$S_2 \xrightarrow[\text{}]{k_2 S_2} S_3$,
$S_2 \xrightarrow[\text{}]{k_3 S_2} S_4$

Find the transfer function for $\frac{S_3}{S_1}$

# Reversible Reaction

$S_1 \xrightarrow[\text{}]{k_1 S_1} S_2$,
$S_2 \xrightarrow[\text{}]{k_2 S_2} S_1$

Find the transfer function for $\frac{S_2}{S_1}$

### Write the system equations

\begin{eqnarray}
\dot{S_1} & = & -k_1 S_1 + k_2 S_2 \\
\dot{S_2} & = & k_1 S_1 - k_2 S_2 \\
\end{eqnarray}

### Find the transfer function

```python
expr = -k1*S1 + k2*S2 - s*S1
tf = sympy.solve(expr, S2)[0] / S1
tf
```

$\displaystyle \frac{k_{1} + s}{k_{2}}$

What is $\frac{k_1}{k_2}$ in terms of first year chemistry?

# Nonlinear

$S_1 \xrightarrow[\text{}]{k_1 S_1 } S_2$,
$S_2 \xrightarrow[\text{}]{k_2 S_1 S_2} S_3$

Find the transfer function for $\frac{S_3}{S_1}$

# Sequential Pathway With a Reversible Reaction

$S_1 \xrightarrow[\text{}]{k_1 S_1} S_2$,
$S_2 \xrightarrow[\text{}]{k_2 S_2} S_3$,
$S_3 \xrightarrow[\text{}]{k_3 S_3} S_2$,
$S_3 \xrightarrow[\text{}]{k_4 S_3} \emptyset$

Find the transfer function for $\frac{S_3}{S_1}$

### Write the state equations

\begin{eqnarray}
\dot{S_1} & = & -k_1 S_1 \\
\dot{S_2} & = & k_1 S_1 - k_2 S_2 + k_3 S_3 \\
\dot{S_3} & = & k_2 S_2 - k_3 S_3 \\
\end{eqnarray}

### Convert the system equations to Laplace Transforms

\begin{eqnarray}
s S_1 (s) & = & -k_1 S_1 (s) \\
s S_2 (s)  & = & k_1 S_1(s) - k_2 S_2(s) + k_3 S_3(s) \\
s S_3(s)  & = & k_2 S_2 (s) - k_3 S_3(s) -k_4 S_3(s) \\
\end{eqnarray}

### Express the output in terms of the input

``sympy`` has its limits. It works best if there is one expression with the two variables of interest.

```python
S2_expr = k1*S1 - k2*S2 + k3*S3 - s*S2
S2_subs = sympy.solve(S2_expr, S2)[0]
S2_subs
```

$\displaystyle \frac{S_{1} k_{1} + S_{3} k_{3}}{k_{2} + s}$

```python
S3_expr = k2*S2 - k3*S3 - k4*S3 - s*S3
new_expr = S3_expr.subs({S2: S2_subs})
result = sympy.solve(new_expr, S3)[0]
result
```

$\displaystyle \frac{S_{1} k_{1} k_{2}}{k_{2} k_{4} + k_{2} s + k_{3} s + k_{4} s + s^{2}}$

```python
tf = result/S1
tf
```

$\displaystyle \frac{k_{1} k_{2}}{k_{2} k_{4} + k_{2} s + k_{3} s + k_{4} s + s^{2}}$

```python
numerator, denominator = sympy.fraction(tf)
numerator, denominator
```

    (k1*k2, k2*k4 + k2*s + k3*s + k4*s + s**2)

```python
# We can find the poles symbolically by solving for s
sim_dct = {k1: 1, k2: 2, k3: 2.5, k4: 1}
sympy.solve(denominator.subs(sim_dct), s)
```

    [-5.10849528301415, -0.391504716985849]

```python
dct = dict(sim_dct)
dct.update({s: 0})
tf.subs(dct)
```

$\displaystyle 1$

What constants can we change to get a DC gain of 4?

## Simulation

```python
MODEL = """
J1: S1 -> S2; k1*S1
J2: S2 -> S3; k2*S2
J3: S3 -> S2; k3*S3
J4: S3 -> ; k4*S3
k1 = 1
k2 = 2
k3 = 2.5
k4 = 1
S1 = 1
S2 = 0
S3 = 0
"""
RR = te.loada(MODEL)
RR.simulate(0, 20)
RR.plot(title="Impulse Response")
```

```python
STEP_MODEL = """
J1: $S1 -> S2; k1*$S1
J2: S2 -> S3; k2*S2
J3: S3 -> S2; k3*S3
J4: S3 -> ; k4*S3
k1 = 1
k2 = 2
k3 = 2.5
k4 = 1
$S1 = 1
S2 = 0
S3 = 0
"""
rr = te.loada(STEP_MODEL)
rr.simulate(0, 20)
rr.plot(title="Step Response")
```

```python
ctlsb = ctl.ControlSBML(MODEL, input_names=["S1"], output_names=["S3"])
tf = ctlsb.makeTransferFunction()
```

```python
tf.dcgain()
```

    0.9999999999999996

```python
tf.pole()
```

    array([-5.10849528, -1.        , -0.39150472])

We got one more pole than solving symbolically beause the symbolic solution simplified the system by eliminating $S_2$.

```python
state_space = ctlsb.makeStateSpace()
times = [0.01*v for v in range(5001)]
result = control.forced_response(tf, times, np.repeat(10, len(times)))
```

```python
tf
```

$$\frac{2}{s^3 + 6.5 s^2 + 7.5 s + 2}$$

```python
plt.plot(result.t, result.outputs)
```

```python
result.y
```

    array([[0.00000000e+00, 3.27974089e-06, 2.58182195e-05, ...,
            9.99999994e+00, 9.99999994e+00, 9.99999994e+00]])

```python
type(tf)
```

    control.xferfcn.TransferFunction


## Fundamentals of mathematics

Let's begin by learning about the basic `SymPy` objects and the
operations we can carry out on them. We'll learn the `SymPy` equivalents
of many math verbs like “to solve” (an equation), “to expand” (an
expression), “to factor” (a polynomial).

```python
from sympy import *
x, y, z, t = symbols('x y z t')
```

### Numbers

In `Python`, there are two types of number objects: `int`s and `float`s.

```python
3         # an int
```

    3

```python
3.0       # a float
```

    3.0

Integer objects in `Python` are a faithful representation of the set of
integers $\mathbb{Z}=\{\ldots,-2,-1,0,1,2,\ldots\}$. Floating point numbers are
approximate representations of the reals $\mathbb{R}$. Regardless of its absolute
size, a floating point number is only accurate to 16 decimals.

Special care is required when specifying rational numbers, because
integer division might not produce the answer you want. In other
words, Python will not automatically convert the answer to a floating
point number, but instead round the answer to the closest integer:

```python
1/7       # int/int gives int
```

    0.14285714285714285

To avoid this problem, you can force `float` division by using the
number `1.0` instead of `1`:

```python
1.0/7     # float/int gives float
```

    0.14285714285714285

This result is better, but it's still only an approximation of the exact
number $\frac{1}{7} \in \mathbb{Q}$, since a `float` has 16 decimals while the decimal
expansion of $\frac{1}{7}$ is infinitely long. To obtain an *exact* representation
of $\frac{1}{7}$ you need to create a `SymPy` expression. You can sympify any
expression using the shortcut function `S()`:

```python
S('1/7')  # = Rational(1,7)
```

$\displaystyle \frac{1}{7}$

Note the input to `S()` is specified as a text string delimited by quotes.
We could have achieved the same result using `S('1')/7` since a `SymPy`
object divided by an `int` is a `SymPy` object.

Except for the tricky `Python` division operator, other math operators
like addition `+`, subtraction `-`, and multiplication `*` work as you would
expect. The syntax `**` is used in `Python` to denote exponentiation:

```python
2**10     # same as S('2^10')
```

    1024

When solving math problems, it's best to work with `SymPy` objects,
and wait to compute the numeric answer in the end. To obtain a
numeric approximation of a `SymPy` object as a `float`, call its `.evalf()`
method:

```python
pi
```

$\displaystyle \pi$

```python
pi.evalf()
```

$\displaystyle 3.14159265358979$

The method `.n()` is equivalent to `.evalf()`. The global `SymPy`
function `N()` can also be used to to compute numerical values. You can
easily change the number of digits of precision of the approximation.
Enter `pi.n(400)` to obtain an approximation of $\pi$ to 400 decimals.

### Symbols

Python is a civilized language so there's no need to define variables
before assigning values to them. When you write `a = 3`, you define a
new name `a` and set it to the value `3`. You can now use the name `a`
in subsequent calculations.

Most interesting `SymPy` calculations require us to define `symbols`,
which are the `SymPy` objects for representing variables and unknowns.
For your convenience, when [live.sympy.org](http://live.sympy.org) starts, it runs the
following commands automatically:

```python
from __future__ import division
from sympy import *
x, y, z, t = symbols('x y z t')
k, m, n = symbols('k m n', integer=True)
f, g, h = symbols('f g h', cls=Function)
```

The first statement instructs python to convert `1/7` to `1.0/7` when
dividing, potentially saving you from any int division confusion. The
second statement imports all the `SymPy` functions. The remaining
statements define some generic symbols `x`, `y`, `z`, and `t`, and several
other symbols with special properties.

Note the difference between the following two statements:

```python
x + 2            # an Add expression
```

$\displaystyle x + 2$

```python
p + 2
```

The name `x` is defined as a symbol, so `SymPy` knows that `x + 2` is an
expression; but the variable `p` is not defined, so `SymPy` doesn't know
what to make of `p + 2`. To use `p` in expressions, you must first define
it as a symbol:

```python
p = Symbol('p')  # the same as p = symbols('p')
p + 2            # = Add(Symbol('p'), Integer(2))
```

$\displaystyle p + 2$

You can define a sequence of variables using the following notation:

```python
a0, a1, a2, a3 = symbols('a0:4')
```

You can use any name you want for a variable, but it's best if you
avoid the letters `Q,C,O,S,I,N` and `E` because they have special uses
in `SymPy`: `I` is the unit imaginary number $i \equiv \sqrt(-1)$, `E` is the base of
the natural logarithm, `S()` is the sympify function, `N()` is used to
obtain numeric approximations, and `O` is used for big-O notation.

The underscore symbol `_` is a special variable that contains the result
of the last printed value. The variable `_` is analogous to the `ans` button
on certain calculators, and is useful in multi-step calculations:

```python
3+3
```

    6

```python
_*2
```

    12

### Expresions

You define `SymPy` expressions by combining symbols with basic math
operations and other functions:

```python
expr = 2*x + 3*x - sin(x) - 3*x + 42
simplify(expr)
```

$\displaystyle 2 x - \sin{\left(x \right)} + 42$

The function `simplify` can be used on any expression to simplify
it. The examples below illustrate other useful `SymPy` functions that
correspond to common mathematical operations on expressions:

```python
factor( x**2-2*x-8 )
```

$\displaystyle \left(x - 4\right) \left(x + 2\right)$

```python
expand( (x-4)*(x+2) )
```

$\displaystyle x^{2} - 2 x - 8$

```python
a, b = symbols('a b')
collect(x**2 + x*b + a*x + a*b, x)  # collect terms for diff. pows of x
```

$\displaystyle a b + x^{2} + x \left(a + b\right)$

To substitute a given value into an expression, call the `.subs()`
method, passing in a python dictionary object `{ key:val, ... }`
with the symbol–value substitutions you want to make:

```python
expr = sin(x) + cos(y)
expr
```

$\displaystyle \sin{\left(x \right)} + \cos{\left(y \right)}$

```python
expr.subs({x:1, y:2})
```

$\displaystyle \cos{\left(2 \right)} + \sin{\left(1 \right)}$

```python
expr.subs({x:1, y:2}).n()
```

$\displaystyle 0.425324148260754$

Note how we used `.n()` to obtain the expression's numeric value.

### Solving equations

The function `solve` is the main workhorse in `SymPy`. This incredibly
powerful function knows how to solve all kinds of equations. In fact
`solve` can solve pretty much any equation! When high school students
learn about this function, they get really angry—why did they spend
five years of their life learning to solve various equations by hand,
when all along there was this `solve` thing that could do all the math
for them? Don't worry, learning math is never a waste of time.

The function `solve` takes two arguments. Use `solve(expr,var)` to
solve the equation `expr==0` for the variable `var`. You can rewrite any
equation in the form `expr==0` by moving all the terms to one side
of the equation; the solutions to $A(x) = B(x)$ are the same as the
solutions to $A(x) - B(x) = 0$.

For example, to solve the quadratic equation $x^2 + 2x - 8 = 0$, use

```python
solve( x**2 + 2*x - 8, x)
```

    [-4, 2]

In this case the equation has two solutions so `solve` returns a list.
Check that $x = 2$ and $x = -4$ satisfy the equation $x^2 + 2x - 8 = 0$.

The best part about `solve` and `SymPy` is that you can obtain symbolic
answers when solving equations. Instead of solving one specific
quadratic equation, we can solve all possible equations of the form
$ax^2 + bx + c = 0$ using the following steps:

```python
a, b, c = symbols('a b c')
solve( a*x**2 + b*x + c, x)
```

    [(-b + sqrt(-4*a*c + b**2))/(2*a), -(b + sqrt(-4*a*c + b**2))/(2*a)]

In this case `solve` calculated the solution in terms of the symbols
`a`, `b`, and `c`. You should be able to recognize the expressions in the
solution—it's the quadratic formula $x_{1,2} = \frac{-b \pm \sqrt{b^2 - 4ac}}{2a}$.

To solve a specific equation like $x^2 + 2x - 8 = 0$, we can substitute
the coefficients $a = 1$, $b = 2$, and $c = -8$ into the general solution to
obtain the same result:

```python
gen_sol = solve( a*x**2 + b*x + c, x)
[ gen_sol[0].subs({'a':1,'b':2,'c':-8}),
  gen_sol[1].subs({'a':1,'b':2,'c':-8}) ]
```

    [2, -4]

To solve a *system of equations*, you can feed `solve` with the list of
equations as the first argument, and specify the list of unknowns you
want to solve for as the second argument. For example, to solve for $x$
and $y$ in the system of equations $x + y = 3$ and $3x - 2y = 0$, use

```python
solve([x + y - 3, 3*x - 2*y], [x, y])
```

    {x: 6/5, y: 9/5}

The function `solve` is like a Swiss Army knife you can use to solve
all kind of problems. Suppose you want to *complete the square* in the
expression $x^2 - 4x + 7$, that is, you want to find constants $h$ and $k$
such that $x^2 -4x + 7 = (x-h)^2 + k$. There is no special “complete the
square” function in `SymPy`, but you can call solve on the equation
$(x - h)^2 + k - (x^2 - 4x + 7) = 0$ to find the unknowns $h$ and $k$:

```python
h, k = symbols('h k')
solve( (x-h)**2 + k - (x**2-4*x+7), [h,k] )
```

    [(2, 3)]

```python
((x-2)**2+3).expand()  # so h = 2 and k = 3, verify...
```

$\displaystyle x^{2} - 4 x + 7$

Learn the basic `SymPy` commands and you'll never need to suffer
another tedious arithmetic calculation painstakingly performed by
hand again!

### Rational functions

By default, `SymPy` will not combine or split rational expressions.
You need to use `together` to symbolically calculate the addition of
fractions:

```python
a, b, c, d = symbols('a b c d')
a/b + c/d
```

$\displaystyle \frac{a}{b} + \frac{c}{d}$

```python
together(a/b + c/d)
```

$\displaystyle \frac{a d + b c}{b d}$

Alternately, if you have a rational expression and want to divide the
numerator by the denominator, use the `apart` function:

```python
apart( (x**2+x+4)/(x+2) )
```

$\displaystyle x - 1 + \frac{6}{x + 2}$

### Exponentials and logarithms

Euler's constant $e = 2.71828\dots$ is defined one of several ways,

$$
e \equiv \lim_{n\to\infty}\left(1+\frac{1}{n}\right)^n
  \equiv \lim_{\epsilon\to 0}(1+\epsilon)^{1/\epsilon}
  \equiv \sum_{n=0}^{\infty}\frac{1}{n!},
$$

and is denoted `E` in `SymPy`. Using `exp(x)` is equivalent to `E**x`.

The functions `log` and `ln` both compute the logarithm base $e$:

```python
log(E**3)  # same as ln(E**3)
```

$\displaystyle 3$

By default, `SymPy` assumes the inputs to functions like `exp` and `log` are
complex numbers, so it will not expand certain logarithmic expressions.
However, indicating to `SymPy` that the inputs are positive real numbers
will make the expansions work:

```python
x, y = symbols('x y')
log(x*y).expand()
```

$\displaystyle \log{\left(x y \right)}$

```python
a, b = symbols('a b', positive=True)
log(a*b).expand()
```

$\displaystyle \log{\left(a \right)} + \log{\left(b \right)}$

### Polynomials

Let's define a polynomial $P$ with roots at $x = 1$, $x = 2$, and $x = 3$:

```python
P = (x-1)*(x-2)*(x-3)
P
```

$\displaystyle \left(x - 3\right) \left(x - 2\right) \left(x - 1\right)$

To see the expanded version of the polynomial, call its `expand` method:

```python
P.expand()
```

$\displaystyle x^{3} - 6 x^{2} + 11 x - 6$

When the polynomial is expressed in it's expanded form $P(x) =
x^3 - 6x^2 + 11x - 6$, we can't immediately identify its roots. This is
why the factored form $P(x) = (x - 1)(x - 2)(x - 3)$ is preferable. To
factor a polynomial, call its `factor` method or simplify it:

```python
P.factor()
```

$\displaystyle \left(x - 3\right) \left(x - 2\right) \left(x - 1\right)$

```python
P.simplify()
```

$\displaystyle \left(x - 3\right) \left(x - 2\right) \left(x - 1\right)$

Recall that the roots of the polynomial $P(x)$ are defined as the
solutions to the equation $P(x) = 0$. We can use the `solve` function
to find the roots of the polynomial:

```python
roots = solve(P,x)
roots
```

    [1, 2, 3]

```python
# let's check if P equals (x-1)(x-2)(x-3)
simplify( P - (x-roots[0])*(x-roots[1])*(x-roots[2]) )
```

$\displaystyle 0$

### Equality checking

In the last example, we used the `simplify` function to check whether
two expressions were equal. This way of checking equality works
because $P = Q$ if and only if $P - Q = 0$. This is the best way to
check if two expressions are equal in `SymPy` because it attempts all
possible simplifications when comparing the expressions. Below is
a list of other ways to check whether two quantities are equal with
example cases where they fail:

```python
p = (x-5)*(x+5)
q = x**2 - 25
```

```python
p == q                      # fail
```

    False

```python
p - q == 0                  # fail
```

    False

```python
simplify(p - q) == 0
```

    True

```python
sin(x)**2 + cos(x)**2 == 1  # fail
```

    False

```python
simplify( sin(x)**2 + cos(x)**2 - 1) == 0
```

    True

### Trigonometry

The trigonometric functions `sin` and `cos` take inputs in radians:

```python
sin(pi/6)
```

$\displaystyle \frac{1}{2}$

```python
cos(pi/6)
```

$\displaystyle \frac{\sqrt{3}}{2}$

For angles in degrees, you need a conversion factor of $\frac{\pi}{180}$[rad/$^\circ$]:

```python
sin(30*pi/180)  # 30 deg = pi/6 rads
```

$\displaystyle \frac{1}{2}$

The inverse trigonometric functions $\sin^{-1}(x) \equiv \arcsin(x)$ and
$\cos^{-1}(x) \equiv \arccos(x)$ are used as follows:

```python
asin(1/2)
```

$\displaystyle 0.523598775598299$

```python
acos(sqrt(3)/2)
```

$\displaystyle \frac{\pi}{6}$

Recall that $\tan(x) \equiv \frac{\sin(x)}{\cos(x)}$. The inverse function of $\tan(x)$ is $\tan^{-1}(x) \equiv \arctan(x) \equiv$ `atan(x)`

```python
tan(pi/6)
```

$\displaystyle \frac{\sqrt{3}}{3}$

```python
atan( 1/sqrt(3) )
```

$\displaystyle \frac{\pi}{6}$

The function `acos` returns angles in the range $[0, \pi]$, while `asin` and
`atan` return angles in the range $[-\frac{\pi}{2},\frac{\pi}{2}]$.

Here are some trigonometric identities that `SymPy` knows:

```python
sin(x) == cos(x - pi/2)
```

    True

```python
simplify( sin(x)*cos(y)+cos(x)*sin(y) )
```

$\displaystyle \sin{\left(x + y \right)}$

```python
e = 2*sin(x)**2 + 2*cos(x)**2
trigsimp(e)
```

$\displaystyle 2$

```python
trigsimp(log(e))
```

$\displaystyle \log{\left(2 \right)}$

```python
trigsimp(log(e), deep=True)
```

$\displaystyle \log{\left(2 \right)}$

```python
simplify(sin(x)**4 - 2*cos(x)**2*sin(x)**2 + cos(x)**4)
```

$\displaystyle \frac{\cos{\left(4 x \right)}}{2} + \frac{1}{2}$

The function `trigsimp` does essentially the same job as `simplify`.

If instead of simplifying you want to expand a trig expression, you
should use `expand_trig`, because the default `expand` won't touch trig
functions:

```python
expand(sin(2*x))       # = (sin(2*x)).expand()
```

$\displaystyle \sin{\left(2 x \right)}$

```python
expand_trig(sin(2*x))  # = (sin(2*x)).expand(trig=True)
```

$\displaystyle 2 \sin{\left(x \right)} \cos{\left(x \right)}$

### Hyperbolic trigonometric functions

The hyperbolic sine and cosine in `SymPy` are denoted `sinh` and `cosh`
respectively and `SymPy` is smart enough to recognize them when
simplifying expressions:

```python
simplify( (exp(x)+exp(-x))/2 )
```

$\displaystyle \cosh{\left(x \right)}$

```python
simplify( (exp(x)-exp(-x))/2 )
```

$\displaystyle \sinh{\left(x \right)}$

Recall that $x = \cosh(\mu)$ and $y = \sinh(\mu)$ are defined as $x$ and $y$
coordinates of a point on the the hyperbola with equation $x^2 - y^2 = 1$
and therefore satisfy the identity $\cosh^2 x - \sinh^2 x = 1$:

```python
simplify( cosh(x)**2 - sinh(x)**2 )
```

$\displaystyle 1$

```python

```

```python

```


# CHEM 1000 - Spring 2022
Prof. Geoffrey Hutchison, University of Pittsburgh

## 1. Functions and Coordinate Sets

Chapter 1 in [*Mathematical Methods for Chemists*](http://sites.bu.edu/straub/mathematical-methods-for-molecular-science/)

By the end of this session, you should be able to:
- Handle 2D polar and 3D spherical coordinates
- Understand area elements in 2D polar coordinates
- Understand volume eleements in 3D spherical coordinates

### X/Y Cartesian 2D Coordinates

We've already been using the x/y 2D Cartesian coordinate set to plot functions.

Beyond `sympy`, we're going to use two new modules:
- `numpy` which lets us create and handle arrays of numbers
- `matplotlib` which lets us plot things

It's a little bit more complicated. For now, you can just consider these as **demos**. We'll go into code (and make our own plots) in the next recitation period.

```python
# import numpy
# the "as np" part is giving a shortcut so we can write "np.function()" instead of "numpy.function()"
# (saving typing is nice)
import numpy as np
# similarly, we import matplotlib's 'pyplot' module
# and "as plt" means we can use "plt.show" instead of "matplotlib.pyplot.show()"
import matplotlib.pyplot as plt

# insert any graphs into our notebooks directly
%matplotlib inline
%config InlineBackend.figure_format = 'retina'
```

```python
# once we've done that import (once) - we just need to create our x/y values
x = np.arange(0, 4*np.pi, 0.1)   # start, stop, resolution
y = np.sin(x) # creates an array with sin() of all the x values

plt.plot(x,y)
plt.show()
```

Sometimes, we need to get areas in the Cartesian xy system, but this is very easy - we simply multiply an increment in x ($dx$) and an increment in y ($dy$).

(Image from [*Mathematical Methods for Chemists*](http://sites.bu.edu/straub/mathematical-methods-for-molecular-science/))

### Polar (2D) Coordinates

Of course, not all functions work well in xy Cartesian coordinates. A function should produce one y value for any x value. Thus, a circle isn't easily represented as $y = f(x)$.

Instead, polar coordinates, use radius $r$ and angle $\theta$. (Image from [*Mathematical Methods for Chemists*](http://sites.bu.edu/straub/mathematical-methods-for-molecular-science/))

As a reminder, we can interconvert x,y into r, theta:

$$
r = \sqrt{x^2 + y^2}
$$

$$
\theta = \arctan \frac{y}{x} = \tan^{-1} \frac{y}{x}
$$

```python
x = 3.0
y = 1.0

r = np.sqrt(x**2 + y**2)
theta = np.arctan(y / x)

print('r =', round(r, 4), 'theta = ', round(theta, 4))
```

Okay, we can't express a circle as an easy $y = f(x)$ expression. Can we do that in polar coordinates? Sure. The radius will be constant, and theta will go from $0 .. 2\pi$.

```python
theta = np.arange(0, 2*np.pi, 0.01) # set up an array of radii from 0 to 2π with 0.01 rad

# create a function r(theta) = 1.5 .. a constant
r = np.full(theta.size, 1.5)

# create a new polar plot
ax = plt.subplot(111, projection='polar')
ax.plot(theta, r, color='blue')

ax.set_rmax(3)
ax.set_rticks([1, 2])  # Less radial ticks
ax.set_rlabel_position(22.5)  # Move radial labels away from plotted line
ax.grid(True)

plt.show()
```

Anything else? Sure - we can create spirals, etc. that are parametric functions in the XY Cartesian coordinates.

```python
r = np.arange(0, 2, 0.01) # set up an array of radii from 0 to 2 with 0.01 resolution

# this is a function theta(r) = 2π * r
theta = 2 * np.pi * r # set up an array of theta angles - spiraling outward .. from 0 to 2*2pi = 4pi

# create a polar plot
ax = plt.subplot(111, projection='polar')
ax.plot(theta, r, color='red')

ax.set_rmax(3)
ax.set_rticks([1, 2])  # Less radial ticks
ax.set_rlabel_position(22.5)  # Move radial labels away from plotted line
ax.grid(True)

plt.show()
```

Just like with xy Cartesian, we will eventually need to consider the area of functions in polar coordinates. (Image from [*Mathematical Methods for Chemists*](http://sites.bu.edu/straub/mathematical-methods-for-molecular-science/))

Note that the area depends on the radius. Even if we sweep out the same $\Delta r$ and $\Delta \theta$ an area further out from the center is larger:

```python
# create a polar plot
ax = plt.subplot(111, projection='polar')

# first arc at r = 1.0
r1 = np.full(20, 1.0)
theta1 = np.linspace(1.0, 1.3, 20)
ax.plot(theta1, r1)

# second arc at r = 1.2
r2 = np.full(20, 1.2)
theta2 = np.linspace(1.0, 1.3, 20)
ax.plot(theta2, r2)

# first radial line at theta = 1.0 radians
r3 = np.linspace(1.0, 1.2, 20)
theta3 = np.full(20, 1.0)
ax.plot(theta3, r3)

# first radial line at theta = 1.3 radians
r4 = np.linspace(1.0, 1.2, 20)
theta4 = np.full(20, 1.3)
ax.plot(theta4, r4)

# smaller box
# goes from r = 0.4-> 0.6
# sweeps out theta = 1.0-1.3 radians
r5 = np.full(20, 0.4)
r6 = np.full(20, 0.6)
r7 = np.linspace(0.4, 0.6, 20)
r8 = np.linspace(0.4, 0.6, 20)
ax.plot(theta1, r5)
ax.plot(theta2, r6)
ax.plot(theta3, r7)
ax.plot(theta4, r8)

ax.set_rmax(1.5)
ax.set_rticks([0.5, 1, 1.5])  # Less radial ticks
ax.set_rlabel_position(-22.5)  # Move radial labels away from plotted line
ax.grid(True)

plt.show()
```

Thus the area element will be $r dr d\theta$. While it's not precisely rectangular, the increments are very small and it's a reasonable approximation.

### 3D Cartesian Coordinates

Of course there are many times when we need to express functions like:

$$ z = f(x,y) $$

These are a standard extension of 2D Cartesian coordinates, and so the volume is simply defined as that of a rectangular solid.

```python
from sympy import symbols
from sympy.plotting import plot3d
x, y = symbols('x y')

plot3d(-0.5 * (x**2 + y**2), (x, -3, 3), (y, -3, 3))
```

### 3D Spherical Coordinates

Much like two dimensions we sometimes need to use spherical coordinates — atoms are spherical, after all.

**WARNING** Some math courses use a different [convention](https://en.wikipedia.org/wiki/Spherical_coordinate_system#Conventions) than chemistry and physics.

- Physics and chemistry use $(r, \theta, \varphi)$ where $\theta$ is the angle down from the z-axis (e.g., latitude)
- Some math courses use $\theta$ as the angle in the XY 2D plane.

(Image from [*Mathematical Methods for Chemists*](http://sites.bu.edu/straub/mathematical-methods-for-molecular-science/))

Where:
- $r$ is the radius, from 0 to $\infty$
- $\theta$ is the angle down from the z-axis
  - e.g., think of N/S latitude on the Earth's surface) from 0° at the N pole to 90° (π/2) at the equator and 180° (π) at the S pole
- $\varphi$ is the angle in the $xy$ plane
  - e.g., think of E/W longitude on the Earth), from 0 to 360° / 0..2π

We can interconvert xyz and $r\theta\varphi$

$$x = r\sin \theta \cos \varphi$$
$$y = r\sin \theta \sin \varphi$$
$$z = r \cos \theta$$

Or vice-versa:

$$
\begin{array}{l}r=\sqrt{x^{2}+y^{2}+z^{2}} \\ \theta=\arccos \left(\frac{z}{r}\right)=\cos ^{-1}\left(\frac{z}{r}\right) \\ \varphi=\tan ^{-1}\left(\frac{y}{x}\right)\end{array}
$$

The code below might look a little complicated. That's okay. I've added comments for the different sections and each line.

You don't need to understand all of it - it's intended to plot the function:

$$ r = |\cos(\theta^2) | $$

```python
# import some matplotlib modules for 3D and color scales
import mpl_toolkits.mplot3d.axes3d as axes3d
import matplotlib.colors as mcolors
cmap = plt.get_cmap('jet') # pick a red-to-blue color map
fig = plt.figure() # create a figure
ax = fig.add_subplot(1,1,1, projection='3d') # set up some axes for a 3D projection

# We now set up the grid for evaluating our function
#  particularly the angle portion of the spherical coordinates
theta = np.linspace(0, np.pi, 100)
phi = np.linspace(0, 2*np.pi, 100)
THETA, PHI = np.meshgrid(theta, phi)

# here's the function to plot
R = np.abs(np.cos(THETA**2))

# now convert R(phi, theta) to x, y, z coordinates to plot
X = R * np.sin(THETA) * np.cos(PHI)
Y = R * np.sin(THETA) * np.sin(PHI)
Z = R * np.cos(THETA)

# set up some colors based on the Z range .. from red to blue
norm = mcolors.Normalize(vmin=Z.min(), vmax=Z.max())
# plot the surface
plot = ax.plot_surface(X, Y, Z, rstride=1, cstride=1, facecolors=cmap(norm(Z)),
    linewidth=0, antialiased=True, alpha=0.4) # no lines, smooth graphics, semi-transparent

plt.show()
```

The volume element in spherical coordinates is a bit tricky, since the distances depend on the radius and angles:
(Image from [*Mathematical Methods for Chemists*](http://sites.bu.edu/straub/mathematical-methods-for-molecular-science/))

$$ dV = r^2 dr \sin \theta d\theta d\phi$$

-------
This notebook is from Prof. Geoffrey Hutchison, University of Pittsburgh
https://github.com/ghutchis/chem1000


# 10 Ordinary Differential Equations (ODEs)

[ODE](http://mathworld.wolfram.com/OrdinaryDifferentialEquation.html)s describe many phenomena in physics. They describe the changes of a **dependent variable** $y(t)$ as a function of a **single independent variable** (e.g. $t$ or $x$).

An ODE of **order** $n$

$$
F(t, y^{(0)}, y^{(1)}, ..., y^{(n)}) = 0
$$

contains derivatives $y^{(k)}(t) \equiv y^{(k)} \equiv \frac{d^{k}y(t)}{dt^{k}}$ up to the $n$-th derivative (and $y^{(0)} \equiv y$).

### Initial and boundary conditions

* $n$ **initial conditions** are needed to *uniquely determine* the solution of a $n$-th order ODE, e.g, initial position and velocities.

* **Boundary conditions** (values of solution on domain boundries) can additionaly restrict solutions but the resulting *eigenvalue problems* are more difficult, e.g, wavefunction goes towards 0 for $\pm\infty$.

### Linear ODEs
A **linear** ODE contains no higher powers than 1 of any of the $y^{(k)}$.

*Superposition principle*: Linear combinations of solutions are also solutions.

#### Example: First order linear ODE

\begin{align}
\frac{dy}{dt} &= f(t)y + g(t)\\
y^{(1)} &= f(t)y + g(t)\\
% y^{(1)} - f(t)y - g(t) &= 0
\end{align}

##### Radioactive decay
$$
\frac{dN}{dt} = -k N
$$

### Non-linear ODEs
**Non-linear** ODEs can contain any powers in the dependent variable and its derivatives.

No superposition of solutions. Often impossible to solve analytically.

#### Example: Second order (general) ODE
\begin{gather}
\frac{d^2 y}{dt^2} + \lambda(t) \frac{dy}{dt} = f\left(t, y, \frac{dy}{dt}\right)\\
\end{gather}

##### Newton's equations of motion
$$
m\frac{d^2 x}{dt^2} = F(x) + F_\text{ext}(x, t) \quad \text{with}
\quad F(x) = -\frac{dU}{dx}
$$
(Force is often derived from a potential energy $U(x)$ and may contain non-linear terms such as $x^{-2}$ or $x^3$.)

## Partial differential equations (PDEs)
* more than one independent variable (e.g. $x$ and $t$)
* partial derivatives
* much more difficult than ODEs

#### Example: Schrödinger equation (Quantum Mechanics)
$$
i\hbar \frac{\partial\psi(\mathbf{x}, t)}{\partial t} = -\frac{\hbar^2}{2m}
   \left(\frac{\partial^2 \psi}{\partial x^2} +
         \frac{\partial^2 \psi}{\partial y^2} +
         \frac{\partial^2 \psi}{\partial z^2}
   \right) + V(\mathbf{x})\, \psi(\mathbf{x}, t)
$$

## Harmonic and anharmonic oscillator
* particle with mass $m$ connected to a spring
* spring described by a harmonic potential or anharmonic ones in the displacements from equilibrium $x$

\begin{align}
U_1(x) &= \frac{1}{2} k x^2, \quad k=1\\
U_2(x) &= \frac{1}{2} k x^2 \left(1 - \frac{2}{3}\alpha x\right), \quad k=1,\ \alpha=\frac{1}{2}\\
U_3(x) &= \frac{1}{p} k x^p, \quad k=1,\ p=6
\end{align}

1. What do these potentials look like? Sketch or plot.
2. Calculate the forces.

#### Potentials

```python
import numpy as np

def U1(x, k=1):
    return 0.5 * k * x*x

def U2(x, k=1, alpha=0.5):
    return 0.5 * k * x*x * (1 - (2/3)*alpha*x)

def U3(x, k=1, p=6):
    return (k/p) * np.power(x, p)
```

```python
import matplotlib
import matplotlib.pyplot as plt
matplotlib.style.use('seaborn-talk')
%matplotlib inline
```

```python
X = np.linspace(-3, 3, 100)
ax = plt.subplot(1,1,1)
ax.plot(X, U1(X), label=r"$U_1$")
ax.plot(X, U2(X), label=r"$U_2$")
ax.plot(X, U3(X), label=r"$U_3$")
ax.set_ylim(None, 10)
ax.legend(loc="best")
```

#### Forces

\begin{align}
F_1(x) &= -kx\\
F_2(x) &= -kx(1 + \alpha x)\\
F_3(x) &= -k x^{p-1}
\end{align}

## ODE Algorithms
Basic idea:

1. Start with initial conditions, $y_0 \equiv y(t=0)$
2. Use $\frac{dy}{dt} = f(t, y)$ (the RHS!) to advance solution a small step $h$ forward in time: $y(t=h) \equiv y_1$
3. Repeat with $y_1$ to obtain $y_2 \equiv y(t=2h)$... and for all future values of $t$.

Possible issues
* small differences: subtractive cancelation and round-off error accumulation
* extrapolation: numerical "solution" can deviate wildly from exact
* possibly need adaptive $h$

### Euler's rule
Simple: forward difference

\begin{align}
f(t, y) = \frac{dy(t)}{dt} &\approx \frac{y(t_{n+1}) - y(t_n)}{h}\\
y_{n+1} &\approx y_n + h f(t_n, y_n) \quad \text{with} \quad y_n := y(t_n)
\end{align}

Error will be $\mathcal{O}(h^2)$ (bad!).

Also: what if we have a second order ODE ?!?! We only used $dy/dt$.

### Convert 2nd order ODE to 2 coupled 1st order ODEs
The 2nd order ODE is
$$
\frac{d^2 y}{dt^2} = f(t, y)
$$

Introduce "dummy" dependent variables $y_i$ with $y_0 \equiv y$ and

\begin{alignat}{1}
\frac{dy}{dt} &= \frac{dy_0}{dt} &=   y_1\\
\frac{d^2y}{dt^2} &= \frac{dy_1}{dt} &= {} f(t, y_0).
\end{alignat}

The first equation defines the velocity $y_1 = v$ and the second one is the original ODE.

### $n$-th oder ODE to $n$ coupled 1st order ODEs

The $n$-th order ODE is
$$
\frac{d^n y}{dt^n} = f(t, y, \frac{d y}{dt}, \frac{d^2 y}{dt^2}, \dots, \frac{d^{n-1} y}{dt^{n-1}})
$$

Introduce "dummy" dependent variables $y^{(i)}$ with $y^{(0)} \equiv y$ and

\begin{align}
\frac{dy^{(0)}}{dt} &=  y^{(1)}\\
\frac{dy^{(1)}}{dt} &=  y^{(2)}\\
\dots & \\
\frac{dy^{(n-1)}}{dt} &=  f(t, y^{(0)}, y^{(1)}, y^{(2)}, \dots, y^{(n-1)}).
\end{align}

### General standard (dynamic) form of ODEs
1 ODE of *any order* $n$ $\rightarrow$ $n$ coupled simultaneous first-order ODEs in $n$ unknowns  $y^{(0)}, \dots, y^{(n-1)}$:

\begin{align}
\frac{dy^{(0)}}{dt} &= f^{(0)}(t, y^{(0)}, \dots, y^{(n-1)})\\
\frac{dy^{(1)}}{dt} &= f^{(1)}(t, y^{(0)}, \dots, y^{(n-1)})\\
\vdots &  \\
\frac{dy^{(n-1)}}{dt} &= f^{(n-1)}(t, y^{(0)}, \dots, y^{(n-1)})\\
\end{align}

In $n$-dimensional vector notation:

\begin{align}
\frac{d\mathbf{y}(t)}{dt} &= \mathbf{f}(t, \mathbf{y})\\
\mathbf{y} &= \left(\begin{array}{c}
  y^{(0)}(t) \\
  y^{(1)}(t) \\
  \vdots \\
  y^{(n-1)}(t)
  \end{array}\right),
\quad
\mathbf{f} = \left(\begin{array}{c}
  f^{(0)}(t, \mathbf{y}) \\
  f^{(1)}(t, \mathbf{y}) \\
  \vdots \\
  f^{(n-1)}(t, \mathbf{y})
  \end{array}\right)
\end{align}

#### Example: Convert Newton's EOMs to standard form

$$
\frac{d^2 x}{dt^2} = m^{-1} F\Big(t, x, \frac{dx}{dt}\Big)
$$

RHS may *not contain any explicit derivatives* but components of $\mathbf{y}$ can represent derivatives.
* position $x$ as first dependent variable $y^{(0)}$ (as usual).
* velocity $dx/dt$ as second dependent variable $y^{(1)}$

\begin{align}
y^{(0)}(t) &:= x(t)\\
y^{(1)}(t) &:= \frac{dx}{dt} = \frac{dy^{(0)}}{dt}
\end{align}

One 2nd order ODE

$$
\frac{d^2 x}{dt^2} = m^{-1} F\Big(t, x, \frac{dx}{dt}\Big)
$$

to two simultaneous 1st order ODEs:

\begin{align}
\frac{dy^{(0)}}{dt} &= y^{(1)}(t)\\
\frac{dy^{(1)}}{dt} &= m^{-1} F\Big(t, y^{(0)}, y^{(1)}\Big)
\end{align}

\begin{align}
\frac{d\mathbf{y}(t)}{dt} &= \mathbf{f}(t, \mathbf{y})\\
\mathbf{y} &= \left(\begin{array}{c}
     y^{(0)} \\
     y^{(1)}
  \end{array}\right) =
  \left(\begin{array}{c}
     x(t) \\
     \frac{dx}{dt}
  \end{array}\right),\\
\mathbf{f} &= \left(\begin{array}{c}
    y^{(1)}(t) \\
    m^{-1} F\Big(t, y^{(0)}, y^{(1)}\Big)
  \end{array}\right) =
  \left(\begin{array}{c}
    \frac{dx}{dt} \\
    m^{-1} F\Big(t, x(t), \frac{dx}{dt}\Big)
  \end{array}\right)
\end{align}

#### Example:  1D harmonic oscillator in standard form
With $F_1 = -k x$:
$$
\frac{d^2 x}{dt^2} = -m^{-1}k x
$$
convert to

\begin{align}
\frac{dy^{(0)}}{dt} &= y^{(1)}(t) \\
\frac{dy^{(1)}}{dt} &= -m^{-1}k y^{(0)}
\end{align}

Force (or derivative) function $\mathbf{f}$ and initial conditions:

\begin{alignat}{3}
f^{(0)}(t, \mathbf{y}) &= y^{(1)},
    &\quad y^{(0)}(0) &= x_0,\\
f^{(1)}(t, \mathbf{y}) &=  -m^{-1} k y^{(0)},
    &\quad y^{(1)}(0) &= v_0.
\end{alignat}

### Euler's rule (standard form)
Given the $n$-dimensional vectors from the ODE standard form

$$
\frac{d\mathbf{y}}{dt} = \mathbf{f}(t, \mathbf{y})
$$
the **Euler rule** amounts to

\begin{align}
\mathbf{f}(t, \mathbf{y}) = \frac{d\mathbf{y}(t)}{dt} &\approx \frac{\mathbf{y}(t_{n+1}) - \mathbf{y}(t_n)}{\Delta t}\\
\mathbf{y}_{n+1} &\approx \mathbf{y}_n + \Delta t \mathbf{f}(t_n, \mathbf{y}_n) \quad \text{with} \quad \mathbf{y}_n := \mathbf{y}(t_n)
\end{align}

## Problem: Numerically integrate the 1D harmonic oscillator with Euler

\begin{alignat}{3}
f^{(0)}(t, \mathbf{y}) &= y^{(1)},
    &\quad y^{(0)}(0) &= x_0,\\
f^{(1)}(t, \mathbf{y}) &=   - \frac{k}{m} y^{(0)},
    &\quad y^{(1)}(0) &= v_0.
\end{alignat}

with $k=1$; $x_0 = 0$ and $v_0 = +1$.

### Explicit implementation:
* Note how in `f_harmonic` we are constructing the force vector of the standard ODE representation
* `y` is the vector of dependents in the standard representation
* We pre-allocate the array for `y` and then assign to individual elements with the
  ```python
  y[:] = ...
  ```
  notation, which has higher performance than creating the array anew every time.

```python
import numpy as np

def F1(x, k=1):
    """Harmonic force"""
    return -k*x

def f_harmonic(t, y, k=1, m=1):
    """Force vector in standard ODE form (n=2)"""
    return np.array([y[1], F1(y[0], k=k)/m])

t_max = 100
h = 0.01
Nsteps = t_max/h
t_range = h * np.arange(Nsteps)
x = np.empty_like(t_range)

y = np.zeros(2)

# initial conditions
x0, v0 = 0.0, 1.0
y[:] = x0, v0

for i, t in enumerate(t_range):
    # store position that corresponds to time t_i
    x[i] = y[0]
    # Euler integrator
    y[:] = y + h * f_harmonic(t, y)
```

Plot the position $x(t)$ (which is $y_0$) against time:

```python
plt.plot(t_range, x)
```

### Modular solution with functions
We can make the Euler integrator a function, which makes the code more readable and modular and we can make the whole integration a function, too. This will allow us to easily run the integration with different initial values or `h` steps.

```python
import numpy as np

def F1(x, k=1):
    """Harmonic force"""
    return -k*x

def f_harmonic(t, y, k=1, m=1):
    """Force vector in standard ODE form (n=2)"""
    return np.array([y[1], F1(y[0], k=k)/m])

def euler(y, f, t, h):
    """Euler integrator.

    Returns new y at t+h.
    """
    return y + h * f(t, y)

def integrate(x0=0, v0=1, t_max=100, h=0.001):
    """Integrate the harmonic oscillator with force F1.

    Note that the spring constant k and particle mass m are currently
    pre-defined.

    Arguments
    ---------
    x0 : float
       initial position
    v0 : float
       initial velocity
    t_max : float
       time to integrate out to
    h : float, default 0.001
       integration time step

    Returns
    -------
    Tuple ``(t, x)`` with times and positions.
    """

    Nsteps = t_max/h
    t_range = h * np.arange(Nsteps)
    x = np.empty_like(t_range)
    y = np.zeros(2)
    # initial conditions
    y[:] = x0, v0

    for i, t in enumerate(t_range):
        # store position that corresponds to time t_i
        x[i] = y[0]
        # Euler integrator
        y[:] = euler(y, f_harmonic, t, h)

    return t_range, x
```

Plot the position as a function of time, $x(t)$.

```python
t, x = integrate(h=0.01)
plt.plot(t, x)
```

Note the increase in amplitude. Explore if smaller $h$ fixes this obvious problem.

```python
t, x = integrate(h=0.001)
plt.plot(t, x)
```

Smaller $h$ improves the integration (but Euler is still a bad algorithm... just run out for longer, i.e., higher `t_max`.)