Before you turn this problem in, make sure everything runs as expected. First, **restart the kernel** (in the menubar, select Kernel$\rightarrow$Restart) and then **run all cells** (in the menubar, select Cell$\rightarrow$Run All).

Make sure you fill in any place that says `YOUR CODE HERE` or "YOUR ANSWER HERE", as well as your name and collaborators below:

```python
NAME = ""
ID = ""
```

---

# Lab 02: Nonlinear Regression and Overfitting

In Lab 01, we explored the construction of linear regression models. Recall the assumptions we make in linear regression:
- $\textbf{x} \in {\cal X} = \mathbb{R}^n$
- $y \in {\cal Y} = \mathbb{R}$
- The $\textbf{x}$ data are drawn i.i.d. from some (unknown) distribution over ${\cal X}$
- There is a linear relationship between $\textbf{x}$ and $y$ with additive constant-variance Gaussian noise, i.e., $y \sim {\cal N}(\theta^\top \textbf{x}, \sigma^2)$,
  where $\theta \in \mathbb{R}^{n+1}$ is unknown and $\textbf{x}$ is an $n+1$-dimensional vector augemented with a constant value of 1 as its first element.

Today, we consider what we might do when the fourth assumption, linearity, does not hold. We introduce a particular form of nonlinear regression,
*polynomial regression*, in which we account for nonlinear relationships between $\mathbf{x}$ and $y$ by performing nonlinear transformations of
the input variables in $\mathbf{x}$.

As an example, if we had a single input variable $x$, linear regression gives us the hypothesis
$$h_\theta(x) = \theta_0 + \theta_1 x .$$
We can add a new "variable" $x^2$, which is a nonlinear transformation of the input $x$:
$$h_\theta(x) = \theta_0 + \theta_1 x + \theta_2 x^2 .$$
The important thing to notice here is that although the hypothesis is *nonlinear* in $x$, allowing us to model a more complex function than
ordinary linear regression, the hypothesis is *linear* in $\theta$, allowing us to use the normal equations to find the optimal $\theta$ as before.

## Polynomial Regression

More generally, polynomial regession is a form of linear regression in which the relationship between the independent variables $\mathbf{x}$ and the dependent
variable $y$ is modelled as a polynomial.

For a single input $x$, the hypothesis in a polynomial regression of degree $d$ is
$$h_\theta(x) = \theta_0 + \theta_1 x + \theta_2 x^2 + \cdots + \theta_d x^d$$
$$h_\theta(x) = \sum_{i=0}^{d} \theta_i x^i$$

For a multivariate input $\mathbf{x}$, we introduce terms corresponding to every degree-$d$
combination of factors. For example, if $n=3$ and $d=2$, we have
$$h_\theta(\mathbf{x}) = \theta_0
                       + \theta_1 x_1 + \theta_2 x_2 + \theta_3 x_3
                       + \theta_4 x_1^2 + \theta_5 x_1 x_2 + \theta_6 x_1 x_3
                       + \theta_7 x_2^2 + \theta_8 x_2 x_3 + \theta_9 x_3^2 .$$

## Example 1: Synthetic data with a quadratic nonlinearity

Let's take a look at how polynomial regression as compared to simple linear regression model works for data with a
simple quadratic nonlinearity.

### Generate a synthetic dataset

First, we generate 100 observations from a ground truth quadratic function with Gaussian noise:

```python
import matplotlib.pyplot as plt
import numpy as np
import random

# please do not change the random seed, or the autograder's result checking will be wrong!

np.random.seed(0)
random.seed(0)
```

```python
# Generate X
m = 100
X = np.random.uniform(-4, 4, (m,1))

# Generate y
a = 0.7
b = 1
c = 2
y = a * X**2 + b * X + c + np.random.randn(m, 1)
```

```python
# Plot
plt.plot(X, y, 'b.')
plt.title('Polynomial regression example')
plt.xlabel('x')
plt.ylabel('y')
plt.show()
```

### Implement the hypothesis function

First, we will use ordinary linear regression:
$$h_\theta(x) = \theta_0 + \theta_1 x$$
Then, we use polynomial regression with $d=2$:
$$h_\theta(x) = \theta_0 + \theta_1 x + \theta_2 x^2 $$
In either case, by letting the input vector $\bf{x} = \begin{bmatrix} 1 \\ x \end{bmatrix}$
or $\bf{x} = \begin{bmatrix} 1 \\ x \\ x^2 \end{bmatrix}$
appropriately, the hypothesis can be written
$$ h_\mathbf{\theta}(\mathbf{x}) = \mathbf{\theta}^\top \mathbf{x} . $$

Let's implement this hypothesis function in Python:

```python
def h(X, theta):
    return X.dot(theta)
```

### Implement the regression function (normal equations)

Recall the normal equations used to find the $\theta$ minimizing $J(\theta)$
when the design matrix $\mathtt{X}_{m\times(n+1)}$ contains one row for each example and
$\mathbf{y}$ is a column vector:
$$\mathbf{\theta} = (\mathtt{X}^\top \mathtt{X})^{-1}\mathtt{X}^\top\mathbf{y}$$

Let's implement the normal equations in Python:

```python
def regression(X, y):
    cov = np.dot(X.T, X)
    cov_inv = np.linalg.inv(cov)
    theta = np.dot(cov_inv, np.dot(X.T, y))
    return theta
```

### Exercise 1.1 (2 points)

Create a Python function to calculate the RMSE (root mean squared error) for a set of predictions
$\hat{\mathbf{y}}$:
$$\mathrm{RMSE}(\mathbf{y},\hat{\mathbf{y}}) = \sqrt{\frac{\sum_{i=1}^{m} \left( y^{(i)}-\hat{y}^{(i)} \right)^2}
{m}}$$

```python
def rmse(y, y_pred):
    # YOUR CODE HERE
    raise NotImplementedError()
    return error
```

```python
print(rmse(np.array([1,1.1,2,-1]), np.array([1.1,1.3,1.5,0.1])))

# Test function: Do not remove
assert np.round(rmse(np.array([1,1.1,2,-0.1]), np.array([1.1,1.3,1.5,0.1])), 5) == np.round(0.29154759474226505, 5), "calculate rmse incorrect"
print("success!")
# End Test function
```

**Expected output:** 0.6144102863722254

### Implement a simple linear model

OK, as stated earlier, let's implement a simple linear model:

```python
# Add intercept column of all 1's
X_aug = np.insert(X, 0, 1, axis=1)

# Print first 5 rows of X
print(X_aug[0:5,:])

# Find optimal parameters
theta_slr = regression(X_aug, y)

# Predict y
y_pred_slr = h(X_aug, theta_slr)

print('Linear regression RMSE: %f' % rmse(y, y_pred_slr))
```

### Exercise 1.2 (2 points)

From the simple linear model above, create another linear model using a **polynomial** model with degree $d=2$.
You need to implement these steps:
 - Create the design matrix $\mathtt{X}$ as numpy matrix X_aug similarly to how we set up
   X_aug above.
 - Find the optimal solution $\theta$ as numpy vector theta_pr similarly to how we set up
   theta_slr above.

    Hint here!

Use the np.insert() function to insert a column of $x^2$ values as the last column of X_aug.

```python
# 1. Add constant column and x^2 column
# 2. Find optimal parameters

# YOUR CODE HERE
raise NotImplementedError()
```

```python
# Predict y
y_pred_pr = h(X_aug, theta_pr)
print(X_aug[0:5,:])
print('Polynomial regression RMSE: %f' % rmse(y, y_pred_pr))

# Test function: Do not remove
assert np.array_equal(np.round(theta_pr.T), np.round([[1.90932595, 1.02311816, 0.71747835]])), "theta_pr are incorrect"
assert np.round(X_aug[10,1] ** 2, 5) == np.round(X_aug[10,2], 5), "X_aug are incorrect"
assert np.round(rmse(y, y_pred_pr) ** 2 * y.shape[0], 5) == np.round(np.dot((y - y_pred_pr).T, y - y_pred_pr), 5), "RMSE incorrect"
print("success!")
# End Test function
```

**Expected output** \
[[ 1.          0.39050803  0.15249652]\
 [ 1.          1.72151493  2.96361366]\
 [ 1.          0.82210701  0.67585993]\
 [ 1.          0.35906546  0.12892801]\
 [ 1.         -0.61076161  0.37302974]]\
Polynomial regression RMSE: 0.986690

### Compare the two different models using RMSE

We see that the degree 2 polynomial fit is much better, reducing average error from 3.4 to 0.99.

To further visualize the performance of our model, we should plot the predictions vs. the observed data.

### Exercise 1.3 (2 points)

This one is a bit tricky.

We'd like to write a function get_predictions that works for any model
degree depending on what $\theta$ it is passed. The function should take as input
a vector of scalar $x$ values along with a set of parameters $\theta$. It should
output a vector of predictions $\hat{\mathbf{y}}$.

Your get_predictions function needs to construct an appropriate design
matrix $\mathtt{X}$ then use the hypothesis function we already wrote earlier.

    Hint here!

1. The code below already converts the input vector x to a 2D $m\times 1$ matrix.
1. Use np.insert to
   1. Insert a column of 1's in front of $\mathbf{x}$
   2. Insert a column of $x^2$ values, $x^3$ values, etc., according to the
      length of $\theta$. Use a while loop for this.
2. Use h() to get $\hat{y}$.

```python
def get_predictions(x, theta):
    # Change the shape of x to support the function
    x = np.array([x]).T

    # YOUR CODE HERE
    raise NotImplementedError()

    return y_hat
```

```python
x_series = np.linspace(-4, 4, 1000)
y_series_slr = get_predictions(x_series, theta_slr)
y_series_pr = get_predictions(x_series, theta_pr)

print("y_series_slr:", y_series_slr[2:5].T)
print("y_series_pr:", y_series_pr[2:5].T)

# Test function: Do not remove
assert np.round(get_predictions(np.array([1, 9, 2, -9]), theta_slr).T, 5) is not None, "predict from theta_slr is incorrect"
assert np.round(get_predictions(np.array([1, 1, 0.1, 2]), theta_pr).T, 5) is not None, "predict from theta_pr is incorrect"
print("success!")
# End Test function
```

**Expected output**:\
y_series_slr: [[2.72462183 2.73101513 2.73740842]]\
y_series_pr: [[9.0812643  9.04632656 9.01147497]]

### Plot x against y with the two regression models

Now that we have a working get_predictions(), we can
plot the data with the linear and quadratic model:

```python
plt.plot(X[:,0], y, 'c.', label='observations')
plt.plot(x_series, y_series_slr, 'b-', label='linear model (d=1)')
plt.plot(x_series, y_series_pr, 'r-', label='polynomial model (d=2)')
plt.title('Simple linear regression vs. polynomial regression (degree 2)')
plt.xlabel('x')
plt.ylabel('y')
plt.legend()
plt.show()
```

From the plot, we see clearly that the quadratic model is a much better fit to the data.

### Compare models using goodness of fit

Besides RMSE, let's also get the $R^2$ for our two models. Recall the formula for $R^2$:
\begin{align}
\ R^2 = 1 - \frac{\sum_{i=1}^{m} \left( y^{\left(i\right)}-\hat{y}^\left(i\right) \right)^2}
{\sum_{i=1}^{m} \left( y^{\left(i\right)}-\bar{y}^\left(i\right) \right)^2}
\end{align}

### Exercise 1.4 (2 points)

Fill in the function r_squared() using the equation above.

    Hint here!
    Use the np.square() function to square each of the elements of a vector.

```python
def r_squared(y, y_pred):
    # YOUR CODE HERE
    raise NotImplementedError()
    return r_sqr
```

```python
print('Fit of simple linear regression model: %.4f' % r_squared(y, y_pred_slr))
print('Fit of polynomial regression model: %.4f' % r_squared(y, y_pred_pr))

# Test function: Do not remove
assert np.round(r_squared(np.array([1, 2, 3]), np.array([1, 2, 3]))) == np.round(1.0), "r_squared is incorrect"
assert np.round(r_squared(y, y_pred_pr), 4) == np.round(0.9353, 4), "r_squared is incorrect"
print("success!")
# End Test function
```

**Expected output:**\
Fit of simple linear regression model: 0.2254\
Fit of polynomial regression model: 0.9353

So we see again the superior fit of the quadratic model using $R^2$ (0.94 vs. 0.23).

### Compare models using residual histograms

Next, let's look at another useful analysis: histograms of each model's residuals. Rather than
summarizing the residuals with RMSE or $R^2$, we'll need a function to calculate a vector of
residuals. Then we'll be able to make histograms.

### Exercise 1.5 (2 points)

Fill in function residual_error() to find the residual error vector $\mathbf{y} - \hat{\mathbf{y}}$.

Once we have that function, we can calculate error_slr for the simple linear regression
and error_pr for the polynomial regression.

```python
def residual_error(y, y_pred):
    # YOUR CODE HERE
    raise NotImplementedError()
    return error

error_slr = residual_error(y, y_pred_slr)
error_pr = residual_error(y, y_pred_pr)
```

```python
# Plot distribution of residual error for each model
print("error_slr sample:", error_slr[0:5, 0].T)
print("error_pr sample:", error_pr[0:5, 0].T)

plt.hist(error_slr, bins=10, label = 'Linear')
plt.hist(error_pr, bins=10, label = 'Polynomial')
plt.xlabel('Error')
plt.ylabel('Frequency')
plt.title('Residual error distribution')
plt.legend()
plt.show()

# Test function: Do not remove
assert np.array_equal(np.round(get_predictions(np.array([1, 9, 2, -9]), theta_slr).T),
                      np.round([[6.70364883, 13.09055058, 7.50201155, -1.27997835]])), "predict from theta_slr is incorrect"
assert np.array_equal(np.round(get_predictions(np.array([0, 7, 1.5, -0.3]), theta_pr).T),
                      np.round([[2.34050076, 42.14663283, 5.3284002, 2.10566904]])), "predict from theta_pr is incorrect"
print("success!")
# End Test function
```

**Expected output:**\
error_slr sample: [-4.88494741 -0.58280848 -2.8007543  -5.27887921 -2.27906541]\
error_pr sample: [-1.49521216  0.67105966  0.15715854 -1.86746535  1.14869785]

The residual plot again shows clearly how much better the polynomial model is than the linear model.

## Example 2: Sales data

Next, let's model some real data, in particular,
monthly sales data from Kaggle using polynomial regression with varying degree.

We will observe the effects of varying the degree of the polynomial regression fit on the prediction accuracy.

However, as discussed in class,
as models become more complex, we will encounter the issue of *overfitting*, in which a too-powerful
model starts to model the noise in the specific training set rather than the overall trend.

To ensure that we're not fitting the noise in the training set, we will split the data into seaparte train and test/validation datasets.
The training dataset will consist of 60% of the original observations, and the test dataset will consist of the remaining 40% of the observations.

For various polynomial degrees, we'll estimate optimal parameters $\theta$, from the
training set, then we'll use the test/validation dataset to measure the accuracy of the optimized model.

First, let's read the data from the CSV file and set up variables X_data, y_data.

```python
# Import CSV
data = np.genfromtxt('MonthlySales_data.csv',delimiter = ',', dtype=str)

# Extract headers
headers = data[0,:]
print("Headers:", headers)

# Extract raw data
data = np.array(data[1:,:], dtype=float);
mean = np.mean(data,axis=0)
std = np.std(data,axis=0)
data_norm = (data-mean)/std

# Extract y column from raw data
y_index = np.where(headers == 'sale amount')[0][0];
y_data = data[:,y_index];

# Extract x column (just the month) from raw data
month_index = np.where(headers == 'month')[0][0]
# print(year_index, month_index)
X_data = data[:,[month_index]];
m = X_data.shape[0]
n = X_data.shape[1]
X_data = X_data.reshape(m, n)

print('Extracted %d monthly sales records' % m)
print(X_data.shape)
print(y_data.shape)
```

### Plot the data

Although year and month are discrete variables, they are also ordinal, so they can be
treated as real values. Let's plot sales month against sales amount as a scatter plot, and
we'll see the discrete nature of the data:

```python
fig = plt.figure()
xx1 = X_data[:,0]
yy1 = y_data

plt.plot(xx1, yy1, 'b.')
plt.xlim(0, 13)
plt.xlabel('Month')
plt.ylabel('Sales amount')
plt.title('Sample monthly sales data')
plt.show()
```

### Partition the data

Next let's split the overall dataset into subsets for training and validation (test).

### Exercise 1.6 (2 points)

Partition X_data and y_data into training and test datasets
 - Let the training set be 60% of the dataset
 - Let the rest be the test set
 - Shuffle the dataset before splitting it to ensure a similar distribution in the two subsets

You can use the [random.shuffle() function](https://www.w3schools.com/python/ref_random_shuffle.asp)</link> to shuffle the indices of the dataset.

```python
percent_train = .6

def partition(X, y, percent_train):
    # Create a list of indices into X and y
    idx = np.arange(0,y.shape[0])
    random.seed(1412)   # just make sure the shuffle always the same please do not remove
    # On your own, do the following:
    # 1. shuffle the idx list
    # 2. Create lists of indices train_idx and test_idx for the train and test sets
    # 3. Set variables X_train, y_train, X_test, and y_test using those index lists

    # YOUR CODE HERE
    raise NotImplementedError()

    return idx, X_train, y_train, X_test, y_test
```

```python
idx, X_train, y_train, X_test, y_test = partition(X_data, y_data, percent_train)
print(X_train.shape)
print(y_train.shape)
print(X_test.shape)
print(y_test.shape)
print(idx[5:9])

# Test function: Do not remove
assert not np.array_equal(np.round(X_data[0:144, :], 3), np.round(X_train,3)), "X_train must be shuffled!"
assert not np.array_equal(np.round(X_data[144:, :], 3), np.round(X_test,3)), "X_test must be shuffled!"
assert not np.array_equal(np.round(y_data[0:144], 3), np.round(y_train,3)), "y_train must be shuffled!"
assert not np.array_equal(np.round(y_data[144:], 3), np.round(y_test,3)), "y_test must be shuffled!"
assert np.array_equal(idx[5:9], [26, 75, 51, 162])
print("success!")
# End Test function
```

**Expected output:**\
(144, 1)\
(144,)\
(96, 1)\
(96,)\
[ 26  75  51 162]

### Set up for polynomial regression

Next, let's implement the transformation of a variable $x$ into the
expanded list $\begin{bmatrix} x & x^2 & \cdots & x^d \end{bmatrix}$.

### Exercise 1.7 (2 points)

Fill in function x_polynomial() with code to output
a row vector consisting of the elements $x, x^2, \ldots, x^d$, where
when $d$ is the degree of the polynomial.

```python
def x_polynomial(x, d):
    # YOUR CODE HERE
    raise NotImplementedError()
    return X
```

```python
print(x_polynomial(np.array([[3],[2]]), 5))
print(x_polynomial(np.array([[3],[2]]), 5).shape)

Xi_train = x_polynomial(X_train, 1)
Xi_test = x_polynomial(X_test, 1)

# Test function: Do not remove
assert x_polynomial(np.array([[2],[3]]), 5).shape[1] == 5 + 1, "Size of polynomial incorrect"
assert np.array_equal(np.round(x_polynomial(np.array([[2],[3]]), 5), 3),
                      np.round([[1, 2, 4, 8, 16, 32], [1, 3, 9, 27, 81, 243]],3)), "Polynomial are wrong."
print("success!")
# End Test function
```

**Expected output:**\
[[  1.   3.   9.  27.  81. 243.]\
 [  1.   2.   4.   8.  16.  32.]]\
(2, 6)

### Write the cost function

Next let's implmeent to cost function for a given set of parameters $\theta$.

### Exercise 1.8 (2 points)

Fill in function cost() with appropriate code. Use a constant of $\frac{1}{2m}$ out front.

```python
def cost(theta, X, y):
    # YOUR CODE HERE
    raise NotImplementedError()
    return J
```

```python
# calculate theta
theta = regression(Xi_train, y_train)

# calculate cost in train
J_train = cost(theta, Xi_train, y_train)

y_pred_test = h(Xi_test, theta)
J_test = cost(theta, Xi_test, y_test)

print("J_train:", J_train)
print("J_test:", J_test)

# Test function: Do not remove
assert type(J_train) == np.float64, "Cost function size must be 1"
assert np.round(J_train, 3) == np.round(174395635.44334993, 3), "Cost function for train set is wrong"
assert np.round(J_test, 3) == np.round(196382485.91395777, 3), "Cost function for test set is wrong"
print("success!")
# End Test function
```

**Expected output:**\
J_train: 174395635.44334993\
J_test: 196382485.91395777

### Try models of varying degree

Next we'll build multiple polynomial regression models with different degree, using sales
month as the independent variable and sales amount as the dependent variable.

```python
max_degree = 5

J_train = np.zeros(max_degree)
J_test = np.zeros(max_degree)

# Initalize plots for predictions and loss
fig, ax = plt.subplots(1,2)
fig.set_figheight(5)
fig.set_figwidth(20)
fig.subplots_adjust(left=.2, bottom=None, right=None, top=None, wspace=.2, hspace=.2)
plt1 = plt.subplot(1,2,1)
plt2 = plt.subplot(1,2,2)
plt2.plot(X_train, y_train, 'c.', label='observations')

for i in range(1, max_degree+1):
    # Fit model on training data and get cost for training and test data
    Xi_train = x_polynomial(X_train, i)
    Xi_test = x_polynomial(X_test, i);
    theta = regression(Xi_train, y_train)
    J_train[i-1] = cost(theta, Xi_train, y_train)
    y_pred_test = h(Xi_test, theta)
    J_test[i-1] = cost(theta, Xi_test, y_test)

    # Plot
    x_series = np.linspace(0, 13, 1000)
    y_series = get_predictions(x_series, theta)
    plt2.plot(x_series, y_series, '-', label='degree ' + str(i) + ' (test accuracy ' + str(r_squared(y_test, y_pred_test)) + ')')

plt1.plot(np.arange(1, max_degree + 1, 1), J_train, '-', label='train')
plt1.plot(np.arange(1, max_degree + 1, 1), J_test, '-', label='test')
plt1.set_title('Loss vs polynomial degree')
plt1.set_xlabel('polynomial degree')
plt1.set_ylabel('loss')
plt1.grid(axis='both', alpha=.25)
plt1.legend()

plt2.set_title('Predicted monthly sales')
plt2.set_xlabel('Month')
plt2.set_ylabel('Sales ($)')
plt2.grid(axis='both', alpha=.25)
plt2.legend()
plt.show()
```

Take some time to undserstand the code. You should see that training loss falls as the degree of the polynomial increases. However, depending on your particular train/test split of the data, you may observe at $d=4$ or $d=5$ that test loss starts to flatten out or even increase. This is the phenomenon of overfitting!

If you don't see any evidence of overfitting, you might regenerate the test/train splits (comment out the seed
setting in the partition function and re-run the rest of the cells, but don't forget to put the seed back before
turning in your solution!).

You may also increase max_degree to a point. However, without normalization of the data, the matrix $\texttt{X}^\top\texttt{X}$ we invert in the solution to the normal equations will become numerically close to singularity, and you will observe unstable solutions. The result is usually a parameter vector $\theta$ that is suboptimal that gives poor results on both the training set and test set.

If you want to evaluate the numerial stability of the correlation matrix $\texttt{X}^\top\texttt{X}$, try this code:

```python
corr = Xi_train.T.dot(Xi_train)
print('Correlation matrix:', corr)
cond = np.linalg.cond(corr)
print('Condition number: %0.5g' % cond)
```

Read more about the condition number on <link>[Wikipedia](https://en.wikipedia.org/wiki/Condition_number)</link>. Roughly speaking, if our condition number is $10^k$, we may lose up to $k$ digits of accuracy in the inverse of the matrix. If $k=12$ as above, then we have an extremely poorly conditioned problem, because the IEEE 64 bit floating point representation of reals we're using in Python only has around 16 digits of accuracy (see <link>[Wikipedia's page on IEEE floating point numbers](https://en.wikipedia.org/wiki/IEEE_754)</link>).

One way to improve the numerical conditioning of the problem is normalization. If the values of the variables we
are correlating in this matrix have relatively small positive and negative values, the condition number of the correlation matrix will be much smaller and you'll get better results.

## In-lab exercises

During the lab session, you should perform the following exercises:
1. Add the `year` variable from the monthly sales dataset to your simple linear regression model and quantify
   whether including it improves test set performance. Show
   the observations and predictions in a 3D surface plot.
2. Develop polynomial regression models of degree 2 and 3 based on the two input variables. Show results
   as 3D surface plots and discuss whether you observe overfitting
   or not.

### Exercise 2.1 (2 points)

Import **MonthlySales_data.csv** file into data_csv and extract **headers**  at the top of data_csv into headers_csv.

```python
# YOUR CODE HERE
raise NotImplementedError()
```

```python
print(headers_csv)
print(data_csv[:5])

# Test function: Do not remove
assert type(data_csv[0,0]) == np.float64, "You must remove the header"
assert headers_csv.shape[0] == 3, "Headers must have 3 values"
assert type(headers_csv[0]) == np.str_, "Headers must be string"
assert np.round(data_csv[30, 2], 3) == np.round(2.222027e+04, 3), "Data is incorrect"
print("success!")
# End Test function
```

**Expected output**:\
['year' 'month' 'sale amount']\
[[1.995000e+03 1.000000e+00 1.238611e+04]\
 [1.995000e+03 2.000000e+00 1.532923e+04]\
 [1.995000e+03 3.000000e+00 5.800217e+04]\
 [1.995000e+03 4.000000e+00 5.130520e+04]\
 [1.995000e+03 5.000000e+00 1.645247e+04]]

### Exercise 2.2 (2 points)

- Extract **sale amount** column into y_csv
- Extract **year** and **month** columns into X_csv by use **year** at column index 0 and **month** at column index 1

```python
# Extract y column from raw data
# Extract x column (year and month) from raw data
# YOUR CODE HERE
raise NotImplementedError()
```

```python
m = X_csv.shape[0]
n = X_csv.shape[1]
X_csv = X_csv.reshape(m, n)
print('Extracted %d sales records' % m)
print('number of x set:', n)

# Test function: Do not remove
assert m == 240, "Sales records incorrect"
assert n == 2, "Need to extract 2 columns of X set"
assert np.max(X_csv[:,0]) == 2014 and np.min(X_csv[:,0]) == 1995, "Year is filled wrong column"
assert np.max(X_csv[:,1]) == 12 and np.min(X_csv[:,1]) == 1, "Month is filled wrong column "
print("success")
# End Test function
```

**Expected output**:\
Extracted 240 sales records\
number of x set: 2

### Exercise 2.3 (2 points)

Plot a 3D graph using the mpl_toolkits.mplot3d library.

    Hint here!
    Refer to the [matplotlib documentation page](https://matplotlib.org/stable/gallery/mplot3d/scatter3d.html)
    for an example.

```python
# Plot the data
from mpl_toolkits.mplot3d import Axes3D

fig = plt.figure()
# 1. Set plot graph as 3D
ax = fig.add_subplot(projection='3d')

# 2. Extract data
# extract year at x-axis
# extract month at y-axis
# extract sale amount at z-axis
x_year = None
y_month = None
z_sale = None

# 3. plot by using scatter

# 4. set x, y, z label
# YOUR CODE HERE
raise NotImplementedError()

plt.show()
```

```python
# Test function: Do not remove
assert ax.get_xbound()[1] >= 2014 and ax.get_xbound()[0] <= 1995, "Year is filled wrong column"
assert ax.get_ybound()[1] >= 12 and ax.get_ybound()[0] <= 1, "Month is filled wrong column"
assert ax.get_zbound()[1] >= 100000 and ax.get_zbound()[0] <= 0, "Year is filled wrong column"
assert 'year' in ax.get_xlabel().lower(), "x-axis label is incorrect"
assert 'month' in ax.get_ylabel().lower(), "y-axis label is incorrect"
assert 'sale' in ax.get_zlabel().lower(), "y-axis label is incorrect"
print("success")
# End Test function
```

**Expected output:**\

### Exercise 2.4 (2 points)

Extract 60% of the data to the training set and the remaining 40% to the test set with shuffling.

You can use the partitions function we already made or create a new function. Make sure
that you use random.seed(1412) to make sure that the result is the same as the expect result.
Place the resulting data in variables idx, X_train, y_train, X_test, y_test.

```python
idx, X_train, y_train, X_test, y_test = None, None, None, None, None
# YOUR CODE HERE
raise NotImplementedError()
```

```python
print(X_train.shape)
print(y_train.shape)
print(X_test.shape)
print(y_test.shape)
print(idx[5:9])

# Test function: Do not remove
assert not np.array_equal(np.round(X_csv[0:144, :], 3), np.round(X_train,3)), "X_train must be shuffled!"
assert not np.array_equal(np.round(X_csv[144:, :], 3), np.round(X_test,3)), "X_test must be shuffled!"
assert not np.array_equal(np.round(y_csv[0:144], 3), np.round(y_train,3)), "y_train must be shuffled!"
assert not np.array_equal(np.round(y_csv[144:], 3), np.round(y_test,3)), "y_test must be shuffled!"
assert np.array_equal(idx[5:9], [26, 75, 51, 162])
print("success!")
# End Test function
```

**Expected output**:\
(144, 2)\
(144,)\
(96, 2)\
(96,)\
[ 26  75  51 162]

### Exercise 2.5 (2 points)

1. Create Xi_train, Xi_Test. X sets must be polynomial of $n=1$.
2. Calculate theta
2. Calculate y_pred_test
2. Calculate cost function $J$ from train and test set

```python
Xi_train, Xi_test = None, None
theta = None
y_pred_test = None
J_train, J_test = None, None

# YOUR CODE HERE
raise NotImplementedError()
```

```python
print("Xi_train[:3]:", np.round(Xi_train[:3], 2))
print("Xi_test[:3]:", np.round(Xi_test[:3], 2))
print("theta:", theta)
print("y_pred_test[:5]:", np.round(y_pred_test[:5].T, 2))
print("J_train:", J_train)
print("J_test:", J_test)

# Test function: Do not remove
assert np.array_equal(np.round(theta, 3), np.round([5.74503812e+05, -2.83158807e+02, 6.37579347e+03],3)), "Regression theta is incorrect"
assert np.round(J_train, 0) == np.round(172968387.44854635, 0), "Train cost is incorrect"
assert np.round(J_test, 0) == np.round(204275431.7643744, 0), "Test cost is incorrect"
print("success")
# End Test function
```

**Expected output**:\
Xi_train[:3]: [[1.000e+00 2.003e+03 1.100e+01]\
 [1.000e+00 2.004e+03 3.000e+00]\
 [1.000e+00 2.002e+03 6.000e+00]]\
Xi_test[:3]: [[1.000e+00 2.008e+03 1.000e+01]\
 [1.000e+00 1.997e+03 5.000e+00]\
 [1.000e+00 2.006e+03 1.100e+01]]\
theta: [5.74503812e+05 -2.83158807e+02  6.37579347e+03]\
y_pred_test[:5]: [69678.86 40914.64 76620.97 79169.4  48852.53]\
J_train: 172968387.44854635\
J_test: 204275431.7643744

### Exercise 2.6 (2 points)

Create a mesh of grid points in order to obtain a surface plot later.

    Hint here!
    Create a mesh grid using [numpy.meshgrid()](https://numpy.org/doc/stable/reference/generated/numpy.meshgrid.html)

```python
# 1. Create mesh grid x_mesh, y_mesh
#    Hint: this step do in input X dataset only (year, and month series)
# 1.1 use numpy.linspace() to generate x_series and y_series
#     - do x_series in between min(year) - 1 to max(year) + 1
#     - do y_series in between min(month) - 1 to max(month) + 1
#     - num_linspace = 100
# 1.2 use numpy.meshgrid() to generate x_mesh, and y_mesh
# 1.3 merge x_mesh and y_mesh to be xy_mesh
num_linspace = 100
x_series, y_series = None, None
x_mesh, y_mesh, xy_mesh = None, None, None

# 2. predict output from xy_mesh to be z_series
#    Hint: use mesh_predictions function instead of get_prediction
def mesh_predictions(x, theta):
    x = np.insert(x, 0, 1, axis=x.ndim-1)
    theta = theta.reshape(-1,1)
    y = x@theta
    return y
z_series = None

# YOUR CODE HERE
raise NotImplementedError()
```

```python
print("xy_mesh.shape", xy_mesh.shape)
print("z_series.shape", z_series.shape)
#print("xy_mesh", xy_mesh)
#print("z_series", z_series)

# Test function: Do not remove
assert xy_mesh.shape == (num_linspace, num_linspace, 2), "mesh shape is incorrect"
assert z_series.shape == (num_linspace, num_linspace), "z_series is incorrect"
print("success")
# End Test function
```

**Expected output**:\
xy_mesh.shape (100, 100, 2)\
z_series.shape (100, 100)

### Exercise 2.6 (2 points)

Make a surface plot for theta with the dataset points from xy_mesh and z_series
variables created above.

    Hint here!
    You can use the [Axes3D.plot_surface() function](https://matplotlib.org/2.0.2/mpl_toolkits/mplot3d/tutorial.html).

```python
fig = plt.figure()
# 1. Set plot graph as 3D
ax = fig.add_subplot(projection='3d')

# 2. Extract data
# extract year at x-axis
# extract month at y-axis
# extract sale amount at z-axis
x_year = None
y_month = None
z_sale = None

# 3. plot by using scatter
# 4. set x, y, z label
#    Hint: In these 3, 4 steps, you can copy Exercise 2.3
# 5. Plot surface from x_mesh, y_mesh, and z_series

# YOUR CODE HERE
raise NotImplementedError()

plt.show()
```

```python
# Test function: Do not remove
assert ax.get_xbound()[1] >= 2014 and ax.get_xbound()[0] <= 1995, "Year is filled wrong column"
assert ax.get_ybound()[1] >= 12 and ax.get_ybound()[0] <= 1, "Month is filled wrong column"
assert ax.get_zbound()[1] >= 100000 and ax.get_zbound()[0] <= 0, "Year is filled wrong column"
assert 'year' in ax.get_xlabel().lower(), "x-axis label is incorrect"
assert 'month' in ax.get_ylabel().lower(), "y-axis label is incorrect"
assert 'sale' in ax.get_zlabel().lower(), "y-axis label is incorrect"
print("success")
# End Test function
```

**Expect result:**

### Exercise 2.7 (20 points)

Develop polynomial regression models of degree 2 and 3 based on the two input variables. Show results as 3D surface plots and discuss whether you observe overfitting or not.

```python
# Your code here
```

```python

```

## Take-home exercise (50 points)

Using the dataset you played with for the take-home exercise in Lab 01, perform the same analysis. You won't be able to visualize the model well, as you will have more
than two inputs, but try to give some idea of the performance of the model visually. Also, depending on the number of variables in your dataset, you may not be able to
increase the polynomial degree beyond 2. Discuss whether the polynomial model is better than the linear model and whether you observe overfitting.

Insert your code, explanation, and results here.

## To turn in

Before the next lab, turn in a brief report in the form of a Jupyter notebook documenting your work in the lab and the take-home exercise, along with your observations and discussion.

```python

```


# Inverted logistic regression
Using Bayesian modelling to invert a logistic regression model.

```python
import numpy as np
import pymc3 as pm
import seaborn as sns
import theano.tensor as tt
import matplotlib.pyplot as plt
from scipy.special import softmax
from sklearn.linear_model import LogisticRegression
```

Let's simulate some data: 800 ($N$) observations with 50 ($P$) features (independent variables) and 6 classes ($K$), e.g., categorical emotions. We'll simulate it as a proper multinomial logistic regression model:

\begin{equation}
p(y | X) = \mathrm{softmax}(X\beta + \alpha)
\end{equation}

```python
N = 800
P = 50
K = 6

X = np.random.normal(0, 1, size=(N, P))
α = np.random.normal(0, 0.001, size=K)
β = np.random.normal(0, 0.001, size=(P, K))
mu = X @ β + α
p_y = softmax(mu, axis=1)

# Make discrete labels
y = np.array([np.random.multinomial(1, p_y[i, :]).argmax() for i in range(N)])
```

Now, suppose we fit a logistic regression model using scikit-learn (note that we *could* also use *pymc3* to fit a Bayesian logistic regression model, but we don't do that here because it's a lot slower).

```python
lr = LogisticRegression()
lr.fit(X, y)
β_hat = lr.coef_.T
α_hat = lr.intercept_
```

Okay cool, now suppose we want to know the most probable set of features ($X$) given a particular target class ($y = c$), i.e., $p(X | y = c)$. Given that we already know the generative model ($p(y | X)$), we can invert this model by putting a prior on the features ($p(X)$):

\begin{equation}
p(X | y = c) \propto p(y = c | X)p(X)
\end{equation}

In `pymc3`:

```python
with pm.Model() as inverted_lr:
    # Important: we set the SD of the normal prior to 10 times the empirical SD
    X_ = pm.Normal('X_', mu=0, sd=10*X.std(axis=0), shape=(K, P))

    # Use the previously estimated parameters from the LR model!
    mu_ = tt.dot(X_, β_hat) + α_hat
    p_ = tt.nnet.softmax(mu_)

    # Do this separately for all classes [0, 1, 2, 3, 4, 5]
    y_ = pm.Categorical('y_', p=p_, observed=range(K))
    trace = pm.sample()
```

    /Users/lukas/software/anaconda3/lib/python3.8/site-packages/pymc3/sampling.py:466: FutureWarning: In an upcoming release, pm.sample will return an `arviz.InferenceData` object instead of a `MultiTrace` by default. You can pass return_inferencedata=True or return_inferencedata=False to be safe and silence this warning.
      warnings.warn(
    Auto-assigning NUTS sampler...
    Initializing NUTS using jitter+adapt_diag...
    Multiprocess sampling (2 chains in 2 jobs)
    NUTS: [X_]

<div>
    <style>
        /* Turns off some styling */
        progress {
            /* gets rid of default border in Firefox and Opera. */
            border: none;
            /* Needs to be in here for Safari polyfill so background images work as expected. */
            background-size: auto;
        }
        .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {
            background: #F44336;
        }
    </style>
  <progress value='4000' class='' max='4000' style='width:300px; height:20px; vertical-align: middle;'></progress>
  100.00% [4000/4000 00:13<00:00 Sampling 2 chains, 0 divergences]
</div>

    Sampling 2 chains for 1_000 tune and 1_000 draw iterations (2_000 + 2_000 draws total) took 20 seconds.

Now we have a posterior distribution of all $P$ features for all $K$ classes. Let's plot the posterior from the first feature for all 6 classes (including the MAP estimate):

```python
X_map = pm.find_MAP(model=inverted_lr)['X_']
fig, axs = plt.subplots(ncols=K, constrained_layout=True, figsize=(15, 5), sharex=True, sharey=True)
for i, ax in enumerate(axs):
    sns.kdeplot(trace['X_'][:, i, 0], ax=ax)
    ax.axvline(X_map[i, 0], ls='--', c='k')
    ax.set_title(f"Class = {i}")
ax.set_ylim(0, 0.05)
sns.despine()
```

Just as a check, let's check whether indeed the MAP estimate would reliably lead to the expected class ($p(y | X^{*}$), i.e., a sort of confusion matrix:

```python
y_opt = softmax(X_map @ β_hat + α_hat, axis=1)
fig, ax = plt.subplots(figsize=(8, 8), constrained_layout=True)
mapp = ax.imshow(y_opt, vmin=0, vmax=1)
cbar = plt.colorbar(mapp)
cbar.ax.set_ylabel(r"$p(y | X^{*}$)", fontsize=20)
ax.set_xlabel('True label ($y$)', fontsize=15)
ax.set_ylabel('Predicted label ($\hat{y}$)', fontsize=15)
fig.show()
```


Enzyme Kinetics

In this notebook, we're going to implement some basic enzyme kinetics notions in Python, enzyme kinetics play a major role in Neuroscience as they dictate how ion channels and molecular interaction networks in intracellular signaling are regulated. We're going to use differential equations to express those kinetics because they are considered as the mathematical objects of scientific modeling.

Table of contents

1  Zero and First Order Reactions

2  Enzymatic Equilibrium

3  Michaelis-Menten-Henri Equation

Let's start by importing some libraries that we'll use mainly here and in the rest of the Notebooks.

```python
import numpy as np # A Python package for scientific computing
import sympy as sp # A Python library for symbolic mathematics
import matplotlib.pyplot as plt # A data visualization Library in Python
```

# Zero and First Order Reactions

- Considering a chemical reaction catalyzed by an enzyme for a reactant $A$ that transforms into a product $B$  :

$$A \xrightarrow{enzyme} B$$

> If the concentration of the enzyme is far less than the concentration of $A$, we can say that the enzyme is saturated and the depletion of the reactant $A$ or the appearance of the product $B$ is constant and is independant from the concentration of the reactant $A$.
$$\frac{d[B]}{dt} = - \frac{d[A]}{dt} = k $$
$k$ being the rate constant of the reaction

- We can see that this expression is equivalent to :
$$\boxed{- \frac{d[A]}{dt} = k[A]^0 = k}$$
>The rate of the reaction $\frac{d[A]}{dt}$ does not depend on the concentration of reactants,but only on the rate constant $k$, in what we call a zero-order chemical reaction. This is a differential equation, solving it consists of finding the function that gives the conentration of the reactant in function of time $A(t)$, it is easy to solve at hand but sometimes differential equations get complicated really fast, this is why we  use computers.

- If the rate depends linearly upon the concentration of the reactant $A$, the equation becomes :
$$\boxed{- \frac{d[A]}{dt} = k[A]}$$
> The first order reaction, this too is considered as a differential equation and can be solved the same way as the previous one.

- In order to solve differential equations using python, we can use two different approaches :

>1 - Analytical methods; by using Python's Sympy library for symbolic mathematics.

2 - Numerical Methods; by implementing Euler's method which is used for solving ordinary differential equations (ODEs) and then we can use Python's NumPy Library and its useful N-dimensional array objects that are the cosidered the basis of scientific computing.

Zero-order

**Analytical Method**

> Let's start first by initialising Sympy's symbol objects which we will be able to manipulate

```python
k, t, C1 = sp.symbols(
    'k t C1')  # Rate constant k , time t and arbitrary integration constant C1
```

> The reactant concentration $A$ should be initialised as Sympy Function Object and it should be in the differentiated form with respect to time $t$

```python
A = sp.Function('A')
dAdt = A(t).diff(t)
```

> Now we can write the zero order equation

```python
zero_order = sp.Eq(-dAdt, k)
zero_order
```

$\displaystyle - \frac{d}{d t} A{\left(t \right)} = k$

> The next step is to solve this differential equation, this can be easily done by using SymPy's function dsolve that takes the equation and the variable to be solved to as parameters.

```python
analytic_gen_sol_zero = sp.dsolve(zero_order, A(t))
analytic_gen_sol_zero
```

$\displaystyle A{\left(t \right)} = C_{1} - k t$

> This is the general solution to our differential equation, in order to calculate the arbitrary constant C1 we can initialize $t=0$ in our general solution to find a particular one.

```python
C1_zero = analytic_gen_sol_zero.subs(
    t, 0)  # This substitutes t with the value of 0 in our general solution
C1_zero
```

$\displaystyle A{\left(0 \right)} = C_{1}$

> The arbitrary constant $C_1$ is the concentration of the reactant at time 0 (the initial condition)

```python
analytic_gen_sol_zero = analytic_gen_sol_zero.subs(C1, A(0))
analytic_gen_sol_zero
```

$\displaystyle A{\left(t \right)} = - k t + A{\left(0 \right)}$

> This is the formula for the general solution of the differential equation, in order to find a particular solution let's consider that the rate constant $k=2$ and the initial concentration of the reactant $A(0) = 10$

```python
analytic_par_sol_zero = analytic_gen_sol_zero.subs({A(0): 10, k: 2})
analytic_par_sol_zero
```

$\displaystyle A{\left(t \right)} = 10 - 2 t$

> Since this is an algebraic SymPy expression, we cannot numerically evaluate it, we have to transform it into a lambda function, we can use SymPy's lambdify function to achieve that.

```python
analytic_sol_zero = sp.lambdify(
    t, analytic_par_sol_zero.rhs,
    "numpy")  # We took the right hand side with .rhs method from our solution
```

**Numerical Method**

> In order to numerically solve differential equations, we have to implement the Euler's method, let's proceed by writing a python function that will work with every differential equation. *For a detailed explanation of Euler's Method, check out Steven Strogatz's Nonlinear dynamics and Chaos*

```python
def euler(init_cond, *constants,  equation, dt=0.01, Tmax=10):
    """Euler's Method for solving ODEs.
    init_cond :  The initial condition to start solving from.
    constants : a variable length argument with constants in the ODE
    equation : the differential equation to be solved.
    dt : the time interval between every step.
    Tmax : the maximal amount of time
    """
    for step in range(int(Tmax / dt)):  # How many steps to take
        init_cond += dt * equation(init_cond, *constants)  # Euler's method
        yield init_cond  # A generator python expression
```

> This Euler function with only 4 lines will take any kind of differential equation and will spit out a generator object that is the solution for a certain array of time. Now we can write the zero-order differential equation as a Lambda function in Python and pass it to our Euler function.

```python
zero_order = lambda c,k : -k
```

 > Likewise, Let's take $A(0) = 10$ as our initial used concentration (starting condition) and $k=2$

```python
Ao, const = 10, 2
```

> So the numerical solution is  :

```python
numeric_sol_zero = list(
    euler(Ao, const, equation=zero_order
          ))  # Transform generator into list to visualize with matplotlib
```

**Plotting the solutions**

Now that we have both the solution, let's visualize them.

```python
def plot_solutions(t, numeric, analytic, title):
    """This function plots the numerical and the analytical solutions
    to differential equation for a given time vector t
    """
    fig, ax = plt.subplots(1, 2, figsize=(10, 4), dpi=150)
    plt.subplot(121)  # Numerical Solution Plot
    plt.plot(t, numeric, 'b', label='$A(t)$')
    plt.title('Numerical Solution')
    plt.legend()
    plt.subplot(122)  # Analytical Solution Plot
    plt.plot(t, analytic(time), 'g', label='$A(t)$')
    plt.xlabel('Time', position=(-0.1, 0))
    plt.title('Analytical Solution')
    plt.suptitle(title)
    plt.legend()
```

```python
time = np.arange(0, 10, 0.01)  # Same vector used for euler's method
plot_solutions(time, numeric_sol_zero, analytic_sol_zero, "Zero-order reaction $A(0)=10$")
```

First-order

**Analytical Method**

> We're going to use the same steps taken for zero order

```python
first_order = sp.Eq(-dAdt, k*A(t))
first_order
```

$\displaystyle - \frac{d}{d t} A{\left(t \right)} = k A{\left(t \right)}$

> After the first order equation has been initialised, let's find the general solution with SymPy

```python
analytic_gen_sol_first = sp.dsolve(first_order, A(t))
analytic_gen_sol_first
```

$\displaystyle A{\left(t \right)} = C_{1} e^{- k t}$

> As always, we'll take $t=0$ to find the arbitrary constant

```python
C1_1 = analytic_gen_sol_first.subs(t,0)
C1_1
```

$\displaystyle A{\left(0 \right)} = C_{1}$

> So the solution is :

```python
analytic_gen_sol_first = analytic_gen_sol_first.subs(C1, A(0))
analytic_gen_sol_first
```

$\displaystyle A{\left(t \right)} = A{\left(0 \right)} e^{- k t}$

> Now to find a particular solution, let's take $A(0) = 10$ and $k=2$

```python
analytic_par_sol_first  = analytic_gen_sol_first.subs({A(0) : 10, k : 2})
analytic_par_sol_first
```

$\displaystyle A{\left(t \right)} = 10 e^{- 2 t}$

> And finally we transform it into a lambda function :

```python
analytic_sol_first = sp.lambdify(t, analytic_par_sol_first.rhs, "numpy")
```

**Numerical Method**

> Let's initialize a Python function for the First order reaction

```python
first_order = lambda c,k : -k*c
```

> And the solution, using Euler's Method and with the same starting condition and rate constant, will be :

```python
numeric_sol_first = list(euler(Ao, const, equation=first_order))
```

**Plotting the solutions**

And now let's visulaize the solutions

```python
plot_solutions(
    time, numeric_sol_first, analytic_sol_first,
    'First-order reaction $A(0)=10$')  #Same time vector used earlier
```

# Enzymatic Equilibrium

- Let's consider now a molecule that passes from a chemical conformation to another one in a reversible manner, like an ion channel that opens and closes, this reaction will be :

$$A \overset{\alpha}{\underset{\beta}\rightleftharpoons} B$$

- $[A] + [B] = c_0$ is always constant, and taking into account the law of conservation of mass we can see that :
$$\frac{d[B]}{dt} = \alpha[A] - \beta[B]$$

- If we take into account the fraction of each conformation where $f_\alpha + f_\beta = 1$ :

$$\frac{df_\beta}{dt} = \alpha f_\alpha - \beta f_\beta$$

- Considering $f_\alpha = 1 - f_\beta$, we get :
$$\frac{df}{dt} = \alpha(1 - f) - \beta f = \alpha - (\alpha + \beta)f$$

- Finally, consider $\tau = \frac{1}{\alpha+\beta}$ and $f_\infty = \frac{\alpha}{\alpha+\beta}$ :

$$\boxed{\tau\frac{df}{dt} = f_\infty - f(t)}$$

> This too is a differential equation, now we're going to solve it similairly to zero and first order reactions.

**Analytical Method**
> We're going to use SymPy's algebraic notation, let's initialize our constants and functions.

```python
tau, f_infty= sp.symbols('tau f_\infty')
f = sp.Function('f')
dfdt = f(t).diff(t)
```

>Let's see our equation :

```python
equilibrium = sp.Eq(tau*dfdt, f_infty - f(t))
equilibrium
```

$\displaystyle \tau \frac{d}{d t} f{\left(t \right)} = f_\infty - f{\left(t \right)}$

> Now it's time to see the general solution :

```python
analytic_gen_sol_eq = sp.dsolve(equilibrium, f(t))
analytic_gen_sol_eq
```

$\displaystyle f{\left(t \right)} = f_\infty + e^{\frac{C_{1} - t}{\tau}}$

> Let's find the arbitrary constant $C_1$

```python
eq_0 = analytic_gen_sol_eq.subs(t,0) # Considering that t=0
C1_eq = sp.Eq(sp.solve(eq_0,C1)[0], C1) # Solving the equation t=0 to find C1
C1_eq
```

$\displaystyle \tau \log{\left(- f_\infty + f{\left(0 \right)} \right)} = C_{1}$

> And the solution is :

```python
analytic_gen_sol_eq = analytic_gen_sol_eq.subs(C1,C1_eq.lhs)
analytic_gen_sol_eq
```

$\displaystyle f{\left(t \right)} = f_\infty + e^{\frac{- t + \tau \log{\left(- f_\infty + f{\left(0 \right)} \right)}}{\tau}}$

> It would be interesting to visualize the solution $f(t)$, so let's assign some values for our constants

```python
analytic_par_sol_eq = analytic_gen_sol_eq.subs({f_infty : 6, tau : 2, f(0) : 0})
analytic_par_sol_eq
```

$\displaystyle f{\left(t \right)} = 6 - 6 e^{- \frac{t}{2}}$

> And the final step is always to transform the SymPy expression into a lambda function :

```python
analytic_sol_eq = sp.lambdify(t, analytic_par_sol_eq.rhs, "numpy")
```

**Numerical Method**

> We should first create a python function (anonymous function) for our equilibrium reaction :

```python
eq = lambda f,f_oo,tau : (f_oo - f)/ tau
```

> Let's assign some values for our constants

```python
f_const, finfty_const, tau_const = 0, 6, 2
```

> And the numerical solution using Euler's method will be :

```python
numeric_sol_eq = list(euler(f_const, finfty_const, tau_const, equation=eq))
```

**Plotting the solutions**

```python
plot_solutions(time, numeric_sol_eq, analytic_sol_eq,
               'Chemical equilibrium $f(0) = 0$')
```

# Michaelis-Menten-Henri Equation

- Considering the following reaction :
$$ S + E \overset{k_1}{\underset{k_{-1}}\rightleftharpoons} ES \overset{k_2}{\rightarrow} P + E$$

> Whereas S : Substrate, E : Enzyme, ES : Enzyme-Substrate Complex, P : Product

**The Michaelis-Menten equation** is :

$$\boxed{ v = \frac{d[P]}{dt} = \frac{V_{max}[S]}{K_m + [S]}}$$
> Where $V_{max} = k_2[E]$ and $K_m$ is the Michaelis constant, it is the concentration of the substrate $[S]$ when the initial rate $v$ is equal to $\frac{V_{max}}{2}$

- If we wanted to graphically determine the constants $K_m$ and $V_{max}$, we're going to use **the Lineweaver-Burk representation** of this equation which we obtain by inversing the earlier equation :

$$ \boxed{\frac{1}{v} = \frac{K_m}{V_{max}}\frac{1}{[S]} + \frac{1}{V_{max}}}$$

> Let's first start by implementing our two equations :

```python
def mich_ment(substrate, vmax, km) :
    """This function returns the rate of the reaction v
    from the Michaelis-Menten equation.
    Substrate : an array of substrate concentrations
    vmax, km : constants of the MM equation, type int"""
    return (vmax*substrate)/(km + substrate)

def line_burk(substrate, vmax, km) :
    """The Lineweaver-Burk representation, it sends back 1/v
    Substrate : an array of substrate concentrations
    vmax, km : constants of the MM equation, type int"""
    return (km/(vmax*substrate)) + (1/vmax)
```

> Now let's take a look at how they are grapically represented :

```python
s = np.arange(0.1,10, 0.1) # Substrate concentrations between 0.1 and 10

fig, ax = plt.subplots(1, 2, figsize = (12,5), dpi = 150)
plt.subplot(121) # Michaelis-Menten plot
plt.plot(s, mich_ment(s,vmax = 6, km= 1)) # v-s curve
plt.axhline(y = 6, linestyle = '--', color = 'y', label = '$V_{max}$') # Vmax
plt.axvline(x = 1, linestyle = '--', color = 'r')
plt.plot(1,3, 'og' , label = '$K_m = 50 \% V{max}$') # Km
plt.title('The Michaelis-Menten equation')
plt.xlabel(r'$[S]$')
plt.ylabel(r'$v$', rotation = 0)
plt.legend()
plt.subplot(122) # Lineweaver-Burk plot
plt.plot(1/s, line_burk(s,vmax = 6, km= 1)) # 1/v - 1/s curve
plt.title('The Lineweaver-Burk representation')
plt.axvline(x = 1, linestyle = '--', color = 'r')
plt.plot(1,1/3, 'og' , label = r'$\frac{1}{K_m}$') # 1/Km,  1/Vmax = 1/3
plt.xlabel(r'$\frac{1}{[S]}$')
plt.ylabel(r'$\frac{1}{v}$', rotation = 0)
plt.legend()
```


# Tutorial Overview
### This tutorial is divided into 4 parts; they are:

- Expected Value
- Variance
- Covariance
- Covariance Matrix

# Expected Value

## Finite case

Let $X$ be a random variable with a finite number of finite outcomes $x_1, x_2, \ldots, x_k$ occurring with probabilities $p_1, p_2, \ldots, p_k,$ respectively. The 'expectation' of $X$ is defined as

$\operatorname{E}[X] =\sum_{i=1}^k x_i\,p_i=x_1p_1 + x_2p_2 + \cdots + x_kp_k.$

Since all probabilities $p_i$ add up to 1 ($p_1 + p_2 + \cdots + p_k = 1$), the expected value is the weighted average, with $p_i$’s being the weights.

## Dice Roll Example

```python
import numpy
import matplotlib.pyplot as plt
```

```python
N = 100000
```

```python
roll = numpy.zeros(N, dtype=int)
```

```python
expectation = numpy.zeros(N)
```

```python
for i in range(N):
    roll[i] = numpy.random.randint(1, 7)
```

```python
for i in range(1, N):
    expectation[i] = numpy.mean(roll[0:i])
```

```python
plt.plot(expectation)
plt.title("Expectation of a dice roll");
```

# Variance

## Definition
The variance of a random variable $X$ is the expected value of the squared deviation from the Expected value|mean of $X$
$$\mu = \operatorname{E}[X]$$:
$$ \operatorname{Var}(X) = \operatorname{E}\left[(X - \mu)^2 \right]. $$

This definition encompasses random variables that are generated by processes that are discrete random variable|discrete, continuous random variable|continuous, Cantor distribution|neither, or mixed. The variance can also be thought of as the covariance of a random variable with itself:

$$\operatorname{Var}(X) = \operatorname{Cov}(X, X).$$
The variance is also equivalent to the second cumulant of a probability distribution that generates $X$. The variance is typically designated as $\operatorname{Var}(X)$, $\sigma^2_X$, or simply $\sigma^2$ (pronounced "sigma squared"). The expression for the variance can be expanded:
$$\begin{align}
\operatorname{Var}(X) &= \operatorname{E}\left[(X - \operatorname{E}[X])^2\right] \\[4pt]
&= \operatorname{E}\left[X^2 - 2X\operatorname{E}[X] + \operatorname{E}[X]^2\right] \\[4pt]
&= \operatorname{E}\left[X^2\right] - 2\operatorname{E}[X]\operatorname{E}[X] + \operatorname{E}[X]^2 \\[4pt]
&= \operatorname{E}\left[X^2 \right] - \operatorname{E}[X]^2
\end{align}$$

In other words, the variance of $X$ is equal to the mean of the square of $X$ minus the square of the mean of $X$.  This equation should not be used for computations using floating point arithmetic because it suffers from catastrophic cancellation if the two components of the equation are similar in magnitude. There exist Algorithms for calculating variance|numerically stable alternatives.

## Dice Roll Example again

```python
N = 1000
```

```python
roll = numpy.zeros(N, dtype=int)
```

```python
variance = numpy.zeros(N)
```

```python
for i in range(N):
    roll[i] = numpy.random.randint(1, 7)
```

```python
for i in range(1, N):
    variance[i] = numpy.var(roll[0:i])
```

```python
plt.plot(variance)
plt.title("Variance of a dice roll");
```

# Covariance

## Definition
For two jointly distributed real real-valued random variables $X$ and $Y$ with finite second moments, the covariance is defined as the expected value (or mean) of the product of their deviations from their individual expected values
$$\operatorname{cov}(X,Y) = \operatorname{E}{\big[(X - \operatorname{E}[X])(Y - \operatorname{E}[Y])\big]}$$

where $\operatorname{E}[X]$ is the expected value of $X$, also known as the mean of $X$. The covariance is also sometimes denoted $\sigma_{XY}$ or $\sigma(X,Y)$, in analogy to variance.  By using the linearity property of expectations, this can be simplified to the expected value of their product minus the product of their expected values:
$$
\begin{align}
\operatorname{cov}(X, Y)
&= \operatorname{E}\left[\left(X - \operatorname{E}\left[X\right]\right) \left(Y - \operatorname{E}\left[Y\right]\right)\right] \\
&= \operatorname{E}\left[X Y - X \operatorname{E}\left[Y\right] - \operatorname{E}\left[X\right] Y + \operatorname{E}\left[X\right] \operatorname{E}\left[Y\right]\right] \\
&= \operatorname{E}\left[X Y\right] - \operatorname{E}\left[X\right] \operatorname{E}\left[Y\right] - \operatorname{E}\left[X\right] \operatorname{E}\left[Y\right] + \operatorname{E}\left[X\right] \operatorname{E}\left[Y\right] \\
&= \operatorname{E}\left[X Y\right] - \operatorname{E}\left[X\right] \operatorname{E}\left[Y\right],
\end{align}
$$
but this equation is susceptible to Loss of catastrophic cancellation.

The units of measurement of the covariance $\operatorname{cov}(X,Y)$ are those of $X$ times those of $Y$. By contrast, correlation coefficients, which depend on the covariance, are a dimensionless measure of linear dependence. (In fact, correlation coefficients can simply be understood as a normalized version of covariance.)

## Iris Dataset Example

```python
import pandas as pd
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
import seaborn.apionly as sns
from sklearn import datasets

%matplotlib inline
```

```python
iris_data = datasets.load_iris()
```

```python
iris_data.keys()
```

    dict_keys(['data', 'target', 'target_names', 'DESCR', 'feature_names', 'filename'])

```python
df = pd.concat(
    [
        pd.DataFrame(iris_data["data"], columns=iris_data["feature_names"]),
        pd.DataFrame(iris_data["target"], columns=["target"]),
    ],
    axis=1,
)
```

```python
df.head()
```

      sepal length (cm)
      sepal width (cm)
      petal length (cm)
      petal width (cm)
      target

      0
      5.1
      3.5
      1.4
      0.2
      0

      1
      4.9
      3.0
      1.4
      0.2
      0

      2
      4.7
      3.2
      1.3
      0.2
      0

      3
      4.6
      3.1
      1.5
      0.2
      0

      4
      5.0
      3.6
      1.4
      0.2
      0

```python
iris_data["target_names"]
```

    array(['setosa', 'versicolor', 'virginica'], dtype='<U10')

```python
df.target = df.target.apply(lambda x: iris_data["target_names"][x])
```

```python
df.head()
```

      sepal length (cm)
      sepal width (cm)
      petal length (cm)
      petal width (cm)
      target

      0
      5.1
      3.5
      1.4
      0.2
      setosa

      1
      4.9
      3.0
      1.4
      0.2
      setosa

      2
      4.7
      3.2
      1.3
      0.2
      setosa

      3
      4.6
      3.1
      1.5
      0.2
      setosa

      4
      5.0
      3.6
      1.4
      0.2
      setosa

### create scatterplot matrix

```python
fig = sns.pairplot(data=df, hue="target")

plt.show()
```

```python
X = df[df.columns[:-1]].values
X.shape
```

    (150, 4)

## Sample Covariance
measures how two variables differ from their mean
positive covariance: that the two variables are both above or both below their respective means
variables with a positive covariance are positively "correlated" -- they go up or done together
negative covariance: valuables from one variable tends to be above the mean and the other below their mean
in other words, negative covariance means that if one variable goes up, the other variable goes down
$$\sigma_{x,y} = \frac{1}{n-1} \sum_{i=1}^{n}(x_i - \bar{x})(y_i - \bar{y})$$
note that similar to variance, the dimension of the covariance is $unit^2$
covariance can be understood as the "variability due to codependence" whereas the variance is the "independent variability"

```python
x_mean, y_mean = np.mean(X[:, 2:4], axis=0)
```

```python
sum([(x - x_mean) * (y - y_mean) for x, y in zip(X[:, 2], X[:, 3])]) / (X.shape[0] - 1)
```

    1.2956093959731545

# Covariance Matrix

## Definition
Throughout this article, boldfaced unsubscripted $\mathbf{X}$ and $\mathbf{Y}$ are used to refer to random vectors, and unboldfaced subscripted $X_i$ and $Y_i$ are used to refer to scalar random variables.

If the entries in the column vector

$\mathbf{X}=(X_1, X_2, ... , X_n)^{\mathrm T}$

are random variables, each with finite variance and expected value, then the covariance matrix $\operatorname{K}_{\mathbf{X}\mathbf{X}}$ is the matrix whose $(i,j)$ entry is the covariance

$\operatorname{K}_{X_i X_j} = \operatorname{cov}[X_i, X_j] = \operatorname{E}[(X_i - \operatorname{E}[X_i])(X_j - \operatorname{E}[X_j])]$

where the operator $\operatorname{E}$ denotes the expected value (mean) of its argument.

In other words,

$
\operatorname{K}_{\mathbf{X}\mathbf{X}}=
\begin{bmatrix}
 \mathrm{E}[(X_1 - \operatorname{E}[X_1])(X_1 - \operatorname{E}[X_1])] & \mathrm{E}[(X_1 - \operatorname{E}[X_1])(X_2 - \operatorname{E}[X_2])] & \cdots & \mathrm{E}[(X_1 - \operatorname{E}[X_1])(X_n - \operatorname{E}[X_n])] \\ \\
 \mathrm{E}[(X_2 - \operatorname{E}[X_2])(X_1 - \operatorname{E}[X_1])] & \mathrm{E}[(X_2 - \operatorname{E}[X_2])(X_2 - \operatorname{E}[X_2])] & \cdots & \mathrm{E}[(X_2 - \operatorname{E}[X_2])(X_n - \operatorname{E}[X_n])] \\ \\
 \vdots & \vdots & \ddots & \vdots \\ \\
 \mathrm{E}[(X_n - \operatorname{E}[X_n])(X_1 - \operatorname{E}[X_1])] & \mathrm{E}[(X_n - \operatorname{E}[X_n])(X_2 - \operatorname{E}[X_2])] & \cdots & \mathrm{E}[(X_n - \operatorname{E}[X_n])(X_n - \operatorname{E}[X_n])]
\end{bmatrix}
$

### Covariance of Iris dataset Features

```python
numpy.cov(X.T)
```

    array([[ 0.68569351, -0.042434  ,  1.27431544,  0.51627069],
           [-0.042434  ,  0.18997942, -0.32965638, -0.12163937],
           [ 1.27431544, -0.32965638,  3.11627785,  1.2956094 ],
           [ 0.51627069, -0.12163937,  1.2956094 ,  0.58100626]])

```python
covariance_matrix = pd.DataFrame(
    numpy.cov(X.T), columns=iris_data["feature_names"], index=iris_data["feature_names"]
)
```

```python
covariance_matrix
```

      sepal length (cm)
      sepal width (cm)
      petal length (cm)
      petal width (cm)

      sepal length (cm)
      0.685694
      -0.042434
      1.274315
      0.516271

      sepal width (cm)
      -0.042434
      0.189979
      -0.329656
      -0.121639

      petal length (cm)
      1.274315
      -0.329656
      3.116278
      1.295609

      petal width (cm)
      0.516271
      -0.121639
      1.295609
      0.581006

## Heatmap of Covariance Matrix

```python
f, ax = plt.subplots(figsize=(11, 15))
heatmap = sns.heatmap(
    covariance_matrix,
    square=True,
    linewidths=0.5,
    cmap="coolwarm",
    cbar_kws={"shrink": 0.4, "ticks": [-1, -0.5, 0, 0.5, 1]},
    vmin=-1,
    vmax=1,
    annot=True,
    annot_kws={"size": 12},
)

# add the column names as labels
ax.set_yticklabels(covariance_matrix.columns, rotation=0)
ax.set_xticklabels(covariance_matrix.columns)
sns.set_style({"xtick.bottom": True}, {"ytick.left": True})
```

content/week-06/samples/Expectation Variance and Covariance Using NumPy.ipynb


We place conjugate prior distributions on the unknown mean and variance, i.e. the mean also follows a Gaussian distribution while the precision follows a [[gamma distribution]].  In other words:

$$\begin{align}
\mu & \sim \mathcal{N}(\mu_0, (\lambda_0 \tau)^{-1}) \\
\tau & \sim \operatorname{Gamma}(a_0, b_0) \\
\{x_1, \dots, x_N\} & \sim \mathcal{N}(\mu, \tau^{-1}) \\
N &= \text{number of data points}
\end{align}$$

We are given $N$ data points $\mathbf{X} = \{x_{1}, \dots, x_N\}$ and our goal is to infer the [[posterior distribution]] $q(\mu, \tau)=p(\mu,\tau\mid x_{1}, \ldots, x_N)$ of the parameters $\mu$ and $\tau$.

The [[hyperparameter]]s $\mu_0$, $\lambda_0$, $a_0$ and $b_0$ are fixed, given values.  They can be set to small positive numbers to give broad prior distributions indicating ignorance about the prior distributions of $\mu$ and $tau$.

## The joint probability

The [[joint probability]] of all variables can be rewritten as

$$p(\mathbf{X},\mu,\tau) = p(\mathbf{X}\mid \mu,\tau) p(\mu\mid \tau) p(\tau)$$

where the individual factors are

$$\begin{align}
p(\mathbf{X}\mid \mu,\tau) & = \prod_{n=1}^N \mathcal{N}(x_n\mid \mu,\tau^{-1}) \\
p(\mu\mid \tau) & = \mathcal{N}(\mu\mid \mu_0, (\lambda_0 \tau)^{-1}) \\
p(\tau) & = \operatorname{Gamma}(\tau\mid a_0, b_0)
\end{align}$$

where

$$\begin{align}
\mathcal{N}(x\mid \mu,\sigma^2) & = \frac{1}{\sqrt{2\pi\sigma^2}} e^{\frac{-(x-\mu)^2}{2\sigma^2}} \\
\operatorname{Gamma}(\tau\mid a,b) & = \frac{1}{\Gamma(a)} b^a \tau^{a-1} e^{-b \tau}
\end{align}$$

## Factorized approximation
Assume that $q(\mu,\tau) = q(\mu)q(\tau)$, i.e. that the posterior distribution factorizes into independent factors for $\mu$ and $\tau$.  This type of assumption underlies the variational Bayesian method.  The true posterior distribution does not in fact factor this way (in fact, in this simple case, it is known to be a [[Gaussian-gamma distribution]]), and hence the result we obtain will be an approximation.

## Derivation of q($\mu$)

Then

$$\begin{align}
\ln q_\mu^*(\mu) &= \operatorname{E}_{\tau}\left[\ln p(\mathbf{X}\mid \mu,\tau) + \ln p(\mu\mid \tau) + \ln p(\tau)\right] + C \\\\
 &= \operatorname{E}_{\tau}\left[\ln p(\mathbf{X}\mid \mu,\tau)\right] + \operatorname{E}_{\tau}\left[\ln p(\mu\mid \tau)\right] + \operatorname{E}_{\tau}\left[\ln p(\tau)\right] + C \\\\
 &= \operatorname{E}_{\tau}\left[\ln \prod_{n=1}^N \mathcal{N}(x_n\mid \mu,\tau^{-1})\right] + \operatorname{E}_{\tau}\left[\ln \mathcal{N}(\mu\mid \mu_0, (\lambda_0 \tau)^{-1})\right] + C_2 \\\\
 &= \operatorname{E}_{\tau}\left[\ln \prod_{n=1}^N \sqrt{\frac{\tau}{2\pi}} e^{-\frac{(x_n-\mu)^2\tau}{2}}\right] + \operatorname{E}_{\tau}\left[\ln \sqrt{\frac{\lambda_0 \tau}{2\pi}} e^{-\frac{(\mu-\mu_0)^2\lambda_0 \tau}{2}}\right] + C_2 \\\\
 &= \operatorname{E}_{\tau}\left[\sum_{n=1}^N \left(\frac{1}{2}(\ln\tau - \ln 2\pi) - \frac{(x_n-\mu)^2\tau}{2})\right)\right] + \operatorname{E}_{\tau}\left[\frac{1}{2}(\ln \lambda_0 + \ln \tau - \ln 2\pi) - \frac{(\mu-\mu_0)^2\lambda_0 \tau}{2}\right] + C_2 \\\\
 &= \operatorname{E}_{\tau}\left[\sum_{n=1}^N -\frac{(x_n-\mu)^2\tau}{2}\right] + \operatorname{E}_{\tau}\left[-\frac{(\mu-\mu_0)^2\lambda_0 \tau}{2}\right] + \operatorname{E}_{\tau}\left[\sum_{n=1}^N \frac{1}{2}(\ln\tau - \ln 2\pi)\right] + \operatorname{E}_{\tau}\left[\frac{1}{2}(\ln \lambda_0 + \ln \tau - \ln 2\pi)\right] + C_2 \\\\
 &= \operatorname{E}_{\tau}\left[\sum_{n=1}^N -\frac{(x_n-\mu)^2\tau}{2}\right] + \operatorname{E}_{\tau}\left[-\frac{(\mu-\mu_0)^2\lambda_0 \tau}{2}\right] + C_3 \\\\
 &= - \frac{\operatorname{E}_{\tau}[\tau]}{2} \left\{ \sum_{n=1}^N (x_n-\mu)^2 + \lambda_0(\mu-\mu_0)^2 \right\} + C_3
\end{align}$$

In the above derivation, $C$, $C_2$ and $C_3$ refer to values that are constant with respect to $\mu$.  Note that the term $\operatorname{E}_{\tau}[\ln p(\tau)]$ is not a function of $\mu$ and will have the same value regardless of the value of $\mu$.  Hence in line 3 we can absorb it into the constant term at the end.  We do the same thing in line 7.

The last line is simply a quadratic polynomial in $\mu$.  Since this is the logarithm of $q_\mu^*(\mu)$, we can see that $q_\mu^*(\mu)$ itself is a [[Gaussian distribution]].

With a certain amount of tedious math (expanding the squares inside of the braces, separating out and grouping the terms involving $\mu$ and $\mu^2$ and [[completing the square]] over $\mu$), we can derive the parameters of the Gaussian distribution:

$$
\begin{align}
\ln q_\mu^*(\mu) &= - \frac{\operatorname{E}_{\tau}[\tau]}{2} \{ \sum_{n=1}^N (x_n-\mu)^2 + \lambda_0(\mu-\mu_0)^2 \} + C_3 \\
                    &= - \frac{\operatorname{E}_{\tau}[\tau]}{2} \{ \sum_{n=1}^N (x_n^2-2x_n\mu + \mu^2) + \lambda_0(\mu^2-2\mu_0\mu + \mu_0^2) \} + C_3 \\
                    &= - \frac{\operatorname{E}_{\tau}[\tau]}{2} \{ (\sum_{n=1}^N x_n^2)-2(\sum_{n=1}^N x_n)\mu + (\sum_{n=1}^N \mu^2) + \lambda_0\mu^2-2\lambda_0\mu_0\mu + \lambda_0\mu_0^2 \} + C_3 \\
                    &= - \frac{\operatorname{E}_{\tau}[\tau]}{2} \{ (\lambda_0+N)\mu^2 -2(\lambda_0\mu_0 + \textstyle\sum_{n=1}^N x_n)\mu + (\textstyle\sum_{n=1}^N x_n^2) + \lambda_0\mu_0^2 \} + C_3 \\
                    &= - \frac{\operatorname{E}_{\tau}[\tau]}{2} \{ (\lambda_0+N)\mu^2 -2(\lambda_0\mu_0 + \textstyle\sum_{n=1}^N x_n)\mu \} + C_4 \\
                    &= - \frac{\operatorname{E}_{\tau}[\tau]}{2} \left\{ (\lambda_0+N)\mu^2 -2\frac{\lambda_0\mu_0 + \textstyle\sum_{n=1}^N x_n}{\lambda_0+N}(\lambda_0+N) \mu \right\} + C_4 \\
                    &= - \frac{\operatorname{E}_{\tau}[\tau]}{2} \left\{ (\lambda_0+N)\left(\mu^2 -2\frac{\lambda_0\mu_0 + \textstyle\sum_{n=1}^N x_n}{\lambda_0+N} \mu\right) \right\} + C_4 \\
                    &= - \frac{\operatorname{E}_{\tau}[\tau]}{2} \left\{ (\lambda_0+N)\left(\mu^2 -2\frac{\lambda_0\mu_0 + \textstyle\sum_{n=1}^N x_n}{\lambda_0+N} \mu + \left(\frac{\lambda_0\mu_0 + \textstyle\sum_{n=1}^N x_n}{\lambda_0+N}\right)^2 - \left(\frac{\lambda_0\mu_0 + \textstyle\sum_{n=1}^N x_n}{\lambda_0+N}\right)^2\right) \right\} + C_4 \\
                    &= - \frac{\operatorname{E}_{\tau}[\tau]}{2} \left\{ (\lambda_0+N)\left(\mu^2 -2\frac{\lambda_0\mu_0 + \textstyle\sum_{n=1}^N x_n}{\lambda_0+N} \mu + \left(\frac{\lambda_0\mu_0 + \textstyle\sum_{n=1}^N x_n}{\lambda_0+N}\right)^2 \right) \right\} + C_5 \\
                    &= - \frac{\operatorname{E}_{\tau}[\tau]}{2} \left\{ (\lambda_0+N)\left(\mu-\frac{\lambda_0\mu_0 + \textstyle\sum_{n=1}^N x_n}{\lambda_0+N}\right)^2 \right\} + C_5 \\
                    &= - \frac{1}{2} \left\{ (\lambda_0+N)\operatorname{E}_{\tau}[\tau] \left(\mu-\frac{\lambda_0\mu_0 + \textstyle\sum_{n=1}^N x_n}{\lambda_0+N}\right)^2 \right\} + C_5 \\
\end{align}$$

Note that all of the above steps can be shortened by using the formula for the [[Normal distribution#Sum_of_two_quadratics|sum of two quadratics]].

In other words:

$$
\begin{align}
q_\mu^*(\mu) &\sim \mathcal{N}(\mu\mid \mu_N,\lambda_N^{-1}) \\
\mu_N &= \frac{\lambda_0 \mu_0 + N \bar{x}}{\lambda_0 + N} \\
\lambda_N &= (\lambda_0 + N) \operatorname{E}[\tau] \\
\bar{x} &= \frac{1}{N}\sum_{n=1}^N x_n
\end{align}$$

## Derivation of q($\tau$)

The derivation of $q_\tau^*(\tau)$ is similar to above, although we omit some of the details for the sake of brevity.

$$\begin{align}
\ln q_\tau^*(\tau) &= \operatorname{E}_{\mu}[\ln p(\mathbf{X}\mid \mu,\tau) + \ln p(\mu\mid \tau)] + \ln p(\tau) + \text{constant} \\
                    &= (a_0 - 1) \ln \tau - b_0 \tau + \frac{1}{2} \ln \tau + \frac{N}{2} \ln \tau - \frac{\tau}{2} \operatorname{E}_\mu [\sum_{n=1}^N (x_n-\mu)^2 + \lambda_0(\mu - \mu_0)^2] + \text{constant}
\end{align}$$

Exponentiating both sides, we can see that $q_\tau^*(\tau)$ is a [[gamma distribution]].  Specifically:

$$\begin{align}
q_\tau^*(\tau) &\sim \operatorname{Gamma}(\tau\mid a_N, b_N) \\
a_N &= a_0 + \frac{N+1}{2} \\
b_N &= b_0 + \frac{1}{2} \operatorname{E}_\mu \left[\sum_{n=1}^N (x_n-\mu)^2 + \lambda_0(\mu - \mu_0)^2\right]
\end{align}$$

## Algorithm for computing the parameters

Let us recap the conclusions from the previous sections:

$$\begin{align}
q_\mu^*(\mu) &\sim \mathcal{N}(\mu\mid\mu_N,\lambda_N^{-1}) \\
\mu_N &= \frac{\lambda_0 \mu_0 + N \bar{x}}{\lambda_0 + N} \\
\lambda_N &= (\lambda_0 + N) \operatorname{E}[\tau] \\
\bar{x} &= \frac{1}{N}\sum_{n=1}^N x_n
\end{align}$$

and

$$\begin{align}
q_\tau^*(\tau) &\sim \operatorname{Gamma}(\tau\mid a_N, b_N) \\
a_N &= a_0 + \frac{N+1}{2} \\
b_N &= b_0 + \frac{1}{2} \operatorname{E}_\mu \left[\sum_{n=1}^N (x_n-\mu)^2 + \lambda_0(\mu - \mu_0)^2\right]
\end{align}$$

In each case, the parameters for the distribution over one of the variables depend on expectations taken with respect to the other variable.  We can expand the expectations, using the standard formulas for the expectations of moments of the Gaussian and gamma distributions:

$$\begin{align}
\operatorname{E}[\tau\mid a_N, b_N] &= \frac{a_N}{b_N} \\
\operatorname{E}[\mu\mid\mu_N,\lambda_N^{-1}] &= \mu_N \\
\operatorname{E}\left[X^2 \right] &= \operatorname{Var}(X) + (\operatorname{E}[X])^2 \\
\operatorname{E}[\mu^2\mid\mu_N,\lambda_N^{-1}] &= \lambda_N^{-1} + \mu_N^2
\end{align}$$

Applying these formulas to the above equations is trivial in most cases, but the equation for $b_N$ takes more work:

$$\begin{align}
b_N &= b_0 + \frac{1}{2} \operatorname{E}_\mu \left[\sum_{n=1}^N (x_n-\mu)^2 + \lambda_0(\mu - \mu_0)^2\right] \\
 &= b_0 + \frac{1}{2} \operatorname{E}_\mu \left[ (\lambda_0+N)\mu^2 -2(\lambda_0\mu_0 + \textstyle\sum_{n=1}^N x_n)\mu + (\textstyle\sum_{n=1}^N x_n^2) + \lambda_0\mu_0^2 \right] \\
 &= b_0 + \frac{1}{2} \left[ (\lambda_0+N)\operatorname{E}_\mu[\mu^2] -2(\lambda_0\mu_0 + \textstyle\sum_{n=1}^N x_n)\operatorname{E}_\mu[\mu] + (\textstyle\sum_{n=1}^N x_n^2) + \lambda_0\mu_0^2 \right] \\
 &= b_0 + \frac{1}{2} \left[ (\lambda_0+N)(\lambda_N^{-1} + \mu_N^2) -2(\lambda_0\mu_0 + \textstyle\sum_{n=1}^N x_n)\mu_N + (\textstyle\sum_{n=1}^N x_n^2) + \lambda_0\mu_0^2 \right] \\
\end{align}$$

We can then write the parameter equations as follows, without any expectations:

$$\begin{align}
\mu_N &= \frac{\lambda_0 \mu_0 + N \bar{x}}{\lambda_0 + N} \\
\lambda_N &= (\lambda_0 + N) \frac{a_N}{b_N} \\
\bar{x} &= \frac{1}{N}\sum_{n=1}^N x_n \\
a_N &= a_0 + \frac{N+1}{2} \\
b_N &= b_0 + \frac{1}{2} \left[ (\lambda_0+N)(\lambda_N^{-1} + \mu_N^2) -2(\lambda_0\mu_0 + \textstyle\sum_{n=1}^N x_n)\mu_N + (\textstyle\sum_{n=1}^N x_n^2) + \lambda_0\mu_0^2 \right]
\end{align}$$

Note that there are circular dependencies among the formulas for $\mu_N$, $\lambda_N$ and $b_N$.  This naturally suggests an [[expectation-maximization algorithm|EM]]-like algorithm:

+ Compute $\sum_{n=1}^N x_n$ and $\sum_{n=1}^N x_n^2.$ Use these values to compute $\mu_N$ and $a_N.$
+ Initialize $\lambda_N$ to some arbitrary value.
+ Use the current value of $\lambda_N,$ along with the known values of the other parameters, to compute $b_N$.
+ Use the current value of $b_N,$ along with the known values of the other parameters, to compute $\lambda_N$.
+ Repeat the last two steps until convergence (i.e. until neither value has changed more than some small amount).

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
from numpy import random
```

```python
true_mu = 130
true_tau = 0.01
N = 1000000
samples = random.normal(true_mu,1/true_tau,N)
_ = plt.hist(samples)
```

```python
from scipy.stats import norm
import matplotlib.mlab as mlab
```

```python
a_0 = 100
b_0 = 20

lambda_0 = 100
mu_0 = -100

x_ = np.sum(samples)
x__ = np.sum(samples**2)

def VB(samples,lambda_N):
  global x_,x__
  N = samples.shape[0]
  mu_N = (lambda_0*mu_0+x_)/(lambda_0+N)
  a_N = a_0*(N+1)/2.0
  b_N = b_0+(1.0/2)*((lambda_0+N)*(1.0/lambda_N+mu_N**2)-2.0*(lambda_0*mu_0+x_)*mu_N+x__+lambda_0*mu_0**2)
  lambda_N_new = (lambda_0+N) * a_N/b_N
  return lambda_N_new,a_N,b_N

plt.hist(samples,weights=np.repeat(1.0/samples.shape[0], samples.shape[0]),normed=True)
plt.plot(np.arange(-1000,1000,1)+true_mu,mlab.normpdf(np.arange(-1000,1000,1)+true_mu,true_mu,1.0/true_tau),'b',
         linewidth=2)
lamb = 0.001
a_N,b_N = 0,0
for i in range(5):
  lamb_,a_N,b_N = VB(samples, lamb)
  mu_N = (lambda_0*mu_0+x_)/(lambda_0+N)

  print("Tau",a_N/b_N)
  plt.plot(np.arange(-1000,1000,1)+true_mu,norm.pdf(np.arange(-1000,1000,1)+true_mu,mu_N,b_N/a_N),linewidth=2)
  lamb = lamb_
```