# COMP 135 Spring 2019: HW2 Solution Notebook

## Setup comp135_env package imports¶

```python
import os
import numpy as np
import warnings

import sklearn.preprocessing
import sklearn.pipeline
import sklearn.linear_model
import sklearn.metrics
import sklearn.model_selection

from matplotlib import pyplot as plt
import seaborn as sns
```

```python
%matplotlib inline
```

## Setup student-defined imports

```python
# None needed
```

## Load Auto dataset for Problem 1

```python
x_tr_MF = np.loadtxt('data_auto/x_train.csv', delimiter=',', skiprows=1)
x_va_NF = np.loadtxt('data_auto/x_valid.csv', delimiter=',', skiprows=1)
x_te_PF = np.loadtxt('data_auto/x_test.csv', delimiter=',', skiprows=1)
```

```python
y_tr_M = np.loadtxt('data_auto/y_train.csv', delimiter=',', skiprows=1)
y_va_N = np.loadtxt('data_auto/y_valid.csv', delimiter=',', skiprows=1)
y_te_P = np.loadtxt('data_auto/y_test.csv', delimiter=',', skiprows=1)
```

## Problem 1a: Polynomial Degree Selection on Fixed Validation Set

#### **1a(i):** Plot the number of polynomial features $G$ vs. degree $D$

```python
degree_list = [1, 2, 3, 4, 5, 6, 7, 8]
num_poly_feat_list = []
for degree in degree_list:
    poly_transformer = sklearn.preprocessing.PolynomialFeatures(
        degree=degree, include_bias=False)
    # TODO make the plot
```

### 1a(ii): Fit a linear regression model to a polynomial feature transformation of the provided training set of $x$, $y$ values at each of these possible degrees: [1, 2, 3, 4, 5, 6].

```python
degree_list = [1, 2, 3, 4, 5, 6]
err_tr_list = []
err_va_list = []
coef_list = []
for degree in degree_list:
    poly_transformer = sklearn.preprocessing.PolynomialFeatures(
        degree=degree, include_bias=False)
    # TODO fit models and make the plot
    # Be sure to track the coefs of the learned weights
```

### **1a(iii):** Based on this plot, which single degree value do you recommend? Why?

TODO answer here

### **1a(iv):** Report the numerical values of the 5th percentile and 95th percentile of the coefficients observed in your linear regression model for degrees 3, 4, 5, and 6. What seems to be happening?

```python
# TODO answer here
```

### **1a(v):** Comment on the training error observed at degree 6. Based on your formulas from **1a(i)** and your knowledge of linear regression, what **should** the training error be at degree 6? What do you think is happening instead?

TODO answer here

```python

```

# 1b: Linear regression with Rescaled Polynomial Bases

Setup code below to make a pipeline

```python
def make_pipeline(degree=None):
    pipeline = sklearn.pipeline.Pipeline(
        steps=[
         ('rescaler', sklearn.preprocessing.MinMaxScaler()),
         ('poly_transformer', sklearn.preprocessing.PolynomialFeatures(degree=degree, include_bias=False)),
         ('poly_rescaler', sklearn.preprocessing.MinMaxScaler()),
         ('linear_regr', sklearn.linear_model.LinearRegression()),
        ])

    # Define utility method that performs all the feature transforms of the pipeline in order.
    def calc_features(x):
        ''' Maps input features to rescaled polynomial features
        '''
        feat = x
        for step_name, step_tfmr in pipeline.steps[:-1]:
            feat = step_tfmr.transform(feat)
        return feat
    pipeline.calc_features = calc_features

    # Define utility method that accesses the linear regression weights
    pipeline.get_linear_regr_weights = lambda: pipeline.named_steps['linear_regr'].coef_

    # Return the constructed pipeline
    # We can treat it as if it has a 'regression' API
    # e.g. a fit and a predict method
    return pipeline
```

```python
degree_list = [1, 2, 3, 4, 5, 6]
err_tr_list = []
err_va_list = []
pipeline_list = []

# TODO fit model at each degree and track error metrics
```

#### 1b(i): Plot MSE for *rescaled* polynomial features vs. degrees

```python
# TODO make plot here
```

### **1b(ii):** Using this new analysis, which degree do you recommend?

ANSWER HERE

### **1b(iii):** Report the numerical values of the 5th percentile and 95th percentile of the coefficients observed in this most recent linear regression model at all degrees. What seems to be happening? What's different than in **1a**?

```python
# TODO Code here
```

**Answer**: goes here

# 1c: Cross Validation setup

Combine original train/valid splits into one mega dataset

```python
x_trva_LF = np.vstack([x_tr_MF, x_va_NF])
y_trva_L = np.hstack([y_tr_M, y_va_N])
```

### **Coding Step 1/1:** Complete the starter code function

```python
def calc_mean_squared_error_across_k_folds(
        model, x_LF, y_L, K=5, random_state=0):
    ''' Calculate mean squared error on K cross-validation folds

    Args
    ----
    model: sklearn predictor object
        has `fit` and `predict` methods
    x_LF : 2D array, size n_examples x n_features

    y_L : 1D array, size n_examples
    K : int
    random_state : int or np.random.RandomState instance

    Returns
    -------
    mse_scores_K : 1D array, size K
        estimate of MSE on each of the K folds
    '''
    kfold_iterator = sklearn.model_selection.KFold(
        n_splits=K, shuffle=True, random_state=random_state)

    # TODO compute mse on each of the K folds
```

### **1c(i):** Using your `calc_mean_squared_error_across_k_folds` function with 10 folds, make a line plot of the *average* mean-squared-error at degrees 1, 2, 3, 4, 5, 6.

```python
# TODO
```

#### **1c(ii):** Based on this plot, what is your recommended degree? How do your recommendations differ from *1b*?

**Answer**: TODO

#### **1c(iii):** For each $K$, make a scatter plot of the $K$ fold-specific estimates of MSE ($K$ is x-axis, MSE on y-axis). Also draw a line connecting the average MSE across $K$.

```python
degree = 3
K_list = [2, 5, 10, 15, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120, 130, 140]
pipeline = make_pipeline(degree)
err_cv_list = []
avg_score_list = []

# TODO
```

#### **1c(iv):** What happens to the distribution of MSE as you use more and more folds? Is the trend observed in Plot **1c(iii)** what we expect based on your readings from the ISL textbook Chapter 5?

TODO Answer here.

```python

```

# Problem 2: Regularized Linear Regression via L2 and L1 penalties

### Setup Pipeline for Problem 2a: Grid Search for L2

```python
alpha_grid = np.logspace(-9, 6, 31)
```

```python
def make_L2regularized_pipeline(degree=2, alpha=1.0):
    pipeline = sklearn.pipeline.Pipeline(
        steps=[
         ('rescaler', sklearn.preprocessing.MinMaxScaler()),
         ('poly_transformer', sklearn.preprocessing.PolynomialFeatures(degree=degree, include_bias=False)),
         ('poly_rescaler', sklearn.preprocessing.MinMaxScaler()),
         ('linear_regr', sklearn.linear_model.Ridge(alpha=alpha)),
        ])

    # Define utility method that performs all the feature transforms of the pipeline in order.
    def calc_features(x):
        tmp = x
        for step_name, step_tfmr in pipeline.steps[:-1]:
            tmp = step_tfmr.transform(tmp)
        return tmp
    pipeline.calc_features = calc_features

    pipeline.get_linear_regr_weights = lambda: pipeline.named_steps['linear_regr'].coef_
    pipeline.get_linear_regr_bias = lambda: pipeline.named_steps['linear_regr'].intercept_

    return pipeline
```

```python

```

## Problem 2a: Grid Search for L2

### **2a(i):** Train `Ridge` regression across grid of possible L2-penalty strengths $\alpha$. Using *degree 2* polynomial features, plot the MSE vs. regularization strength on train and validation.

```python
# TODO plotting code here
```

### **2a(ii):** Make the same plot at degree 6

```python
# TODO plotting code here
```

### 2a(iii): How does the recommended $\alpha$ change from deg. 2 to 6?

**Answer**: here

### 2a(iv): Plot nonzero entries vs $\alpha$ for degree 6?

CODE HERE

## Problem 2b: Grid search for L1

```python
def make_L1regularized_pipeline(degree=2, alpha=1.0):
    pipeline = sklearn.pipeline.Pipeline(
        steps=[
         ('rescaler', sklearn.preprocessing.MinMaxScaler()),
         ('poly_transformer', sklearn.preprocessing.PolynomialFeatures(degree=degree, include_bias=False)),
         ('poly_rescaler', sklearn.preprocessing.MinMaxScaler()),
         ('linear_regr', sklearn.linear_model.Lasso(alpha=alpha, max_iter=50000)),
        ])

    # Define utility method that performs all the feature transforms of the pipeline in order.
    def calc_features(x):
        tmp = x
        for step_name, step_tfmr in pipeline.steps[:-1]:
            tmp = step_tfmr.transform(tmp)
        return tmp
    pipeline.calc_features = calc_features

    pipeline.get_linear_regr_weights = lambda: pipeline.named_steps['linear_regr'].coef_
    pipeline.get_linear_regr_bias = lambda: pipeline.named_steps['linear_regr'].intercept_

    return pipeline
```

```python
alpha_grid = np.logspace(-9, 6, 31)
degree = 3
err_tr_list = []
err_va_list = []
pipeline_list = []
for alpha in [0.0001]: # TODO loop over all alpha_grid

    # Create pipeline with specified alpha
    pipeline = make_L1regularized_pipeline(alpha=alpha, degree=degree)

    # L1 solvers are a bit more finicky than L2 solvers
    # Here's some code that will show if each optimization has converged
    with warnings.catch_warnings(record=True) as warn_list:
        pipeline.fit(x_tr_MF, y_tr_M)
        solver = pipeline.named_steps['linear_regr']
        print("alpha % .3e | %s | completed %5d iters | gap % 10.3f" % (
            alpha,
            'converged    ' if len(warn_list) == 0 else 'NOT converged',
            solver.n_iter_, solver.dual_gap_))

    # TODO evaluation code here
```

    alpha  1.000e-04 | NOT converged | completed 50000 iters | gap    548.999

```python

```

### 2b(i): Plot MSE vs. $\alpha$ for L1 ("Lasso")

```python
# TODO make plot
```

### 2b(ii): Plot number of non-zero coefficients

```python
# TODO make plot
```

### 2b(iii): Print out non-zero coefficients of learned L1-regularized models

```python
# TODO write code
```

## Problem 3

### 3a: Adding fake observations to a no-bias linear regression

**Question:**
You are given a training dataset for regression $\{x_n, y_n \}_{n=1}^N$. Each feature vector $x_n$ has size $F=4$. Each $y_n$ is a scalar.

You decide to add in the following additional "fake" observations $\tilde{x}$:

\begin{align}
\begin{array}{c c c c}
    ~\sqrt{\lambda}  & 0  & 0 & 0
 \\ 0  & \sqrt{\lambda}  & 0  & 0
 \\ 0  &0 & \sqrt{\lambda} &0
 \\ 0  & 0  & 0  &\sqrt{\lambda}
\end{array}
\end{align}

You also add in 4 "fake" responses $\tilde{y}$, all equal to zero.

You then wish to fit a least squares linear regression model with no bias/intercept term to the combined labeled dataset that is the union of $x$ and $\tilde{x}$, $y$ and $\tilde{y}$.

### **3a(i):** Write down the optimization objective and simplify as much as possible.

### **3a(ii):** Does this "add fake data" process lead to an objectie that looks familiar?

**Solution**:

TODO write using Markdown / LaTeX syntax.

Or paste in an image of a handwritten solution.

```python

```

```python

```


```python
import sympy as sp
import numpy as np
```

```python
x, y = [sp.IndexedBase(e) for e in ['x', 'y']]
m = sp.symbols('m', integer=True)
a, b = sp.symbols('a b', real=True)
i = sp.Idx('i', m)
```

```python
loss = (y[i] - (a*x[i] + b))**2
```

```python
loss
```

$\displaystyle \left(- a {x}_{Idx\left(i, \left( 0, \  m - 1\right)\right)} - b + {y}_{Idx\left(i, \left( 0, \  m - 1\right)\right)}\right)^{2}$

Having defined the loss function using indexed variables we might hope that the
implicit summation of repeated indexes might fall through to derivative
but it looks like this isn't the case.

Below we see taking derivative wrt to fit parameters is only applied
to each point rather than the whole sum, which is incorrect.

```python
sp.solve(loss.diff(a), a)
```

    [(-b + y[i])/x[i]]

```python
sp.solve(loss.diff(b), b)
```

    [-a*x[i] + y[i]]

Try adding explicit summation around the loss expression.  This gives the
correct set of equations for derivatives but a solution can't be found.

```python
sp.diff(sp.Sum(loss, i),a)
```

$\displaystyle \sum_{Idx\left(i, \left( 0, \  m - 1\right)\right)=0}^{m - 1} - 2 \left(- a {x}_{Idx\left(i, \left( 0, \  m - 1\right)\right)} - b + {y}_{Idx\left(i, \left( 0, \  m - 1\right)\right)}\right) {x}_{Idx\left(i, \left( 0, \  m - 1\right)\right)}$

```python
sp.diff(sp.Sum(loss, i), b)
```

$\displaystyle \sum_{Idx\left(i, \left( 0, \  m - 1\right)\right)=0}^{m - 1} \left(2 a {x}_{Idx\left(i, \left( 0, \  m - 1\right)\right)} + 2 b - 2 {y}_{Idx\left(i, \left( 0, \  m - 1\right)\right)}\right)$

```python
sp.solve([sp.diff(sp.Sum(loss, i),a), sp.diff(sp.Sum(loss, i),b)], [a, b])
```

    []

```python
sp.solve([loss.expand().diff(a), loss.expand().diff(b)], [a,b])
```

    {a: -b/x[i] + y[i]/x[i]}

MatrixSymbol seems to be the trick

```python
x_2 = sp.MatrixSymbol('x', m, 1)
y_2 = sp.MatrixSymbol('y', m, 1)
a_2 = sp.MatrixSymbol('a', 1, 1)
b_2 = b*sp.OneMatrix(m, 1)
```

```python
err = y_2 - (x_2*a_2 + b_2)
err
```

$\displaystyle - b \mathbb{1} - x a + y$

```python
objective = (err.T * err)
objective
```

$\displaystyle \left(- b \mathbb{1} - a^{T} x^{T} + y^{T}\right) \left(- b \mathbb{1} - x a + y\right)$

```python
objective.diff(a_2)
```

$\displaystyle - 2 x^{T} \left(- b \mathbb{1} - x a + y\right)$

```python
objective.diff(b)
```

$\displaystyle - \left(- b \mathbb{1} - a^{T} x^{T} + y^{T}\right) \mathbb{1} - \mathbb{1} \left(- b \mathbb{1} - x a + y\right)$

Functions of Matrices e.g. generator of rotations

```python
t = sp.symbols('t', real=True)
g = sp.Matrix([[0, -t], [t, 0]])
```

```python
g
```

```python
sp.exp(g)
```

$\displaystyle \left[\begin{matrix}\cos{\left(t \right)} & - \sin{\left(t \right)}\\\sin{\left(t \right)} & \cos{\left(t \right)}\end{matrix}\right]$

```python

```


# Using Neural networks to solve Ordinary Differential Equations (ODEs)

**Morten Hjorth-Jensen, MSU**

Date: **Feb 26, 2020**

## Example: Exponential decay
An exponential decay of a quantity $g(x)$ is described by the equation

$$
\begin{equation} \label{solve_expdec} \tag{1}
  g'(x) = -\gamma g(x)
\end{equation}
$$

with $g(0) = g_0$ for some chosen initial value $g_0$.

The analytical solution of ([1](#solve_expdec)) is

$$
\begin{equation}
  g(x) = g_0 \exp\left(-\gamma x\right)
\label{_auto1} \tag{2}
\end{equation}
$$

Having an analytical solution at hand, it is possible to use it to compare how well a neural network finds a solution of ([1](#solve_expdec)).

We will use a neural network to solve

$$
\begin{equation} \label{solveode} \tag{3}
g'(x) = -\gamma g(x)
\end{equation}
$$

where $g(0) = g_0$ with $\gamma$ and $g_0$ being some chosen values.

In this example, $\gamma = 2$ and $g_0 = 10$.

## The trial solution
To begin with, a trial solution $g_t(t)$ must be chosen. A general trial solution for ordinary differential equations could be

$$
g_t(x, P) = h_1(x) + h_2(x, N(x, P))
$$

with $h_1(x)$ ensuring that $g_t(x)$ satisfies some conditions and
$h_2(x,N(x, P))$ an expression involving $x$ and the output from the
neural network $N(x,P)$ with $P $ being the collection of the weights
and biases for each layer. For now, it is assumed that the network
consists of one input layer, one hidden layer, and one output layer.

## Reformulating the problem
We wish that our neural network manages to minimize a given cost function.

A reformulation of our equation must therefore be done,
such that it describes the problem a neural network can solve for.

The trial solution

$$
g_t(x, P) = g_0 + x \cdot N(x, P)
$$

has been chosen such that it already solves the condition $g(0) = g_0$. What remains, is to find $P$ such that

$$
\begin{equation} \label{nnmin} \tag{4}
g_t'(x, P) = - \gamma g_t(x, P)
\end{equation}
$$

is fulfilled as *best as possible*.

The left hand side and right hand side of ([4](#nnmin)) must be computed separately, and then the neural network must choose weights and biases, contained in $P$, such that the sides are equal as best as possible.
This means that the absolute or squared difference between the sides must be as close to zero, ideally equal to zero.
In this case, the difference squared shows to be an appropriate measurement of how erroneous the trial solution is with respect to $P$ of the neural network.

This gives the following cost function our neural network must solve for:

$$
\min_{P}\Big\{ \big(g_t'(x, P) - ( -\gamma g_t(x, P) \big)^2 \Big\}
$$

(the notation $\min_{P}\{ f(x, P) \}$ means that we desire to find $P$ that yields the minimum of $f(x, P)$)

or, in terms of weights and biases for the hidden and output layer in our network:

$$
\min_{P_{\text{hidden} }, \ P_{\text{output} }}\Big\{ \big(g_t'(x, \{ P_{\text{hidden} }, P_{\text{output} }\}) - ( -\gamma g_t(x, \{ P_{\text{hidden} }, P_{\text{output} }\}) \big)^2 \Big\}
$$

for an input value $x$.

If the neural network evaluates $g_t(x, P)$ at more values for $x$, say $N$ values $x_i$ for $i = 1, \dots, N$, then the *total* error to minimize becomes

$$
\begin{equation} \label{min} \tag{5}
\min_{P}\Big\{\frac{1}{N} \sum_{i=1}^N  \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 \Big\}
\end{equation}
$$

Letting $\vec x$ be a vector with elements $x_i$ and $c(\vec x, P) = \frac{1}{N} \sum_i  \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2$ denote the cost function, the minimization problem that our network must solve, becomes

$$
\min_{P} c(\vec x, P)
$$

In terms of $P_{\text{hidden} }$ and $P_{\text{output} }$, this could also be expressed as

$$
\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\vec x, \{P_{\text{hidden} }, P_{\text{output} }\})
$$

```python
%matplotlib inline

# # Finding eigenvalues of matrices with neural networks.
# Script for finding the eigenvectors corresponding to the largest eigenvalue of a matrix with a neural network.

import tensorflow.compat.v1 as tf
tf.disable_v2_behavior()
tf.reset_default_graph()
# tf.set_random_seed(343)

# import tensorflow as tf
import numpy as np
from matplotlib import cm
from matplotlib import pyplot as plt
from mpl_toolkits.mplot3d import axes3d
#from lib import compute_dx_dt

matrix_size = 6

A = np.random.random_sample(size=(matrix_size,matrix_size))
A = (A.T + A)/2.0
start_matrix = A

eigen_vals, eigen_vecs =  np.linalg.eig(A)

A = tf.convert_to_tensor(A)
print("A = ", A)

x_0 = tf.convert_to_tensor(np.random.random_sample(size = (1,matrix_size)))
print("x0 = ", x_0)

## The construction phase

num_iter = 10000
num_hidden_neurons = [50]
num_hidden_layers = np.size(num_hidden_neurons)

with tf.variable_scope('dnn'):

    previous_layer = x_0

    for l in range(num_hidden_layers):
        current_layer = tf.layers.dense(previous_layer, num_hidden_neurons[l],activation=tf.nn.sigmoid)
        previous_layer = current_layer

    dnn_output = tf.layers.dense(previous_layer, matrix_size)

with tf.name_scope('loss'):
    print("dnn_output = ", dnn_output)

    x_trial = tf.transpose(dnn_output)
    print("x_trial = ", x_trial)

    temp1 = (tf.tensordot(tf.transpose(x_trial), x_trial, axes=1)*A)
    temp2 = (1- tf.tensordot(tf.transpose(x_trial), tf.tensordot(A, x_trial, axes=1), axes=1))*np.eye(matrix_size)
    func = tf.tensordot((temp1-temp2), x_trial, axes=1)

    print(temp1)
    print(temp2)
    print(func)

    func = tf.transpose(func)
    x_trial = tf.transpose(x_trial)

    loss = tf.losses.mean_squared_error(func, x_trial)

learning_rate = 0.001

with tf.name_scope('train'):
    optimizer = tf.train.GradientDescentOptimizer(learning_rate)
    traning_op = optimizer.minimize(loss)

init = tf.global_variables_initializer()

g_dnn = None

losses = []

with tf.Session() as sess:
    init.run()
    for i in range(num_iter):
        sess.run(traning_op)

        if i % 100 == 0:
            l = loss.eval()
            print("Step:", i, "/",num_iter, "loss: ", l)
            losses.append(l)

    x_dnn = x_trial.eval()
x_dnn = x_dnn.T

# ## Plotting loss over time

plt.plot(losses[:5])
plt.xlabel("Iteration")
plt.ylabel("Loss")

print("Eigenvector NN = \n", (x_dnn/(x_dnn**2).sum()**0.5), "\n")

eigen_val_nn = x_dnn.T @ (start_matrix @ x_dnn) / (x_dnn.T @ x_dnn)

print("Eigenvalue NN = \n", eigen_val_nn, "\n \n")
print("Eigenvector analytic = \n", eigen_vecs)
print("\n")
print("Eigenvalues analytic = \n",eigen_vals)
```

```python

```


## Example: Autoregressive model

In this example we are going to perform an automated Variational Bayesian Inference for autoregressive model that can be represented as following:

$$
\begin{equation}
\begin{aligned}
p(\gamma) &= \mathrm{Gamma}(\gamma|a, b),\\
p(\mathbf{\theta}) &= \mathcal{N}(\mathbf{\theta}|\mathbf{\mu}, \Sigma),\\
p(x_t|\mathbf{x}_{t-1:t-k}) &= \mathcal{N}(x_t|\mathbf{\theta}^{T}\mathbf{x}_{t-1:t-k}, \gamma^{-1}),\\
p(y_t|x_t) &= \mathcal{N}(y_t|x_t, \tau^{-1}),
\end{aligned}
\end{equation}
$$

where $x_t$ is a current state of our system, $\mathbf{x}_{t-1:t-k}$ is a sequence of $k$ previous states, $k$ is an order of autoregression process, $\mathbf{\theta}$ is a vector of transition coefficients, $\gamma$ is a precision of state transition process, $y_k$ is a noisy observation of $x_k$ with precision $\tau$.

For a more rigorous introduction to Bayesian inference in Autoregressive models we refer to [Albert Podusenko, Message Passing-Based Inference for Time-Varying Autoregressive Models](https://www.mdpi.com/1099-4300/23/6/683).

We start with importing all needed packages:

```julia
using Rocket, ReactiveMP, GraphPPL
using Distributions, LinearAlgebra, Random, Plots, BenchmarkTools
```

    ┌ Info: Precompiling ReactiveMP [a194aa59-28ba-4574-a09c-4a745416d6e3]
    └ @ Base loading.jl:1342
    ┌ Info: Precompiling Plots [91a5bcdd-55d7-5caf-9e0b-520d859cae80]
    └ @ Base loading.jl:1342

Lets generate some synthetic dataset, we use a predefined sets of coeffcients for $k$ = 1, 3 and 5 respectively:

```julia
# The following coefficients correspond to stable poles
coefs_ar_1 = [-0.27002517200218096]
coefs_ar_2 = [0.4511170798064709, -0.05740081602446657]
coefs_ar_5 = [0.10699399235785655, -0.5237303489793305, 0.3068897071844715, -0.17232255282458891, 0.13323964347539288];
```

```julia
function generate_ar_data(rng, n, θ, γ, τ)
    order        = length(θ)
    states       = Vector{Vector{Float64}}(undef, n + 3order)
    observations = Vector{Float64}(undef, n + 3order)

    γ_std = sqrt(inv(γ))
    τ_std = sqrt(inv(γ))

    states[1] = randn(rng, order)

    for i in 2:(n + 3order)
        states[i]       = vcat(rand(rng, Normal(dot(θ, states[i - 1]), γ_std)), states[i-1][1:end-1])
        observations[i] = rand(rng, Normal(states[i][1], τ_std))
    end

    return states[1+3order:end], observations[1+3order:end]
end
```

    generate_ar_data (generic function with 1 method)

```julia
# Seed for reproducibility
seed = 123
rng  = MersenneTwister(seed)

# Number of observations in synthetic dataset
n = 500

# AR process parameters
real_γ = 5.0
real_τ = 5.0
real_θ = coefs_ar_5

states, observations = generate_ar_data(rng, n, real_θ, real_γ, real_τ);
```

Lets plot our synthetic dataset:

```julia
plot(first.(states), label = "Hidden states")
scatter!(observations, label = "Observations")
```

Next step is to specify probabilistic model and run inference procedure with `ReactiveMP`. We use `GraphPPL.jl` package to specify probabilistic model and additional constraints for variational Bayesian Inference. We also specify two different models for Multivariate AR with order $k$ > 1 and for Univariate AR (reduces to simple State-Space-Model) with order $k$ = 1.

```julia
@model [ default_factorisation = MeanField() ] function lar_model(T::Type{ Multivariate }, n, order, c, stype, τ)

    # Parameter priors
    γ  ~ GammaShapeRate(1.0, 1.0)
    θ  ~ MvNormalMeanPrecision(zeros(order), diageye(order))

    # We create a sequence of random variables for hidden states
    x = randomvar(n)
    # As well a sequence of observartions
    y = datavar(Float64, n)

    ct = constvar(c)
    # We assume observation noise to be known
    cτ = constvar(τ)

    # Prior for first state
    x0 ~ MvNormalMeanPrecision(zeros(order), diageye(order))

    x_prev = x0

    # AR process requires extra meta information
    meta = ARMeta(Multivariate, order, stype)

    for i in 1:n
        # Autoregressive node uses structured factorisation assumption between states
        x[i] ~ AR(x_prev, θ, γ) where { q = q(y, x)q(γ)q(θ), meta = meta }
        y[i] ~ NormalMeanPrecision(dot(ct, x[i]), cτ)
        x_prev = x[i]
    end

    return x, y, θ, γ
end
```

    lar_model (generic function with 1 method)

```julia
@model [ default_factorisation = MeanField() ] function lar_model(T::Type{Univariate}, n, order, c, stype, τ)

    # Parameter priors
    γ  ~ GammaShapeRate(1.0, 1.0)
    θ  ~ NormalMeanPrecision(0.0, 1.0)

    # We create a sequence of random variables for hidden states
    x = randomvar(n)
    # As well a sequence of observartions
    y = datavar(Float64, n)

    ct = constvar(c)
    # We assume observation noise to be known
    cτ = constvar(τ)

    # Prior for first state
    x0 ~ NormalMeanPrecision(0.0, 1.0)

    x_prev = x0

    # AR process requires extra meta information
    meta = ARMeta(Univariate, order, stype)

    for i in 1:n
        x[i] ~ AR(x_prev, θ, γ) where { q = q(y, x)q(γ)q(θ), meta = meta }
        y[i] ~ NormalMeanPrecision(ct * x[i], cτ)
        x_prev = x[i]
    end

    return x, y, θ, γ
end
```

    lar_model (generic function with 2 methods)

We will use different initial marginals depending on type of our AR process

```julia
function init_marginals!(::Type{ Multivariate }, order, γ, θ)
    setmarginal!(γ, GammaShapeRate(1.0, 1.0))
    setmarginal!(θ, MvNormalMeanPrecision(zeros(order), diageye(order)))
end

function init_marginals!(::Type{ Univariate }, order, γ, θ)
    setmarginal!(γ, GammaShapeRate(1.0, 1.0))
    setmarginal!(θ, NormalMeanPrecision(0.0, 1.0))
end
```

    init_marginals! (generic function with 2 methods)

```julia
function inference(data, order, artype, stype, niter, τ)

    # We build a full graph based on nber of observatios
    n = length(data)

    # Depending on the order of AR process `c` is
    # either a nber or a vector
    c = ReactiveMP.ar_unit(artype, order)

    # Note that to run inference for huge model it might be necessary to pass extra
    # options = (limit_stack_depth = 100,) to limit stack depth during recursive inference procedure
    model, (x, y, θ, γ) = lar_model(artype, n, order, c, stype, τ)

    # We are going to keep `γ` and `θ` estimates for all VMP iterations
    # But `buffer` only last posterior estimates for a sequence of hidden states `x`
    # We also will keep Bethe Free Energy in `fe`
    γ_buffer = keep(Marginal)
    θ_buffer = keep(Marginal)
    x_buffer = buffer(Marginal, n)
    fe       = keep(Float64)

    γsub  = subscribe!(getmarginal(γ), γ_buffer)
    θsub  = subscribe!(getmarginal(θ), θ_buffer)
    xsub  = subscribe!(getmarginals(x), x_buffer)
    fesub = subscribe!(score(Float64, BetheFreeEnergy(), model), fe)

    init_marginals!(artype, order, γ, θ)

    # We update data several times to perform several VMP iterations
    for i in 1:niter
        update!(y, data)
    end

    # It is important to unsubscribe from running observables
    unsubscribe!((γsub, θsub, xsub, fesub))

    return γ_buffer, θ_buffer, x_buffer, fe
end
```

    inference (generic function with 1 method)

```julia
γ, θ, xs, fe = inference(observations, length(real_θ), Multivariate, ARsafe(), 15, real_τ);
```

```julia
p1 = plot(first.(states), label="Hidden state")
p1 = scatter!(p1, observations, label="Observations")
p1 = plot!(p1, first.(mean.(xs)), ribbon = sqrt.(first.(var.(xs))), label="Inferred states", legend = :bottomright)

p2 = plot(mean.(γ), ribbon = std.(γ), label = "Inferred transition precision", legend = :bottomright)
p2 = plot!([ real_γ ], seriestype = :hline, label = "Real transition precision")

p3 = plot(getvalues(fe), label = "Bethe Free Energy")

plot(p1, p2, p3, layout = @layout([ a; b c ]))
```

Lets also plot a subrange of our results:

```julia
subrange = div(n,5):(div(n, 5) + div(n, 5))

plot(subrange, first.(states)[subrange], label="Hidden state")
scatter!(subrange, observations[subrange], label="Observations")
plot!(subrange, first.(mean.(xs))[subrange], ribbon = sqrt.(first.(var.(xs)))[subrange], label="Inferred states", legend = :bottomright)
```

It is also interesting to see where our AR coefficients converge to:

```julia
let
    pθ = plot()

    θms = mean.(θ)
    θvs = var.(θ)

    l = length(θms)

    edim(e) = (a) -> map(r -> r[e], a)

    for i in 1:length(first(θms))
        pθ = plot!(pθ, θms |> edim(i), ribbon = θvs |> edim(i) .|> sqrt, label = "Estimated θ[$i]")
    end

    for i in 1:length(real_θ)
        pθ = plot!(pθ, [ real_θ[i] ], seriestype = :hline, label = "Real θ[$i]")
    end

    plot(pθ, legend = :outertopright, size = (800, 300))
end
```

```julia
println("$(length(real_θ))-order AR inference Bethe Free Energy: ", last(fe))
```

    5-order AR inference Bethe Free Energy: 524.068596670656

We can also run a 1-order AR inference on 5-order AR data:

```julia
γ, θ, xs, fe = inference(observations, 1, Univariate, ARsafe(), 15, real_τ);
```

```julia
println("1-order AR inference Bethe Free Energy: ", last(fe))
```

    1-order AR inference Bethe Free Energy: 535.3776616955989

We can see that, according to final Bethe Free Energy value, in this example 5-order AR process can describe data better than 1-order AR.

We may be also interested in benchmarking our algorithm:

```julia
println("Benchmark for n = $n and AR-$(length(real_θ)) inference");
@benchmark inference($observations, length(real_θ), Multivariate, ARsafe(), 15, real_τ)
```

    Benchmark for n = 500 and AR-5 inference

    BenchmarkTools.Trial: 7 samples with 1 evaluation.
     Range (min … max):  781.312 ms … 844.306 ms  ┊ GC (min … max): 15.83% … 16.68%
     Time  (median):     802.343 ms               ┊ GC (median):    15.40%
     Time  (mean ± σ):   805.185 ms ±  24.054 ms  ┊ GC (mean ± σ):  15.51% ±  1.18%

      █   ▁               ▁            ▁       ▁                  ▁
      █▁▁▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█▁▁▁▁▁▁▁▁▁▁▁▁█▁▁▁▁▁▁▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█ ▁
      781 ms           Histogram: frequency by time          844 ms <

     Memory estimate: 435.26 MiB, allocs estimate: 3474118.


**Implementation**

Our software provides the user with a package he or she can use to evaluate functions of multiple variables at a specified point while at the same time evaluating the partial derivatives of the function at that same point. A class is provided to perform simple calculations, and two functions are provided to handle vector inputs and outputs.

Our strategy has been to create input-variable objects in python that once defined may be used in python calculations as if they were standard python variables of type float. However, when calculations are done with these AutDiff objects, a partial derivative with respect to one of the input variables is calculated and carried along. The object representing the result then contains an attribute that contains the value of the function at the specified point (val) and an attribute that contains a partial derivative with respect to one of the input variables (derv). Which partial derivative is calculated is specified by the user by intiating the derv attribute of one input variable to 1 and the derv attributes of the other input variables to 0.  The calculation can then be repeated to get partial derivatives with respect to other input variables by changing which input variable’s derv attribute is intiated to 1. A function *all_derivatives( )* is supplied to automate this process.

*The AutDiff( ) class*

Most of the functionality of our package is implemented in a single class called AutoDiff( ). Fully functional code for this class can be found in the module 'AutoDiff.py' in the *code* directory. It has two attributes, self.val and self.derv, that respectively hold the value of the function being evaluated and the partial derivative of the function being evaluated up to some point in the calculation.

Autodiff() has many methods. Each is short and designed to replace a standard mathematical operation. Thus, for example there is a method \_\_add\_\_ that will overload the standard addition operator (+) and thus specify how objects of type AutoDiff will be added to scalar values or to each other. And similar methods are included to overload the other standard mathematical operators (- * / ^ +). Also, methods are included to replace the
numpy functions (*exp, ln, log, sin, cos, tan, sqrt, log10, $a^{x}$*) so that these functions may be used as well in any calculations involving AutoDiff objects.

Central to all Autodiff( ) methods is the principle that each method returns a new AutDiff object that will represent the next stage in the function’s evaluation. For example, if x is an AutDiff object, and it is multiplied by the scalar 2 (x\*2), then x’s \_\_mul\_\_ method is called, and it returns a new object that will represent x\*2. The AutoDiff.\_\_mul\_\_ method performs this calculation as follows. It assumes that 2 is another object of type AutoDiff, and trys to multiply the two AutoDiff objects together. It finds that 2 is not an AutoDiff object: it falls to its except block, and then it makes two calculations: 2 * x.val and 2 * x.derv and stores these values in the returned object’s val and derv attributes. If the 2 had been to the right of x (2*x), then the \_\_rmul\_\_ method would have been called, and it would have executed the same operation.

More subtle is the case where two AutoDiff objects, say x and y, are multiplied together x\*y. Here x’s \_\_mul\_\_ method is called. It finds that the two objects are instances of AutoDiff, and it 1) multiplies x.val and y.val together and stores the result as the returned object’s val attribute, and 2) calculates (x.derv\*y.val + y.derv\*x.val) and store this value as the returned object’s derv attribute. Notice in the second calculation the derivatives from both objects x and y are carried along as a sum. Recall, however, that the derv value of either x or y will have been initiated to 0, so one term in the sum (x.derv\*y.val + y.derv\*x.val) will always evaluate to 0. In this way, the algorithm can automatically choose the proper derivative to carry along for subsequent calculations (here either that of x or that of y) when two Autodiff objects collide. This same approach can be used more generally and will be used when overloading the division and power operators.

An example of a elementary mathematical function implemented in AutoDiff( ) is the exponential function. This works as follows. For a given AutoDiff object, say x, when exp(x) is executed, x’s exp() function is called on itself. It uses numpy’s exp() function to evaluate np.exp(x.val) and store this value in the return object’s val attribute. It then evaluates the derivative of exp(x.value) —which in this case is just exp(x.value)— and stores this value in the return object’s derv attribute. This same approach is used for all the elementary functions.

Autdiff objects are fully functional replacements for python float variables such that once the AutoDiff class is defined, these objects can be used to perform forward autodifferentiation on any multi-input, single output function at any specified point.

*The all_derivatives() function*

This function has yet to be implemented. It will allow the user to calculate all partial derivatives at once without having to manually initiate the derv attributes of the input variables several times and each time repeat the calculation.

Psuedo code for this function is listed below. It takes advantage of the AutoDiff( ) class.

    def all_derivatives(func, in_vars, in_vals):
        ***
        Inputs:

            func():   —the function to be evaluated
            in_vars:  —a vector containing the input variables.
            in_vals:  —a vector of values, one for each variable in in_vars that specify
                         a point where the function and its partial derivatives are to be evaluated.
        Returns:

            func():    —the function evaluated
            val:       —the value of func( ) at the input point given by in_vals
            derv_vals: —a vector of partial derivative values, one for each input variable in in_vars.
        ***

        Steps:
        * make a vector diff_vars which  coorepsonds to in_vars but all
          values are AutoDiff objects.
        * Set the derv value of the first element of diff_vars to 1 and the derv
          values of the rest of the diff_vars elements to 0.

        * Create the derv_vals vector that will hold the calculated partial derivatives

        * For i in range( the number of elements in in_vars):
            * Set the derv value of each element of in_vars to 0
            * Set  the derv value of the ith element to 1.
            * Evaluate func()

            * If it is the first time thought the loop:
                  set val to the val of the AutoDiff output object
            * Else:
                * Store derivative of the output AutoDiff object as the ith element of derv_vals

        * Return  func, val, derv_vals

*The multi_func_all_derivatives() function*

This has yet to be implemented. I will handle vector inputs and outputs. It will allow the user to specify: a vector of functions, a vector of input variables, and a vectors of values defining a specific point. It will return: the input vector containing the functions, a vector of output values (one for each function evaluated at the specified point), and a Hessian matrix of partial derivatives containing partial derivatives of all functions evaluated at the specified point. Consider the equation below as an example:

\begin{equation}
\begin{pmatrix} f(x,y) \\ g(x,y)\end{pmatrix} =
\begin{pmatrix} 2xy \\ x+y^2 \end{pmatrix} at \begin{pmatrix} x=a \\ y=b \end{pmatrix}
\end{equation}

Here the multi_func_all_derivatives() function would return:
1) funcs, a vector of the functions that were evaluated ($2xy$,  $x+y^2$)
2) out_vals, a vector of the values of each function after evaluation $(f(a,b), g(a,b))$
3) out_derv_matrix, a matrix containing the partial derivatives evaluated at the point (a, b)

Pseudo code for this function is below. It takes advantge of the AutoDiff( ) class and the all_derivatives( ) function:

    def multi_func_all_derivatives(funcs, in_vars, in_vals):
        ***
        Inputs:
            funcs:    —a vector of input functions
            in_vars: —a vector of input variables
            in_vals:  —a vector of input values specifying a point.

        Returns:
            funcs: 	   —the vector of input functions
            out_vals —a vector of output values, one for each function
            out_derv_matrix —the Hessian matrix of partial derivatives evaluated at the input point
        ***
        Steps:
        •	Create an empty vector out_vals
        •	Create an empty matrix out_derv_matrix
        •	For each function in funcs:
                - Run all_derivatives(func, in_vars, in_vals)
                - Store the returned val in the vector out_vals
                - Store the returned vector derv_vals in the matrix out_derv_matrix

        •	Return funcs, out_vals, out_derv_matrix

*Testing*

We have implemented a series of tests to test the AutoDiff( ) class. They are designed to be run by *pytest* and to provide full code coverage. They can be found in the module 'test_AutoDiff.py'. The tests are divided into the following sections: *Testing Urnary, Testing Addition, Testing Subtraction, Testing Multiplication, Testing Division, Testing Powers, Testing trig functions, Testing exp, log, sqrt, log10, Testing exp, log, sqrt, log10, Testing Complicated functions*. Travis CI and codecov inidicate that all tests are currently passed by our code and our code coverage is 100%.

```python

```


# Sampyl Examples

Here I will have some examples showing how to use Sampyl. This is for version 0.2.2. Let's import it and get started. Sampyl is a Python package used to sample from probability distributions using Markov Chain Monte Carlo (MCMC). This is most useful when sampling from the posterior distribution of a Bayesian model.

Every sampler provided by Sampyl works the same way. Define $ \log P(\theta) $ as a function, then pass it to the sampler class. The class returns a sampler object, which you can then use to sample from $P(\theta)$. For samplers which use the gradient, $\nabla_{\theta} \log P(\theta)$, Sampyl uses [autograd](https://github.com/HIPS/autograd) to automatically calculate the gradients. However, you can pass in your own $\nabla_{\theta} \log P(\theta)$ functions.

Starting out simple, let's sample from a normal distribution.

```python
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import matplotlib.pyplot as plt
import sampyl as smp
from sampyl import np

# Autograd throws some warnings that are useful, but this is
# a demonstration, so I'll squelch them.
import warnings
warnings.filterwarnings('ignore')
```

A normal distribution with mean $\mu$ and variance $\sigma^2$ is defined as:

$$
P(x,\mu, \sigma) = \frac{1}{\sigma \sqrt{2 \pi}} \; \mathrm{Exp}\left( \frac{-(x - \mu)^2}{2\sigma^2} \right)
$$

For numerical stability, it is typically better to deal with log probabilities, $\log{P(\theta)}$. Then for the normal distribution with known mean and variance,

$$
\log{P(x \mid \mu, \sigma)} = -\log{\sigma} - \frac{(x - \mu)^2}{2\sigma^2}
$$

where we can drop constant terms since the MCMC samplers only require something proportional to $\log{P(\theta)}$. We can easily write this as a Python function.

```python
mu, sig = 3, 2
def logp(x):
    return  -np.log(sig) - (x - mu)**2/(2*sig**2)
```

First we'll use a Metropolis-Hastings sampler. Each sampler requires a $\log{P(\theta)}$ function and a starting state. We have included a function to calculate the *maximum a posteriori* (MAP) to find the peak of the distribution for use as the starting state. Then you call the sampler and a chain of samples is returned.

```python
start = smp.find_MAP(logp, {'x':1.})
metro = smp.Metropolis(logp, start)
chain = metro(10000, burn=2000, thin=4)
```

    Progress: [##############################] 10000 of 10000 samples

We can retrieve the chain by accessing the attributes defined by the parameter name(s) of `logp`.

```python
plt.plot(chain.x)
```

```python
_ = plt.hist(chain.x, bins=30)
_ = plt.vlines(mu, 0, 250, linestyles='--')
```

Here we have sampled from a normal distribution with a mean of 3, indicated with the dashed vertical line.

There is also a No-U-Turn Sampler (NUTS), which avoids the random-walk nature of Metropolis samplers. NUTS uses the gradient of $\log{P(\theta)}$ to make intelligent state proposals. You'll notice here that we don't pass in any information about the gradient. Instead, it is calculated automatically with [autograd](https://github.com/HIPS/autograd).

```python
nuts = smp.NUTS(logp, start)
chain = nuts(2100, burn=100)
```

    Progress: [##############################] 2100 of 2100 samples

```python
plt.plot(chain)
```

```python
_ = plt.hist(chain.x, bins=30)
_ = plt.vlines(mu, 0, 250, linestyles='--')
```

## Bayesian estimation of phone call rates

Let's try something a little more complicated. Let's say you run a business and you put an advertisement in the paper. Then, to judge the effectiveness of the ad, you want to compare the number of incoming phone calls per hour before and after the placement of the add. Then we can build a Bayesian model using a Poisson likelihood with exponential priors for $\lambda_1$ and $\lambda_2$.

\begin{align}
P(\lambda_1, \lambda_2 \mid D) &\propto P( D \mid \lambda_1, \lambda_2)\, P(\lambda_1)\, P(\lambda_2) \\
P( D \mid \lambda_1, \lambda_2) &\sim \mathrm{Poisson}(D\mid\lambda_1)\,\mathrm{Poisson}(D\mid\lambda_2) \\
P(\lambda_1) &\sim \mathrm{Exp}(1) \\
P(\lambda_2) &\sim \mathrm{Exp}(1)
\end{align}

This analysis method is known as Bayesian inference or Bayesian estimation. We want to know likely values for $\lambda_1$ and $\lambda_2$. This information is contained in the posterior distribution $P(\lambda_1, \lambda_2 \mid D)$. To infer values for $\lambda_1$ and $\lambda_2$, we can sample from the posterior using our MCMC samplers.

```python
# Fake data for the day before and after placing the ad.
# We'll make the calls increase by 2 an hour. Record data for each
# hour over two work days.
before = np.random.poisson(7, size=16)
after = np.random.poisson(9, size=16)

# Define the log-P function here
def logp(λ1, λ2):
    model = smp.Model()
    # Poisson log-likelihoods
    model.add(smp.poisson(before, rate=λ1),
              smp.poisson(after, rate=λ2))

    # Exponential log-priors for rate parameters
    model.add(smp.exponential(λ1),
              smp.exponential(λ2))

    return model()
```

```python
start = smp.find_MAP(logp, {'λ1':1., 'λ2':1.})
sampler = smp.Metropolis(logp, start)
chain = sampler(10000, burn=2000, thin=4)
```

    Progress: [##############################] 10000 of 10000 samples

Sampling returns a numpy record array which you can use to access samples by name. Variable names are taken directly from the argument list of `logp`.

```python
print(sampler.var_names)
```

    ('λ1', 'λ2')

```python
plt.plot(chain.λ1)
plt.plot(chain.λ2)
```

Now to see if there is a significant difference between the two days. We can find the difference $\delta =  \lambda_2 - \lambda_1$, then find the probability that $\delta > 0$.

```python
delta = chain.λ2 - chain.λ1
_ = plt.hist(delta, bins=30)
_ = plt.vlines(2, 0, 250, linestyle='--')
```

```python
p = np.mean(delta > 0)
effect = np.mean(delta)
CR = np.percentile(delta, (2.5, 97.5))
print("{:.3f} probability the rate of phone calls increased".format(p))
print("delta = {:.3f}, 95% CR = {{{:.3f} {:.3f}}}".format(effect, *CR))
```

    0.981 probability the rate of phone calls increased
    delta = 1.916, 95% CR = {0.154 3.746}

There true difference in rates is two per hour, marked with the dashed line. Our posterior is showing an effect, but our best estimate is that the rate increased by only one call per hour. The 95% credible region is {-0.735 2.743} which idicates that there is a 95% probability that the true effect lies with the region, as it indeed does.

We can also use NUTS to sample from the posterior.

```python
nuts = smp.NUTS(logp, start)
chain = nuts.sample(2100, burn=100)
_ = plt.plot(chain.λ1)
_ = plt.plot(chain.λ2)
```

```python
delta = chain.λ2 - chain.λ1
_ = plt.hist(delta, bins=30)
_ = plt.vlines(2, 0, 250, linestyle='--')
p = np.mean(delta > 0)
effect = np.mean(delta)
CR = np.percentile(delta, (2.5, 97.5))
print("{:.3f} probability the rate of phone calls increased".format(p))
print("delta = {:.3f}, 95% CR = {{{:.3f} {:.3f}}}".format(effect, *CR))
```

## Linear models too

When you build larger models, it would be cumbersome to have to include every parameter as an argument in the logp function. To avoid this, you can declare the size of variables when passing in the starting state.

For instance, with a linear model it would be great to pass the coefficients as one parameter. First, we'll make some fake data, then infer the coefficients.

```python
# Number of data points
N = 200
# True parameters
sigma = 1
true_B = np.array([2, 1, 4])

# Simulated features, including a constant
X = np.ones((N, len(true_B)))
X[:,1:] = np.random.rand(N, 2)*2

# Simulated outcomes with normally distributed noise
y = np.dot(X, true_B) + np.random.randn(N)*sigma

data = np.ones((N, len(true_B) + 1))
data[:, :-1] = X
data[:, -1] = y
```

```python
fig, axes = plt.subplots(figsize=(7,4),ncols=2)
for i, ax in enumerate(axes):
    ax.scatter(X[:,i+1], y)
axes[0].set_ylabel('y')
axes[0].set_xlabel('X1')
axes[1].set_xlabel('X2')
```

```python
fig.savefig('linear_model_data.png')
```

```python
# Here, β is a length 3 array of coefficients
def logp(β, sig):

    model = smp.Model()

    # Estimate from our data and coefficients
    y_hat = np.dot(X, β)

    # Add log-likelihood
    model.add(smp.normal(y, mu=y_hat, sig=sig))

    # Add prior for estimate error
    model.add(smp.exponential(sig))

    # Uniform priors on coefficients
    model.add(smp.uniform(β, lower=-100, upper=100))

    return model()
```

```python
start = smp.find_MAP(logp, {'β': np.ones(3), 'sig': 1.},
                     bounds={'β':(-5, 10), 'sig':(0.01, None)})
sampler = smp.Metropolis(logp, start)
chain = sampler(20000, burn=5000, thin=4)
```

    Progress: [##############################] 20000 of 20000 samples

```python
_ = plt.plot(chain.β)
```

And using NUTS too.

```python
start = smp.find_MAP(logp, {'β': np.ones(3), 'sig': 1.})
nuts = smp.NUTS(logp, start)
chain = nuts.sample(2100, burn=100)
```

    Progress: [##############################] 2100 of 2100 samples

```python
fig, axes = plt.subplots(figsize=(8,5), nrows=2, ncols=2)
for i, (row, param) in enumerate(zip(axes, [chain.β, chain.sig])):
    row[0].plot(param)
    row[0].set_ylabel('Sample value')
    #row[0].set_xlabel('Sample')
    row[0].set_title(['β', 'sig'][i])
    row[1].set_title(['β', 'sig'][i])
    if len(param.shape) > 1:
        for each in param.T:
            row[1].hist(each, alpha=0.8, histtype='stepfilled')
        row[1].set_yticklabels('')
        row[1].vlines([2,1,4], 0, 600, linestyles='--', alpha=0.5)

    else:
        row[1].hist(param, alpha=0.8, histtype='stepfilled')
        row[1].set_yticklabels('')
        row[1].vlines(1, 0, 600, linestyles='--', alpha=0.5)
    #row[1].set_xlabel('Sample value')
fig.tight_layout(pad=0.1, h_pad=1.5, w_pad=1)
```

```python
fig.savefig('linear_model_posterior.png')
```

## Using one logp function for both logp and gradient

You can also use one `logp` function that returns both the logp value and the gradient. To let the samplers know about this, set `grad_logp = True`. I'm also using one argument `theta` as the parameter which contains the five $\beta$ coefficients and $\sigma$.

```python
from autograd import grad
grads = [grad(logp, 0), grad(logp, 1)]
def single_logp(theta):
    b, sig = theta[:3], theta[-1]
    logp_val = logp(b, sig)
    grad_val = np.hstack([each(b, sig) for each in grads])
    return logp_val, grad_val
```

```python
start = {'theta': np.ones(4)}
nuts = smp.NUTS(single_logp, start, grad_logp=True)
chain = nuts.sample(2000, burn=1000, thin=2)
```

    Progress: [##############################] 2000 of 2000 samples

```python
_ = plt.plot(chain.theta[:, :4])
```

## Sampling in parallel

We can make use of our multicore CPUs by running chains in parallel. To do this, simply request the number of chains you want when you call `sample`: `nuts.sample(1000, n_chains=4)`. Each chain is given its own process and the OS decides how to run the processes. Typically this means that each process will run on its own core. So, if you have four cores and four chains, they will all run in parallel. But, if you have two cores, only two will run at a time.

```python
start = smp.find_MAP(logp, {'β': np.ones(3), 'sig': 1.})
nuts = smp.NUTS(logp, start)
chains = nuts.sample(1100, burn=100, n_chains=2)
```

    Progress: [##############################] 1100 of 1100 samples

```python
fig, axes = plt.subplots(figsize=(10,3), ncols=2)
for ax, chain in zip(axes, chains):
    _ = ax.plot(chain.β)
```

The future
----------

* Write a module that makes it easier to build logp functions from distributions
* Add various functions such as autocorrelation, HPD, etc.
* Plots!


Massimo Nocentini

Abstract

In this document we collect a naive type system based on sets.

```python
from itertools import repeat
from sympy import *
#from type_system import *
```

```python
%run ../../src/commons.py
```

```python
%run ./type-system.py
```

---

```python
init_printing()
```

```python
x,y,m,n,t,z = symbols('x y m n t z', commutative=True)
alpha, beta, gamma, eta = symbols(r'\alpha \beta \gamma \eta', commutative=True)
f,g = Function('f'), Function('g')
```

---

```python
bin_tree_gfs = bin_tree(tyvar(alpha)[z]).gf()
bin_tree_gfs
```

```python
bin_tree_gf = bin_tree_gfs[0]
```

```python
bin_tree_gf.series(z, n=10, kernel='ordinary')
```

```python
bin_tree_gf.series(z, n=10, kernel='catalan')
```

```python
occupancy(bin_tree_gf, syms=[alpha], objects='unlike', boxes='unlike').series(z)
```

---

```python
bin_tree_of_boolean_gfs = bin_tree(maybe(tyvar(alpha))[z]).gf()
bin_tree_of_boolean_gfs
```

```python
bin_tree_of_boolean_gf = bin_tree_of_boolean_gfs[0]
```

```python
occupancy(bin_tree_of_boolean_gf, syms=[alpha], objects='unlike', boxes='unlike').series(z,n=6, kernel='ordinary')
```

```python

```


# Naive Bayes Classifier

### The following few portions will be referencing resources mainly from Kilian Weinberger's materials from CS4780 [1] and Tom M.Mitchell's Machine Learning book [2] .

The primary goal of classification is to predict the label $y$ from a set of given features $X$. Given that we do not know the underlying distribution of $P(X,Y)$, we can only draw  $n$ samples from this distribution into our training set $D$, then estimate $P(Y|X)$ by the following according the conditional probability formula:

$$
\hat{P}(y|\mathbf{x}) =
\frac{\hat{P}(y,\mathbf{x})}{P(\mathbf{x})} =
\frac{\sum_{i=1}^{n} \frac{I(\mathbf{x}_i = \mathbf{x} \wedge {y}_i = y)}{n}} { \sum_{i=1}^{n} \frac{I(\mathbf{x}_i = \mathbf{x})}{n}} =
\frac{\sum_{i=1}^{n} I(\mathbf{x}_i = \mathbf{x} \wedge {y}_i = y)}{ \sum_{i=1}^{n} I(\mathbf{x}_i = \mathbf{x})}
$$

where $I(\mathbf{x}_i = \mathbf{x} \wedge {y}_i = y) = 1$ if $x_i = x$ and $y_i = y$, and $0$ otherwise.

The problem with estimating probabilities this way is that it can be difficult when $x$ exists in a high dimensional space, or when x is continuous.

#### Example:

If we want to determine whether a person in a university is a student, it is easy to estimate $P(x)$ and $P(y,x)$ if the number of features we want to examine is small.
If we were to only use 'age over or under 35' as a feature (1-D space binary feature), $P(x)$ can be estimated from our sample of $n$ samples simply by counting the number of people under the age of 35, and divide it by the number of people sampled $n$. Similarly, $P(y,x)$ can be estimated by counting the number of people with age less than 35 given that they are students.

However, if the number of features were to be added, estimating probabilities can be difficult. Take the extreme, we use each pixels in the person's potrait to determine if the person is a student. We will not be able to estimate $P(x)$ because we will not be able to achieve a standard for the pixel intensities a student should have. Given that it is pretty unlikely that no 2 people will look alike, $P(x)$ will be 0 and $P(y,x)$ will be 0. Similarly, if we were to use an actual age (down to the exact birth date) as a feature instead of an over under age as a feature, it will also be unlikely that someone will be found in our dataset.

To approach this dilemma, we use Naive Bayes to simplify the problem.

## Naive Bayes

From Bayes rule,

$$
P(y | \mathbf{x}) = \frac{P(\mathbf{x} | y)P(y)}{P(\mathbf{x})}
$$

It is also known as:

$$
\text {Posterior} = \frac{\text{Likelihood} * \text{Prior}} {\text{Evidence}}
$$

Estimating $P(y)$ can be easily done. Using the example from above, we can estimate the number of people that are students ($c$) simply by going through the dataset and counting how many of them are actually students $(y_i = c)$:

$$
P(y = c)  = \frac{\sum_{i=1}^{n} I(y_i = c)}{n} = \hat\pi_c \tag{1}
$$

The Naive Bayes algorithm makes estimating $P(x|y)$ easy by making the assumption that each feature $X = <X_1, ..., X_n >$ are conditionally independent from each other for the given label $y = c$. For the case of $X = <X_1, X_2 >$, this simplifies $P(\mathbf{x} | y)$ to based on conditional independence:

\begin{align}
    P(X | y=c) &= P(X_1, X_2 | y=c) \\
               &= P(X_1| y=c)P(X_2| y=c)
\end{align}

In general for d-dimensional feature vector,

$$
P(X | y=c) = \prod_{i = 1}^{d} P(X_i | y=c)
$$

The Bayes classifier can then be defined as:

\begin{align}
    h(\mathbf{x}) &= \operatorname*{argmax}_{y=c} P(y | \mathbf{x}) \\
                  &= \operatorname*{argmax}_{y=c} \; \frac{P(\mathbf{x} | y=c)P(y=c)}{P(\mathbf{x})} \\
                  &= \operatorname*{argmax}_{y=c} \; P(\mathbf{x} | y=c) P(y=c) && \text{($P(\mathbf{x})$ removed because it does not depend on $y$)} \\
                  &= \operatorname*{argmax}_{y=c} \; \prod_{i=1}^{d} P(x_i | y=c) P(y=c) && \text{(by the Naive Bayes assumption)}\\
                  &= \operatorname*{argmax}_{y=c} \; \sum_{i = 1}^{d} \log(P(x_i | y=c)) + \log(P(y=c)) && \text{(as log is a monotonic function)}
\end{align}

In reality, we know that Naive Bayes assumption of conditionally independent features often do not hold true. Going back to the example above, 2 possible features that are not conditionally independent, given that the chosen person is a student, are net worth and age. Students who are younger typically have a lower net worth.

Assuming that Naive Bayes assumption is true, estimating $log(P(x_i | y))$ is easier given that only one dimension for feature will need to be considered.

### Case #1: Categorical features

Case #1 involves features that each take on $J$ possible discrete values, for a given label $y = c$.

Example: Given that person chosen is a student, feature 1: gender (J=2 for binary, gender when born, not currently identifying as), feature 2: faculty (J=10-20? depending on university).

$$
\theta_{ijc} = P(X_i = x_{ij} | y = c)\\
\text{ and }\sum_{j=1}^{J} \theta_{ijc} = 1
$$

where $\theta_{ijc}$ refers to the probability of feature i taking on value j, given that the label is c.

Probability $\theta_{ijc}$ can be estimated as:

$$
\hat\theta_{ijc} = \frac{\sum_{i=1}^{n} I(y_i = c) I(x_{ij} = j) + l}{\sum_{i=1}^{n} I(y_i = c) + lJ}
$$

where $l$ refers to the smoothing parameter.

$$
\text{Estimate} = \begin{cases}
                      \text{MLE} & l=0 \\
                      \text{Laplace smoothing} & l=+1 \\
                      \text{MAP} & l>0,l\neq +1
                   \end{cases}
$$

The Naive Bayes classifier comes to:

\begin{align}
h(\mathbf{x}) &= \operatorname*{argmax}_{y=c} \; \prod_{i=1}^{d} P(x_i | y=c) P(y=c) \\
              &\propto \operatorname*{argmax}_{y=c} \; \prod_{i=1}^{d} \hat\theta_{ijc} \hat\pi_c
\end{align}

### Case #2: Multinomial features

Case #2 involves features that each take on its count/frequency for a given label $y = c$.

Example: In word documents, the feature vector is really just a dictionary, where the index of this vector refers to a particular word in the english dictionary, and the value at that index refers to the frequency in which that word occurs in a document. Given that the documents that have been given the label as 'confidential', we can identify commonly encounted words as features belonging to 'confidential' documents.

$$
\hat\theta_{jc} = \frac{\sum_{i = 1}^{n} I(y_i = c) x_{ij} + l}{\sum_{i=1}^{n} I(y_i = c) m_i + l \cdot d }
$$

where $x_{ij}$ refers to the number of times a particular word has appeared in the $i^{th}$ document, and $m_i$ refers to the total number of words drawn from that same document. $\hat\theta_{jc}$ refers to the estimated probability that a given word i will appear given that it's document has been classified as $y = c$.

For a given document i, given that $m_i$ words are drawn, the number of counts of a particular word j:

$$
x_{ij} \in \{0, 1, 2, ..., m\}
$$

The multinomial Naive Bayes classifier comes to:

$$
h(\mathbf{x}) \propto \operatorname*{argmax}_{y=c} \; \prod_{i=1}^{d} \hat\theta_{ijc} \hat\pi_c
$$

### Case #3: Continuous features (Gaussian Naive Bayes)

Case #3 involves features that take on continuous values, with each feature $j$ assumed to be drawn its own independent Gaussian distribution with its own mean $\mu_{jc}$ and variance $\sigma_{jc}^2$

$$
P(x_j \mid y=c) = \mathcal{N}\left(\mu_{jc}, \sigma^{2}_{jc}\right) = \frac{1}{\sqrt{2 \pi} \sigma_{jc}} e^{-\frac{1}{2} \left(\frac{x_j - \mu_{jc}}{\sigma_{jc}}\right)^2} \tag{2}
$$

The parameters that requires estimatation are the mean and the variance:

\begin{align}
\mu_{jc} &\leftarrow \frac{1}{n_c} \sum_{i = 1}^{n} I(y_i = c) x_{ij} && \text{where $n_c = \sum_{i=1}^{n} I(y_i = c)$} \\
\sigma_{jc}^2 &\leftarrow \frac{1}{n_c} \sum_{i=1}^{n} I(y_i = c)(x_{ij} - \mu_{jc})^2
\end{align}

The multinomial Naive Bayes classifier comes to:

$$
h(\mathbf{x}) \propto \operatorname*{argmax}_{y=c} \; \prod_{j=1}^{d}\frac{1}{\sqrt{2 \pi} \sigma_{jc}} e^{-\frac{1}{2} \left(\frac{x_j - \mu_{jc}}{\sigma_{jc}}\right)^2}  \hat\pi_c
$$

### GNB algorithm below adapted from: https://github.com/eriklindernoren/ML-From-Scratch/blob/master/mlfromscratch/supervised_learning/naive_bayes.py

```python
import numpy as np
import math

class GaussianNaiveBayes:

    """
    In general, Naive Bayes need to find Likelihood(2), Prior(1) terms to get the Posterior term
    For GNB, this values will be dependent on the mean and var of each feature
    """

    def find_likelihood(self, param, c, x):
        """
        Likelihood = Normal distribution formula
        """
        mean, var = param[0], param[1]
        numer = math.exp(-0.5 * ((x - mean)**2 / var))
        denom = math.sqrt(2 * math.pi * var)

        return numer / denom

    def find_prior(self, c):
        """
        Prior = Number of samples with y == c / Total number of samples
        """
        return np.mean(self.y == c)

    def fit(self, X, y):
        self.x = X
        self.y = y
        self.n = len(y)

        self.dict = {}

        # Find the different possible labels
        #Assumes that each label has at least a sample collected
        self.labels = np.unique(y)

        #Go through each label
        #For that label, get the samples where y == that label
        for c in self.labels:
            X_c = X[np.where(self.y == c)]

            #X_c consists of a matrix where each row correspond to the sample with label == c
            #and each column corresponding to the different features
            #Need to calculate the mean and variance of each feature, or the mean and variance of each column
            #Each dictionary key corresponds to a label
            #The index of each element in the dictionary value corresponds to a feature
            #Each element is stored as (mean, feature)
            for feature in X_c.T:
                if c not in self.dict: #If first feature
                    self.dict[c] = [(np.mean(feature), np.var(feature))]
                else:
                    self.dict[c].append((np.mean(feature), np.var(feature)))

    def predict(self, X):

        y_pred = []

        for sample in X:

            posteriors = []

            for c in self.labels:

                prior = self.find_prior(c)

                for i in range(len(self.dict[c])):
                    likelihood = self.find_likelihood(self.dict[c][i], c, sample[i])
                    prior *= likelihood

                posteriors.append(prior)

            y_pred.append(np.argmax(posteriors))

        return np.array(y_pred)
```

```python
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np

from sklearn.datasets import make_classification, make_blobs
from scipy.stats import multivariate_normal

X, y = make_blobs(n_samples=100, n_features=2, centers=2, cluster_std=2.0, random_state=123)

with plt.style.context('seaborn-darkgrid'):
    fig = plt.figure(figsize=(12,10))

    plt.scatter(x=X[:][y == 0][:, 0], y=X[:][y == 0][:, 1], edgecolor='k', linewidth=1, label='y = 0')
    plt.scatter(x=X[:][y == 1][:, 0], y=X[:][y == 1][:, 1], edgecolor='k', linewidth=1, label='y = 1')

    plt.legend()
```

```python
model = GaussianNaiveBayes()
model.fit(X, y)
```

### Decision boundary plot

```python
fig, ax = plt.subplots(figsize=(12, 10))

x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5
y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5

h = 0.02 #Step size in mesh
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))

Z = model.predict(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)

cset = ax.contourf(xx, yy, Z, colors=['tab:blue', 'tab:orange'], levels=np.arange(Z.max() + 2) - 0.5, alpha=0.2)

ax1 = plt.scatter(x=X[:][y == 0][:, 0], y=X[:][y == 0][:, 1], edgecolor='k', linewidth=1, label='y=0')
ax2 = plt.scatter(x=X[:][y == 1][:, 0], y=X[:][y == 1][:, 1], edgecolor='k', linewidth=1, label='y=1')
ax3 = ax.contour(xx, yy, Z, cset.levels, colors='k', linewidths=1, antialiased=True)
plt.title('Decision Boundary', fontsize=15)
plt.legend()
```

### References
[1] CS4780/CS5780: Machine Learning for Intelligent Systems. Kilian Weinberger, 2017. https://www.cs.cornell.edu/courses/cs4780/2018fa/lectures/lecturenote05.html

[2] Chapter 3 Generative and Discriminative Classifiers: Naive Bayes and Logisitic Regression. Tom M. Mitchell, 2017. Machine Learning. http://www.cs.cmu.edu/∼tom/mlbook/NBayesLogReg.pdf


# Algorithms Exercise 2

## Imports

```python
%matplotlib inline
from matplotlib import pyplot as plt
import seaborn as sns
import numpy as np
```

## Peak finding

Write a function `find_peaks` that finds and returns the indices of the local maxima in a sequence. Your function should:

* Properly handle local maxima at the endpoints of the input array.
* Return a Numpy array of integer indices.
* Handle any Python iterable as input.

```python
def find_peaks(a):
    """Find the indices of the local maxima in a sequence."""
    maxima = np.array([])

    if a[0] > a[1]:
        maxima = np.append(maxima, 0)

    for n in range (1, len(a) - 1):
        if a[n] > a[n-1] and a[n] > a[n+1]:
            maxima = np.append(maxima, n)

    if a[len(a) - 1] > a[len(a) - 2]:
        maxima = np.append(maxima, len(a) - 1)

    return maxima
```

```python
p1 = find_peaks([2,0,1,0,2,0,1])
assert np.allclose(p1, np.array([0,2,4,6]))
p2 = find_peaks(np.array([0,1,2,3]))
assert np.allclose(p2, np.array([3]))
p3 = find_peaks([3,2,1,0])
assert np.allclose(p3, np.array([0]))
```

Here is a string with the first 10000 digits of $\pi$ (after the decimal). Write code to perform the following:

* Convert that string to a Numpy array of integers.
* Find the indices of the local maxima in the digits of $\pi$.
* Use `np.diff` to find the distances between consequtive local maxima.
* Visualize that distribution using an appropriately customized histogram.

```python
from sympy import pi, N
pi_digits_str = str(N(pi, 10001))[2:]
```

```python
# YOUR CODE HERE
pi_digits_array = np.array([])
for n in range(0, 10000):
    pi_digits_array = np.append(pi_digits_array, pi_digits_str[n])

    #there may be a more efficient way to do this with arrays, but I couldn't
    #find it, and I only need to do it once anyway and it only takes ~2 seconds

pi_maxima = find_peaks(pi_digits_array)
pi_diff = np.diff(pi_maxima)

plt.hist(pi_diff, bins=15, range=(0,15), color="g")
plt.xlabel("Seperation")
```

```python
assert True # use this for grading the pi digits histogram
```


# Air in a tank

A spherical metal tank stores hot air, initially at 200°C and 350 psi.
The tank is made out of stainless steel, with specific heat capacity 471 J/kg-K and density 7902 kg/m$^3$. The tank wall is 0.35 mm thick and initially at 20°C; the inner radius is 10 cm and the outer wall is insulated.

Treat the air in the tank as an ideal gas and the stainless steel tank wall as an incompressible substance.

Eventually the air reaches thermal equilibrium with the tank wall. Find the final temperature of the air and the tank, and the final pressure of the air.

```python
import numpy as np
from scipy import optimize
import cantera as ct

from pint import UnitRegistry
ureg = UnitRegistry()
Q_ = ureg.Quantity
```

```python
# input data
tank_specific_heat = Q_(471, 'J/(kg*K)')
tank_density = Q_(7902, 'kg/m^3')
tank_radius = Q_(10, 'cm')
tank_thickness = Q_(0.35, 'mm')

tank_temp1 = Q_(20, 'degC')

air_temp1 = Q_(200, 'degC')
air_pres1 = Q_(350, 'psi')

# use an ideal gas mixture to represent air
air1 = ct.Solution('air.cti')
air1.TPX = (
    air_temp1.to('K').magnitude,
    air_pres1.to('Pa').magnitude,
    'O2:1.0, N2:3.76'
    )
```

```python
air_volume = 4.0 * np.pi * tank_radius**3 / 3.0
air_mass = air_volume / Q_(air1.v, 'm^3/kg')

tank_volume = (4. * np.pi / 3.) * (
    (tank_radius + tank_thickness)**3 - tank_radius**3
    )
tank_mass = tank_volume * tank_density
```

During the equilibration process, the specific volume of the air does not change:
\begin{equation}
v_{a,2} = v_{a,1} \;.
\end{equation}

We can perform an energy balance on the system that contains both the tank and air:

$$
0 = m_t ( u_{t,2} - u_{t,1} ) + m_a ( u_{a,2} - u_{a,1} ) \\
0 = m_t c_{\text{steel}} (T_{t,2} - T_{t,1}) - m_a ( u_{a,2} - u_{a,1} )
$$

and we know that at equilibrium the temperatures are the same: $T_{t,2} = T_{a,2}$.

So, we fundamentally have **one** unknown, the final temperature, which we can solve using the final version of the energy balance:

```python
def tank_energy(T2, air_mass, air1, tank_mass, tank_specific_heat, tank_temp1):
    '''Returns value of energy balance equation based on input T2

    This equation should return zero, if the correct T2 is input.
    '''
    # the temperatures are the same
    tank_temp2 = Q_(T2, 'K')
    air_temp2 = Q_(T2, 'K')

    air_specific_volume2 = Q_(air1.v, 'm^3/kg')
    air_density2 = 1.0 / air_specific_volume2

    air2 = ct.Solution('air.cti')
    air2.TDX = (
        air_temp2.to('K').magnitude,
        air_density2.to('kg/m^3').magnitude,
        'O2:1.0, N2:3.76'
        )

    return (
        tank_mass * tank_specific_heat * (tank_temp2 - tank_temp1) +
        air_mass * (Q_(air2.u, 'J/kg') - Q_(air1.u, 'J/kg'))
        ).to('J').magnitude
```

```python
# find the root of the equation, with an initial guess of 300 K
sol = optimize.root(
    tank_energy, 300.0,
    args=(air_mass, air1, tank_mass, tank_specific_heat, tank_temp1)
    )

# the solution is the first element of the sol.x list
temp2 = Q_(sol.x[0], 'K')
print(f'Final temperature: {temp2.to("degC"): .2f}')

air2 = ct.Solution('air.cti')
air2.TDX = (
    temp2.to('K').magnitude,
    air1.density,
    'O2:1.0, N2:3.76'
    )
air_pres2 = Q_(air2.P, 'Pa')
print(f'Final air pressure: {air_pres2.to("psi"): .2f}')
```

    Final temperature: 64.83 degree_Celsius
    Final air pressure: 250.01 pound_force_per_square_inch

## Alternate solution

If we wanted, we could also solve this problem with a larger number of equations and unknowns, although as seen above this is not necessary. For example, we can treat $v_{a,2}$, $T_{a,2}$, $T_{t,2}$, and $u_{a,2}$ as the four unknowns, with the equations

$$
v_{a,2} = v_{a,1} \\
T_{a,2} = T_{t,2} \\
u_{a,2} = f \left( T_{a,2}, v_{a,2} \right)
$$

being added to the one used above.

```python
def tank_energy2(x, air_mass, air1, tank_mass, tank_specific_heat, tank_temp1):
    '''Returns value of energy balance equation based on input T2

    This equation should return zero, if the correct values are input.

    x: T_t2, v_a2, T_a2, u_a2
    '''
    tank_temp2 = Q_(x[0], 'K')
    air_specific_volume2 = Q_(x[1], 'm^3/kg')
    air_temp2 = Q_(x[2], 'K')
    air_internal_energy2 = Q_(x[3], 'J/kg')

    air_density2 = 1.0 / air_specific_volume2

    air2 = ct.Solution('air.cti')
    air2.TDX = (
        air_temp2.to('K').magnitude,
        air_density2.to('kg/m^3').magnitude,
        'O2:1.0, N2:3.76'
        )

    return [
        (tank_mass * tank_specific_heat * (tank_temp2 - tank_temp1) +
         air_mass * (Q_(air2.u, 'J/kg') - Q_(air1.u, 'J/kg'))
         ).to('J').magnitude,
        (air_specific_volume2 - Q_(air1.v, 'm^3/kg')).to('m^3/kg').magnitude,
        (air_temp2 - tank_temp2).to('K').magnitude,
        (air_internal_energy2 - Q_(air2.u, 'J/kg')).to('J/kg').magnitude
        ]

# now we have to give guesses for all four variables
sol = optimize.root(
    tank_energy2, [300.0, air1.v, 300.0, air1.u],
    args=(air_mass, air1, tank_mass, tank_specific_heat, tank_temp1)
    )

# the solution is the first element of the sol.x list
tank_temp2 = Q_(sol.x[0], 'K')
print(f'Final temperature of tank: {tank_temp2.to("degC"): .2f}')
air_temp2 = Q_(sol.x[2], 'K')
print(f'Final temperature of air: {air_temp2.to("degC"): .2f}')

air2 = ct.Solution('air.cti')
air2.TDX = (
    air_temp2.to('K').magnitude,
    1.0 / sol.x[1],
    'O2:1.0, N2:3.76'
    )
air_pres2 = Q_(air2.P, 'Pa')
print(f'Final air pressure: {air_pres2.to("psi"): .2f}')
```

    Final temperature of tank: 64.83 degree_Celsius
    Final temperature of air: 64.83 degree_Celsius
    Final air pressure: 250.01 pound_force_per_square_inch

As expected, we get the same answer as before, without any real benefits. In fact, this is a bit more work, since we had to specify four guess values instead of one.


### Example 3: Diffusion

In this example we will look at the diffusion equation and how to handle second-order derivatives. For this we will introduce Devito's `.laplace` short-hand expression and demonstrate it using the examples from step 7 of the original tutorial.

So, the equation we are now trying to implement is

$$\frac{\partial u}{\partial t} = \nu \frac{\partial ^2 u}{\partial x^2} + \nu \frac{\partial ^2 u}{\partial y^2}$$

To discretize this equation we will use central differences and reorganizing the terms yields

\begin{align}
u_{i,j}^{n+1} = u_{i,j}^n &+ \frac{\nu \Delta t}{\Delta x^2}(u_{i+1,j}^n - 2 u_{i,j}^n + u_{i-1,j}^n) \\
&+ \frac{\nu \Delta t}{\Delta y^2}(u_{i,j+1}^n-2 u_{i,j}^n + u_{i,j-1}^n)
\end{align}

As usual, we establish our baseline experiment by re-creating some of the original example runs. So let's start by defining some parameters.

```python
from examples.cfd import plot_field, init_hat
import numpy as np
%matplotlib inline

# Some variable declarations
nx = 31
ny = 31
nt = 17
nu = .05
dx = 2. / (nx - 1)
dy = 2. / (ny - 1)
sigma = .25
dt = sigma * dx * dy / nu
```

We now set up the diffusion operator as a separate function, so that we can re-use if for several runs.

```python
def diffuse(u, nt):
    for n in range(nt + 1):
        un = u.copy()
        u[1:-1, 1:-1] = (un[1:-1,1:-1] +
                        nu * dt / dx**2 * (un[1:-1, 2:] - 2 * un[1:-1, 1:-1] + un[1:-1, 0:-2]) +
                        nu * dt / dy**2 * (un[2:,1: -1] - 2 * un[1:-1, 1:-1] + un[0:-2, 1:-1]))
        u[0, :] = 1
        u[-1, :] = 1
        u[:, 0] = 1
        u[:, -1] = 1
```

Now let's take this for a spin. In the next two cells we run the same diffusion operator for a varying nuper of timesteps to see our "hat function" dissipate to varying degrees.

```python
#NBVAL_IGNORE_OUTPUT

# Initialise u with hat function
u = np.empty((nx, ny))
init_hat(field=u, dx=dx, dy=dy, value=2.)
plot_field(u, zmax=2.5)

diffuse(u, nt=10)
plot_field(u, zmax=2.5)
```

```python
#NBVAL_IGNORE_OUTPUT

# Initialise u with hat function
u = np.empty((nx, ny))
init_hat(field=u, dx=dx, dy=dy, value=2.)

diffuse(u, nt=50)
plot_field(u, zmax=2.5)
```

Excellent. Now for the Devito part we need to note one important detail to our previous examples: we now have a second-order derivative. So, when creating our `TimeFunction` object we need to tell it about our spatial discretisation by setting `space_order=2`. Only then can we use the shorthand notation `u.dx2` and `u.dx2` to denote second derivatives.

```python
from devito import Grid, TimeFunction
from sympy import Eq, solve
from sympy.abc import a

# Initialize `u` for space order 2
grid = Grid(shape=(nx, ny), extent=(2., 2.))
u = TimeFunction(name='u', grid=grid, space_order=2)

# Create an equation with second-order derivatives
eq = Eq(u.dt, a * (u.dx2 + u.dy2))
stencil = solve(eq, u.forward)[0]
eq_stencil = Eq(u.forward, stencil)

print(eq_stencil)
```

    Eq(u(t + dt, x, y), (a*dt*h_x**2*(-2*u(t, x, y) + u(t, x, y - h_y) + u(t, x, y + h_y)) + a*dt*h_y**2*(-2*u(t, x, y) + u(t, x - h_x, y) + u(t, x + h_x, y)) + h_x**2*h_y**2*u(t, x, y))/(h_x**2*h_y**2))

Now, there is another trick here! Note how the above formulation explicitly uses `u.dx2` and `u.dy2` to denote the laplace operator, which makes this equation dependent on the spatial dimension. We can instead use the notation `u.laplace` to denote all second derivatives in space, allowing us to reuse this code for 2D and 3D examples.

```python
eq = Eq(u.dt, a * u.laplace)
stencil = solve(eq, u.forward)[0]
eq_stencil = Eq(u.forward, stencil)

print(eq_stencil)
```

    Eq(u(t + dt, x, y), (a*dt*h_x**2*(-2*u(t, x, y) + u(t, x, y - h_y) + u(t, x, y + h_y)) + a*dt*h_y**2*(-2*u(t, x, y) + u(t, x - h_x, y) + u(t, x + h_x, y)) + h_x**2*h_y**2*u(t, x, y))/(h_x**2*h_y**2))

Great. Now all that is left is to pu it all together to build the operator and use it on our examples. For illustration purposes we will do this in one cell, including update equation and boundary conditions.

```python
#NBVAL_IGNORE_OUTPUT
from sympy import Eq, solve
from devito import Operator, Constant

# Reset our data field and ICs
init_hat(field=u.data[0], dx=dx, dy=dy, value=2.)

# Create an operator with seond-order derivatives
a = Constant(name='a')
eq = Eq(u.dt, a * u.laplace)
stencil = solve(eq, u.forward)[0]
eq_stencil = Eq(u.forward, stencil)

# Create boundary condition expressions
x, y = grid.dimensions
t = grid.stepping_dim
bc = [Eq(u.indexed[t+1, 0, y], 1.)]  # left
bc += [Eq(u.indexed[t+1, nx-1, y], 1.)]  # right
bc += [Eq(u.indexed[t+1, x, ny-1], 1.)]  # top
bc += [Eq(u.indexed[t+1, x, 0], 1.)]  # bottom

op = Operator([eq_stencil] + bc)
op(time=10, dt=dt, a=nu)

plot_field(u.data[0], zmax=2.5)
```

And now let's use the same operator again to show the more diffused field. In fact, instead of resetting the previously used object `u`, we can also create a new `TimeFunction` object and tell our operator to use this.

```python
#NBVAL_IGNORE_OUTPUT
u2 = TimeFunction(name='u2', grid=grid, space_order=2)
init_hat(field=u2.data[0], dx=dx, dy=dy, value=2.)

op(u=u2, time=50, dt=dt, a=nu)

plot_field(u2.data[0], zmax=2.5)
```


# Ficha Técnica

Disciplina - Algebra linear e Processos Dinâmicos.

Conteúdo -

- Resolução de sistemas lineares.

- Representação matricial

- Operações com linhas em matrizes.

Autores:

- Anderson Borba (anderson.aborba@professores.ibmec.edu.br) e

- Luiz Barboza (luiz.cbjunior@professores.ibmec.edu.br).

Endereço eletrônico: http://quant-ibmec.group/calculos/

# Sistemas de equações lineares:
Um sistema linear de equações com $m$ equações e $n$ incógnitas é um conjunto de equações do tipo:
\begin{equation}\nonumber
    \left\{
    \begin{array}{ccccccccc}
        a_{11}x_1&+& a_{12}x_2&+& \cdots&+& a_{1n}x_{n}&=&b_1 \\
        a_{21}x_1&+& a_{22}x_2&+& \cdots&+& a_{2n}x_{n}&=&b_2 \\
        \vdots & \vdots& && & &\vdots &&\vdots \\
        a_{m1}x_1&+& a_{m2}x_2&+& \cdots&+& a_{mn}x_{n}&=&b_n \\
    \end{array}
    \right.
\end{equation}
A solução do sistema linear é uma $n-$upla de números $(x_1,x_2,\cdots,x_n)$ que satisfaça simultaneamento a $m$ equações.

# Sistema linear na forma matricial:
\begin{equation}\nonumber
    \left[
    \begin{array}{cccc}
        a_{11}& a_{12}& \cdots& a_{1n} \\
        a_{21}& a_{22}& \cdots& a_{2n} \\
        \vdots & \vdots&  &\vdots \\
        a_{m1}& a_{m2}& \cdots& a_{mn} \\
    \end{array}
    \right]
    \left[
    \begin{array}{c}
        x_1 \\
        x_2 \\
        \vdots \\
        x_n \\
    \end{array}
    \right]=
    \left[
    \begin{array}{c}
        b_1 \\
        b_2 \\
        \vdots \\
        b_n \\
    \end{array}
    \right]
\end{equation}
Podemos escrever $Ax=b$.

EX 1:
\begin{equation}\nonumber
    \left\{
    \begin{array}{cccccccc}
         x_1&+& 4x_2&+& 3x_3&=& 1\\
        2x_1&-& 5x_2&-& 4x_3&=& 4\\
         x_1&-& 3x_2&-& 3x_3&=& 5\\
    \end{array}
    \right.,
    \left[
    \begin{array}{rrr|r}
        1&  4&  3& 1\\
        2&  5&  4& 4\\
        1& -3& -2& 5\\
    \end{array}
    \right]
\end{equation}

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.linalg as la

A = np.array([[1.,4.,3.,1],[2.,5.,4.,4],[1.,-3.,-2.,5]])
print(A)
```

    [[ 1.  4.  3.  1.]
     [ 2.  5.  4.  4.]
     [ 1. -3. -2.  5.]]

Definindo a operação $L_2=>L_2-\frac{a_{21}}{a_{11}}L_1$, isto é, $L_2=>L_2-\frac{2}{1}L_1$

```python
E1 = np.array([[1,0,0],[-2,1,0],[0,0,1]])
print(E1)
```

    [[ 1  0  0]
     [-2  1  0]
     [ 0  0  1]]

\begin{equation}\nonumber
    \left\{
    \begin{array}{cccccccc}
         x_1&+& 4x_2&+& 3x_3&=& 1\\
        0x_1&-& 3x_2&-& 2x_3&=& 2\\
         x_1&-& 3x_2&-& 2x_3&=& 5\\
    \end{array}
    \right.,
    \left[
    \begin{array}{rrr|r}
        1&  4&  3& 1\\
        0&  -3&  -2& 2\\
        1& -3& -2& 5\\
    \end{array}
    \right]
\end{equation}

```python
E1 @ A
```

    array([[ 1.,  4.,  3.,  1.],
           [ 0., -3., -2.,  2.],
           [ 1., -3., -2.,  5.]])

Definindo a operação $L_3=>L_3-\frac{a_{31}}{a_{31}}L_1$, isto é, $L_3=>L_3-\frac{1}{1}L_1$

```python
E2 = np.array([[1,0,0],[0,1,0],[-1,0,1]])
print(E2)
```

    [[ 1  0  0]
     [ 0  1  0]
     [-1  0  1]]

\begin{equation}\nonumber
    \left\{
    \begin{array}{cccccccc}
         x_1&+& 4x_2&+& 3x_3&=& 1\\
        0x_1&-& 3x_2&-& 2x_3&=& 2\\
        0x_1&-& 7x_2&-& 5x_3&=& 4\\
    \end{array}
    \right.,
    \left[
    \begin{array}{rrr|r}
        1&  4&  3& 1\\
        0&  -3&  -2& 2\\
        0& -7& -5& 4\\
    \end{array}
    \right]
\end{equation}

```python
E2 @ E1 @ A

```

    array([[ 1.,  4.,  3.,  1.],
           [ 0., -3., -2.,  2.],
           [ 0., -7., -5.,  4.]])

Definindo a operação $L_3=>L_3-\frac{a_{32}}{a_{22}}L_2$, isto é, $L_3=>L_3-\frac{7}{3}L_2$

```python
E3 = np.array([[1,0,0],[0,1,0],[0,-7/3,1]])
print(E3)
```

    [[ 1.          0.          0.        ]
     [ 0.          1.          0.        ]
     [ 0.         -2.33333333  1.        ]]

Encontramos a matriz triangular superior equivalente:
\begin{equation}\nonumber
    \left\{
    \begin{array}{cccccccc}
         x_1&+& 4x_2&+& 3x_3&=& 1\\
        0x_1&-& 3x_2&-& 2x_3&=& 2\\
        0x_1&-& 0x_2&-& \frac{1}{3}x_3&=&-\frac{2}{3} \\
    \end{array}
    \right.,
    \left[
    \begin{array}{rrr|r}
        1&  4&  3& 1\\
        0&  -3&  -2& 2\\
        0& 0& -\frac{1}{3}& -\frac{2}{3}\\
    \end{array}
    \right]
\end{equation}

```python
E = E3 @ E2 @ E1
print(E)
```

    [[ 1.          0.          0.        ]
     [-2.          1.          0.        ]
     [ 3.66666667 -2.33333333  1.        ]]

```python
EQ = E @ A
print(EQ)
```

    [[ 1.00000000e+00  4.00000000e+00  3.00000000e+00  1.00000000e+00]
     [ 0.00000000e+00 -3.00000000e+00 -2.00000000e+00  2.00000000e+00]
     [ 0.00000000e+00  4.44089210e-16 -3.33333333e-01 -6.66666667e-01]]

Usamos a matriz equivalente E para encontrar o vetor solução aproximado $(x_1,x_2,x_3)$.

```python
x3=EQ[2,3]/EQ[2,2]
print(x3)
```

    1.9999999999999973

```python
x2=- 1/3 *(2 + 2 * x3)
print(x2)
```

    -1.9999999999999982

```python
x1= 1 - 4 * x2 - 3 * x3
print(x1)
```

    3.000000000000001

Encontramos a solução numéricamente $(x_1,x_2,x_3) = (3.000000000000001,-1.9999999999999982,1.9999999999999973)$. A solução exata é $(x_1,x_2,x_3)=(3,-2,2)$, assim podemos afirmar que a solução encontrada numéricamente aproxima a solução exata encontrada de forma algébrica.

Ex 2: a) Escrever o sistema linear na forma padrão,
\begin{equation}\nonumber
    \left[
    \begin{array}{rrrrr|r}
         1&  1&  2& 2&1& 1\\
         2&  2&  4& 4&3& 1\\
         2&  2&  4& 4&2& 2\\
         3&  5& 8& 6&5& 3\\
     \end{array}
\right]
\end{equation}
b) Resolver usando operações por linhas na forma matricial.

# Referências
- Anton, Howard Rorres, Chris. Álgebra Linear com Aplicações. Editora Bookman- 2012.
- https://www.math.ubc.ca/~pwalls/math-python/linear-algebra/solving-linear-systems/