# Interval estimates

#### Estimation framework, reminder

The framework we consider is the following. We have $N$ data points modelled as a vector $y \in \mathbb{R}^N$. We have a model for the data, that is the data is assumed to have a distribution $ p(y|\theta)$ for a certain parameter $\theta$ that we wish to estimate. The function $\theta \rightarrow p(y|\theta)$ is called the likelihood.

For instance, we have $N$ data points, independent statistically, each data point is assumed to follow a Gaussian distribution of mean $\mu$ and variance $\sigma_k^2$, denoted by $Y_k \sim G(\mu, \sigma_k^2)$. Let us suppose that the $\sigma_k^2$ are known, so that the only unknown parameter is $\mu$ (it plays the role of $\theta$ in the definition of the likelihood). The likelihood of the data point $Y_k$ is then
$$p(y_k|\mu)  = \frac{1}{\sqrt{2\pi} \sigma_k} \mathrm{e}^{-\frac{1}{2\sigma_k^2} (y_k - \mu)^2}$$
Since all the $Y_k$ are independent, the likelihood of the data $Y$ is the product of the likelihoods of the $Y_k$,
$$p(y|\mu)  = \prod\limits_{k=1}^N  p(y_k|\mu) =\prod\limits_{k=1}^N \frac{1}{\sqrt{2\pi} \sigma_k} \mathrm{e}^{-\frac{1}{2\sigma_k^2} (y_k - \mu)^2} = \frac{1}{\sqrt{2\pi}^N \prod\limits_{k=1}^N \sigma_k} \mathrm{e}^{-\frac{1}{2} \sum\limits_{k=1}^N \frac{(y_k - \mu)^2}{\sigma_k^2} } $$

Furthermore, we might assume that the parameter itself has a prior distrbution. That is, the probability of $\theta$ before seeing the data is $p(\theta)$. In the example above, we could assume that $p(\mu)$ is a Gaussian distribution of mean $0$ and variance $\sigma_\mu$

#### Point estimates, reminder

In the previous lesson we have studied the point estimates. In the point estimates view, we have an estimator $\hat{\theta}:y \rightarrow \hat{\theta}(y)$ that takes as argument the data $y$ and outputs a value wanted to be close to  $\theta$. The error bar is then given as the variance or square root mean squared error ($\sqrt{\mathrm{MSE}}$) of $\hat{\theta}$.

Some point estimates ignore the prior distributions, while some take it into account. The most common estimators that do not involve the prior are the maximum likelihood and least square estimates. When the Likelihood of the data is Gaussian and the covariance is known, they are equivalent. In the example above, the maximum likelihood estimate is
$$\hat{\mu}_{ML} = \arg \max_\mu  p(y|\mu)  = \frac{\sum\limits_{k=1}^N \frac{y_k}{\sigma_k^2} }{\sum\limits_{k=1}^N \frac{1}{\sigma_k^2}} $$

If we assume a prior on $\mu$, $p(\mu)$, the common estimators are the mean, median and a posteriori, that are
$$ \hat{\theta}_{\mathrm{mean}} = \int_{-\infty}^\infty \mu p(\mu|y) \mathrm{d} \mu  =\int_{-\infty}^\infty \mu \frac{p(y|\mu) p(\mu)   }{p(y)}  \mathrm{d} \mu  $$
$$ \hat{\theta}_{\mathrm{median}} = \mathrm{median}(p(\mu|y)) $$
$$ \hat{\theta}_{\mathrm{mode}} = \mathrm{mode}(p(\mu|y)) $$
where the mode is the argument that maximizes the a function, $\mathrm{mode}(p(\mu|y)) = \arg \max_\mu p(\mu|y)$.

In the example above, $$\hat{\theta}_{\mathrm{mean}} = \hat{\theta}_{\mathrm{median}} = \hat{\theta}_{\mathrm{mode}} = \frac{\sum\limits_{k=1}^N \frac{y_k}{\sigma_k^2} }{\frac{1}{\sigma_\mu^2} +\sum\limits_{k=1}^N \frac{1}{\sigma_k^2}} $$.

If the model is correct, the posterior mean and median have respectively minimal mean squared error and mean absolute error.

#### Interval estimates

In this spreadsheet, we change the viewpoint of the estimation. Instead of aiming at finding an estimator that is optimal in a certain sense, we consider the question: how likely is it that the true value of the parameters lie in a certain interval ?

''Likely'' is a loose term that needs clarifications. There are two main ways of constructing interval estimates: the confidence intervals and the credible intervals, which have different properties.

## Confidence interval

A confidence interval is constructed in the following way. Given a likelihood $p(y|\theta)$ and data $y$, a confidence interval is constructed by choosing a probability $\alpha$, and two functions of the data $l_\alpha(y)$ and $u_\alpha(y)$ such that
$$ \mathrm{Pr}\left\{  \theta \in [l_\alpha(y),   u_\alpha(y) ] \;  | \; \theta \right\} = \alpha $$

We first consider an example where we construct a confidence interval for the weighted mean of independent Gaussian variables.
$$\hat{\mu}_{ML} = \arg \max_\mu  p(y|\mu)  = \frac{\sum\limits_{k=1}^N \frac{y_k}{\sigma_k^2} }{\sum\limits_{k=1}^N \frac{1}{\sigma_k^2}} $$

$\hat{\mu}_{ML}$ has a Gaussian distribution of variance $\sigma_{\hat{\mu}}^2 = \frac{1}{\sum\limits_{k=1}^N \frac{1}{\sigma_k^2}}$.

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import minimize
import scipy.special as sp
```

```python
# Number of simulations
Nsim = 100000
N = 10 # Number of data points
mean_error = 1 #Mean value of the error bars
mu = 1
#alpha = 4
errors_sigma = mean_error*np.random.chisquare(4,size=N)/4 #Generage values of the error bars
k = 3
conditions = np.ones(Nsim, dtype=bool)

for i in range(Nsim):

    y = mu + np.random.randn(N)*errors_sigma

    mu_estim = np.sum(y/errors_sigma**2) / np.sum(1/errors_sigma**2)
    sigma_estim = 1/ np.sqrt(np.sum(1/errors_sigma**2))

    u = mu_estim  + k*sigma_estim
    l = mu_estim  - k*sigma_estim
    condition = (mu <= u) * (mu >= l)

    conditions[i] = condition

print('The true value of the parameter is in the interval in', np.sum(conditions) /Nsim*100, '% of the trials')

```

    The true value of the parameter is in the interval in 99.724 % of the trials

### Question 1

Compute analytically the probability that the true parameter is in the confidence interval if $l = \hat{\mu} - k \sigma_\hat{\mu}$ and $u = \hat{\mu} + k \sigma_\hat{\mu}$ for $k = 1, 2, 3$.

Does the value of the confidence interval depend on the number of data points ?

Compute the centered confidence interval that gives an inclusion probability  $ \alpha = 50\%$

Given that the distribution of data point is Gaussian with mean $\mu$ and error $\sigma$, we know that, $\alpha$ can be calculated using the following definition,

$$\alpha = \int_{\mu-k\sigma}^{\mu + k\sigma} \frac{1}{\sqrt{2\pi} \sigma} e^{-\frac{(y-\mu)^2}{2\sigma^2}} dy$$

Making a change of variables, $\left(\frac{y-\mu}{\sigma}\right)^2 = u^2$, we can write above equation as,

\begin{equation*}
    \begin{split}
        \alpha &= \frac{1}{\sqrt{2\pi}\sigma} \int_{-k}^{k} e^{-\frac{1}{2}u^2} \sigma du \\
        &= \sqrt{\frac{2}{\pi}} \int_{0}^{k} e^{-\frac{1}{2}u^2} du
    \end{split}
\end{equation*}

In the last line we used the fact that the integral is the even function around the limit. To solve this integration we can make another change of variable $\frac{u^2}{2} = x$,

\begin{equation*}
    \begin{split}
        \alpha &= \sqrt{\frac{2}{\pi}} \int_{0}^{k^2/2} e^{-x} \frac{1}{\sqrt{2x}} dx \\
        &= \frac{1}{\sqrt{\pi}} \int_0^{k^2/2} x^{-1/2} e^{-x} dx \\
        &= \frac{1}{\sqrt{\pi}} \left[-\sqrt{\pi} (1 - erf(\sqrt{x}) \right]^{k^2/2}_0
    \end{split}
\end{equation*}

In the last line we used the integral tables to find the value of given integration. Solving last equation one would get that,

$$\alpha = erf\left(\frac{k}{\sqrt{2}}\right)$$

We can check this formula using scipy as follows,

```python
k1 = np.array([1,2,3])
alpha = sp.erf(k1/np.sqrt(2))

print('For k=1, the probability that the true value would be in interval is ', alpha[0])
print('For k=2, the probability that the true value would be in interval is ', alpha[1])
print('For k=3, the probability that the true value would be in interval is ', alpha[2])
```

    For k=1, the probability that the true value would be in interval is  0.6826894921370859
    For k=2, the probability that the true value would be in interval is  0.9544997361036416
    For k=3, the probability that the true value would be in interval is  0.9973002039367398

This probability would not depend on the number of data points. (In the last calculation, we didn't use number of data points anywhere, we just used the PDF of the data).

Now, we want to calculate the centered confidence interval that gives the inclusion probability $\alpha=0.5$. That means we want to compute $k$ for given alpha which can be done using the inverse error function.

$$k = \sqrt{2} \cdot erf^{-1}(\alpha)$$

We can calculate this using the scipy.

```python
alpha1 = 0.5
kk = np.sqrt(2)*sp.erfinv(alpha1)
print('The confidence interval that gives the inclusion probability of 50% would be at about '
      + str(np.around(kk,2)) +
      '-sigma from the center')
```

    The confidence interval that gives the inclusion probability of 50% would be at about 0.67-sigma from the center

### Question 2

We now consider another example. Suppose we observe
$$Y = \theta + \epsilon$$ where $\epsilon$ follows an exponential distribution
$f(\epsilon) = \frac{1}{\lambda} \exp(-\frac{\epsilon}{\lambda})$ and $\theta$ is the parameter to estimate.

Given the data $y$, construct confidence intervals for  68.27, 95.45 and 99.73 $\%$ for $\theta$ of the form $[y - x_\alpha ,y]$. In other words, find $x_\alpha$ such that $\theta \in [y - x_\alpha ,y]$ with a probability $\alpha$.

Check your calculations with a simulation as above.

The likelihood function for the given exponential distribution would be,

$$p(y|\theta) = \frac{1}{\lambda}\exp{\left(-\sum_k \frac{y_k}{\lambda}\right)}$$

We can calculate the Maximum Likelihood estimate of $\hat{\lambda}$ as follows,

\begin{equation}
    \begin{split}
        \log p(y|\theta) &= - \log \lambda - \sum_k \frac{y_k}{\lambda} \\
        \Rightarrow \frac{d \log p(y|\theta)}{d\lambda} &= -\frac{1}{\lambda} + \sum_k \frac{y_k}{\lambda^2} \\
        \Rightarrow 0 &= -1 + \sum_k \frac{y_k}{\lambda} \\
        \Rightarrow \hat{\lambda} &= \sum_k y_k
    \end{split}
\end{equation}

Now, we want to find confidence interval for this distribution. We can do so as we did in the previous case. Let, $\alpha$ be probability with which true value of $\lambda$ lies in the given interval $(0,k)$. Then,

\begin{equation}
    \begin{split}
        \alpha &= \int_0^k \frac{1}{\lambda} e^{-x/\lambda} dx \\
        &= \frac{1}{\lambda} \left( \frac{e^{-x/\lambda}}{-1/\lambda} \right)_0^k \\
        &= 1 - e^{-k/\lambda}
    \end{split}
\end{equation}

Here, $\lambda$ would be the ML estimate of the parameter. Using above formula, we can find the interval $(0,k)$ for which the $\hat{\lambda}$ would be in the interval would be $\alpha$.

$$k = \lambda \ln{(1-\alpha)}$$

```python
# Number of simulations
Nsim = 100000
N = 10 # Number of data points

mu = 2
alpha = 0.95

conditions1 = np.ones(Nsim, dtype=bool)

for i in range(Nsim):

    y = np.random.exponential(mu,N)

    mu_estim = np.sum(y)

    l = 0
    u = mu_estim*np.log(1-alpha)
    condition = (mu <= u) * (mu >= l)

    conditions1[i] = condition

print('The true value of the parameter is in the interval in', np.sum(conditions1) /Nsim*100, '% of the trials')
```

    The true value of the parameter is in the interval in 0.0 % of the trials

### Question 3

We now consider another example. Suppose we observe
$$Y = \theta + \epsilon$$ where $\epsilon$ follows a gamma distribution of parameters $\alpha, \beta$.

Given the data $y$, construct confidence intervals for  68.27, 95.45 and 99.73 $\%$ for $\theta$ of the form $[y - x_\alpha ,y]$. In other words, find $x_\alpha$ such that $\mu \in [y - x_\alpha ,y]$ with a probability $\alpha$.

Check your calculations with a simulation as above.

```python

```


```python
from sympy import *
# Use pretty printing for SymPy `expr`essions
init_printing()

def noop(*args, **kwargs):
    pass
if __name__ != '__main__':
    display = noop
```

# Linear ODEs: example with complex eigenvalues

In this lecture we solve the linear ODE $\dot x=Ax$, where $A=\begin{bmatrix}6&-5\\13&-10\end{bmatrix}$.
I will solve this ODE manually verifying each step using [Python] library [SymPy].

[Python]: https://www.python.org "Python programming language"
[SymPy]: https://www.sympy.org "Symboic math in Python"

## Eigenvalues and eigenvectors

The eigenvalues of this matrix are $-2\pm i$, and the corresponding eigenvectors are $\begin{pmatrix}8\pm i\\13\end{pmatrix}$.

```python
A = Matrix(2, 2, [6, -5, 13, -10])
A.eigenvects()
```

## Normal form
Therefore, $A = PCP^{-1}$, where $P=\begin{bmatrix}8&-1\\13&0\end{bmatrix}$, $C=\begin{bmatrix}-2&-1\\1&-2\end{bmatrix}$. In general, if $v=\begin{bmatrix}v_1\\v_2\end{bmatrix}$ is an eigenvector of $A$ with eigenvalue $λ$, then $C=\begin{bmatrix}\Re\lambda&-\Im\lambda\\\Im\lambda&\Re\lambda\end{bmatrix}$, $P=\begin{bmatrix}\Re v_1&-\Im v_1\\\Re v_2&-\Im v_2\end{bmatrix}$.

```python
(λ, m, (v,)) = A.eigenvects()[1]
v *= 13 # get rid of the denominator; it's still an eigenvector
assert (A * v).expand() == (λ * v).expand()
P = re(v).row_join(-im(v))
C = Matrix(2, 2, [re(λ), -im(λ), im(λ), re(λ)])
assert P * C * P ** -1 == A
λ, v, P, C
```

## Formula for the solution

### Solution of the normalized equation
Recall that the solution of $\dot y=\begin{pmatrix}a&-b\\b&a\end{pmatrix}y$, $y(0)=c$, is given by $y(t)=e^{at}\begin{pmatrix}\cos(bt) & -\sin(bt)\\\sin(bt) & \cos(bt)\end{pmatrix}c$, hence solutions of $\dot y=Cy$ are given by $y(t)=e^{-2t}\begin{pmatrix}\cos t&-\sin t\\\sin t&\cos t\end{pmatrix}y(0)$.

```python
var('c0 c1') # Coordinates of $x(0)$
var('t') # Time
c = Matrix([c0, c1])
M = exp(re(λ) * t) * Matrix(2, 2, [cos(im(λ) * t), -sin(im(λ) * t), sin(im(λ) * t), cos(im(λ) * t)])
y = M * c
assert y.diff(t) == (C * y).expand()
M, y
```

### Solution of the original equation
Solutions of $\dot x=Ax$, $x(0)=c$, are given by
$$
x(t)=PM(t)P^{-1}c=e^{-2t}\begin{bmatrix}8\sin t+\cos t&-5\sin t\\13\sin t&\cos t-8\sin t\end{bmatrix}x(0).
$$

```python
x = P * M * P ** -1 * c
assert x.diff(t).expand() == (A * x).expand()
assert x.subs(t, 0) == c
display(P * M * P ** -1, x)
```


# 07 Numbers and Errors: Sine Series Problem

* *Computational Physics*: Ch 2.5

## Set up

Pull changes in PHY494-resources
```
cd ~/PHY494-resources
git pull
```
Copy the the problem notebook with the whole directory as your work dir:
```
cd
cp -r ~/PHY494-resources/07_numbers ~/PHY494
```
Work on the `07-problem-sine-series.ipynb` notebook:
```
~/PHY494
jupyter notebook
```

## Problem: Summing Series: sin(x)

Evaluate the $\sin$ function from its series representation
$$
\sin x = x - \frac{x^3}{3!} + \frac{x^5}{5!} - \frac{x^7}{7!} + \dots
$$

A naive algorithm is to sum the series up to the $N$th term:
$$
\sin x \approx \sum_{n=1}^N \frac{(-1)^{n-1} x^{2n-1}}{(2n - 1)!}
$$

Problems:

- How to decide when to stop summing?
- Division of large terms (overflows!)
- Powers and factorials are very expensive to compute.

Better approach: Build up series terms $a_n$ using previous term $a_{n-1}$ through a recursion:

\begin{align}
a_n &= a_{n-1} \times q_n\\
a_n &= \frac{(-1)^{n-1} x^{2n-1}}{(2n - 1)!} = \frac{(-1)^{n-2} x^{2n-3}}{(2n - 3)!} \frac{-x^2}{(2n - 1)(2n - 2)}\\
a_n & = a_{n-1} \frac{-x^2}{(2n - 1)(2n - 2)}
\end{align}

Accuracy of this approach? Not clear in absolute terms but we can make the assumption that the error is approximately the last term in the sum, $a_N$. Hence we can strive to make the relative error smaller than the machine precision
$$
\left| \frac{a_N}{\sum_{n=1}^N a_n} \right| < \epsilon_m
$$

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
def sin_series(x, eps=1e-16):
    """Calculate sin(x) to precision eps.

    Arguments
    ---------
    x : float
        argument of sin(x)
    eps : float, optional
        desired precision

    Returns
    -------
    func_value, N

    where func_value = sin(x) and N is the number
    of terms included in the approximation.
    """
    # ....
    # add your code here
    return sumN, n-1
```

Test the implementation against the "exact" numpy function `np.sin()`.

Report
1. `x`
2. maximum `n`
3. `sin_series(x)`
4. relative error `abs(sin_series(x) - sin(x))/abs(sin(x))`

Plot against `x` the quantities above and also `sin(x)`.
* `x` - `sin_series(x)`
* `x` - `sin(x)`
* `x` - max `n`
* `x` - relative error (semilogy plot!)

For a range of numbers look at `Xsmall` and `Xlarge`:

```python
Xsmall = np.pi*np.arange(-2, 2.05, 0.05)
Xlarge = np.pi*np.arange(-50, 50.05, 0.1)
```

Implementation of the test:

```python
def test_sin(x):
    y, nmax = sin_series(x)
    y0 = np.sin(x)
    delta = y - y0
    if y0 != 0:
        relative_error = delta/y0
    else:
        relative_error = 0
    # print(x, y, y0, delta, relative_error)
    #return x, y, y0, delta, relative_error
    return x, nmax, y, relative_error
```

```python
def test_plot_sine(X, filename="sine_error.pdf"):
    results = np.array([test_sin(x) for x in X])

    fig = plt.figure(figsize=(8, 10))

    ax1 = fig.add_subplot(3,1,1)
    ax1.plot(results[:, 0], results[:, 2], 'k-', lw=1, label="series")
    ax1.plot(results[:, 0], np.sin(results[:, 0]), 'g--', lw=2, label="sin x")
    ax1.legend(loc="best")

    ax2 = fig.add_subplot(3,1,2)
    ax2.plot(results[:, 0], results[:, 1], label="max n")
    ax2.legend(loc="best")

    ax3 = fig.add_subplot(3,1,3)
    ax3.semilogy(results[:, 0], results[:, 3], label="rel.error")
    ax3.legend(loc="best")

    fig.suptitle("sine series approximation")

    fig.savefig(filename)
    print("saved to file {0}".format(filename))
```

Now test the two ranges of numbers and write to different files:

```python
test_plot_sine(Xsmall, filename="sine_error_Xsmall.pdf")
```

```python
# ...
```

```python

```


# Mathematical Theory and Intuition behind Our Improved Model

This markdown explains the mathematical theory behind our model. Note that the actual implementation might be different. For simplicity, we are also not including all the datasets we used (otherwise the equations might be too long).

Let $i$ denote the $i$th date since the beginning of our dataset and $n$ denote the cutoff date for training-testing . Let the range from $n+1$ to $m$ be the range for testing (forecast).

For a given date $i$, let $\textbf{asp}(i)$ be the actual index for S\&P500; $\textbf{fbsp}(i)$ be the index for S\&P500 bases on a training set of the first $i-1$ many days' market data of S\&P500; $\textbf{div}(i)$ be the dividend yield rates of S\&P500; $\textbf{eps}(i)$ be the dividen yield rates of S\&P500; $\textbf{tby}(i)$ be the 10-year treasury bond yield; $\textbf{ffr}(i)$ be the fed fund rates;  $\textbf{fta}(i)$ be the fed total assets.

## Linear Model

\begin{equation}
    \textbf{fb}_1(i,a,b,c,d,e):=\textbf{fbsp}(i)+a*\textbf{tby}(i)+b*\textbf{ffr}(i)+c*\textbf{fta}(i)+d*\textbf{div}(i)+e*\textbf{eps}(i)
\end{equation}

\begin{equation}
    \textbf{E}_n(a,b,c,d,e):= \frac{1}{n-1000}\sum_{i=1000}^{n}
    (\textbf{fb}_1(i,a,b,c,d,e)-\textbf{asp}(i))^2
\end{equation}

attains its minimum. Namely, finding

\begin{equation}
    (a_n,b_n,c_n,d_n,e_n):=\text{argmin} \textbf{E}_n(a,b,c,d,e)
\end{equation}

Note that it doesn't make sense to start with $i=1$ since the fbprophet itself need to use the first $i-1$ of the data for training

## Nonlinear Model

All notations are the same as above.

Here is a different model (nonlinear revision of fbprophet). In this model, we will the dividend yield rate of S\&P 500.

First, what is the dividend yield rate? Dividend is the money that publicly listed companies pay you (usually four times a year) for holding their stocks (until the so-called ex-dividend dates). It's like the interest paid to your saving accounts from the banks. Some companies pay while some don't, especially for the growth tech stocks. From my experience, the impact of bond rates and fed fund rates on the stock market will change when they rise above or fall below the dividend yield rate}. Stock prices fall when those rates rise above the dividend yield rate of SP500 (investor are selling their stocks to buy bonds or save more money in their bank account!).

Based on this idea, it might be useful to consider the differences of those rates and the dividend yield rate of SP500.

Normally an increase in the federal fund rate will result in an increase in bank loan interest rate, which will in turn result in an decrease in the net income of S\&P500-listed companies since they have to pay a higher interest when borrowing money from banks. Based on this thought, I believe it is reasonable to make a correction to $c*\textbf{eps}(i)$ by replacing the term by $c*\textbf{eps}(i)(1+d*\textbf{ffr}(i))$. If my intuition is correct, the generated constant $d$ from the optimization should be a negative number.

$$\textbf{fb}_2(i,a,b,c,d,e):= \textbf{fbsp}(i)*\big[1+a*(\textbf{div}(i)-\textbf{tby}(i))\big]\big[1+b*(\textbf{div}(i)-\textbf{ffr}(i))\big]\\
+c*\textbf{eps}(i)(1+d*\textbf{ffr}(i)+e*\textbf{fta}(i))$$

and consider

$$E_n(a,b,c,d,e) := \frac{1}{n-1000}\sum_{i=1000}^n(\textbf{fb}_2(i,a,b,c,d,e)-\textbf{asp}(i))^2$$

Now find (by approximation, SGD, etc.) $(a_n,b_n,c_n,d_n,e_n):=\text{argmin} E_n(a,b,c,d,e)$

Using $(a_n,b_n,c_n,d_n,e_n)$ as constants, our output will be $\textbf{fb}_2(i,a_n,b_n,c_n,d_n,e_n)$

### The actual implementation of the nonlinear model

For the actual implementation of the nonlinear model, we threw away the higher order terms (the products of three things) since they are relatively smaller quantities

### The mathematical theory behind our nonlinear model:

#### The Taylor's theorem for multivariate functions

Let $f :\mathbb{R}^n → \mathbb R$ be a $k$-times-differentiable function at the point $a ∈ \mathbb{R}^n$. Then there exists $h_a:\mathbb{R}^n → \mathbb R$ such that:

<math>\begin{align}
& f(\boldsymbol{x}) = \sum_{|\alpha|\leq k} \frac{D^\alpha f(\boldsymbol{a})}{\alpha!} (\boldsymbol{x}-\boldsymbol{a})^\alpha  + \sum_{|\alpha|=k} h_\alpha(\boldsymbol{x})(\boldsymbol{x}-\boldsymbol{a})^\alpha, \\
& \mbox{and}\quad \lim_{\boldsymbol{x}\to \boldsymbol{a}}h_\alpha(\boldsymbol{x})=0.
\end{align}</math>

In our model, we think of $\textbf{asp}$ as a function of $\textbf{fbsp}, \textbf{tby}, \textbf{div}, \textbf{ffr}, \textbf{fta}$, etc. Say

$$\textbf{asp}:=F(\textbf{fbsp}, \textbf{tby}, \textbf{div}, \textbf{ffr}, \textbf{fta},\cdots)$$

With the assumption that $F$ here is regular, we can apply the Taylor's theorem for multivariate functions from above to $F$. Ideally, we have to consider all possible products in our implementation. But in our implementation, we chose to make a balance between our intuitive nonlinear model and Taylor's theorem.

## A faster computation scheme
One drawback of the models we proposed about is that we have to call fbprophet thousands of times when we implement it.

Here is a method that reduces the number of times calling fbprophet:

Say in the training process we are considering from i=1,000 to 11,000. Namely based on the current scheme, we have to call fbprophet for 10,000 times.

Instead of this, we make a break-down 10,000=100*100 as follows:

For i=1,000 to 1,100, we only use the first 999 dates for training;

For i=1,100 to 1,200, we only use the first 1,099 dates for training;

.............................................

For i=10,900 to 11,000, we only use the first 10,899 dates for training;

In this way, it seems that we only need to call fbprophet for 100 times. And this doesn't seem harm the accuracy too much.

```python
import pandas as pd
import numpy as np

## For plotting
import matplotlib.pyplot as plt
from matplotlib import style
import datetime as dt
import seaborn as sns
sns.set_style("whitegrid")
```

```python
path = '../Data/dff1.csv'
```

```python
df= pd.read_csv(path, parse_dates=['ds'])
# df = df.rename(columns = {"Date":"ds","Close":"y"})
df = df[['ds', 'y','fbsp', 'diff','tby', 'ffr', 'fta', 'eps', 'div', 'une', 'wti', 'ppi',
       'rfs']]
# df
```

```python
df['fbsp_tby'] = df['fbsp'] * df['tby']
df['fbsp_ffr'] = df['fbsp'] * df['ffr']
df['fbsp_div'] = df['fbsp'] * df['div']
df['eps_tby'] = df['eps'] * df['tby']
df['eps_ffr'] = df['eps'] * df['ffr']
df['eps_div'] = df['eps'] * df['div']
```

```python
# cutoff between test and train data
cutoff = len(df) - 252
df_train = df[:cutoff].copy()
df_test = df[cutoff:].copy()
print(cutoff)
```

    2300

```python
df_train.columns
```

    Index(['ds', 'y', 'fbsp', 'diff', 'tby', 'ffr', 'fta', 'eps', 'div', 'une',
           'wti', 'ppi', 'rfs', 'fbsp_tby', 'fbsp_ffr', 'fbsp_div', 'eps_tby',
           'eps_ffr', 'eps_div'],
          dtype='object')

```python
#possible_features = ['tby', 'ffr', 'fta', 'eps', 'div', 'une', 'wti',
#      'ppi', 'rfs', 'fbsp_tby', 'fbsp_ffr', 'fbsp_div', 'eps_tby',
#      'eps_ffr', 'eps_div']

possible_features = ['tby', 'ffr', 'fta', 'div', 'une',
      'ppi', 'rfs', 'fbsp_tby', 'fbsp_ffr', 'fbsp_div']

from itertools import chain, combinations

def powerset(iterable):
    #"powerset([1,2,3]) --> () (1,) (2,) (3,) (1,2) (1,3) (2,3) (1,2,3)"
    s = list(iterable)
    return chain.from_iterable(combinations(s, r) for r in range(len(s)+1))

#print(list(powerset(possible_features)))
```

```python
len(possible_features)
```

    10

```python
from statsmodels.regression.linear_model import OLS

reg_new = OLS((df_train['diff']).copy(),df_train[possible_features].copy()).fit()
print(reg_new.params)

#from the output, we can see it's consistent with sklearn output
```

    tby         142.277390
    ffr          85.508342
    fta           0.000068
    div         366.196032
    une        -139.915289
    ppi          -0.651339
    rfs           0.003782
    fbsp_tby     -0.060019
    fbsp_ffr      0.017499
    fbsp_div     -0.462135
    dtype: float64

```python
new_coef = reg_new.params
new_possible_feats = new_coef[abs(new_coef)>0].index

power_feats = list(powerset(new_possible_feats))
power_feats.remove(())

power_feats = [ list(feats) for feats in power_feats]
len(power_feats)

```

    1023

```python
AIC_scores = []
parameters = []

for feats in power_feats:
    tmp_reg = OLS((df_train['diff']).copy(),df_train[feats].copy()).fit()
    AIC_scores.append(tmp_reg.aic)
    parameters.append(tmp_reg.params)

Min_AIC_index = AIC_scores.index(min(AIC_scores))
Min_AIC_feats = power_feats[Min_AIC_index]
Min_AIC_params  = parameters[Min_AIC_index]
print(Min_AIC_feats)
print(Min_AIC_params)
```

    ['tby', 'ffr', 'fta', 'div', 'une', 'rfs', 'fbsp_tby', 'fbsp_div']
    tby         151.672502
    ffr         133.514484
    fta           0.000063
    div         359.410771
    une        -144.152409
    rfs           0.003479
    fbsp_tby     -0.064792
    fbsp_div     -0.447041
    dtype: float64

```python
len(Min_AIC_feats)
```

    8

```python
###After selecting the best features, we report the testing error, and make the plot
AIC_df_test = df_test[Min_AIC_feats]
AIC_pred_test = AIC_df_test.dot(Min_AIC_params)+df_test.fbsp

AIC_df_train = df_train[Min_AIC_feats]
AIC_pred_train = AIC_df_train.dot(Min_AIC_params)+ df_train.fbsp

```

```python
from sklearn.metrics import mean_squared_error as MSE

mse_train = MSE(df_train.y, AIC_pred_train)
mse_test = MSE(df_test.y, AIC_pred_test)

#compare with fbprophet()

fb_mse_train = MSE(df_train.y, df_train.fbsp)
fb_mse_test = MSE(df_test.y, df_test.fbsp)

print(mse_train,fb_mse_train)

print(mse_test,fb_mse_test)
```

    3021.3722993387496 22303.56360854362
    39273.94024676501 15247.912341091065

```python
df_train.ds
```

    0      2009-12-15
    1      2009-12-16
    2      2009-12-17
    3      2009-12-18
    4      2009-12-21
              ...
    2295   2019-02-20
    2296   2019-02-21
    2297   2019-02-22
    2298   2019-02-25
    2299   2019-02-26
    Name: ds, Length: 2300, dtype: datetime64[ns]

```python
plt.figure(figsize=(18,10))

# plot the training data
plt.plot(df_train.ds,df_train.y,'b',
            label = "Training Data")

plt.plot(df_train.ds, AIC_pred_train,'r-',
            label = "Improved Fitted Values by Best_AIC")

# # plot the fit
plt.plot(df_train.ds, df_train.fbsp,'g-',
            label = "FB Fitted Values")

# # plot the forecast
plt.plot(df_test.ds, df_test.fbsp,'g--',
            label = "FB Forecast")
plt.plot(df_test.ds, AIC_pred_test,'r--',
            label = "Improved Forecast by Best_AIC")
plt.plot(df_test.ds,df_test.y,'b--',
            label = "Test Data")

plt.legend(fontsize=14)

plt.xlabel("Date", fontsize=16)
plt.ylabel("SP&500 Close Price", fontsize=16)

plt.show()
```

```python
plt.figure(figsize=(18,10))
plt.plot(df_test.y,label="Training Data")
plt.plot(df_test.fbsp,label="FB Forecast")
plt.plot(AIC_pred_test,label="Improved Forecast by Best_AIC")
plt.legend(fontsize = 14)
plt.show()
```

```python

```

```python
column = ['tby', 'ffr', 'fta', 'eps', 'div', 'une',
                                                  'wti', 'ppi', 'rfs', 'fbsp_tby', 'fbsp_ffr', 'fbsp_div', 'eps_tby', 'eps_ffr', 'eps_div']
```

```python
from sklearn import preprocessing
df1_train = df_train[['diff', 'tby', 'ffr', 'fta', 'eps', 'div', 'une', 'wti', 'ppi', 'rfs', 'fbsp_tby', 'fbsp_ffr', 'fbsp_div', 'eps_tby', 'eps_ffr', 'eps_div']]

X = preprocessing.scale(df1_train)
from statsmodels.regression.linear_model import OLS

reg_new = OLS((X[:,0]).copy(),X[:,1:].copy()).fit()
print(reg_new.params)
```

    [ 1.50405129  1.03228322  0.27409454  1.17073571  0.31243092 -0.75747342
      0.46988206 -0.39944639  2.10369448 -0.69112943 -2.1804296  -2.38576385
     -1.14196633  1.41832903 -0.34501927]

```python
# Before Covid
# pd.Series(reg_new.params, index=['tby', 'ffr', 'fta', 'eps', 'div', 'une',
#                                                   'wti', 'ppi', 'rfs', 'fbsp_tby', 'fbsp_ffr', 'fbsp_div', 'eps_tby', 'eps_ffr', 'eps_div'] )
```

```python
# before covid
coef1 = [ 1.50405129,  1.03228322,  0.27409454,  1.17073571,  0.31243092,
       -0.75747342,  0.46988206, -0.39944639,  2.10369448, -0.69112943,
       -2.1804296 , -2.38576385, -1.14196633,  1.41832903, -0.34501927]
# include covid
coef2 = [ 0.65150054,  1.70457239, -0.1573802 , -0.18007979, -0.15221931,
       -0.62326075,  0.45065894, -0.38972706,  2.87210843, -1.17604495,
       -4.92858316, -2.15459111,  0.11418468,  2.74829778,  0.55520382]
```

```python
# Include Covid
# pd.Series( np.append( ['coefficients (before covid)'], np.round(coef1,3)),  index= np.append(['features'], column) )

```

```python
index1 = ['10 Year U.S Treasury Bond Yield Rates (tby)', 'Federal Funds Rates (ffr)',
        'Federal Total Assets (fta)', 'Earning-Per-Share of S&P 500 (eps)', 'Dividend Yield of S&P 500 (div)',
        'Unemployment Rates (une) ', 'West Texas Intermediate oil index (wit)', 'Producer Price Index (ppi)',
         'Retail and Food Services Sales (rfs)',
         'fbsp_tby', 'fbsp_ffr', 'fbsp_div', 'eps_tby', 'eps_ffr', 'eps_div'
        ]
```

```python
len(index1)
```

    15

```python
pd.Series(coef2, index =index1)
```

    10 Year U.S Treasury Bond Yield Rates (tby)    0.651501
    Federal Funds Rates (ffr)                      1.704572
    Federal Total Assets (fta)                    -0.157380
    Earning-Per-Share of S&P 500 (eps)            -0.180080
    Dividend Yield of S&P 500 (div)               -0.152219
    Unemployment Rates (une)                      -0.623261
    West Texas Intermediate oil index (wit)        0.450659
    Producer Price Index (ppi)                    -0.389727
    Retail and Food Services Sales (rfs)           2.872108
    fbsp_tby                                      -1.176045
    fbsp_ffr                                      -4.928583
    fbsp_div                                      -2.154591
    eps_tby                                        0.114185
    eps_ffr                                        2.748298
    eps_div                                        0.555204
    dtype: float64

```python
df3 = pd.DataFrame(coef1, index = index1, columns = ['coefficients (before covid)'])
df3['coefficients (include covid)'] =pd.Series(coef2, index =index1)
df3
```

      coefficients (before covid)
      coefficients (include covid)

      10 Year U.S Treasury Bond Yield Rates (tby)
      1.504051
      0.651501

      Federal Funds Rates (ffr)
      1.032283
      1.704572

      Federal Total Assets (fta)
      0.274095
      -0.157380

      Earning-Per-Share of S&P 500 (eps)
      1.170736
      -0.180080

      Dividend Yield of S&P 500 (div)
      0.312431
      -0.152219

      Unemployment Rates (une)
      -0.757473
      -0.623261

      West Texas Intermediate oil index (wit)
      0.469882
      0.450659

      Producer Price Index (ppi)
      -0.399446
      -0.389727

      Retail and Food Services Sales (rfs)
      2.103694
      2.872108

      fbsp_tby
      -0.691129
      -1.176045

      fbsp_ffr
      -2.180430
      -4.928583

      fbsp_div
      -2.385764
      -2.154591

      eps_tby
      -1.141966
      0.114185

      eps_ffr
      1.418329
      2.748298

      eps_div
      -0.345019
      0.555204

```python

```

Code/Scenario2 including the covid period without wti, eps.ipynb


In this notebook, you will implement the forward longitudinal vehicle model. The model accepts throttle inputs and steps through the longitudinal dynamic equations. Once implemented, you will be given a set of inputs that drives over a small road slope to test your model.

The input to the model is a throttle percentage $x_\theta \in [0,1]$ which provides torque to the engine and subsequently accelerates the vehicle for forward motion.

The dynamic equations consist of many stages to convert throttle inputs to wheel speed (engine -> torque converter -> transmission -> wheel). These stages are bundled together in a single inertia term $J_e$ which is used in the following combined engine dynamic equations.

\begin{align}
    J_e \dot{\omega}_e &= T_e - (GR)(r_{eff} F_{load}) \\ m\ddot{x} &= F_x - F_{load}
\end{align}

Where $T_e$ is the engine torque, $GR$ is the gear ratio, $r_{eff}$ is the effective radius, $m$ is the vehicle mass, $x$ is the vehicle position, $F_x$ is the tire force, and $F_{load}$ is the total load force.

The engine torque is computed from the throttle input and the engine angular velocity $\omega_e$ using a simplified quadratic model.

\begin{align}
    T_e = x_{\theta}(a_0 + a_1 \omega_e + a_2 \omega_e^2)
\end{align}

The load forces consist of aerodynamic drag $F_{aero}$, rolling friction $R_x$, and gravitational force $F_g$ from an incline at angle $\alpha$. The aerodynamic drag is a quadratic model and the friction is a linear model.

\begin{align}
    F_{load} &= F_{aero} + R_x + F_g \\
    F_{aero} &= \frac{1}{2} C_a \rho A \dot{x}^2 = c_a \dot{x}^2\\
    R_x &= N(\hat{c}_{r,0} + \hat{c}_{r,1}|\dot{x}| + \hat{c}_{r,2}\dot{x}^2) \approx c_{r,1} \dot{x}\\
    F_g &= mg\sin{\alpha}
\end{align}

Note that the absolute value is ignored for friction since the model is used for only forward motion ($\dot{x} \ge 0$).

The tire force is computed using the engine speed and wheel slip equations.

\begin{align}
    \omega_w &= (GR)\omega_e \\
    s &= \frac{\omega_w r_e - \dot{x}}{\dot{x}}\\
    F_x &= \left\{\begin{array}{lr}
        cs, &  |s| < 1\\
        F_{max}, & \text{otherwise}
        \end{array}\right\}
\end{align}

Where $\omega_w$ is the wheel angular velocity and $s$ is the slip ratio.

We setup the longitudinal model inside a Python class below. The vehicle begins with an initial velocity of 5 m/s and engine speed of 100 rad/s. All the relevant parameters are defined and like the bicycle model, a sampling time of 10ms is used for numerical integration.

```python
from math import sin, atan2
import sys
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.image as mpimg

class Vehicle():
    def __init__(self):

        # ==================================
        #  Parameters
        # ==================================

        #Throttle to engine torque
        self.a_0 = 400
        self.a_1 = 0.1
        self.a_2 = -0.0002

        # Gear ratio, effective radius, mass + inertia
        self.GR = 0.35
        self.r_e = 0.3
        self.J_e = 10
        self.m = 2000
        self.g = 9.81

        # Aerodynamic and friction coefficients
        self.c_a = 1.36
        self.c_r1 = 0.01

        # Tire force
        self.c = 10000
        self.F_max = 10000

        # State variables
        self.x = 0
        self.v = 5
        self.a = 0
        self.w_e = 100
        self.w_e_dot = 0

        self.sample_time = 0.01

    def reset(self):
        # reset state variables
        self.x = 0
        self.v = 5
        self.a = 0
        self.w_e = 100
        self.w_e_dot = 0
```

Implement the combined engine dynamic equations along with the force equations in the cell below. The function $\textit{step}$ takes the throttle $x_\theta$ and incline angle $\alpha$ as inputs and performs numerical integration over one timestep to update the state variables. Hint: Integrate to find the current position, velocity, and engine speed first, then propagate those values into the set of equations.

```python
class Vehicle(Vehicle):
    def step(self, throttle, alpha):
        # ==================================
        #  Implement vehicle model here
        # ==================================
        x_dot = self.v
        v_dot = self.a
        w_e_dot = self.w_e_dot
        self.x += x_dot * self.sample_time
        self.v += v_dot * self.sample_time
        self.w_e += w_e_dot * self.sample_time
        omega_w = self.GR * self.w_e # Wheel speed from engine speed
        s = omega_w * self.r_e / self.v - 1 # Slip ratio
        F_x = self.c * s if abs(s) < 1 else self.F_max
        F_g = self.m * self.g * sin(alpha)
        R_x = self.c_r1 * self.v
        F_aero = self.c_a * self.v**2
        F_load = F_aero + R_x + F_g
        T_e = throttle * (self.a_0 + self.a_1 * self.w_e + self.a_2 * self.w_e**2)
        self.a = (F_x - F_load) / self.m
        self.w_e_dot = (T_e - self.GR * self.r_e * F_load) / self.J_e
```

Using the model, you can send constant throttle inputs to the vehicle in the cell below. You will observe that the velocity converges to a fixed value based on the throttle input due to the aerodynamic drag and tire force limit. A similar velocity profile can be seen by setting a negative incline angle $\alpha$. In this case, gravity accelerates the vehicle to a terminal velocity where it is balanced by the drag force.

```python
sample_time = 0.01
time_end = 100
model = Vehicle()

t_data = np.arange(0,time_end,sample_time)
v_data = np.zeros_like(t_data)

# throttle percentage between 0 and 1
throttle = 0.2

# incline angle (in radians)
alpha = 0

for i in range(t_data.shape[0]):
    v_data[i] = model.v
    model.step(throttle, alpha)

plt.plot(t_data, v_data)
plt.show()
```

We will now drive the vehicle over a slope as shown in the diagram below.

To climb the slope, a trapezoidal throttle input is provided for the next 20 seconds as shown in the figure below.

The vehicle begins at 20% throttle and gradually increases to 50% throttle. This is maintained for 10 seconds as the vehicle climbs the steeper slope. Afterwards, the vehicle reduces the throttle to 0.

In the cell below, implement the ramp angle profile $\alpha (x)$ and throttle profile $x_\theta (t)$ and step them through the vehicle dynamics. The vehicle position $x(t)$ is saved in the array $\textit{x_data}$. This will be used to grade your solution.

```python
time_end = 20
t_data = np.arange(0,time_end,sample_time)
x_data = np.zeros_like(t_data)
v_data = np.zeros_like(t_data)
w_e_data = np.zeros_like(t_data)

# reset the states
model.reset()

# ==================================
#  Learner solution begins here
# ==================================
def angle(i, alpha, x):
    if x < 60:
        alpha[i] = np.arctan(3/60)
    elif x < 150:
        alpha[i] = np.arctan(9/90)
    else:
        alpha[i] = 0

throttle = np.zeros_like(t_data)
alpha = np.zeros_like(t_data)

#throttle depends on time and alpha depends on distance travelled (model.x)
for i in range(t_data.shape[0]):
    if t_data[i] < 5:
        throttle[i] = 0.2 + ((0.5 - 0.2)/5)*t_data[i]
        angle(i, alpha, model.x)
    elif t_data[i] < 15:
        throttle[i] = 0.5
        angle(i, alpha, model.x)
    else:
        throttle[i] = ((0 - 0.5)/(20 - 15))*(t_data[i] - 20)
        angle(i, alpha, model.x)

    #call the step function and update x_data array
    model.step(throttle[i], alpha[i])
    x_data[i] = model.x
    v_data[i] = model.v
    w_e_data[i] = model.w_e

# ==================================
#  Learner solution ends here
# ==================================

# Plot x vs t for visualization
plt.plot(t_data, x_data)
plt.show()
```

If you have implemented the vehicle model and inputs correctly, you should see that the vehicle crosses the ramp at ~15s where the throttle input begins to decrease.

The cell below will save the time and vehicle inputs as text file named $\textit{xdata.txt}$. To locate the file, change the end of your web directory to $\textit{/notebooks/Course_1_Module_4/xdata.txt}$

Once you are there, you can download the file and submit to the Coursera grader to complete this assessment.

```python
data = np.vstack([t_data, x_data]).T
np.savetxt('xdata.txt', data, delimiter=', ')
```

Congratulations! You have now completed the assessment! Feel free to test the vehicle model with different inputs in the cell below, and see what trajectories they form. In the next module, you will see the longitudinal model being used for speed control. See you there!

```python
sample_time = 0.01
time_end = 30
model.reset()

t_data = np.arange(0,time_end,sample_time)
x_data = np.zeros_like(t_data)

# ==================================
#  Test various inputs here
# ==================================
for i in range(t_data.shape[0]):

    model.step(0,0)

plt.axis('equal')
plt.plot(x_data, t_data)
plt.show()
```

```python

```

Introduction to Self Driving Cars/week_4/Programming_Exercises/Longitudinal_Vehicle_Model.ipynb


# Robust Kalman filtering for vehicle tracking

We will try to pinpoint the location of a moving vehicle with high accuracy from noisy sensor data. We'll do this by modeling the vehicle state as a discrete-time linear dynamical system. Standard **Kalman filtering** can be used to approach this problem when the sensor noise is assumed to be Gaussian. We'll use **robust Kalman filtering** to get a more accurate estimate of the vehicle state for a non-Gaussian case with outliers.

# Problem statement

A discrete-time linear dynamical system consists of a sequence of state vectors $x_t \in \mathbf{R}^n$, indexed by time $t \in \lbrace 0, \ldots, N-1 \rbrace$ and dynamics equations

\begin{align}
x_{t+1} &= Ax_t + Bw_t\\
y_t &=Cx_t + v_t,
\end{align}

where $w_t \in \mathbf{R}^m$ is an input to the dynamical system (say, a drive force on the vehicle), $y_t \in \mathbf{R}^r$ is a state measurement, $v_t \in \mathbf{R}^r$ is noise, $A$ is the drift matrix, $B$ is the input matrix, and $C$ is the observation matrix.

Given $A$, $B$, $C$, and $y_t$ for $t = 0, \ldots, N-1$, the goal is to estimate $x_t$ for $t = 0, \ldots, N-1$.

# Kalman filtering

A Kalman filter estimates $x_t$ by solving the optimization problem

\begin{array}{ll}
\mbox{minimize} & \sum_{t=0}^{N-1} \left(
\|w_t\|_2^2 + \tau \|v_t\|_2^2\right)\\
\mbox{subject to} & x_{t+1} = Ax_t + Bw_t,\quad t=0,\ldots, N-1\\
& y_t = Cx_t+v_t,\quad t = 0, \ldots, N-1,
\end{array}

where $\tau$ is a tuning parameter. This problem is actually a least squares problem, and can be solved via linear algebra, without the need for more general convex optimization. Note that since we have no observation $y_{N}$, $x_N$ is only constrained via $x_{N} = Ax_{N-1} + Bw_{N-1}$, which is trivially resolved when $w_{N-1} = 0$ and $x_{N} = Ax_{N-1}$. We maintain this vestigial constraint only because it offers a concise problem statement.

This model performs well when $w_t$ and $v_t$ are Gaussian. However, the quadratic objective can be influenced by large outliers, which degrades the accuracy of the recovery. To improve estimation in the presence of outliers, we can use **robust Kalman filtering**.

# Robust Kalman filtering

To handle outliers in $v_t$, robust Kalman filtering replaces the quadratic cost with a Huber cost, which results in the convex optimization problem

\begin{array}{ll}
\mbox{minimize} & \sum_{t=0}^{N-1} \left( \|w_t\|^2_2 + \tau \phi_\rho(v_t) \right)\\
\mbox{subject to} & x_{t+1} = Ax_t + Bw_t,\quad t=0,\ldots, N-1\\
& y_t = Cx_t+v_t,\quad t=0,\ldots, N-1,
\end{array}

where $\phi_\rho$ is the Huber function
$$
\phi_\rho(a)= \left\{ \begin{array}{ll} \|a\|_2^2 & \|a\|_2\leq \rho\\
2\rho \|a\|_2-\rho^2 & \|a\|_2>\rho.
\end{array}\right.
$$

The Huber penalty function penalizes estimation error linearly outside of a ball of radius $\rho$, whereas in standard Kalman filtering, all errors are penalized quadratically. Thus, large errors are penalized less harshly, making this model more robust to outliers.

# Vehicle tracking example

We'll apply standard and robust Kalman filtering to a vehicle tracking problem with state $x_t \in \mathbf{R}^4$, where
$(x_{t,0}, x_{t,1})$ is the position of the vehicle in two dimensions, and $(x_{t,2}, x_{t,3})$ is the vehicle velocity.
The vehicle has unknown drive force $w_t$, and we observe noisy measurements of the vehicle's position, $y_t \in \mathbf{R}^2$.

The matrices for the dynamics are

$$
A = \begin{bmatrix}
1 & 0 & \left(1-\frac{\gamma}{2}\Delta t\right) \Delta t & 0 \\
0 & 1 & 0 & \left(1-\frac{\gamma}{2} \Delta t\right) \Delta t\\
0 & 0 & 1-\gamma \Delta t & 0 \\
0 & 0 & 0 & 1-\gamma \Delta t
\end{bmatrix},
$$

$$
B = \begin{bmatrix}
\frac{1}{2}\Delta t^2 & 0 \\
0 & \frac{1}{2}\Delta t^2 \\
\Delta t & 0 \\
0 & \Delta t \\
\end{bmatrix},
$$

$$
C = \begin{bmatrix}
1 & 0 & 0 & 0 \\
0 & 1 & 0 & 0
\end{bmatrix},
$$
where $\gamma$ is a velocity damping parameter.

# 1D Model
The recurrence is derived from the following relations in a single dimension. For this subsection, let $x_t, v_t, w_t$ be the vehicle position, velocity, and input drive force. The resulting acceleration of the vehicle is $w_t - \gamma v_t$, with $- \gamma v_t$ is a damping term depending on velocity with parameter $\gamma$.

The discretized dynamics are obtained from numerically integrating:
$$
\begin{align}
x_{t+1} &= x_t + \left(1-\frac{\gamma \Delta t}{2}\right)v_t \Delta t + \frac{1}{2}w_{t} \Delta t^2\\
v_{t+1} &= \left(1-\gamma\right)v_t + w_t \Delta t.
\end{align}
$$

Extending these relations to two dimensions gives us the dynamics matrices $A$ and $B$.

## Helper Functions

```
import matplotlib
import matplotlib.pyplot as plt
import numpy as np

def plot_state(t,actual, estimated=None):
    '''
    plot position, speed, and acceleration in the x and y coordinates for
    the actual data, and optionally for the estimated data
    '''
    trajectories = [actual]
    if estimated is not None:
        trajectories.append(estimated)

    fig, ax = plt.subplots(3, 2, sharex='col', sharey='row', figsize=(8,8))
    for x, w in trajectories:
        ax[0,0].plot(t,x[0,:-1])
        ax[0,1].plot(t,x[1,:-1])
        ax[1,0].plot(t,x[2,:-1])
        ax[1,1].plot(t,x[3,:-1])
        ax[2,0].plot(t,w[0,:])
        ax[2,1].plot(t,w[1,:])

    ax[0,0].set_ylabel('x position')
    ax[1,0].set_ylabel('x velocity')
    ax[2,0].set_ylabel('x input')

    ax[0,1].set_ylabel('y position')
    ax[1,1].set_ylabel('y velocity')
    ax[2,1].set_ylabel('y input')

    ax[0,1].yaxis.tick_right()
    ax[1,1].yaxis.tick_right()
    ax[2,1].yaxis.tick_right()

    ax[0,1].yaxis.set_label_position("right")
    ax[1,1].yaxis.set_label_position("right")
    ax[2,1].yaxis.set_label_position("right")

    ax[2,0].set_xlabel('time')
    ax[2,1].set_xlabel('time')

def plot_positions(traj, labels, axis=None,filename=None):
    '''
    show point clouds for true, observed, and recovered positions
    '''
    matplotlib.rcParams.update({'font.size': 14})
    n = len(traj)

    fig, ax = plt.subplots(1, n, sharex=True, sharey=True,figsize=(12, 5))
    if n == 1:
        ax = [ax]

    for i,x in enumerate(traj):
        ax[i].plot(x[0,:], x[1,:], 'ro', alpha=.1)
        ax[i].set_title(labels[i])
        if axis:
            ax[i].axis(axis)

    if filename:
        fig.savefig(filename, bbox_inches='tight')
```

## Problem Data

We generate the data for the vehicle tracking problem. We'll have $N=1000$, $w_t$ a standard Gaussian, and $v_t$ a standard Guassian, except $20\%$ of the points will be outliers with $\sigma = 20$.

Below, we set the problem parameters and define the matrices $A$, $B$, and $C$.

```
n = 1000 # number of timesteps
T = 50 # time will vary from 0 to T with step delt
ts, delt = np.linspace(0,T,n,endpoint=True, retstep=True)
gamma = .05 # damping, 0 is no damping

A = np.zeros((4,4))
B = np.zeros((4,2))
C = np.zeros((2,4))

A[0,0] = 1
A[1,1] = 1
A[0,2] = (1-gamma*delt/2)*delt
A[1,3] = (1-gamma*delt/2)*delt
A[2,2] = 1 - gamma*delt
A[3,3] = 1 - gamma*delt

B[0,0] = delt**2/2
B[1,1] = delt**2/2
B[2,0] = delt
B[3,1] = delt

C[0,0] = 1
C[1,1] = 1
```

# Simulation

We seed $x_0 = 0$ (starting at the origin with zero velocity) and simulate the system forward in time. The results are the true vehicle positions `x_true` (which we will use to judge our recovery) and the observed positions `y`.

We plot the position, velocity, and system input $w$ in both dimensions as a function of time.
We also plot the sets of true and observed vehicle positions.

```
sigma = 20
p = .20
np.random.seed(6)

x = np.zeros((4,n+1))
x[:,0] = [0,0,0,0]
y = np.zeros((2,n))

# generate random input and noise vectors
w = np.random.randn(2,n)
v = np.random.randn(2,n)

# add outliers to v
np.random.seed(0)
inds = np.random.rand(n) <= p
v[:,inds] = sigma*np.random.randn(2,n)[:,inds]

# simulate the system forward in time
for t in range(n):
    y[:,t] = C.dot(x[:,t]) + v[:,t]
    x[:,t+1] = A.dot(x[:,t]) + B.dot(w[:,t])

x_true = x.copy()
w_true = w.copy()

plot_state(ts,(x_true,w_true))
plot_positions([x_true,y], ['True', 'Observed'],[-4,14,-5,20],'rkf1.pdf')
```

# Kalman filtering recovery

The code below solves the standard Kalman filtering problem using CVXPY. We plot and compare the true and recovered vehicle states. Note that the recovery is distorted by outliers in the measurements.

```
%%time
from cvxpy import *

x = Variable(shape=(4, n+1))
w = Variable(shape=(2, n))
v = Variable(shape=(2, n))

tau = .08

obj = sum_squares(w) + tau*sum_squares(v)
obj = Minimize(obj)

constr = []
for t in range(n):
    constr += [ x[:,t+1] == A*x[:,t] + B*w[:,t] ,
                y[:,t]   == C*x[:,t] + v[:,t]   ]

Problem(obj, constr).solve(verbose=True)

x = np.array(x.value)
w = np.array(w.value)

plot_state(ts,(x_true,w_true),(x,w))
plot_positions([x_true,y], ['True', 'Noisy'], [-4,14,-5,20])
plot_positions([x_true,x], ['True', 'KF recovery'], [-4,14,-5,20], 'rkf2.pdf')
```

# Robust Kalman filtering recovery

Here we implement robust Kalman filtering with CVXPY. We get a better recovery than the standard Kalman filtering, which can be seen in the plots below.

```
%%time
from cvxpy import *
x = Variable(shape=(4, n+1))
w = Variable(shape=(2, n))
v = Variable(shape=(2, n))

tau = 2
rho = 2

obj = sum_squares(w)
obj += sum(tau*huber(norm(v[:,t]),rho) for t in range(n))
obj = Minimize(obj)

constr = []
for t in range(n):
    constr += [ x[:,t+1] == A*x[:,t] + B*w[:,t] ,
                y[:,t]   == C*x[:,t] + v[:,t]   ]

Problem(obj, constr).solve(verbose=True)

x = np.array(x.value)
w = np.array(w.value)

plot_state(ts,(x_true,w_true),(x,w))
plot_positions([x_true,y], ['True', 'Noisy'], [-4,14,-5,20])
plot_positions([x_true,x], ['True', 'Robust KF recovery'], [-4,14,-5,20],'rkf3.pdf')
```


```python
# Solving an expression in python

my_value = 2 * (3 + 2)**2 / 5 - 4

print(my_value)
```

    6.0

```python
# making use of parenthesis for clarity in python

my_value = 2 * ((3 + 2)**2 / 5) - 4

print(my_value) # prints 6.0
```

    6.0

```python
# Functions are expressions that define relationships between two or more variables.
# More specifically, a function takes input variables (also called domain variables or independent variables),
# plugs them into an expression, and then results in an output variable (also called dependent variable).

def f(x):
    return 2 * x + 1

x_values = [0, 1, 2, 3]

for x in x_values:
    y = f(x)
    print(y)

```

    1
    3
    5
    7

```python
# Charting a linear function in  python using Sympy (Sympy comes auto bundled in Anaconda)
from matplotlib import *
from sympy import *

x = symbols('x')
f = 2*x + 1
plot(f)
```

```python
x = symbols('x')
f = x**2 + 1
plot(f)
```

```python
from sympy.plotting import plot3d

x, y = symbols('x y')
f = 2*x + 3*y
plot3d(f)
```

```python
# A summation is expressed as a sigma  and adds elements together.
# Summation in python

summation = sum(2*i for i in range(1,6))
print(summation)
```

    30

```python
# Exponents multiplies a number by itself a specified number of times
# .The base is the variable or value we are exponentiating,
# and the exponent is the number of times we multiply the base value.
# using sympy to print exponent expression

x = symbols('x')
expr = x**2 / x**5
print(expr) # x**(-3)
```

    x**(-3)

```python
# A logarithm is a math function that finds a power for a specific number and base.
# Using the log function in python

from math import log

# 2 raised to what power gives me 8?
x = log(8, 2)

print(x)
```

    3.0

```python
# There is a special number that shows up quite a bit in math called Euler’s number e.
# It is a special number much like Pi π, and is approximately 2.71828.
# e is used a lot because it mathematically simplifies a lot of problems.

# Calculating continuous interest in python, A = P*e**rt

from math import exp

p = 100 # principal, starting amount
r = .20 # interest rate, by year
t = 2.0 # time, number of years

a = p * exp(r*t)

print(a)
```

    149.18246976412703

```python
# When we use e as our base for a logarithm, we call it a natural logarithm.

from math import log

# e raised to what power gives us 10?
x = log(10)

print(x) # prints 2.302585092994046
```

    2.302585092994046

```python
# A derivative tells the slope of a function, and it is useful to measure the rate of change at any point in a function.
# derivative calculator in python

def derivative_x(f, x, step_size):
    m = (f(x + step_size) - f(x)) / ((x + step_size) - x)
    return m

def my_function(x):
    return x**2

slope_at_2 = derivative_x(my_function, 2, .00001)

print(slope_at_2)
```

    4.000010000000827

```python
# Calculating a derivative in Sympy

# Declare 'x' to SymPy
x = symbols('x')

# Now just use Python syntax to declare function
f = x**2

# Calculate the derivative of the function
dx_f = diff(f)
print(dx_f) # prints 2*x
```

    2*x

```python
# Calculating partial derivative with Sympy

# Declare x and y to SymPy
x,y = symbols('x y')

# Now just use Python syntax to declare function
f = 2*x**3 + 3*y**3

# Calculate the partial derivatives for x and y
dx_f = diff(f, x)
dy_f = diff(f, y)

print(dx_f) # prints 6*x**2
print(dy_f) # prints 9*y**2

# plot the function
plot3d(f)
```

```python
#  integral, which finds the area under the curve for a given range.
# Integral approximation in python

def approximate_integral(a, b, n, f):
    delta_x = (b - a) / n
    total_sum = 0

    for i in range(1, n + 1):
        midpoint = 0.5 * (2 * a + delta_x * (2 * i - 1))
        total_sum += f(midpoint)

    return total_sum * delta_x

def my_function(x):
    return x**2 + 1

area = approximate_integral(a=0, b=1, n=5, f=my_function)

print(area)
```

    1.33

```python

```

Maths/Basic Maths and Calculus Review/Basic Maths and Calculus Review.ipynb


```python
import sys, os
sys.path.insert(0, os.path.join(os.pardir, 'src'))
from fe_approx1D_numint import approximate, mesh_uniform, u_glob
from sympy import sqrt, exp, sin, Symbol, lambdify, simplify
import numpy as np
from math import log

x = Symbol('x')
A = 1
w = 1

cases = {'sqrt': {'f': sqrt(x), 'Omega': [0,1]},
         'exp': {'f': A*exp(-w*x), 'Omega': [0, 3.0/w]},
         'sin': {'f': A*sin(w*x), 'Omega': [0, 2*np.pi/w]}}

results = {}
d_values = [1, 2, 3, 4]

for case in cases:
    f = cases[case]['f']
    f_func = lambdify([x], f, modules='numpy')
    Omega = cases[case]['Omega']
    results[case] = {}
    for d in d_values:
        results[case][d] = {'E': [], 'h': [], 'r': []}
        for N_e in [4, 8, 16, 32, 64, 128]:
            try:
                c = approximate(
                    f, symbolic=False,
                    numint='GaussLegendre%d' % (d+1),
                    d=d, N_e=N_e, Omega=Omega,
                    filename='tmp_%s_d%d_e%d' % (case, d, N_e))
            except np.linalg.linalg.LinAlgError as e:
                print((str(e)))
                continue
            vertices, cells, dof_map = mesh_uniform(
                N_e, d, Omega, symbolic=False)
            xc, u, _ = u_glob(c, vertices, cells, dof_map, 51)
            e = f_func(xc) - u
            # Trapezoidal integration of the L2 error over the
            # xc/u patches
            e2 = e**2
            L2_error = 0
            for i in range(len(xc)-1):
                L2_error += 0.5*(e2[i+1] + e2[i])*(xc[i+1] - xc[i])
            L2_error = np.sqrt(L2_error)
            h = (Omega[1] - Omega[0])/float(N_e)
            results[case][d]['E'].append(L2_error)
            results[case][d]['h'].append(h)
        # Compute rates
        h = results[case][d]['h']
        E = results[case][d]['E']
        for i in range(len(h)-1):
            r = log(E[i+1]/E[i])/log(h[i+1]/h[i])
            results[case][d]['r'].append(round(r, 2))

print(results)
for case in results:
    for d in sorted(results[case]):
        print(('case=%s d=%d, r: %s' % \
              (case, d, results[case][d]['r'])))

```

Data Science and Machine Learning/Machine-Learning-In-Python-THOROUGH/EXAMPLES/FINITE_ELEMENTS/INTRO/EXERCICES/19_PD_APPROX_ERROR.ipynb


# One-dimensional Lagrange Interpolation

The problem of interpolation or finding the value of a function at an arbitrary point $X$ inside a given domain, provided we have discrete known values of the function inside the same domain is at the heart of the finite element method. In this notebooke we use Lagrange interpolation where the approximation $\hat f(x)$ to the function $f(x)$ is built like:

\begin{equation}
\hat f(x)={L^I}(x)f^I
\end{equation}

In the expression above $L^I$ represents the $I$ Lagrange Polynomial of order $n-1$ and $f^1, f^2,,...,f^n$ are the $n$ known values of the function. Here we are using the summation convention over the repeated superscripts.

The $I$ Lagrange polynomial is given by the recursive expression:

\begin{equation}
{L^I}(x)=\prod_{J=1, J \ne I}^{n}{\frac{{\left( {x - {x^J}} \right)}}{{\left( {{x^I} - {x^J}} \right)}}}
\end{equation}

in the domain $x\in[-1.0,1.0]$.

We wish to interpolate the function $ f(x)=x^3+4x^2-10 $ assuming we know its value at points $x=-1.0$, $x=1.0$ and $x=0.0$.

```python
from __future__ import division
import numpy as np
from scipy import interpolate
import sympy as sym
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
```

```python
%matplotlib notebook

```

First we use a function to generate the Lagrage polynomial of order $order$ at point $i$

```python
def basis_lagrange(x_data, var, cont):
    """Find the basis for the Lagrange interpolant"""
    prod = sym.prod((var - x_data[i])/(x_data[cont] - x_data[i])
                    for i in range(len(x_data)) if i != cont)
    return sym.simplify(prod)
```

we now define the function $ f(x)=x^3+4x^2-10 $:

```python
fun = lambda x: x**3 + 4*x**2 - 10
```

```python
x = sym.symbols('x')
x_data = np.array([-1, 1, 0])
f_data = fun(x_data)
```

And obtain the Lagrange polynomials using:

```python
basis = []
for cont in range(len(x_data)):
    basis.append(basis_lagrange(x_data, x, cont))
    sym.pprint(basis[cont])

```

    x⋅(x - 1)
    ─────────
        2
    x⋅(x + 1)
    ─────────
        2
       2
    - x  + 1

which are shown in the following plots/

```python
npts = 101
x_eval = np.linspace(-1, 1, npts)
basis_num = sym.lambdify((x), basis, "numpy")  # Create a lambda function for the polynomials
```

```python
plt.figure(figsize=(6, 4))
for k in range(3):
    y_eval = basis_num(x_eval)[k]
    plt.plot(x_eval, y_eval)
```

    <IPython.core.display.Javascript object>

```python
y_interp = sym.simplify(sum(f_data[k]*basis[k] for k in range(3)))
y_interp
```

    4*x**2 + x - 10

Now we plot the complete approximating polynomial, the actual function and the points where the function was known.

```python
y_interp = sum(f_data[k]*basis_num(x_eval)[k] for k in range(3))
y_original = fun(x_eval)

plt.figure(figsize=(6, 4))
plt.plot(x_eval, y_original)
plt.plot(x_eval, y_interp)
plt.plot([-1, 1, 0], f_data, 'ko')
plt.show()
```

    <IPython.core.display.Javascript object>

## Interpolation in 2 dimensions

We can extend the concept of Lagrange interpolation to 2 or more dimensions.
In the case of bilinear interpolation (2×2, 4 vertices) in $[-1, 1]^2$,
the base functions are given by (**prove it**):

\begin{align}
N_0 = \frac{1}{4}(1 - x)(1 - y)\\
N_1 = \frac{1}{4}(1 + x)(1 - y)\\
N_2 = \frac{1}{4}(1 + x)(1 + y)\\
N_3 = \frac{1}{4}(1 - x)(1 + y)
\end{align}

Let's see an example using piecewise bilinear interpolation.

```python
def rect_grid(Lx, Ly, nx, ny):
    u"""Create a rectilinear grid for a rectangle

    The rectangle has dimensiones Lx by Ly. nx are
    the number of nodes in x, and ny are the number of nodes
    in y
    """
    y, x = np.mgrid[-Ly/2:Ly/2:ny*1j, -Lx/2:Lx/2:nx*1j]
    els = np.zeros(((nx - 1)*(ny - 1), 4), dtype=int)
    for row in range(ny - 1):
        for col in range(nx - 1):
            cont = row*(nx - 1) + col
            els[cont, :] = [cont + row, cont + row + 1,
                              cont + row + nx + 1, cont + row + nx]
    return x.flatten(), y.flatten(), els
```

```python
def interp_bilinear(coords, f_vals, grid=(10, 10)):
    """Piecewise bilinear interpolation for rectangular domains"""
    x_min, y_min = np.min(coords, axis=0)
    x_max, y_max = np.max(coords, axis=0)
    x, y = np.mgrid[-1:1:grid[0]*1j,-1:1:grid[1]*1j]
    N0 = (1 - x) * (1 - y)
    N1 = (1 + x) * (1 - y)
    N2 = (1 + x) * (1 + y)
    N3 = (1 - x) * (1 + y)
    interp_fun = N0 * f_vals[0] + N1 * f_vals[1] + N2 * f_vals[2] + N3 * f_vals[3]
    interp_fun = 0.25*interp_fun
    x, y = np.mgrid[x_min:x_max:grid[0]*1j, y_min:y_max:grid[1]*1j]
    return x, y, interp_fun
```

```python
def fun(x, y):
    """Monkey saddle function"""
    return y**3 +  3*y*x**2
```

```python
x_coords, y_coords, els = rect_grid(2, 2, 4, 4)
nels = els.shape[0]
z_coords = fun(x_coords, y_coords)
z_min = np.min(z_coords)
z_max = np.max(z_coords)
```

```python
fig = plt.figure(figsize=(6, 6))
ax = fig.add_subplot(111, projection='3d')
x, y = np.mgrid[-1:1:51j,-1:1:51j]
z = fun(x, y)
surf = ax.plot_surface(x, y, z, rstride=1, cstride=1, linewidth=0, alpha=0.6,
                      cmap="viridis")
plt.colorbar(surf, shrink=0.5, aspect=10)
ax.plot(x_coords, y_coords, z_coords, 'ok')
for k in range(nels):
    x_vals = x_coords[els[k, :]]
    y_vals = y_coords[els[k, :]]
    coords = np.column_stack([x_vals, y_vals])
    f_vals = fun(x_vals, y_vals)
    x, y, z = interp_bilinear(coords, f_vals, grid=[4, 4])
    inter = ax.plot_wireframe(x, y, z, color="black", cstride=3, rstride=3)
plt.xlabel(r"$x$", fontsize=18)
plt.ylabel(r"$y$", fontsize=18)
ax.legend([inter], [u"Interpolation"])
plt.show();
```

    <IPython.core.display.Javascript object>

```python

```

The next cell change the format of the Notebook.

```python
from IPython.core.display import HTML
def css_styling():
    styles = open('../styles/custom_barba.css', 'r').read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>


# Loss Functions

```python
%matplotlib inline
from matplotlib import pyplot as plt
import mxnet as mx
from mxnet import nd, autograd
import numpy as np
import math

x = nd.arange(-5, 5, 0.01)
x.attach_grad()
```

## $l_2$ loss

$$
\begin{align}
l(y,y') = \frac{1}{2}(y-y')^2
\end{align}
$$

```python
with autograd.record():
    l = 0.5 * x**2
l.backward()
p = nd.exp(-l)
p = 100 * p / p.sum()
plt.figure(figsize=(10, 5))
plt.plot(x.asnumpy(), l.asnumpy())
plt.plot(x.asnumpy(), x.grad.asnumpy())
plt.plot(x.asnumpy(), 10*p.asnumpy())
plt.show()
```

## $l_1$ loss

$$
\begin{align}
l(y,y') = |y-y'|
\end{align}
$$

```python
with autograd.record():
    l = nd.abs(x)
l.backward()
p = nd.exp(-l)
p = 100 * p / p.sum()
plt.figure(figsize=(10, 5))
plt.plot(x.asnumpy(), l.asnumpy())
plt.plot(x.asnumpy(), x.grad.asnumpy())
plt.plot(x.asnumpy(), 10*p.asnumpy())
plt.show()
```

## Huber's Robust loss

$$
\begin{align}
l(y,y') = \begin{cases}
|y-y'| - \frac{1}{2} & \text{ if } |y-y'| > 1 \\
\frac{1}{2} (y-y')^2 & \text{ otherwise}
\end{cases}
\end{align}
$$

```python
with autograd.record():
    tmp = nd.abs(x) > 1
    l = tmp * (nd.abs(x) - 0.5) + (1-tmp) * (0.5 * x**2)
l.backward()
p = nd.exp(-l)
p = 100 * p / p.sum()
plt.figure(figsize=(10, 5))
plt.plot(x.asnumpy(), l.asnumpy())
plt.plot(x.asnumpy(), x.grad.asnumpy())
plt.plot(x.asnumpy(), 10*p.asnumpy())
plt.show()
```

Many more loss functions, e.g.
* $\epsilon$-insensitive loss $\mathrm{max}(0, |y-y'| - \epsilon)$
* Huber's robust loss with width adjustment
* Log-normal loss, i.e. $(\log y - \log y')^2$

```python

```


Before you turn this problem in, make sure everything runs as expected. First, **restart the kernel** (in the menubar, select Kernel$\rightarrow$Restart) and then **run all cells** (in the menubar, select Cell$\rightarrow$Run All).

Make sure you fill in any place that says `YOUR CODE HERE` or "YOUR ANSWER HERE", as well as your name and collaborators below:

```python
NAME = "Prabal CHowdhury"
COLLABORATORS = ""
```

---

# Differentiation: Forward, Backward, And Central
---

## Task 1: Differentiation

We have already learnt about *forward differentiation*, *backward diferentiation* and *central differentiation*. In this part of the assignment we will write methods to calculate this values and check how they perform.

The equations are as follows,

\begin{align}
\text{forward differentiation}, f^\prime(x) \simeq \frac{f(x+h)-f(x)}{h} \tag{4.6} \\
\text{backward differentiation}, f^\prime(x) \simeq \frac{f(x)-f(x-h)}{h} \tag{4.7} \\
\text{central differentiation}, f^\prime(x) \simeq \frac{f(x+h)-f(x-h)}{2h} \tag{4.8}
\end{align}

## Importing libraries

```python
import numpy as np
import matplotlib.pyplot as plt
from numpy.polynomial import Polynomial
```

Here, `forward_diff(f, h, x)`, `backward_diff(f, h, x)`, and `central_diff(f, h, x)` calculates the *forward differentiation*, *backward differentiation* and *central differentiation* respectively. finally the `error(f, f_prime, h, x)` method calculates the different values for various $h$ and returns the errors.

Later we will run some code to test out performance. The first one is done for you.

```python
def forward_diff(f, h, x):
    return (f(x+h) - f(x)) / h
```

```python
def backward_diff(f, h, x):
    # --------------------------------------------
    # YOUR CODE HERE
    return (f(x-h) - f(x)) / h
    # --------------------------------------------
```

```python
def central_diff(f, h, x):
    # --------------------------------------------
    # YOUR CODE HERE
    return (f(x+h) - f(x-h)) / (2*h)
    # --------------------------------------------
```

```python
def error(f, f_prime, h, x):
    Y_correct = f_prime(x)
    f_error = np.array([])
    b_error = np.array([])
    c_error = np.array([])

    for h_i in h:
        # for different values of h (h_i)
        # calculate the error at the point x for  (i) forward method
        #                                         (ii) backward method
        #                                         (ii) central method
        # the first one is done for you
        f_error_h_i = forward_diff(f, h_i, x) - Y_correct
        f_error = np.append(f_error, f_error_h_i)

        b_error_h_i = backward_diff(f, h_i, x) - Y_correct
        b_error = np.append(b_error, b_error_h_i)

        c_error_h_i = central_diff(f, h_i, x) - Y_correct
        c_error = np.append(c_error, c_error_h_i)

    return f_error, b_error, c_error
```

## Plot1
Polynomial and Actual Derivative Function

```python
fig, ax = plt.subplots()
ax.axhline(y=0, color='k')

p = Polynomial([2.0, 1.0, -6.0, -2.0, 2.5, 1.0])
data = p.linspace(domain=[-2.4, 1.5])
ax.plot(data[0], data[1], label='Function')

p_prime = p.deriv(1)
data2 = p_prime.linspace(domain=[-2.4, 1.5])
ax.plot(data2[0], data2[1], label='Derivative')

ax.legend()
```

```python
h = 1
fig, bx = plt.subplots()
bx.axhline(y=0, color='k')

x = np.linspace(-2.0, 1.3, 50, endpoint=True)
y = forward_diff(p, h, x)
bx.plot(x, y, label='Forward; h=1')
y = backward_diff(p, h, x)
bx.plot(x, y, label='Backward; h=1')
y = central_diff(p, h, x)
bx.plot(x, y, label='Central; h=1')

data2 = p_prime.linspace(domain=[-2.0, 1.3])
bx.plot(data2[0], data2[1], label='actual')

bx.legend()

```

```python
h = 0.1
fig, bx = plt.subplots()
bx.axhline(y=0, color='k')

x = np.linspace(-2.2, 1.3, 50, endpoint=True)
y = forward_diff(p, h, x)
bx.plot(x, y, label='Forward; h=0.1')
y = backward_diff(p, h, x)
bx.plot(x, y, label='Backward; h=0.1')
y = central_diff(p, h, x)
bx.plot(x, y, label='Central; h=0.1')

data2 = p_prime.linspace(domain=[-2.2, 1.3])
bx.plot(data2[0], data2[1], label='actual')

bx.legend()

```

```python
h = 0.01
fig, bx = plt.subplots()
bx.axhline(y=0, color='k')

x = np.linspace(-2.2, 1.3, 50, endpoint=True)
y = forward_diff(p, h, x)
bx.plot(x, y, label='Forward; h=0.01')
y = backward_diff(p, h, x)
bx.plot(x, y, label='Backward; h=0.01')
y = central_diff(p, h, x)
bx.plot(x, y, label='Central; h=0.01')

data2 = p_prime.linspace(domain=[-2.2, 1.3])
bx.plot(data2[0], data2[1], label='actual')

bx.legend()

```

```python
fig, bx = plt.subplots()
bx.axhline(y=0, color='k')

h = np.array([1., 0.55, 0.3, .17, 0.1, 0.055, 0.03, 0.017, 0.01])
err = error(p, p_prime, h, 2.0)

bx.plot(h, err[0], label='Forward')
bx.plot(h, err[1], label='Backward')
bx.plot(h, err[2], label='Central')
bx.legend()

```

```python

```


```python
import pyprob
from pyprob import Model
from pyprob.distributions import Normal

import torch
import numpy as np
import math
import matplotlib.pyplot as plt
fig = plt.figure();
```

    <Figure size 432x288 with 0 Axes>

# Defining the model
First, we define the model as a probabilistic program inheriting from `pyprob.Model`. Models inherit from `torch.nn.Module` and can be potentially trained with gradient-based optimization (not covered in this example).

The `forward` function can have any number and type of arguments as needed.

```python
%matplotlib inline
class GaussianUnknownMean(Model):
    def __init__(self):
        super().__init__(name='Gaussian with unknown mean') # give the model a name
        self.prior_mean = 1
        self.prior_std = math.sqrt(5)
        self.likelihood_std = math.sqrt(2)

    def forward(self): # Needed to specify how the generative model is run forward
        # sample the (latent) mean variable to be inferred:
        mu = pyprob.sample(Normal(self.prior_mean, self.prior_std)) # NOTE: sample -> denotes latent variables

        # define the likelihood
        likelihood = Normal(mu, self.likelihood_std)

        # Lets add two observed variables
        # -> the 'name' argument is used later to assignment values:
        pyprob.observe(likelihood, name='obs0') # NOTE: observe -> denotes observable variables
        pyprob.observe(likelihood, name='obs1')

        # return the latent quantity of interest
        return mu

model = GaussianUnknownMean()
```

# Finding the correct posterior analytically
Since all distributions are gaussians in this model, we can analytically compute the posterior and we can compare the true posterior to the inferenced one.

Assuming that the prior and likelihood are $p(x) = \mathcal{N}(\mu_0, \sigma_0)$ and $p(y|x) = \mathcal{N}(x, \sigma)$ respectively and, $y_1, y_2, \ldots y_n$ are the observed values, the posterior would be $p(x|y) = \mathcal{N}(\mu_p, \sigma_p)$ where,
$$
\begin{align}
\sigma_{p}^{2} & = \frac{1}{\frac{n}{\sigma^2} + \frac{1}{\sigma_{0}^{2}}} \\
\mu_p & = \sigma_{p}^{2} \left( \frac{\mu_0}{\sigma_{0}^{2}} + \frac{n\overline{y}}{\sigma^2} \right)
\end{align}
$$
The following class implements computing this posterior distribution. We also implement some helper functions and variables for plotting the correct posterior and prior.

```python
def plot_function(min_val, max_val, func, *args, **kwargs):
    '''
    Apply a function repetitively to tuples within a vector
    '''
    x = np.linspace(min_val,max_val,int((max_val-min_val)*50))
    plt.plot(x, np.vectorize(func)(x), *args, **kwargs)

def get_dist_pdf(dist):
    return lambda x: math.exp(dist.log_prob(x))

class CorrectDistributions:
    def __init__(self, model):
        self.prior_mean = model.prior_mean
        self.prior_std = model.prior_std
        self.likelihood_std = model.likelihood_std
        self.prior_dist = Normal(self.prior_mean, self.prior_std)

    @property
    def observed_list(self):
        return self.__observed_list

    @observed_list.setter
    def observed_list(self, new_observed_list):
        self.__observed_list = new_observed_list
        self.construct_correct_posterior()

    def construct_correct_posterior(self):
        n = len(self.observed_list)
        # As per the analytical method we construct the posterior given by the formula
        posterior_var = 1/(n/self.likelihood_std**2 + 1/self.prior_std**2)
        posterior_mu = posterior_var * (self.prior_mean/self.prior_std**2 + n*np.mean(self.observed_list)/self.likelihood_std**2)
        self.posterior_dist = Normal(posterior_mu, math.sqrt(posterior_var))

    def prior_pdf(self, model, x):
        p = Normal(model.prior_mean,model.prior_stdd)
        return math.exp(p.log_prob(x))

    def plot_posterior(self, min_val, max_val):
        if not hasattr(self, 'posterior_dist'):
            raise AttributeError('observed values are not set yet, and posterior is not defined.')
        plot_function(min_val, max_val, get_dist_pdf(self.posterior_dist), label='correct posterior', color='orange')

    def plot_prior(self, min_val, max_val):
        plot_function(min_val, max_val, get_dist_pdf(self.prior_dist), label='prior', color='green')

correct_dists = CorrectDistributions(model)
```

# Prior distribution
We inspect the prior distribution to see if it behaves in the way we intended. First we construct an `Empirical` distribution with forward samples from the model.

Note: Extra arguments passed to `prior_distribution` will be forwarded to model's `forward` function.

```python
prior = model.prior_distribution(num_traces=1000)
```

    Time spent  | Time remain.| Progress             | Trace     | Traces/sec
    0d:00:00:00 | 0d:00:00:00 | #################### | 1000/1000 | 1,621.42

We can plot a histogram of these samples that are held by the `Empirical` distribution.

```python
# Compare our assumed prior against the actual prior from the model - sanity check
prior.plot_histogram(show=False, alpha=0.75, label='empirical prior')
correct_dists.plot_prior(min(prior.values_numpy()),max(prior.values_numpy()))
plt.legend();
```

# Posterior inference with importance sampling
For a given set of observations, we can get samples from the posterior distribution.

```python
correct_dists.observed_list = [4, 5] # Observations
# sample from posterior (5000 samples)
posterior = model.posterior_distribution(
                                         num_traces=5000, # the number of samples estimating the posterior
                                         inference_engine=pyprob.InferenceEngine.IMPORTANCE_SAMPLING, # specify which inference engine to use
                                         observe={'obs0': correct_dists.observed_list[0],
                                                  'obs1': correct_dists.observed_list[1]} # assign values to the observed values
                                         )
```

    Time spent  | Time remain.| Progress             | Trace     | Traces/sec
    0d:00:00:02 | 0d:00:00:00 | #################### | 5000/5000 | 1,807.42

Regular importance sampling uses proposals from the prior distribution. We can see this by plotting the histogram of the posterior distribution without using the importance weights. As expected, this is the same as the prior distribution.

```python
posterior_unweighted = posterior.unweighted()
posterior_unweighted.plot_histogram(show=False, alpha=0.75, label='empirical proposal')
correct_dists.plot_prior(min(posterior_unweighted.values_numpy()),
                         max(posterior_unweighted.values_numpy()))
correct_dists.plot_posterior(min(posterior_unweighted.values_numpy()),
                             max(posterior_unweighted.values_numpy()))
plt.legend();
```

When we do use the weights, we end up with the correct posterior distribution. The following shows the sampled posterior with the correct posterior (orange curve).

```python
posterior.plot_histogram(show=False, alpha=0.75, bins=50, label='inferred posterior')
correct_dists.plot_posterior(min(posterior.values_numpy()),
                             max(posterior_unweighted.values_numpy()))
plt.legend();
```

In practice, it is advised to use methods of the `Empirical` posterior distribution instead of dealing with the weights directly, which ensures that the weights are used in the correct way.

For instance, we can get samples from the posterior, compute its mean and standard deviation, and evaluate expectations of a function under the distribution:

```python
print(posterior.sample())
```

    tensor(4.1385)

```python
print(posterior.mean)
```

    tensor(3.9196)

```python
print(posterior.stddev)
```

    tensor(0.9020)

```python
print(posterior.expectation(lambda x: torch.sin(x)))
```

    tensor(-0.4711)

# Inference compilation
Inference compilation is a technique where a deep neural network is used for parameterizing the proposal distribution in importance sampling (https://arxiv.org/abs/1610.09900). This neural network, which we call inference network, is automatically generated and trained with data sampled from the model.

We can learn an inference network for our model.

```python
model.learn_inference_network(num_traces=20000,
                              observe_embeddings={'obs0' : {'dim' : 32},
                                                  'obs1': {'dim' : 32}},
                              inference_network=pyprob.InferenceNetwork.LSTM)
```

    Creating new inference network...
    Observable obs0: observe embedding not specified, using the default FEEDFORWARD.
    Observable obs0: embedding depth not specified, using the default 2.
    Observable obs1: observe embedding not specified, using the default FEEDFORWARD.
    Observable obs1: embedding depth not specified, using the default 2.
    Observe embedding dimension: 64
    Train. time | Epoch| Trace     | Init. loss| Min. loss | Curr. loss| T.since min | Traces/sec
    New layers, address: 16__forward__mu__Normal__1, distribution: Normal
    Total addresses: 1, distribution types: 1, parameters: 1,643,583
    0d:00:00:31 | 1    | 20,032    | +2.21e+00 | +1.15e+00 | +1.27e+00 | 0d:00:00:10 | 638.0

We now construct the posterior distribution using samples from inference compilation, using the trained inference network.

A much smaller number of samples are enough (200 vs. 5000) because the inference network provides good proposals based on the given observations. We can see that the proposal distribution given by the inference network is doing a job much better than the prior, by plotting the posterior samples without the importance weights, for a selection of observations.

```python
# sample from posterior (200 samples)
posterior = model.posterior_distribution(
    num_traces=200, # the number of samples estimating the posterior
    inference_engine=pyprob.InferenceEngine.IMPORTANCE_SAMPLING_WITH_INFERENCE_NETWORK, # specify which inference engine to use
    observe={'obs0': correct_dists.observed_list[0],
             'obs1': correct_dists.observed_list[1]} # assign values to the observed values
)
```

    Time spent  | Time remain.| Progress             | Trace   | Traces/sec
    0d:00:00:00 | 0d:00:00:00 | #################### | 200/200 | 316.34

```python
posterior_unweighted = posterior.unweighted()
posterior_unweighted.plot_histogram(
    show=False, bins=50, alpha=0.75, label='empirical proposal')

correct_dists.plot_posterior(min(posterior.values_numpy()),
                             max(posterior.values_numpy()))
plt.legend();
```

We can see that the proposal distribution given by the inference network is already a good estimate to the true posterior which makes the inferred posterior a much better estimate than the prior, even using much less number of samples.

```python
posterior.plot_histogram(show=False, bins=50, alpha=0.75, label='inferred posterior')
correct_dists.plot_posterior(min(posterior.values_numpy()),
                             max(posterior.values_numpy()))
plt.legend();
```

Inference compilation performs amortized inference which means, the same trained network provides proposal distributions for any observed values.

We can try performing inference using the same trained network with different observed values.

```python
correct_dists.observed_list = [8, 10] # New observations

posterior = model.posterior_distribution(
    num_traces=200,
    inference_engine=pyprob.InferenceEngine.IMPORTANCE_SAMPLING_WITH_INFERENCE_NETWORK, # specify which inference engine to use
    observe={'obs0': correct_dists.observed_list[0],
          'obs1': correct_dists.observed_list[1]}
)
```

    Time spent  | Time remain.| Progress             | Trace   | Traces/sec
    0d:00:00:00 | 0d:00:00:00 | #################### | 200/200 | 362.70

```python
posterior_unweighted = posterior.unweighted()
posterior_unweighted.plot_histogram(show=False, bins=50, alpha=0.75, label='empirical proposal')

correct_dists.plot_posterior(min(posterior.values_numpy()),
                       max(posterior.values_numpy()))
plt.legend();
```

```python
posterior.plot_histogram(show=False, bins=50, alpha=0.75, label='inferred posterior')
correct_dists.plot_posterior(min(posterior.values_numpy()),
                       max(posterior.values_numpy()))
plt.legend();

```

```python

```


### Minimum graph coloring/ Minimum chromatic number

Problem definition: https://www8.cs.umu.se/kurser/TDBAfl/VT06/algorithms/COMPEND/COMPED11/NODE13.HTM#SECTION00021500000000000000

Assignment based ILP-formulation (e.g. see here: https://arxiv.org/abs/1706.10191):
Let a graph $G=(V,E)$ with vertex set $V$ and edge set $E$ be given.
We model the decision via variables $x_{vi}\in\{0,1\}$, that take value $1$ if vertex $v\in V$ is assigned color $i$ and $0$ otherwise.

Further, let $H$ be an upper bound for the chromatic number, i.e. $H\le|V|$, but usually smaller as it could come from, e.g., the greedy coloring algorithm (https://en.wikipedia.org/wiki/Greedy_coloring). $w_{i}$ takes value $1$ if color $i=1,\ldots H$ is used in the assignment and $0$ otherwise.

\begin{align}
    &\min \sum_{1\le i \le H}w_{i} \text{ (minimize the total number of colors used) }\\
    &\text{s.t.} \\
    &\sum_{i=1}^{H} x_{vi} = 1\;\forall v\in V \text{ (make sure every vertex gets exactly one color) } \\
    &x_{ui}+x_{vi}\le w_{i}\;\forall(u,v)\in E, i=1,\ldots,H\text{ (make sure no two neighboring vertices get the same color) } \\
    &x_{vi},w_{i}\in\{0,1\}\;\forall v\in V, i=1,\ldots, H \text{ (assigning a color or not is a binary decision) }
\end{align}

```python
import gurobipy as gp
import networkx as nx
```

```python
from graphilp.partitioning.min_vertex_coloring import *
from graphilp.partitioning.heuristics import vertex_coloring_greedy
```

```python
from graphilp.imports import networkx as imp_nx
```

#### Create test graphs

```python
#create cycle graphs as test cases. we know odd cycles have chromatic number 3 and even cycles have chromatic number 2
G_odd_init = nx.cycle_graph(n=5)
G_even_init = nx.cycle_graph(n=4)
```

```python
#create ILPGraph objects
G_odd = imp_nx.read(G_odd_init)
G_even = imp_nx.read(G_even_init)
```

```python
#create test models
m_odd = create_model(G_odd)
m_even = create_model(G_even)
```

```python
#run optimization
m_odd.optimize()
```

```python
m_even.optimize()
```

#### Inspect solutions

```python
color_assignment_even, node_to_col_even = extract_solution(G_even, m_even)
```

```python
#visualize solution
nx.draw_circular(G_even.G, node_color=list(node_to_col_even.values()))
```

```python
color_assignment_odd, node_to_col_odd = extract_solution(G_odd, m_odd)
```

```python
nx.draw_circular(G_odd.G, node_color=list(node_to_col_odd.values()))
```

```python
col_to_node, node_to_col = vertex_coloring_greedy.get_heuristic(G_even)
```

```python
nx.draw_circular(G_even.G, node_color=list(node_to_col.values()))
```


# Part 1 - Scalars and Vectors

For the questions below it is not sufficient to simply provide answer to the questions, but you must solve the problems and show your work using python (the NumPy library will help a lot!) Translate the vectors and matrices into their appropriate python  representations and use numpy or functions that you write yourself to demonstrate the result or property.

## 1.1 Create a two-dimensional vector and plot it on a graph

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
vector = np.array([1,6])
plt.arrow(0, 0, *vector, head_width=.1, head_length=0.1)
plt.xlim(0,max(vector)+1)
plt.ylim(0,max(vector)+1);
```

## 1.2 Create a three-dimensional vecor and plot it on a graph

```python
from mpl_toolkits.mplot3d import Axes3D

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

vector = np.array([3, 2, 3])

ax.quiver(*(0,0,0), *vector)
ax.set_xlim([0, max(vector) + 1])
ax.set_ylim([0, max(vector) + 1])
ax.set_zlim([0, max(vector) + 1])
ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_zlabel('Z')

plt.show();
```

## 1.3 Scale the vectors you created in 1.1 by $5$, $\pi$, and $-e$ and plot all four vectors (original + 3 scaled vectors) on a graph. What do you notice about these vectors?

```python
vector = np.array([1,6])
quiver = np.array([vector, vector*5, vector*np.pi, vector*-np.exp(1)])

for v in quiver:
    plt.arrow(0, 0, *v, head_width=1, head_length=1)

plt.xlim(quiver.min()-1,quiver.max()+1)
plt.ylim(quiver.min()-1,quiver.max()+1);
```

## 1.4 Graph vectors $\vec{a}$ and $\vec{b}$ and plot them on a graph

\begin{align}
\vec{a} = \begin{bmatrix} 5 \\ 7 \end{bmatrix}
\qquad
\vec{b} = \begin{bmatrix} 3 \\4 \end{bmatrix}
\end{align}

```python
a = np.array([5, 7])
b = np.array([3, 4])
quiver = np.array([a, b])

for v in quiver:
    plt.arrow(0, 0, *v, head_width=0.5, head_length=0.5)

plt.xlim(0,quiver.max()+1)
plt.ylim(0,quiver.max()+1);
```

## 1.5 find $\vec{a} - \vec{b}$ and plot the result on the same graph as $\vec{a}$ and $\vec{b}$. Is there a relationship between vectors $\vec{a} \thinspace, \vec{b} \thinspace \text{and} \thinspace \vec{a-b}$

```python
a = np.array([5, 7])
b = np.array([3, 4])
quiver = np.array([a, b, a-b])

for v in quiver:
    plt.arrow(0, 0, *v, head_width=0.5, head_length=0.5)

plt.xlim(0,quiver.max()+1)
plt.ylim(0,quiver.max()+1);
```

$\vec{a-b}$ is the length and direction connecting the tips of $\vec{a}$ and $\vec{b}$

## 1.6 Find $c \cdot d$

\begin{align}
\vec{c} = \begin{bmatrix}7 & 22 & 4 & 16\end{bmatrix}
\qquad
\vec{d} = \begin{bmatrix}12 & 6 & 2 & 9\end{bmatrix}
\end{align}

```python
np.dot(np.array([ 7, 22,  4, 16]),
       np.array([12,  6,  2,  9]))
```

    368

##  1.7 Find $e \times f$

\begin{align}
\vec{e} = \begin{bmatrix} 5 \\ 7 \\ 2 \end{bmatrix}
\qquad
\vec{f} = \begin{bmatrix} 3 \\4 \\ 6 \end{bmatrix}
\end{align}

```python
np.cross(np.array([5, 7, 2]),
         np.array([3, 4, 6]))
```

    array([ 34, -24,  -1])

## 1.8 Find $||g||$ and then find $||h||$. Which is longer?

\begin{align}
\vec{e} = \begin{bmatrix} 1 \\ 1 \\ 1 \\ 8 \end{bmatrix}
\qquad
\vec{f} = \begin{bmatrix} 3 \\3 \\ 3 \\ 3 \end{bmatrix}
\end{align}

```python
(np.linalg.norm(np.array([1, 1, 1, 8])),
 np.linalg.norm(np.array([3, 3, 3, 3])))
```

    (8.18535277187245, 6.0)

## 1.9 Show that the following vectors are orthogonal (perpendicular to each other):

\begin{align}
\vec{g} = \begin{bmatrix} 1 \\ 0 \\ -1  \end{bmatrix}
\qquad
\vec{h} = \begin{bmatrix} 1 \\ \sqrt{2} \\ 1 \end{bmatrix}
\end{align}

```python
np.dot(np.array([1,        0, -1]),
       np.array([1, 2**(1/2),  1]))
```

    0.0

# Part 2 - Matrices

## 2.1 What are the dimensions of the following matrices? Which of the following can be multiplied together? See if you can find all of the different legal combinations.
\begin{align}
A = \begin{bmatrix}
1 & 2 \\
3 & 4 \\
5 & 6
\end{bmatrix}
\qquad
B = \begin{bmatrix}
2 & 4 & 6 \\
\end{bmatrix}
\qquad
C = \begin{bmatrix}
9 & 6 & 3 \\
4 & 7 & 11
\end{bmatrix}
\qquad
D = \begin{bmatrix}
1 & 0 & 0 \\
0 & 1 & 0 \\
0 & 0 & 1
\end{bmatrix}
\qquad
E = \begin{bmatrix}
1 & 3 \\
5 & 7
\end{bmatrix}
\end{align}

```python
A = np.array([[1, 2],
              [3, 4],
              [5, 6]])

B = np.array([[2, 4, 6]])

C = np.array([[9, 6,  3],
              [4, 7, 11]])

D = np.array([[1, 0, 0],
              [0, 1, 0],
              [0, 0, 1]])

E = np.array([[1, 3],
              [5, 7]])

quiver = [A, B, C, D, E]

# You can multipy any two matrices where the number of columns of the first matrix
# is equal to the number of rows of the second matrix.

for M1 in quiver:
    for M2 in quiver:
        if M1 is M2:
            continue
        if M1.shape[1] == M2.shape[0]:
            print('Can multiply:', M1, M2, sep='\n')
            print('\n')
        if M1.shape[0] == M2.shape[1]:
            print('Can multiply:', M2, M1, sep='\n')
            print('\n')
```

    Can multiply:
    [[2 4 6]]
    [[1 2]
     [3 4]
     [5 6]]

    Can multiply:
    [[1 2]
     [3 4]
     [5 6]]
    [[ 9  6  3]
     [ 4  7 11]]

    Can multiply:
    [[ 9  6  3]
     [ 4  7 11]]
    [[1 2]
     [3 4]
     [5 6]]

    Can multiply:
    [[1 0 0]
     [0 1 0]
     [0 0 1]]
    [[1 2]
     [3 4]
     [5 6]]

    Can multiply:
    [[1 2]
     [3 4]
     [5 6]]
    [[1 3]
     [5 7]]

    Can multiply:
    [[2 4 6]]
    [[1 2]
     [3 4]
     [5 6]]

    Can multiply:
    [[2 4 6]]
    [[1 0 0]
     [0 1 0]
     [0 0 1]]

    Can multiply:
    [[ 9  6  3]
     [ 4  7 11]]
    [[1 2]
     [3 4]
     [5 6]]

    Can multiply:
    [[1 2]
     [3 4]
     [5 6]]
    [[ 9  6  3]
     [ 4  7 11]]

    Can multiply:
    [[ 9  6  3]
     [ 4  7 11]]
    [[1 0 0]
     [0 1 0]
     [0 0 1]]

    Can multiply:
    [[1 3]
     [5 7]]
    [[ 9  6  3]
     [ 4  7 11]]

    Can multiply:
    [[1 0 0]
     [0 1 0]
     [0 0 1]]
    [[1 2]
     [3 4]
     [5 6]]

    Can multiply:
    [[2 4 6]]
    [[1 0 0]
     [0 1 0]
     [0 0 1]]

    Can multiply:
    [[ 9  6  3]
     [ 4  7 11]]
    [[1 0 0]
     [0 1 0]
     [0 0 1]]

    Can multiply:
    [[1 2]
     [3 4]
     [5 6]]
    [[1 3]
     [5 7]]

    Can multiply:
    [[1 3]
     [5 7]]
    [[ 9  6  3]
     [ 4  7 11]]

## 2.2 Find the following products: CD, AE, and BA. What are the dimensions of the resulting matrices? How does that relate to the dimensions of their factor matrices?

```python
# the result of multiplying two matrices will have the shape of
# the first matrix's rows and the second matrix's columns

CD = np.matmul(C, D)
print(CD)

AE = np.matmul(A, E)
print(AE)

BA = np.matmul(B, A)
print(BA)
```

    [[ 9  6  3]
     [ 4  7 11]]
    [[11 17]
     [23 37]
     [35 57]]
    [[44 56]]

## 2.3  Find $F^{T}$. How are the numbers along the main diagonal (top left to bottom right) of the original matrix and its transpose related? What are the dimensions of $F$? What are the dimensions of $F^{T}$?

\begin{align}
F =
\begin{bmatrix}
20 & 19 & 18 & 17 \\
16 & 15 & 14 & 13 \\
12 & 11 & 10 & 9 \\
8 & 7 & 6 & 5 \\
4 & 3 & 2 & 1
\end{bmatrix}
\end{align}

```python
F = np.array([[20, 19, 18, 17],
              [16, 15, 14, 13],
              [12, 11, 10,  9],
              [ 8,  7,  6,  5],
              [ 4,  3,  2,  1]])
print(F)

print(np.transpose(F))
```

    [[20 19 18 17]
     [16 15 14 13]
     [12 11 10  9]
     [ 8  7  6  5]
     [ 4  3  2  1]]
    [[20 16 12  8  4]
     [19 15 11  7  3]
     [18 14 10  6  2]
     [17 13  9  5  1]]

# Part 3 - Square Matrices

## 3.1 Find $IG$ (be sure to show your work) 😃

\begin{align}
G=
\begin{bmatrix}
12 & 11 \\
7 & 10
\end{bmatrix}
\end{align}

The product of any square matrix with its corresponding identity matrix is the same martrix.

## 3.2 Find $|H|$ and then find $|J|$.

\begin{align}
H=
\begin{bmatrix}
12 & 11 \\
7 & 10
\end{bmatrix}
\qquad
J=
\begin{bmatrix}
0 & 1 & 2 \\
7 & 10 & 4 \\
3 & 2 & 0
\end{bmatrix}
\end{align}

```python
H = np.array([[12, 11],
              [ 7, 10]])
np.linalg.det(H)
```

    43.0

```python
J = np.array([[ 0, 1, 2],
              [ 7,10, 4],
              [ 3, 2, 0]])
np.linalg.det(J)
```

    -19.999999999999996

## 3.3 Find $H^{-1}$ and then find $J^{-1}$

```python
np.linalg.inv(H)
```

    array([[ 0.23255814, -0.25581395],
           [-0.1627907 ,  0.27906977]])

```python
np.linalg.inv(J)
```

    array([[ 0.4 , -0.2 ,  0.8 ],
           [-0.6 ,  0.3 , -0.7 ],
           [ 0.8 , -0.15,  0.35]])

## 3.4 Find $HH^{-1}$ and then find $J^{-1}J$. Is $HH^{-1} == J^{-1}J$? Why or Why not?

```python
np.matmul(H, np.linalg.inv(H))
```

    array([[ 1.00000000e+00,  0.00000000e+00],
           [-2.22044605e-16,  1.00000000e+00]])

```python
np.matmul(np.linalg.inv(J), J)
```

    array([[ 1.00000000e+00, -4.44089210e-16,  0.00000000e+00],
           [ 6.66133815e-16,  1.00000000e+00,  0.00000000e+00],
           [-1.11022302e-16,  0.00000000e+00,  1.00000000e+00]])

No, they are not equal. They don't have the same dimensions.

# Stretch Goals:

A reminder that these challenges are optional. If you finish your work quickly we welcome you to work on them. If there are other activities that you feel like will help your understanding of the above topics more, feel free to work on that. Topics from the Stretch Goals sections will never end up on Sprint Challenges. You don't have to do these in order, you don't have to do all of them.

- Write a function that can calculate the dot product of any two vectors of equal length that are passed to it.
- Write a function that can calculate the norm of any vector
- Prove to yourself again that the vectors in 1.9 are orthogonal by graphing them.
- Research how to plot a 3d graph with animations so that you can make the graph rotate (this will be easier in a local notebook than in google colab)
- Create and plot a matrix on a 2d graph.
- Create and plot a matrix on a 3d graph.
- Plot two vectors that are not collinear on a 2d graph. Calculate the determinant of the 2x2 matrix that these vectors form. How does this determinant relate to the graphical interpretation of the vectors?