## Maximum Likelihood Estimation

Maximum likelihood estimation is one of the key techniques employed in statistical signal processing for a wide variety of applications from signal detection to parameter estimation. In the following, we consider a simple experiment and work through the details of maximum likelihood estimation to ensure that we understand the concept in one of its simplest applications.

### Setting up the Coin Flipping Experiment

Suppose we have coin and want to estimate the probability of heads ($p$) for it. The coin is Bernoulli distributed:

$$ \phi(x)= p^x (1-p)^{(1-x)} $$

where $x$ is the outcome, *1* for heads and *0* for tails. The $n$ independent flips, we have the likelihood:

$$ \mathcal{L}(p|\mathbf{x})= \prod_{i=1}^n p^{ x_i }(1-p)^{1-x_i} $$

This is basically notation. We have just substituted everything into $ \phi(x)$ under the independent-trials assumption.

The idea of *maximum likelihood* is to maximize this as the function of $p$ after plugging in all of the $x_i$ data. This means that our estimator, $\hat{p}$ , is a function of the observed $x_i$ data, and as such, is a random variable with its own distribution.

### Simulating the Experiment

We need the following code to simulate coin flipping.

```
%matplotlib inline
from __future__ import division
from scipy.stats import bernoulli
import numpy as np

p_true=1/2 # this is the value we will try to estimate from the observed data
fp=bernoulli(p_true)

def sample(n=10):
    'simulate coin flipping'
    return fp.rvs(n)# flip it n times

xs = sample(100) # generate some samples
```

Now, we can write out the likelihood function using `sympy`

```
import sympy
from sympy.abc import x, z
p=sympy.symbols('p',positive=True)

L=p**x*(1-p)**(1-x)
J=np.prod([L.subs(x,i) for i in xs]) # objective function to maximize
```

Below, we find the maximum using basic calculus. Note that taking the `log` of $J$ makes the maximization problem tractable but doesn't change the extrema.

```
logJ=sympy.expand_log(sympy.log(J))
sol=sympy.solve(sympy.diff(logJ,p),p)[0]

x=linspace(0,1,100)
plot(x,map(sympy.lambdify(p,logJ,'numpy'),x),sol,logJ.subs(p,sol),'o',
                                          p_true,logJ.subs(p,p_true),'s',)
xlabel('$p$',fontsize=18)
ylabel('Likelihood',fontsize=18)
title('Estimate not equal to true value',fontsize=18)
```

Note that our estimator $\hat{p}$ (red circle) is not equal to the true value of $p$ (green square), but it is at the maximum of the likelihood function. This may sound disturbing, but keep in mind this estimate is a function of the random data; and since that data can change, the ultimate estimate can likewise change. I invite you to run this notebook a few times to observe  this. Remember that the estimator is a *function* of the data and is thus also a *random variable*, just like the data is.

Let's write some code to empirically examine the behavior of the maximum likelihood estimator using a simulation of multiple trials. All we're doing here is combining the last few blocks of code.

```
def estimator_gen(niter=10,ns=100):
    'generate data to estimate distribution of maximum likelihood estimator'
    out=[]
    x=sympy.symbols('x',real=True)
    L=   p**x*(1-p)**(1-x)
    for i in range(niter):
        xs = sample(ns) # generate some samples from the experiment
        J=np.prod([L.subs(x,i) for i in xs]) # objective function to maximize
        logJ=sympy.expand_log(sympy.log(J))
        sol=sympy.solve(sympy.diff(logJ,p),p)[0]
        out.append(float(sol.evalf()))
    return out if len(out)>1 else out[0] # return scalar if list contains only 1 term

etries = estimator_gen(100) # this may take awhile, depending on how much data you want to generate
hist(etries) # histogram of maximum likelihood estimator
title('$\mu=%3.3f,\sigma=%3.3f$'%(mean(etries),std(etries)),fontsize=18)
```

Note that the mean of the estimator ($\mu$) is pretty close to the true value, but looks can be deceiving. The only way to know for sure is to check if the estimator is unbiased, namely, if

$$ \mathbb{E}(\hat{p}) = p $$

Because this problem is simple, we can solve for this in general noting that since $x=0$ or $x=1$, the terms in the product of $\mathcal{L}$ above are either $p$, if $x_i=1$ or $1-p$ if $x_i=0$. This means that we can write

$$ \mathcal{L}(p|\mathbf{x})= p^{\sum_{i=1}^n x_i}(1-p)^{n-\sum_{i=1}^n x_i} $$

with corresponding log as

$$ J=\log(\mathcal{L}(p|\mathbf{x})) =  \log(p)  \sum_{i=1}^n x_i +   \log(1-p) \left(n-\sum_{i=1}^n x_i\right)$$

Taking the derivative of this gives:

$$  \frac{dJ}{dp} = \frac{1}{p}\sum_{i=1}^n x_i + \frac{(n-\sum_{i=1}^n x_i)}{p-1} $$

and solving this leads to

$$  \hat{p} = \frac{1}{ n} \sum_{i=1}^n x_i $$

This is our *estimator* for $p$. Up til now, we have been using `sympy` to solve for this based on the data $x_i$ but now we have it generally and don't have to solve for it again. To check if this estimator is biased, we compute its expectation:

$$ \mathbb{E}\left(\hat{p}\right) =\frac{1}{n}\sum_i^n \mathbb{E}(x_i) = \frac{1}{n} n \mathbb{E}(x_i) $$

by linearity of the expectation and where

$$\mathbb{E}(x_i)  = p$$

Therefore,

$$ \mathbb{E}\left(\hat{p}\right) =p $$

This means that the esimator is unbiased. This is good news. We almost always want our estimators to be unbiased. Similarly,

$$ \mathbb{E}\left(\hat{p}^2\right) = \frac{1}{n^2} \mathbb{E}\left[\left(  \sum_{i=1}^n x_i \right)^2 \right]$$

and where

$$ \mathbb{E}\left(x_i^2\right) =p$$

and by the independence assumption,

$$ \mathbb{E}\left(x_i x_j\right) =\mathbb{E}(x_i)\mathbb{E}( x_j) =p^2$$

Thus,

$$ \mathbb{E}\left(\hat{p}^2\right) =\left(\frac{1}{n^2}\right) n
\left[
p+(n-1)p^2
\right]
$$

So, the variance of the estimator, $\hat{p}$ is the following:

$$ \sigma_\hat{p}^2 = \mathbb{E}\left(\hat{p}^2\right)- \mathbb{E}\left(\hat{p}\right)^2  = \frac{p(1-p)}{n} $$

Note that the $n$ in the denominator means that the variance asymptotically goes to zero as $n$ increases (i.e. we consider more and more samples). This is good news also because it means that more and more coin flips leads to a better estimate of the underlying $p$.

Unfortunately, this formula for the variance is practically useless because we have to know $p$ to compute it and $p$ is the parameter we are trying to estimate in the first place! But, looking at $ \sigma_\hat{p}^2 $, we can immediately notice that if $p=0$, then there is no estimator variance because the outcomes are guaranteed to be tails. Also, the maximum of this variance, for whatever $n$, happens at $p=1/2$. This is our worst case scenario and the only way to compensate is with more samples (i.e. larger $n$).

All we have computed is the mean and variance of the estimator. In general, this is insufficient to characterize the underlying probability density of $\hat{p}$, except if we somehow knew that  $\hat{p}$ were normally distributed. This is where the powerful [*central limit theorem*](http://mathworld.wolfram.com/CentralLimitTheorem.html) comes in. The form of the estimator, which is just a mean estimator, implies that we can apply this theorem and conclude that  $\hat{p}$ is normally distributed. However, there's a wrinkle here: the theorem tells us that  $\hat{p}$ is asymptotically normal, it doesn't quantify how many samples $n$ we need to approach this asymptotic paradise. In our simulation this is no problem since we can generate as much data as we like, but in the real world, with a costly experiment, each sample may be precious. In the following, we won't apply this theorem and instead proceed analytically.

### Probability Density for the Estimator

To write out the full density for $\hat{p}$, we first have to ask what is the probability that the estimator will equal a specific value and the tally up all the ways that could happen with their corresponding probabilities. For example, what is the probability that

$$  \hat{p} = \frac{1}{n}\sum_{i=1}^n x_i  = 0 $$

This can only happen one way: when $x_i=0 \hspace{0.5em} \forall i$. The probability of this happening can be computed from the density

$$ f(\mathbf{x},p)= \prod_{i=1}^n \left(p^{x_i} (1-p)^{1-x_i}  \right) $$

$$ f\left(\sum_{i=1}^n x_i  = 0,p\right)= \left(1-p\right)^n  $$

Likewise, if $\lbrace x_i \rbrace$ has one $i^{th}$  value equal to one, then

$$ f\left(\sum_{i=1}^n x_i  = 1,p\right)= n p \prod_{i=1}^{n-1} \left(1-p\right)$$

where the $n$ comes from the $n$ ways to pick one value equal to one from the $n$ elements $x_i$. Continuing this way, we can construct the entire density as

$$ f\left(\sum_{i=1}^n x_i  = k,p\right)= \binom{n}{k} p^k  (1-p)^{n-k}  $$

where the term on the left is the binomial coefficient of $n$ things taken $k$ at a time. This is the binomial distribution and it's not the density for $\hat{p}$, but rather for $n\hat{p}$. We'll leave this as-is because it's easier to work with below. We just have to remember to keep track of the $n$ factor.

#### Confidence Intervals

Now that we have the full density for $\hat{p}$, we are ready to ask some meaningful questions. For example,

$$ \mathbb{P}\left( | \hat{p}-p | \le \epsilon p \right) $$

Or, in words, what is the probability we can get within $\epsilon$ percent of the true value of $p$. Rewriting,

$$ \mathbb{P}\left(  p - \epsilon p \lt \hat{p} \lt p + \epsilon p \right) = \mathbb{P}\left(  n p - n \epsilon p \lt \sum_{i=1}^n x_i   \lt n p + n \epsilon p \right)$$

Let's plug in some live numbers here for our worst case scenario where $p=1/2$. Then, if $\epsilon = 1/100$, we have

$$ \mathbb{P}\left( \frac{99 n}{200} \lt \sum_{i=1}^n x_i   \lt \frac{101 n}{200} \right)$$

Since the sum in integer-valued, we need $n> 100$ to even compute this. Thus, if $n=101$ we have

$$ \mathbb{P}\left( \frac{9999}{200} \lt \sum_{i=1}^{101} x_i   \lt \frac{10201}{200} \right) = f\left(\sum_{i=1}^{101} x_i  = 50,p\right)= \binom{101}{50} (1/2)^{50}  (1-1/2)^{101-50} = 0.079$$

This means that in the worst-case scenario for $p=1/2$, given $n=101$ trials, we will only get within 1% of the actual $p=1/2$ about 8% of the time. If you feel disappointed, that only means you've been paying attention. What if the coin was really heavy and it was costly to repeat this 101 times? Then, we would be within 1% of the actual value only 8% of the time. Those odds are terrible.

Let's come at this another way: given I could only flip the coin 100 times, how close could I come to the true underlying value with high probability (say, 95%)? In this case we are seeking to solve for $\epsilon$. Plugging in gives,

$$ \mathbb{P}\left(  50 - 50 \epsilon \lt \sum_{i=1}^{100} x_i   \lt 50 + 50 \epsilon  \right) = 0.95$$

which we have to solve for $\epsilon$. Fortunately, all the tools we need to solve for this are already in `scipy`.

```
import scipy.stats

b=scipy.stats.binom(100,.5) # n=100, p = 0.5, distribution of the estimator \hat{p}

f,ax= subplots()
ax.stem(arange(0,101),b.pmf(arange(0,101))) # heres the density of the sum of x_i

g = lambda i:b.pmf(arange(-i,i)+50).sum() # symmetric sum the probability around the mean
print 'this is pretty close to 0.95:%r'%g(10)
ax.vlines( [50+10,50-10],0 ,ax.get_ylim()[1] ,color='r',lw=3.)

```

The two vertical lines in the plot show how far out from the mean we have to go to accumulate 95% of the probability. Now, we can solve this as

$$ 50 + 50 \epsilon = 60 $$

which makes $\epsilon=1/5$ or 20%. So, flipping 100 times means I can only get within 20% of the real $p$ 95% of the time in the worst case scenario (i.e. $p=1/2$).

```
b=scipy.stats.bernoulli(.5) # coin distribution
xs = b.rvs(100) # flip it 100 times
phat = mean(xs) # estimated p

print abs(phat-0.5) < 0.5*0.20 # did I make it w/in interval 95% of the time?
```

    True

Let's keep doing this and see if we can get within this interval 95% of the time.

```
out=[]
b=scipy.stats.bernoulli(.5) # coin distribution
for i in range(500): # number of tries
    xs = b.rvs(100) # flip it 100 times
    phat = mean(xs) # estimated p
    out.append(abs(phat-0.5) < 0.5*0.20 ) # within 20%

print 'Percentage of tries within 20 interval = %3.2f'%(100*sum(out)/float(len(out) ))
```

    Percentage of tries within 20 interval = 96.20

Well, that seems to work. Now we have a way to get at the quality of the estimator, $\hat{p}$.

## Summary

In this section, we explored the concept of maximum likelihood estimation using a coin flipping experiment both analytically and numerically with the  scientific Python tool chain. There are two key points to remember. First, maximum likelihood estimation produces a function of the data that is itself a random variable, with its own statistics and distribution. Second, it's worth considering how to analytically derive the density function of the estimator rather than relying on canned packages to compute confidence intervals wherever possible. This is especially true when data is hard to come by and the approximations made in the central limit theorem are therefore harder to justify.

### References

This [IPython notebook](www.ipython.org) is available for [download](https://github.com/unpingco/Python-for-Signal-Processing/blob/master/Maximum_likelihood.ipynb). I urge you to experiment with the calculations for different parameters. As always, corrections and comments are welcome!


[sympy](https://docs.sympy.org/latest/index.html)

```python
# pip install sympy
# Jacobian https://docs.sympy.org/latest/modules/matrices/matrices.html

from sympy import sin, cos, Matrix
from sympy.abc import rho, phi
X = Matrix([rho*cos(phi), rho*sin(phi), rho**2])
Y = Matrix([rho, phi])
X.jacobian(Y)
```

$\displaystyle \left[\begin{matrix}\cos{\left(\phi \right)} & - \rho \sin{\left(\phi \right)}\\\sin{\left(\phi \right)} & \rho \cos{\left(\phi \right)}\\2 \rho & 0\end{matrix}\right]$

```python
X = Matrix([rho*cos(phi), rho*sin(phi)])
X.jacobian(Y)
```

$\displaystyle \left[\begin{matrix}\cos{\left(\phi \right)} & - \rho \sin{\left(\phi \right)}\\\sin{\left(\phi \right)} & \rho \cos{\left(\phi \right)}\end{matrix}\right]$

```python

```

```python
from sympy import MatrixSymbol, Matrix,diff
X = MatrixSymbol('X', 3, 3)
Y = MatrixSymbol('Y', 3, 3)
Z = (X.T*X).I*Y
```

```python
Z = (X.T*Y)*X
Z.diff(X).simplify()

# from sympy import derive_by_array
# derive_by_array(Z, X)

n=5
A = MatrixSymbol("A", n, n)
x = MatrixSymbol("x", n, 1)
diff(A.T*x, x)

```

$\displaystyle A$

```python
# https://github.com/sympy/sympy/issues/5858
x = MatrixSymbol("x", n, n)
diff(A.T*x, x)
```

$\displaystyle \frac{\partial}{\partial x} A^{T} x$


# SymPy Expressions

SymPy expressions reason about mathematics and generate numeric code.

```
from sympy import *
from sympy.abc import x, y, z
init_printing(use_latex='mathjax')
```

### Operations on SymPy objects create expressions

```
x + y
```

$$x + y$$

```
type(x + y)
```

    sympy.core.add.Add

### These expressions can be somewhat complex

```
expr = sin(x)**2 + 2*cos(x)
expr
```

$$\sin^{2}{\left (x \right )} + 2 \cos{\left (x \right )}$$

### We generate numeric code from these expressions

```
ccode(expr)  # C
```

    'pow(sin(x), 2) + 2*cos(x)'

```
fcode(expr)  # Fortran
```

    '      sin(x)**2 + 2*cos(x)'

```
jscode(expr) # JavaScript
```

    'Math.pow(Math.sin(x), 2) + 2*Math.cos(x)'

```
latex(expr)  # Even LaTeX
```

    '\\sin^{2}{\\left (x \\right )} + 2 \\cos{\\left (x \\right )}'

### We also reason about expressions

```
expr
```

$$\sin^{2}{\left (x \right )} + 2 \cos{\left (x \right )}$$

```
expr.diff(x)
```

$$2 \sin{\left (x \right )} \cos{\left (x \right )} - 2 \sin{\left (x \right )}$$

```
expr.diff(x).diff(x)
```

$$- 2 \sin^{2}{\left (x \right )} + 2 \cos^{2}{\left (x \right )} - 2 \cos{\left (x \right )}$$

### And then can generate code

```
ccode(expr.diff(x).diff(x))
```

    '-2*pow(sin(x), 2) + 2*pow(cos(x), 2) - 2*cos(x)'

### Combining reasoning and code generation gives us more efficient code

```
expr.diff(x).diff(x)
```

$$- 2 \sin^{2}{\left (x \right )} + 2 \cos^{2}{\left (x \right )} - 2 \cos{\left (x \right )}$$

```
simplify(expr.diff(x).diff(x))
```

$$- 2 \cos{\left (x \right )} + 2 \cos{\left (2 x \right )}$$

```
ccode(simplify(expr.diff(x).diff(x)))  # Faster code
```

    '-2*cos(x) + 2*cos(2*x)'

# Final Thoughts

We combine high-level reasoning with low-level code generation.

Blaze does the same thing, just swap out calculus and trig with relational and linear algebra.


# Lab 4
## Introduction
The Euler method is a method for numerically solving a DE of the form
\begin{align}
\frac{\mathrm{d}y}{\mathrm{d}x} = f(x,y).
\end{align}

It is often necessary to solve DEs this way as analytical solutions are the exception
rather than the rule.

Euler’s method works by approximating small segments of the curve solution to the DE
with the straight-line tangent or slope of the curve. As long as we keep the segments
small enough, they will approximately match what the actual curve looks like. It requires us to ”know” an initial value $y(x_0) = y_0$ so we can start the calculation.

To calculate the first segment we start off with our known start point $(x_0, y_0)$, and calculate the end point, $(x_1, y_1)$. We can define $\Delta x$ to be some constant small distance so that we always increment the $x$ value by the same amount. Then, $\Delta y = m \Delta x$ and $(x_1, y_1)=(x_0, y_0)+(\Delta x, m\Delta x)$.

But, we also know that $m$, the slope of the line, is given by $\mathrm{d}y/\mathrm{d}x$, i.e., $f(x, y)$ evaluated at $(x_0, y_0)$. So actually, $\Delta y = f(x_0, y_0) \Delta x$.

The final step is to calculate the new point: the point at the end of the first line segment. This point is then given by $(x_1, y_1) = (x_0 + \Delta x, y_0 + f(x_0, y_0) \Delta x)$.

We then do it again to calculate $(x_2, y_2)$ using $(x_1, y_1)$ as our starting point. We then do it again to calculate $(x_3, y_3)$ using $(x_2, y_2)$ as our starting point and so on.

**Summary:** The Euler method for evaluating a DE of the form
\begin{align}
\frac{\mathrm{d}y}{\mathrm{d}x} = f(x,y).
\end{align}
involves the iterative calculation of
\begin{align}
x_{n+1} &= x_n + \Delta x\\
\text{and}\quad y_{n+1} &= y_n + f(x_n,y_n)\Delta x.
\end{align}

### Implementation

First import the necessary functions from NumPy and SciPy and set up Plotly.

```python
from numpy import arange, empty, exp
from plotly import graph_objs as go
```

Now let's write a function that implements Euler's method. We will model it on `scipy.integrate.odeint`. We will make slight changes to the parameters because we want to input $\Delta x$. Note that the string (delimited by triple quotes) immediately after the function definition is a _docstring_. It tells us what the function does and is good programming practice. The prodigious comments in the function body are not generally necessary but are included for you.

```python
def euler(func, y0, x0, xn, Dx):
    """
    Integrate an ordinary differential equation using Euler's method.

    Solves the initial value problem for systems of first order ode-s::
        dy/dx = func(y, x).

    Parameters
    ----------
    func : callable(y, x)
        Computes the derivative of y at x.
    y0 : float
        Initial condition on y.
    x0 : float
        Initial condition on x.
    xn : float
        Upper limit to value of x.
    Dx : float
        x increment.

    Returns
    -------
    x : float
        Array containing the value of x for each value of x0 + n * Dx,
        where n ranges from zero to floor( (xn - x0) / Dx ).
    y : float
        Array containing the value of y for each value of x.
    """
    x = arange(x0, xn, Dx)  # Create the x array
    y = empty(len(x))  # Create an empty y array of the same length as x
    y[0] = y0  # Set the first value of y to y0
    for n in range(len(x) - 1):  # Loop to populate the rest of the values of y
        y[n+1] = y[n] + func(y[n], x[n]) * Dx  # Euler's method

    return x, y  # Return x and y as a pair
```

First try solving
\begin{align}
\frac{\mathrm{d}y}{\mathrm{d}x} = y.
\end{align}
for $y(0)=1$ for $x$ between 1 and 5 and using $\Delta x=1$.

```python
def diff_eq(y, x):
    return y

x, y = euler(diff_eq, 1, 0, 5.01, 1)
```

Why was `xn` set to 5.01 rather than 5?

We know that the analytic solution to the above IVP is $y=\mathrm{e}^x$, so calculate that as well.

```python
x_analytic = arange(0, 5.01, 0.1)
y_analytic = exp(x_analytic)
```

Now plot them both for comparison.

```python
fig = go.Figure()
fig.add_trace(go.Scatter(x=x, y=y,
                         name='Euler'))
fig.add_trace(go.Scatter(x=x_analytic,
                         y=y_analytic,
                         name='Truth'))
fig.show('png')
```

Reproduce the comparison plot below but with $\Delta x=0.1$.

```python
def diff_eq(y, x):
    return y

x, y = euler(diff_eq, 1, 0, 5.01, 0.1)
x_analytic = arange(0, 5.01, 0.1)
y_analytic = exp(x_analytic)
fig = go.Figure()
fig.add_trace(go.Scatter(x=x, y=y,
                         name='Euler'))
fig.add_trace(go.Scatter(x=x_analytic,
                         y=y_analytic,
                         name='Truth'))
fig.show('png')
```

It is possible to quantify the error in the Euler solution compared to the analytic solution. To do this you need to re-calculate the analytic solution at the same $x$ points as you calculated your Euler solution. Then you can do a Mean Squared Error (MSE) comparison between the two.

```python
x, y = euler(diff_eq, 1, 0, 5.01, 1)
y_analytic = exp(x)
((y_analytic - y)**2).mean()
```

    2533.317105161909

Note that `((y_analytic - y)**2)` returned an `array` object, and then we called the `mean` method that was _bound_ to that object.

What is the MSE if $\Delta x = 0.1$?

```python
x, y = euler(diff_eq, 1, 0, 5.01, 0.1)
y_analytic = exp(x)
((y_analytic - y)**2).mean()
```

    88.60637343780924

## Exercises

In this lab you will try Euler's method for a couple of differential equations.

1.  a. Consider the IVP
\begin{align}
\frac{\mathrm{d}y}{\mathrm{d}x} = 2x\quad\text{where}\quad y(-2)=4.
\end{align}
Calculate the Euler approximation on the interval $x=[-2,2]$ using a step size of $\Delta x = 0.5$. On the same figure, plot your approximation and the analytic solution.

```python
def diff_eq(y, x):
    return 2*x

x, y = euler(diff_eq, 4, -2, 2.01, 0.5)
x_analytic = arange(-2.01, 2.01, 0.5)
y_analytic = (x_analytic**2)
fig = go.Figure()
fig.add_trace(go.Scatter(x=x, y=y,
                         name='Euler'))
fig.add_trace(go.Scatter(x=x_analytic,
                         y=y_analytic,
                         name='Truth'))
fig.show('png')
```

1.  b. Calculate the mean squared error (MSE) of the approximation.

```python
x, y = euler(diff_eq, 4, -2, 2.01, 0.5)
y_analytic = exp(x)
((y_analytic - y)**2).mean()
```

    9.51151128124883

1.  c. Reproduce your plot from 1a except with $\Delta x=0.1$.

```python
def diff_eq(y, x):
    return 2*x

x, y = euler(diff_eq, 4, -2, 2.01, 0.1)
x_analytic = arange(-2.01, 2.01, 0.1)
y_analytic = exp(x_analytic)
fig = go.Figure()
fig.add_trace(go.Scatter(x=x, y=y,
                         name='Euler'))
fig.add_trace(go.Scatter(x=x_analytic,
                         y=y_analytic,
                         name='Truth'))
fig.show('png')
```

1.  d. Recalculate the MSE.

```python
x, y = euler(diff_eq, 4, -2, 2.01, 0.1)
y_analytic = exp(x)
((y_analytic - y)**2).mean()
```

    4.21824682630169

2.  a. The following is the DE for the arrow problem from class.
\begin{align}
\frac{\mathrm{d}y}{\mathrm{d}t} = 294\mathrm{e}^{-0.04t}-245\quad\text{where}\quad y(0)=0
\end{align}
Calculate the Euler approximation to the solution on the interval  $t=[0,10]$ with $\Delta t=0.5$. Plot your approximation and the analytic solution on the same figure.

```python
from scipy.integrate import odeint

def diff_eq(y, x):
    return 294*exp(-0.04*x) - 245

x, y = euler(diff_eq, 0, 0, 10.01, 0.5)
x_analytic = arange(0, 10.01, 0.1)
y_analytic = odeint(diff_eq, 0, x_analytic).flatten()
fig = go.Figure()
fig.add_trace(go.Scatter(x=x, y=y,
name='Euler'))
fig.add_trace(go.Scatter(x=x_analytic,
y=y_analytic,
name='Truth'))
fig.show('png')

```

2.  b. Calculate the MSE of the approximation.

```python
x, y = euler(diff_eq, 0, 0, 10.01, 0.5)
y_analytic = (-7350*exp(-0.04*x)) - 245*x + 7350
((y_analytic - y)**2).mean()
```

    221.1226824539981

2.  c. Reproduce your plot from 2a except with $\Delta t=0.1$.

```python

x, y = euler(diff_eq, 0, 0, 10.01, 0.1)
x_analytic = arange(0, 10.01, 0.1)
y_analytic = odeint(diff_eq, 0, x_analytic).flatten()
fig = go.Figure()
fig.add_trace(go.Scatter(x=x, y=y,
name='Euler'))
fig.add_trace(go.Scatter(x=x_analytic,
y=y_analytic,
name='Truth'))
fig.show('png')
```

2.  d. Recalculate the MSE.

```python
x, y = euler(diff_eq, 0, 0, 10.01, 0.1)
y_analytic = (-7350*exp(-0.04*x)) - 245*x + 7350
((y_analytic - y)**2).mean()
```

    8.673112166785117

```python

```

```python

```

```python

```

```python

```


```python
# linear algebra concerns itself with linear systems, but represents them through vector spaces and matrices.

```

```python
#  vector is an arrow in space with a specific direction and length, often representing a piece of data.
# It is the central building block to linear algebra, including matrices and linear transformations.

# Declaring a vector in Python using a list

v = [3, 2]
print(v)
```

    [3, 2]

```python
# Declaring a vector in Python using NumPy

import numpy as np
v = np.array([3, 2])
print(v)

```

    [3 2]

```python
# Declaring a 3-dimensional vector in Python using NumPy

v = np.array([4, 1, 2])
print(v)

```

    [4 1 2]

```python
# Adding two vectors in Python using NumPy

from numpy import array
v = array([3,2])
w = array([2,-1])

# sum the vectors
v_plus_w = v + w

# display summed vector
print(v_plus_w)
```

    [5 1]

```python
# Scaling a number in Python using NumPy

v = array([3,1])

# scale the vector
scaled_v = 2.0 * v

# display scaled vector
print(scaled_v)
```

    [6. 2.]

```python
# A matrix is a collection of vectors
# , can have multiple rows and columns,
# and is a convenient way to package data.
```

```python
# Matrix vector multiplication in NumPy

# compose basis matrix with i-hat and j-hat
basis = array(
    [[3, 0],
     [0, 2]]
 )

# declare vector v
v = array([1,1])

# create new vector
# by transforming v with dot product
new_v = basis.dot(v)

print(new_v)
```

    [3 2]

```python
# When thinking in terms of basis vectors,
# I prefer to break out the basis vectors and then compose them together into a matrix.
# Just note you will need to transpose, or swap the columns and rows.
# This is because NumPy’s array() function will do the opposite orientation we want,
# populating each vector as a row rather than a column.

# Declare i-hat and j-hat
i_hat = array([2, 0])
j_hat = array([0, 3])

# compose basis matrix using i-hat and j-hat
# also need to transpose rows into columns
basis = array([i_hat, j_hat]).transpose()

# declare vector v
v = array([1,1])

# create new vector
# by transforming v with dot product
new_v = basis.dot(v)

print(new_v)
```

    [2 3]

```python
# Transforming a vector using NumPy

# Declare i-hat and j-hat
i_hat = array([2, 0])
j_hat = array([0, 3])

# compose basis matrix using i-hat and j-hat
# also need to transpose rows into columns
basis = array([i_hat, j_hat]).transpose()

# declare vector v 0
v = array([2,1])

# create new vector
# by transforming v with dot product
new_v = basis.dot(v)

print(new_v)
```

    [4 3]

```python
# Transformation 1
i_hat1 = array([0, 1])
j_hat1 = array([-1, 0])
transform1 = array([i_hat1, j_hat1]).transpose()

# Transformation 2
i_hat2 = array([1, 0])
j_hat2 = array([1, 1])
transform2 = array([i_hat2, j_hat2]).transpose()

# Combine Transformations
combined = transform2 @ transform1

# Test
print("COMBINED MATRIX:\n {}".format(combined))

v = array([1, 2])
print(combined.dot(v))
```

    COMBINED MATRIX:
     [[ 1 -1]
     [ 1  0]]
    [-1  1]

```python
#  Determinants describe how much a sampled area in a vector space changes in scale with linear transformations,
#     and this can provide helpful information about the transformation

from numpy.linalg import det

i_hat = array([3, 0])
j_hat = array([0, 2])

basis = array([i_hat, j_hat]).transpose()

determinant = det(basis)

print(determinant)
```

    6.0

```python
# testing for a 0 determinant is highly helpful to determine if a transformation has linear dependence.
# When you encounter this you will likely find a difficult or unsolvable problem on your hands.

i_hat = array([-2, 1])
j_hat = array([3, -1.5])

basis = array([i_hat, j_hat]).transpose()

determinant = det(basis)

print(determinant)
```

    0.0

```python
from sympy import *

# 4x + 2y + 4z = 44
# 5x + 3y + 7z = 56
# 9x + 3y + 6z = 72

A = Matrix([
    [4, 2, 4],
    [5, 3, 7],
    [9, 3, 6]
])

# dot product between A and its inverse
# will produce identity function
inverse = A.inv()
identity = inverse * A

# prints Matrix([[-1/2, 0, 1/3], [11/2, -2, -4/3], [-2, 1, 1/3]])
print("INVERSE: {}".format(inverse))

# prints Matrix([[1, 0, 0], [0, 1, 0], [0, 0, 1]])
print("IDENTITY: {}".format(identity))
```

    INVERSE: Matrix([[-1/2, 0, 1/3], [11/2, -2, -4/3], [-2, 1, 1/3]])
    IDENTITY: Matrix([[1, 0, 0], [0, 1, 0], [0, 0, 1]])

```python
# 4x + 2y + 4z = 44
# 5x + 3y + 7z = 56
# 9x + 3y + 6z = 72
from numpy import array

from numpy.linalg import inv

A = array([[4, 2, 4],[5, 3, 7],[9, 3, 6]])

B = array([44,56,72])

X = inv(A).dot(B)

print(X)
```

    [ 2. 34. -8.]

```python
# Performing eigendecomposition in NumPy

from numpy import array, diag
from numpy.linalg import eig, inv

A = array([
    [1, 2],
    [4, 5]
])

eigenvals, eigenvecs = eig(A)

print("EIGENVALUES")
print(eigenvals)
print("\nEIGENVECTORS")
print(eigenvecs)
```

    EIGENVALUES
    [-0.46410162  6.46410162]

    EIGENVECTORS
    [[-0.80689822 -0.34372377]
     [ 0.59069049 -0.9390708 ]]

```python
# Decomposing and recomposing a matrix in NumPy

```


# Least Squares Regression

We're going to tackle one of the most simple and yet powerful tool of Machine Learning, the Least Squares estimator. Given a function in the shape of

\begin{equation}
\hat y = w^T\Phi\left(x\right) + b \label{linear}
\end{equation}

where $\hat y, b \in \mathbb{R}$ and $\Phi$ is a mapping from our input space to a given feature space such that $\Phi: \mathbb{R}^m \to \mathbb{R}^n$, we want to estimate the optimal $w$ which minimizes

\begin{equation}
e = \sum^{N}_{i = 1} \left(y - \hat y\right)^2 \label{sse}
\end{equation}

## Problem: fit the a straight line through a number of points

This is the most basic and common use case, so we need to see it in action. Let's generate a number of random points between $\left[0, 1\right]$ according to the following model

\begin{equation}
y = 2x + 1 + e
\end{equation}

where $e \sim \mathcal{N} \left(0, 0.2^2\right)$. In this case $w \in \mathbb{R}$ and is equal to $2$, $x \in \mathcal{U} \left(0, 1\right)$ and $b=1$

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

n = 100
x = np.random.uniform(size=n)
e = np.random.normal(scale=0.2, size=n)
y = 2*x + 1 + e

#
plt.scatter(x=x, y=y)
plt.show()

```

### How to find the best $w$ and $b$?

Let's derive this one time so that you understand the most basic concepts of all these algorithms. We are interested in finding the weight vector $w$ which minimizes the sum of the square error

\begin{equation}
\underset{w}{\operatorname{argmin}} e = \sum^{N}_{i = 1} \left(y_i - w^T \Phi(x_i) - b \right)^2
\end{equation}

We can augment $w$ such that $w_0 = b$ and $w \in \mathbb{R}^{n+1}$ and rerite the expression above as

\begin{equation}
\underset{w}{\operatorname{argmin}} e = \sum^{N}_{i = 1} \left(y_i - w^T \left[\begin{matrix}\Phi(x_i) \\ 1 \end{matrix} \right] \right)^2
\end{equation}

Since the equation is linear, it has a closed-form solution. Let's start by finding the $w$ which gives as a null gradient. We're also going to make use of an important property of matrix derivatives, in this case applied to vectors

\begin{equation}
\frac{d x^Ta}{dx} =  a
\end{equation}

for $a, x \in \mathbb{R}^n$.

\begin{eqnarray}
\frac{d}{dw} \sum^{N}_{i = 1} \left(y_i - w^T \left[\begin{matrix}\Phi(x_i) \\ 1 \end{matrix} \right] \right)^2  & = & 0 \\
\sum^{N}_{i = 1} \frac{d}{dw} \left(y_i - w^T \left[\begin{matrix}\Phi(x_i) \\ 1 \end{matrix} \right] \right)^2 & = & 0 \\
\sum^{N}_{i = 1} -2 \left[\begin{matrix}\Phi(x_i) \\ 1 \end{matrix} \right] \left(y_i - w^T \left[\begin{matrix}\Phi(x_i) \\ 1 \end{matrix} \right] \right) & = & 0 \\
\sum^{N}_{i = 1} \left[\begin{matrix}\Phi(x_i) \\ 1 \end{matrix} \right] \left(\left[\begin{matrix}\Phi(x_i)^T & 1 \end{matrix} \right] w - y_i\right) & = & 0 \\
\sum^{N}_{i = 1} \left[\begin{matrix}\Phi(x_i) \\ 1 \end{matrix} \right] \left[\begin{matrix}\Phi(x_i)^T & 1 \end{matrix} \right] w & = & \sum^{N}_{i = 1} \left[\begin{matrix}\Phi(x_i) \\ 1 \end{matrix} \right] y_i \\
\sum^{N}_{i = 1} \left[\begin{matrix}\Phi(x_i)\Phi(x_i)^T & \Phi(x_i) \\ \Phi(x_i)^T & 1 \end{matrix} \right] w & = & \sum^{N}_{i = 1} \left[\begin{matrix}\Phi(x_i) \\ 1 \end{matrix} \right] y_i \\
 \left[\begin{matrix}\sum^{N}_{i = 1} \Phi(x_i)\Phi(x_i)^T & \sum^{N}_{i = 1} \Phi(x_i) \\ \sum^{N}_{i = 1} \Phi(x_i)^T & N \end{matrix} \right] w & = & \left[\begin{matrix} \sum^{N}_{i = 1}\Phi(x_i) y_i \\ \sum^{N}_{i = 1}y_i \end{matrix} \right]
\end{eqnarray}

resulting of course in

\begin{equation}
 w = \left[\begin{matrix}\sum^{N}_{i = 1} \Phi(x_i)\Phi(x_i)^T & \sum^{N}_{i = 1} \Phi(x_i) \\ \sum^{N}_{i = 1} \Phi(x_i)^T & N \end{matrix} \right]^{-1}\left[\begin{matrix} \sum^{N}_{i = 1}\Phi(x_i) y_i \\ \sum^{N}_{i = 1}y_i \end{matrix} \right]\\
\end{equation}

Let's apply this proof to our case. First we need to figure what is what. We know the data was generated under the following model

\begin{equation}
y = 2x + 1
\end{equation}

plus some added noise. In this case the $\Phi(x) = x$, $w = 2$ and $b = 1$.

```python
X = np.array([[np.dot(x, x), np.sum(x)],[np.sum(x), n]])
Y = np.array([np.dot(x,y), np.sum(y)])
theta = np.linalg.solve(X,Y)
print(theta)
```

    [1.95328232 0.98646063]

```python
plt.plot(x, y, 'o', label='original data')
plt.plot(x, theta[0]*x + theta[1], 'r', label='fitted line')
plt.legend()
plt.show()
```

Scikit already provides linear regression capabilities among others. So everything we went through could have been replaced by

```python
import sklearn
from sklearn import linear_model

regr = linear_model.LinearRegression()
regr.fit(x.reshape(-1,1), y)

# The coefficients
print('Coefficients: \n', regr.coef_)
print('Intercept: \n', regr.intercept_)
```

    Coefficients:
     [1.95328232]
    Intercept:
     0.9864606337778363

    /usr/local/lib/python3.6/site-packages/scipy/linalg/basic.py:1226: RuntimeWarning: internal gelsd driver lwork query error, required iwork dimension not returned. This is likely the result of LAPACK bug 0038, fixed in LAPACK 3.2.2 (released July 21, 2010). Falling back to 'gelss' driver.
      warnings.warn(mesg, RuntimeWarning)

### Normal Equations

\begin{equation}
 w = \left[\begin{matrix}\sum^{N}_{i = 1} \Phi(x_i)\Phi(x_i)^T & \sum^{N}_{i = 1} \Phi(x_i)^T \\ \sum^{N}_{i = 1} \Phi(x_i) & N \end{matrix} \right]^{-1}\left[\begin{matrix} \sum^{N}_{i = 1}\Phi(x_i) y_i \\ \sum^{N}_{i = 1}y_i \end{matrix} \right]\\
\end{equation}

Can be written in another way commonly know as the **normal equations**. Just like before, consider you have $N$ samples and you stack your features and your target like this

\begin{equation}
X =  \left[\begin{matrix}\Phi(x_1)^T & \dots  & 1 \\
                         \vdots    & \ddots & \vdots \\
                         \Phi(x_N)^T & \dots  & 1 \end{matrix}\right]
\end{equation}

\begin{equation}
\textbf{y} =  \left[\begin{matrix} y_1 \\ \vdots \\ y_N \end{matrix}\right]
\end{equation}

The best $w$ is given by

\begin{equation}
\hat{w} = (X^TX)^{-1} X^T  \textbf{y}
\end{equation}

Let's test it.

```python
X = np.matrix([x, np.ones(len(x))]).T
Y = np.matrix(y).T
print("X shape:\n", X.shape)
print("Y shape:\n", Y.shape)

w = np.linalg.solve(X.T * X, X.T * Y)
print("w:\n", w)
```

    X shape:
     (100, 2)
    Y shape:
     (100, 1)
    w:
     [[1.95328232]
     [0.98646063]]

**Homework**: Prove it!

## Issues

If you run into troubles or find mistakes, bugs, please open an issue on the [issue tracker](https://github.com/SergioRAgostinho/bootstrap-ml/issues).


# Analytical problem
Defining a problem with an explicit mathematical representation is straightforwars.

As an example, consider the following multiobjective optimization problem

\begin{equation}
\begin{aligned}
& \underset{\mathbf x}{\text{min}}
& & x_1^2 - x_2; x_2^2 - 3x_1 \\
& \text{s.t.} & &  x_1 + x_2 \leq 10 \\
& & &  \mathbf{x} \; \in S, \\
\end{aligned}
\end{equation}

where the feasible region is

\begin{equation}
x_i \in \left[-5, 5\right] \; \forall i \;\in \left[1,2\right].
\end{equation}

Begin by importing the necessary classes:

```python
from desdeov2.problem.Problem import ScalarMOProblem
from desdeov2.problem.Objective import ScalarObjective
from desdeov2.problem.Variable import Variable
from desdeov2.problem.Constraint import ScalarConstraint
```

Define the variables:

```python
# Args: name, starting value, lower bound, upper bound
x1 = Variable("x_1", 0, -0.5, 0.5)
x2 = Variable("x_2", 0, -0.5, 0.5)
```

Define the objectives, notice the argument of the callable objective function, it is assumed to be array-like.

```python
# Args: name, callable
obj1 = ScalarObjective("f_1", lambda x: x[0]**2 - x[1])
obj2 = ScalarObjective("f_2", lambda x: x[1]**2 - 3*x[0])
```

Define the constraints. Constraint may depend on objective function as well (second argument to the lambda, notice the underscore). In that case, the objectives should not be defined inline, like above, but as their own function definitions. The constraint should be defined so, that when evaluated, it should return a positive value, if the constraint is adhered to, and a negative, if the constraint is breached.

```python
# Args: name, n of variables, n of objectives, callable
cons1 = ScalarConstraint("c_1", 2, 2, lambda x, _: 10 - (x[0] + x[1]))
```

Finally, put it all together and create the problem.

```python
# Args: list of objevtives, variables and constraints
problem = ScalarMOProblem([obj1, obj2]
                         ,[x1, x2]
                         ,[cons1])
```

Now, the problem is fully specified and can be evaluated and played around with.

```python
import numpy as np

print("N of objectives:", problem.n_of_objectives)
print("N of variables:", problem.n_of_variables)
print("N of constraints:", problem.n_of_constraints)

res1, eval_cons1 = problem.evaluate(np.array([2, 4]))
res2, eval_cons2 = problem.evaluate(np.array([6, 6]))
res3, eval_cons3 = problem.evaluate(np.array([[6, 3], [4,3], [7,4]]))

print("Single feasible decision variables:", res1, "with constraint values", eval_cons1)
print("Single non-feasible decision variables:", res2, "with constraint values", eval_cons2)
print("Multiple decision variables:", res3, "with constraint values", eval_cons3)
```


```python
import numpy as np
import pandas as pd
import statsmodels.api as sm
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
```

# Class 10: Stochastic Time Series Processes

Most models of the business cycle are *stochastic* time series models. That is, the models incorporate randomness as a determinant of equilibrium. Randomness is necessary for two reasons. First, from a practical point of view, any model will be incomplete and will not fit the data perfectly and so randomness in a macroeconomic model is analagous to an error term in an OLS model. Second, from a philosophical perspective, randomness in a macroeconomic model reflects economists' consensus that fundamentally unpredictable forces cause the business cycle.

## Simulating Normal Random Variables with Numpy

Recall that the `numpy.random` module has functions for generating (pseudo) random variables. Learn more about the module by reading the documentation: https://docs.scipy.org/doc/numpy/reference/routines.random.html

We use the `numpy.random.normal()` function to crate arrays of random draws from the normal distribution. The function takes three arguments:
* `loc`: the mean of the distribution (default=0)
* `scale`: the standard deviation of the distribution (default=1)
* `size`: how many to numbers to draw (default = `None`)

The default is to draw numbers from the *standard normal* distribution.

### Example

Draw 500 values each from the $\mathcal{N}(0,1)$ and $\mathcal{N}(0,2^2)$ distributions. Plot.

```python
# Set the seed for the random number generator to 126
np.random.seed(126)

# Create two arrays:
#     x: 500 draws from the normal(0,1) distribution
#     y: 500 draws from the normal(0,4) distribution
x = np.random.normal(loc=0,scale=1,size=500)
y = np.random.normal(loc=0,scale=2,size=500)

# Plot x and y
plt.plot(x,lw=2,alpha = 0.6,label='$\sigma=1$')
plt.plot(y,lw=2,alpha = 0.6,label='$\sigma=2$')
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
plt.grid()
```

## The White Noise Process

In the previous example, we created two variables that stored draws from normal distrbutions with means of zero but with different standard deviations. Both of the variables were simulations of *white noise processes*. A white noise process is a random variable $\epsilon_t$ with constant mean and constant variance. We are concerned only with the zero-mean white noise process and we'll denote that a variable is a zero-mean white noise process with the following shorthand notation:

\begin{align}
\epsilon_t & \sim \text{WN}(0,\sigma^2),
\end{align}

where $\sigma^2$ is the variance of the processes. Strictly speaking, a white noise process can follow any distribution as long as the mean and variance are constant, but we'll concentrate exclusively white noise process drawn from the normal distribution.

## The AR(1) Process

A random variable $y_t$ is an *autoregressive process of order 1*  or AR(1) process if it can be written in the following form:

\begin{align}
y_t & = \rho y_{t-1} + \epsilon_t,
\end{align}

where $\rho$ is a constant and $\epsilon \sim \text{WN}(0,\sigma^2)$. The AR(1) process is the stochastic analog of the first-order difference equation where the random variable $\epsilon_t$ replaces the exogenous variable $w_t$.

### Example

Simulate an AR(1) process for 101 periods ($t = 0,\ldots, 100$) using the following parameter values:

\begin{align}
\rho & = 0.5\\
\sigma &  = 1\\
y_0 & = 0
\end{align}

Plot the simulated values for $y$.

```python
# Set the seed for the random number generator to 126
np.random.seed(126)

# Initialize values for T, y0, rho, and sigma
T = 101
y0 = 0
rho = 0.5
sigma = 1

# Initialize an array of zeros for y
y = np.zeros(T)

# Set the first value of y equal to y0
y[0] = y0

# Create a variable called 'epsilon' equal to an array containing T draws from the normal(0,sigma^2) process
eps= np.random.normal(loc=0,scale=sigma,size=T)

# Iterate over t in range(T-1) to compute y
for t in range(T-1):
    y[t+1] = rho*y[t] + eps[t+1]

# Plot y
plt.plot(y,lw=2)
plt.grid(linestyle=':')
```

The AR(1) process wtih $\rho = 0.5$ seems to fluctuate around the value $y=0$ and so the process appears to be stable.

### Example

Simulate an AR(1) process for 101 periods ($t = 0,\ldots, 100$) using the following parameter values:

\begin{align}
\rho & = 1.5\\
\sigma &  = 1\\
y_0 & = 0
\end{align}

Plot the simulated values for $y$.

```python
# Set the seed for the random number generator to 126
np.random.seed(126)

# Initialize values for T, y0, rho, and sigma
T = 101
y0 = 0
rho = 1.5
sigma = 1

# Initialize an array of zeros for y
y = np.zeros(T)

# Set the first value of y equal to y0
y[0] = y0

# Create a variable called 'epsilon' equal to an array containing T draws from the normal(0,sigma^2) process
eps= np.random.normal(loc=0,scale=sigma,size=T)

# Iterate over t in range(T-1) to compute y
for t in range(T-1):
    y[t+1] = rho*y[t] + eps[t+1]

# Plot y
plt.plot(y,lw=2)
plt.grid(linestyle=':')
```

The AR(1) process wtih $\rho = 1.5$ seems to be approaching infinity in magnitude and so the process appears to be explosive.

In general, like the first-order difference equation, if $|\rho| < 1$, the the AR(1) process is stable. If $|\rho|>1$ then the process is explosive. A special case is the *random walk process* which is an AR(1) process with $\rho = 1$. The random walk process has many important applications including asset pricing theory.

The function in the next cell computes a simulation of an AR(1) process.

```python
# Define a function for simulating an AR(1) process. CELL PROVIDED
def ar1_sim(rho=0,sigma=1,y0=0,T=25):
    '''Funciton for simulating an AR(1) process for T periods

            Args:
            rho (float):   autoregressive parameter
            sigma (float): standard deviation of the white noise process
            y0 (float):    initial value of the process
            T (int):       number of periods to simulate

        Returns:
            numpy array
    '''

    # initialize y array
    y = np.zeros(T)
    y[0] = y0

    # draw random numbers for white noise process
    eps= np.random.normal(loc=0,scale=sigma,size=T-1)
    for t in range(T-1):
        y[t+1] = rho*y[t] + eps[t]

    return y
```

### Example:

Use the `ar1_sim()` function to simulate an AR(1) process for 201 periods ($t = 0,\ldots, 200$) using the following parameter values:

\begin{align}
\rho & = -0.99\\
\sigma &  = 0.5\\
y_0 & = 0
\end{align}

Set the seed for the NumPy random number generator to 126. Plot the simulated values for $y$.

```python
# Set the seed for the random number generator to 126
np.random.seed(126)

# Simulate y and plot
y = ar1_sim(rho=-0.99,sigma=0.5,y0=0,T=201)
plt.plot(y)
plt.grid()
```

## Application: Estimate TFP as an AR(1) process

It is routine to model quarterly fulctuations in TFP as an AR(1) process and we will encounter this in our business cycle models. Here we will fit the following AR(1) model to US data:

\begin{align}
\log\left(A_t/A^{trend}_t\right) & = \rho \log\left(A_{t-1}/A^{trend}_{t-1}\right) + \epsilon_t
\end{align}

where $\log\left(A_t/A^{trend}_t\right) = \log A_t - \log A^{trend}$ is the log-deviation of TFP from its trend and $\epsilon_t$ is a white noise process with mean 0 and variance $\sigma^2$.

### The Data
The file `rbc_data_actual_trend.csv`, available at https://github.com/letsgoexploring/econ126/raw/master/Data/Csv/rbc_data_actual_trend.csv, contains actual and trend data for real GDP per capita, real consumption per capita, real investment per capita, real physical capital per capita, TFP, and hours per capita at quarterly frequency. The GDP, consumption, investment, and capital data are in terms of 2012 dollars. Hours is measured as an index with the value in October 2012 set to 100. All of the data are *real* quantities. That is, there are no *nominal* quantities like money or inflation or a nominal interest rate. The reason is that the first theory that we will encounter is called *real business cycle* or RBC theory and, in that theory, there is no place for nominal quantities. RBC theory seeks to explain fluctuations in real quantities as being primarily due to TFP shocks; i.e., shocks to the production function.

### Objectives

1. Import TFP data (trend and actual) and compute cyclical component as log-deviation from trend.
2. Construct a scatter plot of log-deviation of TFP from trend against *lagged* log-deviation of TFP from trend.
3. Estimate the AR(1) model of log-deviation of TFP from trend to obtain estimates of $\rho$ and $\sigma$.

```python
# Read business_cycle_data_actual_trend.csv into a Pandas DataFrame called 'df' with the first column set as the index and parse_dates=True
df = pd.read_csv('https://github.com/letsgoexploring/econ126/raw/master/Data/Csv/rbc_data_actual_trend.csv',index_col=0,parse_dates=True)
```

```python
# Construct a plot of TFP with its trend with:
#    1. Actual line: blue with lw=1, alpha=0.7, label = 'actual'
#    2. Trend line:  red with lw=3, alpha=0.7, label = 'trend'
plt.plot(df['tfp'],lw=2,alpha=0.6,label='Actual')
plt.plot(df['tfp_trend'],'r',lw=2,alpha=0.6,label='Trend')
plt.title('TFP for US from '+df.index[0].strftime('%B %Y')+' to '+df.index[-1].strftime('%B %Y'))
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
plt.grid()
```

Now we need to compute the log-deviation of TFP from its trend. Note that since $\log(a/b) = \log(a) - \log(b)$, we can write:

\begin{align}
\log A_t - \log A^{trend} & = \log\left(A_t/A^{trend}_t\right)
\end{align}

The term on the right will make the AR(1) model easier to read.

\begin{align}
\log\left(A_t/A^{trend}_t\right) & = \rho \log\left(A_{t-1}/A^{trend}_{t-1}\right) + \epsilon_t
\end{align}

```python
# Create a new column to df called tfp_cycle equal to the log difference between actual TFP and it's trend:
df['tfp_cycle'] = np.log(df['tfp']/df['tfp_trend'])
```

```python
# Plot the log deviation of TFP from its trend (times 100)
plt.plot(df['tfp_cycle'],lw=2,alpha=0.6)
plt.title('TFP for US from '+df.index[0].strftime('%B %Y')+' to '+df.index[-1].strftime('%B %Y'))
plt.grid()
```

```python
# Create a new column to df called tfp_cycle_lag that, for each date, contains values
# in 'tfp_cycle' at the previous date
df['tfp_cycle_lag'] = df['tfp_cycle'].shift()

# Print the first five rows of only the 'tfp_cycle' and 'tfp_cycle_lag' columns. PROVIDED
print(df[['tfp_cycle','tfp_cycle_lag']].head())
```

                tfp_cycle  tfp_cycle_lag
    1948-01-01   0.003250            NaN
    1948-04-01   0.009117       0.003250
    1948-07-01  -0.001220       0.009117
    1948-10-01  -0.005680      -0.001220
    1949-01-01  -0.019553      -0.005680

Notice that there is a missing value inthe `tfp_cycle_lag` column for the first date in the index because there is no prior observation in the `tfp_cycle` column. To proceed with the estimation, we need to get rid of the row with missing values.

```python
# Use dropna() method on df to remove the row iwth the missing value
df= df.dropna()

# Print the first five rows of only the 'tfp_cycle' and 'tfp_cycle_lag' columns. PROVIDED
print(df[['tfp_cycle','tfp_cycle_lag']].head())
```

                tfp_cycle  tfp_cycle_lag
    1948-04-01   0.009117       0.003250
    1948-07-01  -0.001220       0.009117
    1948-10-01  -0.005680      -0.001220
    1949-01-01  -0.019553      -0.005680
    1949-04-01  -0.020787      -0.019553

Next, we should make a scatter plot of log-deviation of TFP from trend against the one-period lag of log-deviation of TFP from trend to see if out AR(1) model is a good idea.

```python
# Construct a scatter plot of log-deviation of TFP from trend against the one-period lag of log-deviation
# of TFP from trend with:
#    1. scatter points sized at least 50, opacity (alpha) no greater than 0.25
#    2. x- and y-axis limits: [-0.04,0.04]
plt.scatter(df['tfp_cycle_lag'],df['tfp_cycle'],s=50,alpha=0.25)
plt.xlabel('Lag log-deviation from trend')
plt.ylabel('Current log-deviation from trend')
plt.title('TFP for US from '+df.index[0].strftime('%B %Y')+' to '+df.index[-1].strftime('%B %Y'))
plt.xlim([-0.04,0.04])
plt.ylim([-0.04,0.04])
plt.grid()
```

Now we use StatsModels to fit the model. First we estimate the AR(1) model to obtain an estimate of $\rho$. Then we find the standard deviation of the residuals of the regression to estimate $\sigma$.

```python
# Create a variable 'X' to be the independent variable of the OLS model. Do not add a constant to X
X = df['tfp_cycle_lag']

# Create a variable 'Y' to be the dependent variable of the OLS model.
Y = df['tfp_cycle']

# Create a variable called 'model' that initializes the OLS model
model = sm.OLS(Y,X)

# Create a variable called 'results' that stores the results of the estimated OLS model
results = model.fit()

# Print output of summary2() method of results
print(results.summary2())
```

                            Results: Ordinary least squares
    ================================================================================
    Model:                  OLS              Adj. R-squared (uncentered): 0.553
    Dependent Variable:     tfp_cycle        AIC:                         -2130.2726
    Date:                   2021-02-04 15:30 BIC:                         -2126.6028
    No. Observations:       290              Log-Likelihood:              1066.1
    Df Model:               1                F-statistic:                 360.3
    Df Residuals:           289              Prob (F-statistic):          1.01e-52
    R-squared (uncentered): 0.555            Scale:                       3.7653e-05
    ------------------------------------------------------------------------------------
                        Coef.      Std.Err.        t        P>|t|      [0.025     0.975]
    ------------------------------------------------------------------------------------
    tfp_cycle_lag       0.7463       0.0393     18.9804     0.0000     0.6689     0.8237
    --------------------------------------------------------------------------------
    Omnibus:                    4.758             Durbin-Watson:               1.834
    Prob(Omnibus):              0.093             Jarque-Bera (JB):            5.687
    Skew:                       -0.127            Prob(JB):                    0.058
    Kurtosis:                   3.637             Condition No.:               1
    ================================================================================

Estimated coefficients are stored as a Pandas `Series` in the `params` attribute of `results`. Index values correspond to names of columns in the dependent variable `X`

```python
# Print the contents of the params method of results
print(results.params)
```

    tfp_cycle_lag    0.746324
    dtype: float64

```python
# Create a variable called 'rho' that equals the value of the estimated coefficient on df['tfp_cycle_lag']
rho = results.params['tfp_cycle_lag']
```

The residuals from the regression are stored in an attribute of `results` called `resid`

```python
# Create a variable called 'sigma' that equals the standard deviation of the residuals of the regression
sigma = results.resid.std()

# Print the value of sigma
print(sigma)
```

    0.00613621877954209

And that's how you estimate the parameters of an AR(1) model of the cyclical component of TFP for the US.

## Application: A Stochastic Solow Growth Model (Optional)

Consider the Solow growth model with written in "per worker" terms:

\begin{align}
y_t & = A_tk_t^{\alpha}\\
k_{t+1} & = i_t + (1-\delta)k_{t}\\
y_t & = c_t + i_t\\
c_t & = (1-s)y_t,
\end{align}

where $y_t$ is output per worker, $k_t$ is capital per worker, $c_t$ is consumption per worker, $i_t$ is investment per worker, and $A_t$ is time-varying TFP. Assume:

\begin{align}
\log A_{t+1} & = \rho \log A_{t} + \epsilon_{t+1}
\end{align}

where $\epsilon_t$ is a white noise process with mean 0 and variance $\sigma^2$. Since capital and TFP are the only two variables, that depend on past value of themselves, they are the two state variables of the model. Therefore, we can simulate simulate the model in two steps. First, simulate $k_t$ and $A_t$ using the following two laws of motion:

\begin{align}
k_{t+1} & = sAk_t^{\alpha} + (1-\delta)k_{t} \label{eqn:capital_solution}\\
\log A_{t+1} & = \rho \log A_{t} + \epsilon_{t+1}. \label{eqn:tfp_solution}
\end{align}

Second, compute simulated values for $y_t$, $c_t$, and $i_t$ using the following static relationships:

\begin{align}
y_t & = A_tk_t^{\alpha}\\
i_t & = sA_tk_t^{\alpha}\\
c_t & = (1-s)A_tk_t^{\alpha},
\end{align}

In the previous example, we estimated $\rho$ and $\sigma$ for the US so let's use those values for this simulation. For the other parameter, use the following values for the simulation:

| $$A_0$$ | $$k_0$$ | $$s$$ | $$\alpha$$ | $$\delta $$ | $$T$$  |
|---------|---------|-------|------------|-------------|--------|
| 1       | 8.43    | 0.1   | 0.35       |  0.025      | 201    |

Where $T$ is the total number of simulation periods (i.e., $t$ will range from $0$ to $200$).

### Function

The function in the next cell simulates a stochastic Solow growth model. It returns a `DataFrame` with columns equal to the log-deviations of the simulated variables relative to the trend (i.e., nonstochastic steady state) implied by the model.

```python
# Define a function that returns a DataFrame of simulated value from the Solow model with exogenous labor and TFP growth. CELL PROVIDED
def solow_stochastic(s,alpha,delta,k0,A0,rho,sigma,T):

    '''Function for computing a simulation of the Solow growth model with exogenous labor and TFP growth.

            y[t]   = A[t]*K[t]^alpha
            k[t+1] = i[t] + (1-delta)*k[t]
            y[t]   = c[t] + i[t]
            c[t]   = (1-s)*y[t]
            A[t+1] = exp(rho*logA[t] + epsilon[t+1])

    Args:
        s (float):      Saving rate
        alpha (float):  Capital share in Cobb-Douglas production function
        delta (float):  Capital depreciation rate
        T (int):        Number of periods to simulate
        k0 (float):     Initial value of capital per worker
        A0 (float):     Initial TFP
        rho (float):    AR coeficient for log A[t]
        sigma (float):  Standard deviation of shock to log A[t]

     Returns:
         Pandas DataFrame
     '''
    # Create epsilon values
    epsilon = np.random.normal(scale=sigma,size=T)

    # Initialize capital values
    capital = np.zeros(T)

    # Set first value of capital equal to k0
    capital[0] = k0

    # Initialize TFP values
    tfp = np.zeros(T)

    # Set first value of TFP equal to A0
    tfp[0] = A0

    # Iterate over t in range(T-1) to update subsequent values in the capital and tfp arrays
    for t in range(T-1):
        capital[t+1] = s*tfp[t]*capital[t]**alpha + (1-delta)*capital[t]
        tfp[t+1] = np.exp(rho*np.log(tfp[t]) + epsilon[t])

    # Compute the values of the other aggregate variables
    output = tfp*capital**alpha
    consumption = (1-s)*output
    investment = s*output

    # Compute steady state (or trend) of endogenous vars
    capital_ss = (s/delta)**(1/(1-alpha))
    tfp_ss = 1
    output_ss = tfp_ss*capital_ss**alpha
    consumption_ss = (1-s)*output_ss
    investment_ss = s*output_ss

    # Put simulated data into a DataFrame
    df = pd.DataFrame({'output_log_dev':np.log(output/output_ss),
                       'consumption_log_dev':np.log(consumption/consumption_ss),
                       'investment_log_dev':np.log(investment/investment_ss),
                       'capital_log_dev':np.log(capital/capital_ss),
                       'tfp_log_dev':np.log(tfp/tfp_ss)})

    # Return the simulated data
    return df
```

### Simulate the Stochastic Solow Model

```python
# CELL PROVIDED
# Set parameters for simulation
alpha=0.35
s = 0.1
delta = 0.025
k0 = 8.43
A0=1
T = 201

np.random.seed(126)
# Simulate the model and store output in a variable called 'solow_df'
solow_df = solow_stochastic(s,alpha,delta,k0,A0,rho,sigma,T)

# Plot
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
solow_df.plot(ax=ax,grid=True)
ax.legend(loc='center left', bbox_to_anchor=(1, 0.5))
```

### Simulation Results

Compute the standard deviation of the simulated data and the correlation coefficients of the simulated data.

```python
# Standard deviations. CELL PROVIDED
solow_df.std()*100
```

    output_log_dev         0.837875
    consumption_log_dev    0.837875
    investment_log_dev     0.837875
    capital_log_dev        0.187482
    tfp_log_dev            0.829267
    dtype: float64

```python
# Correlation coefficients. CELL PROVIDED
solow_df.corr()
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>output_log_dev</th>
      <th>consumption_log_dev</th>
      <th>investment_log_dev</th>
      <th>capital_log_dev</th>
      <th>tfp_log_dev</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>output_log_dev</th>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>0.169656</td>
      <td>0.996955</td>
    </tr>
    <tr>
      <th>consumption_log_dev</th>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>0.169656</td>
      <td>0.996955</td>
    </tr>
    <tr>
      <th>investment_log_dev</th>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>0.169656</td>
      <td>0.996955</td>
    </tr>
    <tr>
      <th>capital_log_dev</th>
      <td>0.169656</td>
      <td>0.169656</td>
      <td>0.169656</td>
      <td>1.000000</td>
      <td>0.092288</td>
    </tr>
    <tr>
      <th>tfp_log_dev</th>
      <td>0.996955</td>
      <td>0.996955</td>
      <td>0.996955</td>
      <td>0.092288</td>
      <td>1.000000</td>
    </tr>
  </tbody>
</table>
</div>

It looks like the stochastic Solow model does a good job replicating the volatility (standard deviation) of ouput and consumption relative to the data. However the simulated investment volatility is too small by about a factor of 6. The stochastic Solow model does capture the correlation between output, consumption, and investment, but it implies perfect correlation which is too much.

## The Random Walk Process (Optional)

The *random walk process* is an AR(1) process with $\rho=1$:

\begin{align}
y_t = y_{t-1} + \epsilon_t
\end{align}

The random walk process has an important place in finance since the evidence suggests that stock prices follow a random walk process.

### Example

Simulate 7 random walk processes for 501 periods. Set $\sigma = 1$. Plot all 7 simulated processes on the same axes.

```python
# CELL PROVIDED
np.random.seed(126)
for i in range(7):
    plt.plot(ar1_sim(rho=1,T=501))

plt.title('Seven random walk processes')
plt.grid()
```


# SymPy demo

If you need to install SymPy:

    pip install sympy

```python
import sympy
sympy.__version__
```

    '1.6.2'

```python
sympy.init_printing(use_latex='mathjax')
```

## Rearrange and simplify equations

We have an expression for Vp in terms of Young's modulus, $E$, and shear modulus, $\mu$:

$$ V_\mathrm{P} = \sqrt{\frac{\mu\,(E-4\mu)}{\rho\,(E-3\mu)}} $$

We need single symbols for quantities, so I'll use $\alpha$ for $V_\mathrm{P}$ and $\beta$ for $V_\mathrm{S}$, and $\gamma$ for their ratio.

```python
alpha, beta, gamma = sympy.symbols("alpha, beta, gamma")
lamda, mu, E, K, M, rho = sympy.symbols("lamda, mu, E, K, M, rho")
```

Now we can use these symbols to make an expression.

```python
from sympy import sqrt

alpha_expr = sqrt((mu * (E - 4*mu)) / (rho * (E - 3*mu)))
alpha_expr
```

$\displaystyle \sqrt{\frac{\mu \left(E - 4 \mu\right)}{\rho \left(E - 3 \mu\right)}}$

```python
print(sympy.latex(alpha_expr))
```

    \sqrt{\frac{\mu \left(E - 4 \mu\right)}{\rho \left(E - 3 \mu\right)}}

We also know that

$$ \mu = \frac{3KE}{9K-E} $$

```python
mu_expr = (3 * K * E) / (9 * K - E)
```

Now we can substitute this into the first expression.

```python
subs = alpha_expr.subs(mu, mu_expr)
subs
```

$\displaystyle \sqrt{3} \sqrt{\frac{E K \left(- \frac{12 E K}{- E + 9 K} + E\right)}{\rho \left(- E + 9 K\right) \left(- \frac{9 E K}{- E + 9 K} + E\right)}}$

This is a bit ugly! Let's simplify it:

```python
from sympy import simplify

simplify(subs)
```

$\displaystyle \sqrt{3} \sqrt{\frac{K \left(E + 3 K\right)}{\rho \left(- E + 9 K\right)}}$

We can get this as LaTeX plain-text if we want.

```python
print(sympy.latex(simplify(subs)))
```

    \sqrt{3} \sqrt{\frac{K \left(E + 3 K\right)}{\rho \left(- E + 9 K\right)}}

## Solve an equation

We'll solve

 $$ x + 2y = 3 $$
 $$ 3x + 4y = 17 $$

We'll re-write this using $x_1$ and $x_2$ instead of $x$ and $y$. They are just the names for the unknowns, but it's easier if we just think of there being one, multi-valued unknown.

 $$ x_1 + 2x_2 = 3 $$
 $$ 3x_1 + 4x_2 = 17 $$

Multi-valued quantities are _vectors_, usually written in bold face: $\mathbf{x} = [x_1, x_2]$.

Now we can rewrite this in the form $\mathbf{A}\mathbf{x} = \mathbf{b}$ (which is analogous to the $\mathbf{G}\mathbf{m} = \mathbf{d}$ form of many geophysical problems). $\mathbf{A}$ is the matrix containing all the parameters, or coefficients, of the variables (the unknowns in this case) on the left-hand side of the equation. And $\mathbf{b}$ is a vector containing the known outputs — the right-hand side of the equation.

 $$  \mathbf{A}\mathbf{x} = \mathbf{b} $$

 $$ \begin{bmatrix} 1, 2 \\ 3, 4 \end{bmatrix}  \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} =  \begin{bmatrix} 3 \\ 17 \end{bmatrix}$$

To multiply a matrix by a vector, we multiply the first row by the vector and add the result. So the first row yields $(1 x_1 + 2 x_2)$ — the first of the two equations we started with. And the second row gives us $(3 x_1 + 4 x_2)$ — the second equation.

So now 'all' we have to do is find the vector $\mathbf{x}$ that satisfies this new algebraic equation:

 $$ \mathbf{A}\mathbf{x} = \mathbf{b} $$
 $$ \Rightarrow \mathbf{x} = \mathbf{A}^{-1} \mathbf{b}$$

There's one catch... That symbol $\mathrm{A}^{-1}$ doesn't mean the reciprocal. It means the inverse. And that's where the fun starts.

We won't go into it now, but the inverse can be hard to compute. Sometimes it's impossible. So mathematicians have come up with lots of other ways to solve this kind of equation. Welcome to the world of **linear algebra**!

### Solve with SymPy

```python
from sympy.solvers import solve
from sympy import symbols

# Define the symbols we are going to use.
x_1, x_2 = symbols('x_1, x_2')

# Define the equations, making them equal zero.
equations = [x_1 + 2*x_2 - 3,
             3*x_1 + 4*x_2 - 17]

# Solve for x_1 and x_2.
solve(equations, (x_1, x_2))
```

$\displaystyle \left\{ x_{1} : 11, \  x_{2} : -4\right\}$

We can also solve it as a linear system.

This requires us to formulate the equations as a single matrix, called an _augmented matrix_. It might look a bit funny, but it's a standard way to solve this kind of problem:

```python
from sympy import Matrix, solve_linear_system

x_1, x_2 = symbols('x_1, x_2')

system = Matrix([[1, 2,  3],
                 [3, 4, 17]])

solve_linear_system(system, x_1, x_2)
```

$\displaystyle \left\{ x_{1} : 11, \  x_{2} : -4\right\}$

### Solve with `np.linalg`

We can solve linear systems without SymPy.

```python
import numpy as np

A = np.array([[1, 2],
              [3, 4]])

b = np.array([3, 17])
```

```python
x = np.linalg.inv(A) @ b

x
```

    array([11., -4.])

We can check that this is actually a solution:

```python
A @ x
```

    array([ 3., 17.])

Indeed, this is the `b` we started with. However, not all matrices are invertible, so this method won't always work. Sometimes we need `solve`:

```python
np.linalg.solve(A, b)
```

    array([11., -4.])

Least squares is another option:

```python
x, *_ = np.linalg.lstsq(A, b, rcond=-1)

x
```

    array([11., -4.])

----

© 2020 Agile Scientific — licensed CC-BY


```python
# Symbolic computation is the task of computation of mathematical objects symbolically,
# meaning that the objects are represented exactly, not approximately.

import math
math.sqrt(100)
```

    10.0

```python
import sympy
sympy.sqrt(100)
```

    10

```python
math.sqrt(3)
```

    1.7320508075688772

```python
sympy.sqrt(3)
```

    sqrt(3)

```python
math.sqrt(8)
```

    2.8284271247461903

```python
sympy.sqrt(8)
```

    2*sqrt(2)

```python
type(math.sqrt(8))
```

    float

```python
type(sympy.sqrt(8))
```

    sympy.core.mul.Mul

```python
# sqrt(3) is an irrational number
# 1.7320508075688772 an only approximation for sqrt(3).
# sqrt(3) will be slightly different if we change the data type, for example, double or long double.
#
# This is how symbolic computation is different from the "normal/standard"
# numerical computation most of us are familiar with
#
# Symbolic computation is also known as computer algebra
```

```python
# Advantages
# * work with equations in their natural mathematical form
# * working with mathematical objects and equations at the fundamental level, without approximations
# * compute derivatives/integrals of an expression
# * useful in prototyping of new (mathematical) models and gain insights
# Disadvantages
# * symbolic computations are significantly slow when compared to numerical computations
# * limited to simple problems, for example, solutions of PDEs.
```

```python
# Why SymPy?
# Why might you want to learn SymPy?

# There are many software for symbolic computations, with Mathematica and Maple being the players.
# Matlab/Octave also supports symbolic computations.
# Others: SageMath, Cadabra

# * Mathematica, Maple and Matlab are not open source
# * Mathematica's syntax and formatting is ugly (my personal opinion)
# * Matlab and Sage are heavy (requires a lot of disk space)
#
# * SymPy is open source; easy to modify and extend/customise
# * SymPy is lightweight.
# * SymPy uses Python; enhance your Python skills (for avid users)
# * Useful for interactive online teaching of courses on basic maths using Jupyter notebooks

```


#Table of Contents

# Lecture 1

This lecture introduces ordinary differential equations, and some techniques for solving first order equations. This notebook uses the computer algebra via Sympy (<http://sympy.org/>) to solve some ODE examples from the lecture notes.

## Solving ordinary differential equations

To use Sympy, we first need to import it and call `init_printing()` to get nicely typeset equations:

```
from sympy import *

# This initialises pretty printing
init_printing()
from IPython.display import display

# This command makes plots appear inside the browser window
%matplotlib inline
```

### Example: car breaking

During braking a car’s velocity is given by $v = v_{0} e^{−t/\tau}$. Calculate the distance travelled.

We first define the symbols in the equation ($t$, $\tau$ abd $v_{0}$), and the function ($x$, for the displacement):

```
t, tau, v0 = symbols("t tau v0")
x = Function("x")
```

Next, we define the differential equation, and print it to the screen:

```
eqn = Eq(Derivative(x(t), t) , v0*exp(-t/(tau)))
display(eqn)
```

The `dsolve` function solves the differential equation symbolically:

```
x = dsolve(eqn, x(t))
display(x)
```

where $C_{1}$ is a constant. As expected for a first-order equation, there is one constant.

SymPy is not yet very good at eliminating constants from initial conditions, so we will do this manually assuming that $x = 0$ and $t = 0$:

```
v0 = symbols('v0')
x = x.subs('C1', v0*tau)
display(x)
```

Specifying values for $v_{0}$ and $\tau$, when can plot the velocity as a function of time:

```
# Specify values for v0 and tau
x = x.subs(v0, 100)
x = x.subs(tau, 2)

# Plot velocity vs time
plot(x.args[1], (t, 0.0, 10.0), xlabel="time", ylabel="velocity")
```

#### Classification

We can ask SymPy to classify our ODE, e.g. show that it is first order):

```
classify_ode(eqn)
```

    ('separable',
     '1st_exact',
     '1st_linear',
     'Bernoulli',
     '1st_power_series',
     'lie_group',
     'nth_linear_constant_coeff_undetermined_coefficients',
     'nth_linear_constant_coeff_variation_of_parameters',
     'separable_Integral',
     '1st_exact_Integral',
     '1st_linear_Integral',
     'Bernoulli_Integral',
     'nth_linear_constant_coeff_variation_of_parameters_Integral')

### Parachutist

Find the variation of speed with time of a parachutist subject to a drag force of $kv^{2}$.

The equations to solve is

$$
\frac{m}{k} \frac{dv}{dt} = \alpha^{2} - v^{2}
$$

where $m$ is mass, $k$ is a prescribed constant, $v$ is the velocity, $t$ is time and $\alpha^{2} = mg/k$ ($g$ is acceleration due to gravity).

We specify the symbols, unknown function $v$ and the differential equation

```
t, m, k, alpha = symbols("t m k alpha")
v = Function("v")
eqn = Eq((m/k)*Derivative(v(t), t), alpha*alpha - v(t)*v(t))
display(eqn)
```

First, let's classify the ODE:

```
classify_ode(eqn)
```

    ('separable', '1st_power_series', 'lie_group', 'separable_Integral')

We see that it is not linear, but it is separable. Using `dsolve` again,

```
v = dsolve(eqn, v(t))
display(v)
```

SymPy can verify that an expression is a solution to an ODE:

```
print("Is v a solution to the ODE: {}".format(checkodesol(eqn, v)))
```

    Is v a solution to the ODE: (True, 0)


# Tutorial for Homework 5 - Interference of Light

```python
# setup by importing some good modules
import sympy as sp
# All calls to sympy require sp. at the beginning
# One could use "from sympy import *", but then it's difficult to know what is sympy
# and what is another module when importing multiple modules.
# print things all pretty
from sympy.abc import *
sp.init_printing(use_latex='mathjax')

```

The sympy module (http://docs.sympy.org) allows one to plot functions without having to digitize the independent variable. Let's take a look at how to use it to plot functions. In the first example, we'll plot cosine, sine, and both together. These functions are built into sympy.  Simply give it the range of x-axis values you want plotted. NOTE: The text in between \$ is LaTeX typesetting.

```python
sp.plot(sp.cos(x), (x,-sp.pi, sp.pi), legend=True, line_color='red', xlabel='$\\theta$', ylabel='Amplitude')
sp.plot(sp.sin(x), (x,-sp.pi, sp.pi), legend=True, line_color='blue', xlabel='$\\theta$', ylabel='Amplitude')
sp.plot(sp.sin(x), sp.cos(x), (x,-sp.pi, sp.pi), legend=True, line_color='red', xlabel='$\\theta$', ylabel='Amplitude')
```

As you can see, once we plot two functions, we lose the ability to decorate them individually. Let's see how we can decorate them individually. To do this we'll need to make the graph an object and show the graph after we decorate it.

```python
#Define the graph as an object p
p = sp.plot(sp.sin(x), sp.cos(x), (x,-sp.pi, sp.pi), show=False, legend=True, xlabel='$\\theta$', ylabel='Amplitude')
p[0].line_color='red' #color the first data element as red
p[1].line_color='blue'#color the second data element as blue
p.show()
```

### Your turn
Using the cell below, try plotting $\cos^2(x)$ and $\sin^2(x)$ on the same graph. HINT: Squaring in Python is done with ```**2```. Make the cosine function purple and the sine function orange.

```python

```

## Diffraction

Now, let's look at a function that is relevant to our homework. The intensity pattern of light diffracting through a single slit is

$$I = I_o \left[\frac{\sin \left(\beta\right)}{\beta}\right]^2$$

To simplify the plotting, we'll define $\beta$ to be our $x$ variable. We will need to create this function since it does not exist in Sympy. We need to define the variables we plan to use as "symbols" for the independent and dependent variables. Once we define them, SymPy will also format them to look like pretty mathematics.

```python
#Define x and I as real-valued symbols
x, I= symbols("x, I", real = True)
# Define I
I = (sp.sin(x)/x)**2
I # This causes it to output formatted nicely. The print command formats it as computer code
```

$$\frac{1}{x^{2}} \sin^{2}{\left (x \right )}$$

```python
#Plot the intensity of diffraction from -10 to 10
sp.plot(I, (x,-10, 10))
```

### Your turn
In the cell below, plot the same diffraction data as above but label the axes with "Intensity" and "$\theta$".

```python

```

From the plot of the diffraction intensity function, we know the intensity equation must have a maximum in the center, i.e., when $\beta\rightarrow0$. L'H$\hat{\rm{o}}$pital's rule says

$$\lim_{\beta\to 0} \frac{\sin\beta}{\beta} = 1$$

We could also take a derivative if the intensity function and find the maximum. More on that later. We get the central maximum at $\beta=0$ since the limit is 1, the intensity is $I_o$ at $\beta = 0$. We read in the textbook that there is a first order minimum when

$$D \sin\left(\theta\right) = m\lambda$$

and $m=1$. Minima will occur when $\beta = m\pi$ in the intensity equation above because $\sin(m\pi)=0$. Rearrarange the destructive interference condition equation above to solve for $m$; multiply by $\pi$ and substitute this into the first equation for $\beta$.

You should get

$$I = I_o \left[\frac{\sin \left(\frac{\pi D\sin\left(\theta\right)}{\lambda}\right)}{\frac{\pi D\sin\left(\theta\right)}{\lambda}}\right]^2$$

This means, we can determine the size of the single slit if we know the wavelength of light we are using. You did this in lab. Let's make a plot of this function by first defining $D=\lambda$ and $\lambda=500$ nm. Recall that $\sin\left(\theta\right)\approx\frac{y}{L}$, where $y$ is the distance away from the central point on the screen, and $L$ is the distance the screen is from the diffraction slit. You should also define $L = 1$ meter.

### Your turn
Below, edit the code to set the known variables and to create the function of diffraction intensity. Then, in the cell below the next one, plot the intensity function. Be sure to decorate your graph with axis labels and colors.

```python
#Create your constants
wl=0 #variable for wavelength
D=0 #variable for slit width
L=0 #variable for screen distance from slit
#Define x and I as real-valued symbols
x, I= symbols("x, I", real = True)
# Define I for real
I = x
I # This causes it to output formatted nicely. The print command formats it as computer code
```

$$x$$

```python
#Plot your intensity function
```

Try changing the wavelength. What happens to the diffraction pattern when $D\gt\lambda$? What happens to the diffraction pattern when $D\lt\lambda$?

## Interference

Next, let's look at interference. We expect interference to have constructive maxima following the relationship

$$d \sin\left(\theta\right) = m\lambda$$

The angle $\theta$ is the same as for diffraction. It is the angle away from a straight traveling beam. In this case $d$ is the spacing between two slits. From this pattern of constructive interference, we get a bright spot when $\theta=0$ and again when $\sin\left(\theta\right) = \lambda$. This means we'll get a bright spot on the screen when

$$\sin\left(\theta\right) \approx \frac{y}{L} = \frac{m\lambda}{d}$$

$$y = \frac{mL\lambda}{d}$$

Thus, we expect repeated constructive interference bright spots every integer multiple of

$$\frac{L\lambda}{d}$$

Since the interference has a bright spot when $\theta = 0$, we can write the intensity function as a cosine. There will be high intensity at the same locations as the conditional equation $d \sin\left(\theta\right) = m\lambda$. We know the cosine function should be maximum (have an argument of $m\pi$) at these locations. We also know that intensity is the square of the electric field of the light. This means the cosine function will be squared. This prevents a negative intensity, which makes no sense.

$$I = I_o \cos^2\left(\alpha\right)$$

Similar to the diffraction arguments, the angle $\alpha = m\pi$ is constructive interference. This gives

$$\alpha = \frac{\pi d \sin\left(\theta\right)}{\lambda}$$

### Your turn
In the cells below, plot the interference intensity function. You will need to define a value for $d$. Let's use $d = 1 \rm{\mu m}$. Be sure to give the function a new name, ```Iint``` for example. Once you get it to plot, try changing $d$ to values between $\lambda/2$ and $2\lambda$. What happens to the constructive interference fringes?

```python
#Define x and I as real-valued symbols
d = 1e-5 #double slit spacing
#wavelength is defined above already. You can redefine it anywhere with wl
# Define your symbols x and Iint
# Define interference intensity
Iint = sp.cos(sp.pi*d*x/500e-9/3)**2
Iint # This causes it to output formatted nicely. The print command formats it as computer code
```

$$\cos^{2}{\left (6.66666666666667 \pi x \right )}$$

```python
#Plot the intensity of an interference pattern.
sp.plot(Iint, (x, -0.5,0.5))
```

## Putting Diffraction and Interference Together
The combined effect of diffraction and interference is the product of the two functions. This gives

$$I = I_o \left[\frac{\sin \left(\frac{\pi D\sin\left(\theta\right)}{\lambda}\right)}{\frac{\pi D\sin\left(\theta\right)}{\lambda}}\right]^2 \cos^2\left(\frac{\pi d \sin\left(\theta\right)}{\lambda}\right)$$

### Your turn
Create a new function (give it a new name such as ```Itot```) for the combined diffraction and interference. Then, plot this function. Also plot the intensity function from diffraction. It should still be defined by ```I``` from above.

Test what happens when $D$ increases or decreases. Test what happens when $d$ increases or decreases. Test what happens when $\lambda$ increases or decreases.

```python
#Plot the combined diffraction and interference function along with the diffraction intensity function.
```

## Derivatives, Maxima, Minima
Another powerful tool in SymPy is its ability to calculate derivatives. It is really simple too! The command ```diff``` will calculate a derivative of any existing function. Below, I differentiate cosine and store it in the symbol dcos.

```python
dcos = sp.diff(sp.cos(x), x)
dcos
```

$$- \sin{\left (x \right )}$$

### Your turn
Try differentiating sine below.

```python

```

When it is a user defined function, the ```(x)``` is not necessary. Here is an example where I define a polynomial function related to the acceleration due to gravity and differentiate it to obtain the velocity function.

```python
t, poly = symbols("t, poly", real = True)
poly = 0.5*(-9.8)*t**2+20*t+1
dpoly = sp.diff(poly, t)
poly, dpoly
```

$$\left ( - 4.9 t^{2} + 20 t + 1, \quad - 9.8 t + 20\right )$$

I can solve for extrema too. To do this I use the SymPy ```solveset``` function. According to SymPy's documentation... The first argument for ```solveset()``` is an equation (equaled to zero by solveset) and the second argument is the symbol that we want to solve the equation for (usually your independent variable). Thus, ```solveset``` finds the roots of a function, i.e., where it is equal to zero.

```python
extrema = sp.solveset(dpoly, t)
extrema
```

$$\left\{2.04081632653061\right\}$$

It's telling me there is a maximum or minimum at $t = 2.04$ seconds. We know this polynomial has no upper and lower limits. As $t\to\pm\infty$, the function goes to $\pm\infty$ depending on the concavity (2nd derivative). Let's plot ```poly``` to see if this extremum is true. First, I will find the zeros (where the function evaluates to zero) and plot over that range. This would correspond to an object going from the ground, upward to maximum height, and then back to the ground. Keep in mind my function implies I threw the object upward from a height of 1 meter.

```python
polyzeros = sp.solveset(poly, t)
polyzeros
```

$$\left\{-0.0494020618888763, 4.1310347149501\right\}$$

```python
sp.plot(poly, (t, -0.0494, 4.131))
```

It does appear that there is a maximum at $t=2.04$ seconds.

### Your turn
Find the derivative of your total intensity function. I have written the function below, but you should already have it defined above.

```python
#Define x and I as real-valued symbols
x, Itot = symbols("x, Itot", real=True)
L = 1
wl = 5e-7
D = 2e-7
dd = 1e-6 #double slit spacing
# Define your symbols x and Iint
# Define interference intensity
Itot = (sp.sin(sp.pi*D*x/(wl*L))/(sp.pi*D*x/(wl*L))*sp.cos(sp.pi*dd*x/(wl*L)))**2
Itot # This causes it to output formatted nicely. The print command formats it as computer code
```

$$\frac{6.25}{\pi^{2} x^{2}} \sin^{2}{\left (0.4 \pi x \right )} \cos^{2}{\left (2.0 \pi x \right )}$$

```python
sp.plot(Itot, (x, -5, 5))
```

```python
Itotdiff = sp.diff(x,x)
Itotdiff
```

$$1$$

Ouch! That is quite a derivative!!! We can try to find the roots using ```solveset```. Give it a shot. To limit the range that Python tries to search, use the argument ```sp.Interval```. NOTE: You'll need to change ```y``` to the appropriate function.

```python
Itotextreme = sp.solveset(y, x, sp.Interval(-0.1,0.1))
Itotextreme
```

$$\emptyset$$

The result is pretty difficult to interpret when the function is so complicated. Let's plot the derivative to see if we can find the global maximum and first two minima. To do this choose a range of x values to plot that will zoom in on the region of the total intensity where you expect these to be.

```python
sp.plot(x, (x,-1,1))
```

## Integrals
You don't have integrals on your homework, but we might as well look at how to evaluate them using SymPy while we are learning about SymPy.

To take the integral of a function is as simple as taking a derivative. The exception is that integrals can be definite or indefinite. Let's look at an example of both using SymPy's ```integrate``` function.

First, we'll take the indefinite integral of cosine.

```python
sp.integrate(sp.cos(x),x)
```

$$\sin{\left (x \right )}$$

Excellent! We could have stored that in a variable ```intcos = sp.integrate(sp.cos(x),x)``` if we wanted to use the integral later. Now, let's integrate over a range (definite integral). We'll continue with the cosine function. Before we do, let's think about what we expect to obtain from the integral. Say we integrate cosine from $-\pi$ to $\pi$. This is one period of cosine. What do you expect to get from the integral?

```python
sp.integrate(sp.cos(x),(x,-1, 1))
```

$$2 \sin{\left (1 \right )}$$

### Your turn
Create a function that describes a projectile's velocity as a function of time. Give the object an initial upward velocity of 10 m/s and downward acceleration due to gravity. Integrate this function to obtain the position vs. time function. First, do this as an indefinite integral. Then, integrate over the range of times that correspond to the object leaving the ground and reaching its maximum height. Do each of these in their own cell. You may want to plot your velocity and position functions to visually inspect that they are what you expect. After you obtain your results, create a markdown cell to explain the results at the bottom. Use LaTeX in your markdown to typeset the equations for velocity and position functions.

```python

```