# A Complete Guide to Matrix Notation and Linear Regression

Let's really understand matrix notation in context of linear regression, from the ground up.

Linear Regression finds the best line, or *hyperplane* $\hat{y}$ in higher dimension, or generally a function $f$:

$$ \hat{y} = f(x) = wx $$

that fits the whole data. This is just a dot product between vector $w$ and a data point $x$ in $d$ dimension:

$$ \hat{y} = w_0 + w_1x_1 + w_2x_2 + ... + w_dx_d $$

Notice that we use $w_0$ as an intercept term, and thus we need to add a dummy dimension with value of "1" ($x_0$) for all data points $x$. Thus, $x$ here is on $d+1$ dimension. Think of it as the y-intercept term $c$ in 2-dimension ($y = mx + c$).

Another way to look at this is that $f(x)$ transforms a data point $x$ on $d+1$ dimension into a predicted scalar value $\hat{y}$ that is close to target $y$:

$$
\begin{bmatrix}
x_0 \\
x_1 \\
\vdots \\
x_d
\end{bmatrix}
\xrightarrow{f}
\hat{y}
\approx y
$$

## The Sum of Squared Error Loss

The best way to solve this is to find $w$ that minimizes the **sum of squared errors (SSE)**, or the "error" between all of predicted value $\hat{y}^i$ and the target $y^i$ of $i^{th}$ data point for $i = 1$ to $n$, writing this as a loss function $L(w)$:

$$
L(w) = \sum_{i=1}^{n} \left( y^i - \hat{y}^i \right)^2 = \sum_{i=1}^{n} \left( y^i - wx^i \right)^2
$$

From now on we refer to a data point (d+1 vector) as $x^i$ and its corresponding target (scalar) as $y^i$. I know it's confusing for the first time, but you'll get used to using superscript for indexing data points.

Surprisingly, the SSE loss is not from someone's intuition, but it's from the assumption that there is **Gaussian noise in our observation** of the underlying linear relationship. We will show how this leads to SSE loss later, but first let's visualize what we're trying to do.

<center>

\

</center>

1. There is a true line, the true linear relationship that we want to discover (blue line).
2. The data points are then observed through noise **deviating from that line, with Gaussian distribution**.
3. Suppose we predict a random line (red), not the best one yet.
4. We calculate the distance or difference between the predicted points (along the line) and the actual data points. This is then **squared and sum up to get sum of squared error**.

> Linear regression is the method to get the line that fits the given data with the minimum sum of squared error.

### How to Find the Optimal Solution
An optimal solution ($w$) for this equation can be found either using *closed-form solution* or via iterative methods like gradient descent.

**A closed-form solution** means we figure out the formula for $w = ...$. Implementing that formula in a program directly solves the problem. The thing is you have to come up with the correct one yourself, by hand.

Do you remember how to find a minimum (or maximum) value for a function? We take the derivative of the function above with respect to $w$, set it to zero, and solve for the $w$ in terms of other parameters. This is like taking a single jump to the optimal value. We do all the hard work for computers.

Luckily we can do this for linear regression, but not all loss functions be solved this way, actually, only a few. In those cases, we use **iterative methods like gradient descent** to search for the solution. In contrast to closed-form solution, we do not jump directly to the optimal answer, instead, we take many steps that lead us near to where the optimal answer lives.

Next let's derive the closed-form solution for linear regression. In order to do that efficiently, we need some matrix notations.

## Going into Matrix Notation
Writing things down in matrix notation makes things much faster in NumPy. **But it's not easy to read matrix notation, especially if you study machine learning on your own.** There're things like dot product, matrix multiplication, transpose and stuff that you need to keep track of in your head. If you're starting out, then please write them on papers, drawing figures as needed to make you understand. It really pays off.

On top of that, these few **key standards** will make our lives with linear algebra easier:

### 1. Always a *column* vector

When you see standalone vectors in a matrix notation formula, assumes it's a column vector. E.g.,

$$x =
\begin{bmatrix}
1 \\
2 \\
3 \\
\end{bmatrix}
$$

and so its transpose is a row vector,

$$x^T =
\begin{bmatrix}
1 & 2 & 3
\end{bmatrix}
$$

Likewise, you should try to make the final result of matrix operation to be a column vector.

Note that the NumPy vector created by `np.zeros`, `np.arange`, etc., is not really a column vector. It has only one dimension `(N,)`. So, you cannot transpose it directly (`x.T` still gives you `x`.) To convert it to a column vector, we use `x.reshape(N,1)` or `x[:, None]`.

### 2. Feature matrix $X$ is rows of data points
Our data points $x^i$ are on $d+1$ dimension, and there is $n$ of them, we store them all in a 2-d matrix $X$:

$$
X = \begin{align}
\underset{n\times d}
{\begin{bmatrix}
\longleftarrow & x^1 & \longrightarrow \\
\longleftarrow & x^2 & \longrightarrow \\
& \vdots & \\
\longleftarrow & x^n & \longrightarrow \\
\end{bmatrix}}
\end{align}
$$

Each row in $X$ is a row vector for each data point. Also note that we use uppercase letter for matrix.

#### 3. Again, $w$ is a column vector
Like the first point, our $w$ will be $d+1$ dimension column vector with w_0 as an intercept term:
$$
w =
\begin{bmatrix}
w_0 \\
w_1 \\
\vdots \\
w_d
\end{bmatrix}
$$

### 4. Dot products of rows in matrix $X$ with vector $w$ is $Xw$
Sometimes we want the dot product of each row in matrix with a vector:
$$
Xw = \underset{n\times (d+1)}
{
    \begin{bmatrix}
    \longleftarrow & x^1 & \longrightarrow \\
    \longleftarrow & x^2 & \longrightarrow \\
    & \vdots & \\
    \longleftarrow & x^n & \longrightarrow \\
    \end{bmatrix}
}
\underset{(d+1) \times 1}
{
    \begin{bmatrix}
    \uparrow \\
    w \\
    \downarrow
    \end{bmatrix}
}
=
\begin{bmatrix}
x^1w \\
x^2w \\
\vdots \\
x^nw
\end{bmatrix}
$$

given that $X$ contains rows of vectors we want to dot product with.

Interestingly, this gives us a column vector of our predictions $\hat{y}$:

$$
\begin{bmatrix}
x^1w \\
x^2w \\
\vdots \\
x^nw
\end{bmatrix} =
\begin{bmatrix}
\hat{y}^1 \\
\hat{y}^2 \\
\vdots \\
\hat{y}^n \\
\end{bmatrix} = \hat{y}
$$

It's also good to remind yourself that it sums along dimension of $x^i$ and $w$:
$$
Xw =
\begin{bmatrix}
\sum_{j=0}^{d} x_j^1w_j \\
\sum_{j=0}^{d} x_j^2w_j \\
\vdots \\
\sum_{j=0}^{d} x_j^nw_j \\
\end{bmatrix}
$$

### 5. Sum of Squared  is $x^Tx$

This is a useful pattern to memorize. Sometimes we want the sum of squared of each element in arbitrary d-dimension vector $x$:

$$
\sum_{j=1}^{d} x_i^2
$$

which is simply $x^Tx$:

$$
x^Tx =
\begin{bmatrix}
x_1 & ... & x_d
\end{bmatrix}
\begin{bmatrix}
x_1 \\
\vdots \\
x_d
\end{bmatrix}
= \sum_{j=1}^{d} x_i^2
$$

Notice that the result of $x^Tx$ is scalar, e.g., a number.

In fancy term, ${\left\lVert x \right\rVert} = \sqrt{\sum_{j=1}^{d} x_i^2}$ is L2-norm (or Euclidean norm) of $x$. So we can write sum of squared as ${\left\lVert x \right\rVert}^2 = \sum_{j=1}^{d} x_i^2$. For now, let's not care what norm actually means.

### Writing SSE Loss in Matrix Notation

Now you're ready, let's write the above SSE loss function in matrix notation. If you look at $L(w)$ closely, it's a sum of squared of vector $y - \hat{y}$. This means we can kick-off by applying our fifth trick:

$$
L(w) = {\left\lVert y - \hat{y} \right\rVert}^2
$$

Next we just have to find $y - \hat{y}$. First we encode target $y$ in a long column vector of shape `[n, 1]`:

$$
y =
\begin{bmatrix}
y^1 \\
y^2 \\
\vdots \\
y^n
\end{bmatrix}
$$

(Remember that we use superscript for indexing the $i^{th}$ data point.)

Next we encode each of our predicted values ($\hat{y}^i$) in a column vector $\hat{y}$. Since $\hat{y}^i$ is a dot product between $w$ and each of $x^i$, we can apply [4](#4.-Dot-products-of-rows-in-matrix-$X$-with-vector-$w$-is-$Xw$):

$$
\begin{align}
y - \hat{y} &=
\begin{bmatrix}
y^1 \\
y^2 \\
\vdots \\
y^n
\end{bmatrix}
- \begin{bmatrix}
\hat{y}^1 \\
\hat{y}^2 \\
\vdots \\
\hat{y}^n
\end{bmatrix} && \text{(Error between target and predicted)} \\ &=
\begin{bmatrix}
y^1 \\
y^2 \\
\vdots \\
y^n
\end{bmatrix}
- \begin{bmatrix}
x^1w \\
x^2w \\
\vdots \\
x^nw
\end{bmatrix}
&& \text{(Predicted is a dot product of $w$ and each of data point $x^i$)} \\ &=
\underset{n\times 1}
{
    \begin{bmatrix}
    y^1 \\
    y^2 \\
    \vdots \\
    y^n
    \end{bmatrix}
}
- \underset{n\times (d+1)}
{
    \begin{bmatrix}
    \longleftarrow & x^1 & \longrightarrow \\
    \longleftarrow & x^2 & \longrightarrow \\
    & \vdots & \\
    \longleftarrow & x^n & \longrightarrow \\
    \end{bmatrix}
}
\underset{(d+1)\times 1}
{
    \begin{bmatrix}
    \uparrow \\
    w \\
    \downarrow
    \end{bmatrix}
} && \text{(Separate them out)} \\ &=
y - Xw && \text{(Encode in matrix/vector form)}
\end{align}
$$

Putting all together we get our loss function for linear regression:

$$
L(w) = {\left\lVert y - Xw \right\rVert}^2
$$

In NumPy code, we can compute $L(w) = (y - Xw)^T(y - Xw)$.

There's no intuitive way to come up with this nice formula the first time you saw it. You have to work it out and put things together yourself. Then you'll start to memorize the pattern and it'll become easier.

## Deriving a Closed-form Solution

To do that, we'll take derivative of $L(w)$ with respect to $w$, set to zero and solve for $w$.

Writing matrix notation is already hard, taking derivative of it is even harder. I recommend writing out partial derivatives to see what happens. For $L(w) = L_w$, we have to take derivative with respect to each dimension of $w$:

$$
\nabla L_w =
\begin{bmatrix}
\frac{\partial L}{\partial w_0} \\
\frac{\partial L}{\partial w_1} \\
\vdots \\
\frac{\partial L}{\partial w_d} \\
\end{bmatrix}
=
\begin{bmatrix}
\frac{\partial \sum_{i=1}^{n} \left( y^i - wx^i \right)^2}{\partial w_0} \\
\frac{\partial \sum_{i=1}^{n} \left( y^i - wx^i \right)^2}{\partial w_1} \\
\vdots \\
\frac{\partial \sum_{i=1}^{n} \left( y^i - wx^i \right)^2}{\partial w_d}
\end{bmatrix}
=
\begin{bmatrix}
\frac{\partial \sum_{i=1}^{n} \left( y^i - wx^i \right)^2}{\partial w_0} \\
\frac{\partial \sum_{i=1}^{n} \left( y^i - wx^i \right)^2}{\partial w_1} \\
\vdots \\
\frac{\partial \sum_{i=1}^{n} \left( y^i - wx^i \right)^2}{\partial w_d}
\end{bmatrix}
=
\underset{(d+1) \times 1}
{
\begin{bmatrix}
-2\sum_{i=1}^{n} x^i_0 \left( y^i - wx^i \right) \\
-2\sum_{i=1}^{n} x^i_1 \left( y^i - wx^i \right) \\
\vdots \\
-2\sum_{i=1}^{n} x^i_d \left( y^i - wx^i \right) \\
\end{bmatrix}
}
$$

Looks like we might be able to apply our fourth point ($Xw$, but in this case $w$ is $(y - Xw)$. But unlike our fourth point, we now sum along data points ($n$) instead of dimensions ($d$). For this, we want each row of $X$ to be one given dimension along all data points instead of one data point with all dimensions, and thus we use $X^T$ instead of $X$. Finally, here's the full derivative in matrix notation:

$$
\nabla L_w = -2X^T(y-Xw)
$$

Setting to zero and solve:

$$
\begin{align}
0 &= -2X^T(y-Xw) \\
&= X^T(y-Xw)     \\
&= X^Ty - X^TXw
\end{align}
$$

Move $X^TX$ to other side and we get a closed-form solution:

$$
\begin{align}
X^TXw &= X^Ty    \\
w &= (X^TX)^{-1}X^Ty
\end{align}
$$

In NumPy, this is:
```python
w = np.linalg.inv(X.T @ X) @ X @ y
```

## A NumPy Example

```python
import numpy as np
import matplotlib.pyplot as plt
np.random.seed(1)
```

We will create a fake dataset from the underlying equation $y = 2x + 7$:

```python
def true_target(x):
  return 2*x + 7
```

In practical settings, there is no way we know this exact equation. We only get **observed** targets, and there's some **noise** on it. The reason is that it's impossible to measure any data out there in the world perfectly:

```python
def observed_target(x):
  """Underlying data with Gaussian noise added"""
  normal_noise = np.random.normal() * 3
  return true_target(x) + normal_noise
```

### Creating data points

Next, make 50 data points, observations and targets:

```python
N = 50

# Features, X is [1,50]
X = np.random.rand(N).reshape(N, 1) * 10

# Observed targets
y = np.array([observed_target(x) for x in X]).reshape(N, 1)
```

Adding dummy dimension term to each $x^i$:

```python
# Append 1 for intercept term later
X = np.hstack([np.ones((N, 1)), X])
```

Note that it **doesn't matter** here whether we add it to the front or back, it will simply reflect correspondingly in our solution $w$.

### Visualize our data points with respect to the true line

```python
# For plotting
features = X[:,1:] # exclude the intercept for plotting
target = y
true_targets = true_target(X[:,1:])

plt.scatter(features, target, s=10, label='Observed data points')
plt.plot(features, true_targets, c='blue', label='True target line y = 2x + 7', alpha=0.3)

plt.xlabel('Feature')
plt.ylabel('Target')
plt.legend(loc='best')
plt.title('True and observed data points')
plt.show()
```

### Compute a closed-form solution

Our goal is to get the line that is closest to that true target (blue) line as possible, without the knowledge of its existence. For this we use linear regression to fit observed data points by following the formula from the previous section:

```python
w = np.linalg.inv(X.T @ X) @ X.T @ y
```

To predict, we compute $\hat{y} = xw$ for each data point $x^i$. Here we predict the training set (`X`) itself:

```python
predicted = X @ w # y_hat
```

To predict a set of new points, you just make it the same format as `X`, e.g., rows of data points.

### Visualize best fit line vs. true target line

```python
plt.scatter(features, target, s=10, label='Data points')
plt.plot(features, true_targets, c='blue', label='True target line', alpha=0.3)
plt.plot(features, predicted, c='red', label='Best fit line')

plt.xlabel('Feature')
plt.ylabel('Target')
plt.legend(loc='best')
plt.show()
```

That's pretty close.

### Understanding the result

And our $w$ is:

```python
print(w)
```

    [[7.00426874]
     [2.08162643]]

Since we append ones in front of each data point $x$, `w[0]` will be the intercept term and `w[1]` will be the slope.
So our predicted line will be in the format of `y = w[1]*x + w[0]`.
Recall the *true* equation $y = 2x + 7$, you can see that we almost got the true slope (2):

```python
print("Our slope is", w[1][0])
```

    Our slope is 2.081626431082087

The intercept seems a little off, but that's okay because our data is in a big range ($x \in [0, 50], y \in [7, 107]$). If we normalize the data into $[0, 1]$ range, expect it to be much closer.

Below is our sum of squared error for the best fit line. Note that the number doesn't mean anything much, apart from that this is the least possible loss we would get from any lines that try to fit the data:

```python
diff = (y - X @ w)
loss = diff.T @ diff
print(loss)
```

    [[368.25248566]]

If you don't want intermediate variable, you can use `np.linalg.norm`, but to get the sum of squared loss, you have to square that after:

```python
loss = np.linalg.norm(y - X @ w, ord=2) ** 2
print(loss)
```

    368.252485661978

### Visualize the loss surface

Let's confirm that our solution is really the one with lowest loss by seeing the loss surface.

Our loss function $L(w)$ depends on two dimensions of $w$, e.g., `w[0]` and `w[1]`. If we plot $L(w)$ over possible values of `w[0]` and `w[1]`, the minimum of $L(w)$ should be near `w = [5.17,2.066]`, which is our solution.

To plot that out, first we have to create all possible values of w[0] and w[1] in a grid.

```python
from mpl_toolkits.mplot3d import Axes3D

# Ranges of w0 and w1 to see, centering at the true line
spanning_radius = 10
w0range = np.arange(7-spanning_radius, 7+spanning_radius, 0.05)
w1range = np.arange(2-spanning_radius, 2+spanning_radius, 0.05)
w0grid, w1grid = np.meshgrid(w0range, w1range)

range_len = len(w0range)
print("Number of values in each axis:", range_len)
```

    Number of values in each axis: 400

This means we'll look into a total of 400*400 = 160,000 values of `w`. We have to calculate loss for each pair of `w0, w1`:

```python
# Make [w0, w1] in (2, 14400) shape
all_w0w1_values = np.hstack([w0grid.flatten()[:,None], w1grid.flatten()[:,None]]).T

# Compute all losses, reshape back to grid format
all_losses = (np.linalg.norm(y - (X @ all_w0w1_values), axis=0, ord=2) ** 2).reshape((range_len, range_len))
```

Then, we can plot the loss surface (with minimum at the red point):

```python
fig = plt.figure(figsize=(10,6))
ax = fig.gca(projection='3d')

ax.plot_surface(w0grid, w1grid, all_losses, alpha=0.5, cmap='RdBu')
ax.contour(w0grid, w1grid, all_losses, offset=0, alpha=1, cmap='RdBu')
ax.scatter(w[0], w[1], loss, lw=3, c='red', s=100, label="Minimum point (5.9,2.2)")

ax.legend(loc='best')
ax.set_xlabel('w[0]')
ax.set_ylabel('w[1]')
ax.set_zlabel('L(w)')
ax.set_xticks(np.arange(7-spanning_radius, 7+spanning_radius, 2))
ax.set_yticks(np.arange(2-spanning_radius, 2+spanning_radius, 2))
ax.set_zticks([loss])
plt.show()
```

You can notice the bowl **centers** at the solution.

## Using sklearn

Using `sklearn` for linear regression is very simple (if you already understand all the concepts above).

```python
from sklearn.linear_model import LinearRegression
```

First we create the classifier `clf`. If `fit_intercept` is `True` (default), then it adds the dummy '1' to the `X`. But we already did that manually, so set it to `False` here.

```python
clf = LinearRegression(fit_intercept=False)
```

Then fit the data:

```python
clf.fit(X,y)
```

    LinearRegression(copy_X=True, fit_intercept=False, n_jobs=None,
             normalize=False)

Check the $w$ learned, it's the same as ours:

```python
print(clf.coef_)
```

    [[7.16920731 2.05342609]]

Previously we use `X @ w` to predict data. For `sklearn` we can use `clf.predict`:

```python
predicted = clf.predict(X)
```

And the result is the same:

```python
plt.figure()
plt.scatter(X[:,1:], y, s=10, label='Data points')
plt.plot(X[:,1:], true_targets, c='blue', label='True line', alpha=0.3)
plt.plot(X[:,1:], predicted, c='red', label='Best fit line')
plt.legend(loc='best')
plt.show()
```


# Visualizing stationary paths of a  functional

Last revised: 02-Feb-2019 by Dick Furnstahl [furnstahl.1@osu.edu]

Consider the functional

$\begin{align}
  S = \int_{x_1}^{x_2} f[y(x), y'(x), x] \, dx
\end{align}$

with $y_1 = y(x_1)$ and $y_2 = y(x_2)$ fixed.  We denote by $y^*(x)$ the path that minimizes $S$ (or, more generally, makes it stationary).  Then we consider the class of candidate paths $y(x)$ given by

$\begin{align}
  y(x) = y^*(x) + \alpha \eta(x)
\end{align}$

where $\eta(x)$ is some function that vanishes at the endpoints: $\eta(x_1) = \eta(x_2) = 0$.  We can derive the Euler-Lagrange equations by minimizing $S(\alpha)$ with respect to $\alpha$.

Here we visualize this problem by considering a particular $S$, choosing among some possible $\eta(x)$ definitions, and seeing how $S$ is minimized with respect to $\alpha$.  We will also allow for an incorrect determination of $y^*(x)$, in which case we expect that the minimum alpha will give us a reasonable reproduction of the true $y^*(x)$.  The variation of $\alpha$ and the choice of functions will be made using widgets from `ipywidgets`.

## Looking at a plot of the functional evaluation versus $\alpha$

\[We'll use `%matplotlib notebook` so that we can modify figures without redrawing them.\]

```python
%matplotlib notebook
```

```python
import numpy as np
import matplotlib.pyplot as plt

import ipywidgets as widgets
from IPython.display import display

```

### Functional from Taylor problem 6.9

This problem states: "Find the equation of the path from the origin $O$ to the point $P(1,1)$ in the $xy$ plane that makes the integral $\int_O^P (y'^2 + y y' + y^2)$ stationary.  The answer from solving the Euler-Lagrange equation is $y^*(x) = \sinh(x)/\sinh(1)$.

```python
def y_star(x):
    """Path that minimizes the functional in Taylor problem 6.9."""
    return np.arcsinh(x)
```

```python
delta_x = 0.001
x_pts = np.arange(1. , 5., delta_x)

fig = plt.figure(figsize=(6,3),
                 num='Visualizing stationary paths of a functional')
ax1 = fig.add_subplot(1,2,1)
ax2 = fig.add_subplot(1,2,2)

def setup_figure():
    ax1.set_title('Show paths')
    ax1.plot(x_pts, y_star(x_pts), color='black', lw=2)
    ax1.set_xlabel('x')
    ax1.set_ylabel('y(x)')

    ax2.set_title('Evaluate functional')
    ax2.set_xlabel(r'$\alpha$')
    ax2.set_ylabel('functional')
    ax2.set_xlim(-0.4, 0.4)
    #ax2.set_ylim(1.5, 3.)
    #ax2.axvline(0., color='black', alpha=0.3)
    ax2.axhline(evaluate_functional(x_pts, 1 + y_star(x_pts)),
                                    color='black', alpha=0.3)

    fig.tight_layout()

def evaluate_functional(x_pts, y_pts):
    """Given arrays of x and y points, evaluate the functional from 6.9."""
    # The numpy gradient function takes the derivative of an array y_pts
    #  that is a function of points x in x_pts.
    y_deriv_pts = np.gradient(y_pts, x_pts)
    f = y_deriv_pts**2 + y_pts * y_deriv_pts + y_pts**2

    # Use the numpy trapezoid rule (trapz) to do the integral over f.
    return np.trapz(f, x_pts)

def make_path(alpha, ax1_passed, ax2_passed,
              base_function='exact', eta_function='arcsine'):
    """Given a base function, which may be the exact y^*(x) or a guess that
       is not correct, generate and plot the path corresponding to adding
       alpha*eta(x) to the base function, with eta(x) chosen among some
       functions that vanish at the endpoints in x.
    """
    # map x_pts to zero to 1 (it may already be there)
    x_mapped_pts = (x_pts - x_pts[0]) / (x_pts[-1] - x_pts[0])

    # Choices for the base function
    if (base_function == 'exact'):
        base = lambda x : y_star(x)
    elif (base_function == 'guess 1'):
        base = lambda x : np.arcsinh(2.*x)
    elif (base_function == 'guess 2'):
        base = lambda x : x**3

    if (eta_function == 'arcsine'):
        eta = lambda x : np.arcsin(np.pi * x)
    elif (eta_function == 'parabola'):
        eta = lambda x : 4. * x * (1. - x)

    y_new_pts = base(x_pts) + alpha * eta(x_mapped_pts)

    ax1_passed.plot(x_pts, y_new_pts, color='red', lw=1)
    ax2_passed.plot(alpha, evaluate_functional(x_pts, y_new_pts), '.',
                    color='red')

def reset_graph(event):
    ax1.clear()
    ax2.clear()
    setup_figure()

button = widgets.Button(
    description='reset graph'
)
button.on_click(reset_graph)

widgets.interact(make_path,
                 alpha=widgets.FloatSlider(min=-1., max=1., step=.05,
                                           value=0.0, description=r'$\alpha$',
                                           continuous_update=False),
                 ax1_passed=widgets.fixed(ax1),
                 ax2_passed=widgets.fixed(ax2),
                 base_function=widgets.Dropdown(options=['exact', 'guess 1',
                                                         'guess 2'],
                                                value='exact',
                                                description='base function'),
                 eta_function=widgets.Dropdown(options=['arcsine', 'parabola'],
                                               value='arcsine',
                                               description=r'$\eta(x)$')
                )

setup_figure()
button
```

    <IPython.core.display.Javascript object>

    interactive(children=(FloatSlider(value=0.0, continuous_update=False, description='$\\alpha$', max=1.0, min=-1…

    Button(description='reset graph', style=ButtonStyle())

```python

```


# Sympy

```python
import math

math.sqrt(9)
```

    3.0

```python
math.sqrt(8)
```

    2.8284271247461903

```python
import sympy
result = sympy.sqrt(8)
# result = 2*sqrt(2)
result
```

$\displaystyle 2 \sqrt{2}$

```python
result / 2
```

$\displaystyle \sqrt{2}$

# Símbolos en sympy

```python
# Representar la expresión x + 2y

expression = x + 2*y
```

```python
from sympy import symbols

x, y = symbols('x y')
expression = x + 2*y
expression

```

$\displaystyle x + 2 y$

```python
expression + 1

```

$\displaystyle x + 2 y + 1$

```python
expression - x

```

$\displaystyle 2 y$

# Integrales
El método principal de este módulo es `integrate()`

integrate(f, x) retorna la integral indefinida

integrate(f, (x, a, b)) retorna la integral definida entre `b` y `a`

```python
from sympy import *
x = Symbol('x')

expr = 2 * (x**3)
expr

```

$\displaystyle 2 x^{3}$

```python
integrate(expr, x)
```

$\displaystyle \frac{x^{4}}{2}$

```python
expr = x**2 * exp(x) * cos(x)
expr
```

$\displaystyle x^{2} e^{x} \cos{\left(x \right)}$

```python
integrate(expr, x)
```

$\displaystyle \frac{x^{2} e^{x} \sin{\left(x \right)}}{2} + \frac{x^{2} e^{x} \cos{\left(x \right)}}{2} - x e^{x} \sin{\left(x \right)} + \frac{e^{x} \sin{\left(x \right)}}{2} - \frac{e^{x} \cos{\left(x \right)}}{2}$

```python

```


Take the $\alpha$-divergence...

\begin{align}
D_\alpha(p||q) = \frac{1}{\alpha(1-\alpha)} \int_x \alpha p(x) +(1-\alpha)q(x)-p(x)^{\alpha}q(x)^{1-\alpha}dx
\end{align}

We are going to examine a modified form of this equation. Notice the $p(x)^{\alpha}$ in the final term. We will replace this with a linear term...

\begin{align}
J_\alpha(p||q) = \frac{1}{\alpha(1-\alpha)} \int_x \alpha p(x) +(1-\alpha)q(x)-p(x)q(x)^{1-\alpha}dx
\end{align}

Let $q(x)$ be the pdf for a Gaussian, $\mathcal{N}(\mu, V)$, and $p(x)$ be unknown. We would like to show that minimizing $\frac{\partial J}{\partial \mu_q}$ results in $q(x)$ fitting to one of the modes of $p(x)$. Let us begin by evaluating the partial derivative.

\begin{align}
\frac{\partial J_\alpha}{\partial \mu_q} &= \frac{1}{\alpha} \int_x q_{\mu}(x)\bigg(1-\frac{p(x)}{q(x)^{\alpha}}\bigg)dx \\
 \frac{\partial J_\alpha}{\partial \mu_q} &= \frac{1}{\alpha} \int_x q(x)\Sigma^{-1}(x-\mu_q)\bigg(1-\frac{p(x)}{q(x)^{\alpha}}\bigg)dx
\end{align}

After observing that $\int_x q_{\mu}(x) dx = 0$, via...

\begin{align}
\int_x q_{\mu}(x) dx &= \int_x q(x) \Sigma^{-1} (x-\mu_q)dx \\
&= \Sigma^{-1} \bigg(\int_x q(x) x dx - \mu_q \int_x q(x) dx \bigg)\\
&= \Sigma^{-1} \big(\mu_q - \mu_q \big) \\
&= 0\\
\end{align}

The equation reduces to...

\begin{align}
\frac{\partial J_\alpha}{\partial \mu_q} = -\frac{\Sigma^{-1}}{\alpha} \int_x p(x)q(x)^{1-\alpha}(x-\mu_q) dx
\end{align}

Our objective is to identify the values of $\mu_q$ that make $\frac{\partial J_\alpha}{\partial \mu}$ equal zero. In other words...

\begin{align}
F(\mu_q) = \int_x p(x)q(x)^{1-\alpha}(x-\mu_q) dx = 0
\end{align}

Recall that $\mu_q$ is also used in the definition of $q(x)$, therefore we cannot isolate it with simple algebra. To help us, let us represent $p(x)$ exactly as an infinite Gaussian mixture model

\begin{align}
p(x) = \sum_{i=0}^\infty \pi_i p_i(x)
\end{align}

where each $p_i(x)$ is $\mathcal{N}(\mu_i, V_i)$. Then our objective function becomes

\begin{align}
F(\mu_q) & = \int_x  \sum_{i=0}^\infty \pi_i p_i(x) q(x)^{1-\alpha}(x-\mu_q) dx \\
& = \sum_{i=0}^\infty \pi_i \int_x p_i(x) q(x)^{1-\alpha}(x-\mu_q) dx \\
& = \sum_{i=0}^\infty \pi_i \int_x g_i(x)(x-\mu_q) dx \\
& = \sum_{i=0}^\infty \pi_i \bigg ( \int_x g_i(x)x dx - \mu_q\int_x g_i(x) dx \bigg ) \\
& = \sum_{i=0}^\infty \pi_i \bigg ( \mathbb{E}_{g_i} \big[ x \big ] - \mu_q\int_x g_i(x) dx \bigg )
\end{align}

where $g_i(x) = p_i(x)q(x)^{1-\alpha}$, a product of two gaussians. We see, that if we can determine the functional form of $g_i(x)$ along with its integral and expectation, we will have removed the $\mu_q$ term from the definition of $q(x)$.

If we look at equations (10), (11), and (12) from [Products and Convolutions of Gaussian Probability Density Functions](http://www.tina-vision.net/docs/memos/2003-003.pdf) we see that $g_i(x)$ is a scaled gaussian pdf with mean and covariance...

\begin{align}
\mu_{g_i} & = (1 + (1-\alpha)V_q V_{p_i}^{-1})\mu_{p_i} +  (1-\alpha)((1 - \alpha + V_{p_i} V_q^{-1})\mu_q \\
V_{g_i}^{-1} & = V_{p_i}^{-1} + (1-\alpha)V_q^{-1}
\end{align}

$\mathbb{E}_{p} \bigg[ \bigg ]$

```python

```

```python

```

dissertation/Minimizing J finds the peaks of P.ipynb


# Recursion

A very powerful concept in programming is *recursion*, or functions which call themselves. A simple example is the factorial function, which can be defined as:

$$
x_n =
\begin{cases}
1          &\text{if }n = 0 \\
n x_{n-1}, &\text{if }n > 0
\end{cases}
$$

Note that:

- There is a *base case*, $n=0$, which defines the output directly
- For higher $n$, the output is defined *in terms of lower values*

This can be implemented in a Julia function using an if-statement and a call to the function itself:

```julia
function recursive_factorial(n)
    if n == 0
        return 1
    else
        return n*recursive_factorial(n-1)
    end
end
```

    recursive_factorial (generic function with 1 method)

```julia
recursive_factorial(7)
```

    5040

Note the way this will be evaluated in Julia: Eventually it will call the function with $n=0$ which will return 1, this will be multiplied by 2, etc, and last it will multiply by 7. This can be demonstrated by printing additional information:

```julia
function recursive_factorial_info(n)
    if n == 0
        println("n == 0, returning 1")
        return 1
    else
        println("n == ", n, ", calling itself with parameter ", n-1)
        output = n*recursive_factorial_info(n-1)
        println("n == ", n, ", finished calling itself, multiplying and returning ", n, "! = ", output)
        return output
    end
end
```

    recursive_factorial_info (generic function with 1 method)

```julia
recursive_factorial_info(7)
```

    n == 7, calling itself with parameter 6
    n == 6, calling itself with parameter 5
    n == 5, calling itself with parameter 4
    n == 4, calling itself with parameter 3
    n == 3, calling itself with parameter 2
    n == 2, calling itself with parameter 1
    n == 1, calling itself with parameter 0
    n == 0, returning 1
    n == 1, finished calling itself, multiplying and returning 1! = 1
    n == 2, finished calling itself, multiplying and returning 2! = 2
    n == 3, finished calling itself, multiplying and returning 3! = 6
    n == 4, finished calling itself, multiplying and returning 4! = 24
    n == 5, finished calling itself, multiplying and returning 5! = 120
    n == 6, finished calling itself, multiplying and returning 6! = 720
    n == 7, finished calling itself, multiplying and returning 7! = 5040

    5040

This example is highly hypothetical - the factorial function is easier and more efficient to implement using for-loops. But for more complex problems, recursion can be an essential tool.

## Example: The Ackermann function

Think Julia, Exercise 6-5: The Ackermann function, $A(m, n)$, is defined:

$$
\begin{equation} {A(m, n) = \begin{cases} n+1& \textrm{if}\ m = 0 \\ A(m-1, 1)& \textrm{if}\ m > 0\ \textrm{and}\ n = 0 \\ A(m-1, A(m, n-1))& \textrm{if}\ m > 0\ \textrm{and}\ n > 0. \end{cases}} \end{equation}
$$

See https://en.wikipedia.org/wiki/Ackermann_function. Write a function named `ack` that evaluates the Ackermann function. Use your function to evaluate `ack(3, 4)`, which should be 125. What happens for larger values of m and n?

This example is not an obvious for-loop like the factorial function, but very easy to implement using recursion:

```julia
function ack(m,n)
    if m == 0
        return n + 1
    elseif m > 0 && n == 0
        return ack(m-1,1)
    else
        return ack(m-1, ack(m,n-1))
    end
end
```

    ack (generic function with 1 method)

```julia
ack(3,4)
```

    125

Again, let's print some additional information to try to understand how the recursive function calls itself:

```julia
function ack_info(m,n)
    function printme()
        print("ack(", m, ",", n, "): ")
    end
    if m == 0
        output = n + 1
        printme()
        println("Case 1: returning n + 1 = ", output)
        return output
    elseif m > 0 && n == 0
        printme()
        println("Case 2: calling itself with parameters m-1,1 == ", m-1, ",", 1)
        output = ack_info(m-1,1)
        printme()
        println("Case 2: finished calling itself, returning with output ", output)
        return output
    else
        printme()
        println("Case 3: calling itself for new n-value with parameters m,n-1 == ", m, ",", n-1)
        newn = ack_info(m,n-1)
        printme()
        println("Case 3: finished calling itself for new n-value == ", newn)
        printme()
        println("Case 3: calling itself with parameters m-1, A(m,n-1) == ", m-1, ",", newn)
        output = ack_info(m-1,newn)
        printme()
        println("Case 3: finished calling itself, returning ", output)
        return output
    end
end
```

    ack_info (generic function with 1 method)

```julia
ack_info(2,1)
```

    ack(2,1): Case 3: calling itself for new n-value with parameters m,n-1 == 2,0
    ack(2,0): Case 2: calling itself with parameters m-1,1 == 1,1
    ack(1,1): Case 3: calling itself for new n-value with parameters m,n-1 == 1,0
    ack(1,0): Case 2: calling itself with parameters m-1,1 == 0,1
    ack(0,1): Case 1: returning n + 1 = 2
    ack(1,0): Case 2: finished calling itself, returning with output 2
    ack(1,1): Case 3: finished calling itself for new n-value == 2
    ack(1,1): Case 3: calling itself with parameters m-1, A(m,n-1) == 0,2
    ack(0,2): Case 1: returning n + 1 = 3
    ack(1,1): Case 3: finished calling itself, returning 3
    ack(2,0): Case 2: finished calling itself, returning with output 3
    ack(2,1): Case 3: finished calling itself for new n-value == 3
    ack(2,1): Case 3: calling itself with parameters m-1, A(m,n-1) == 1,3
    ack(1,3): Case 3: calling itself for new n-value with parameters m,n-1 == 1,2
    ack(1,2): Case 3: calling itself for new n-value with parameters m,n-1 == 1,1
    ack(1,1): Case 3: calling itself for new n-value with parameters m,n-1 == 1,0
    ack(1,0): Case 2: calling itself with parameters m-1,1 == 0,1
    ack(0,1): Case 1: returning n + 1 = 2
    ack(1,0): Case 2: finished calling itself, returning with output 2
    ack(1,1): Case 3: finished calling itself for new n-value == 2
    ack(1,1): Case 3: calling itself with parameters m-1, A(m,n-1) == 0,2
    ack(0,2): Case 1: returning n + 1 = 3
    ack(1,1): Case 3: finished calling itself, returning 3
    ack(1,2): Case 3: finished calling itself for new n-value == 3
    ack(1,2): Case 3: calling itself with parameters m-1, A(m,n-1) == 0,3
    ack(0,3): Case 1: returning n + 1 = 4
    ack(1,2): Case 3: finished calling itself, returning 4
    ack(1,3): Case 3: finished calling itself for new n-value == 4
    ack(1,3): Case 3: calling itself with parameters m-1, A(m,n-1) == 0,4
    ack(0,4): Case 1: returning n + 1 = 5
    ack(1,3): Case 3: finished calling itself, returning 5
    ack(2,1): Case 3: finished calling itself, returning 5

    5

This also shows that the function calls itself for the same parameter values many times, suggesting that this could be done more efficiently by storing values that have already been computed.

## Example: The Greatest Common Divisor (GCD)

Think Julia, Exercise 6-8: The greatest common divisor (GCD) of $a$ and $b$ is the largest number that divides both of them with no remainder.

One way to find the GCD of two numbers is based on the observation that if $r$ is the remainder when $a$ is divided by $b$, then `gcd(a, b) = gcd(b, r)`. As a base case, we can use `gcd(a, 0) = a`.

```julia
function my_gcd(a,b)
    if a == 0
        return b
    elseif b == 0
        return a
    else
        return my_gcd(b, a % b)
    end
end
```

    my_gcd (generic function with 1 method)

```julia
factor = 123_456_789
prime1 = 67_867_979
prime2 = 86_028_121
my_gcd(prime1*factor, prime2*factor)
```

    123456789

## Example: Recursive triangles

Consider the shape obtained by the following algorithm:
```
    if the triangle is big enough
        Connect the midpoints.
        Color the interior triangle mauve.
        Draw smaller versions of the same shape in each of the 3 remaining interior triangles
    else
        Color the whole triangle yellow.
    end
```

We implement it as follows. "Big enough" is determined by keeping track of the level (depth) of recursion.

```julia
using PyPlot

function drawTriangle(x, y, level)
    # Draw recursively colored triangles.
    # x,y are 3-vectors that define the vertices of a triangle.

    if level == 0
        # Recursion limit (depth) reached
        fill(x, y, "y") # Color whole triangle yellow
    else
        # Draw the triangle...
        plot(x[[1,2,3,1]], y[[1,2,3,1]], "k", linewidth=0.5)
        # Determine the midpoints...
        a = (x + x[[2,3,1]]) / 2
        b = (y + y[[2,3,1]]) / 2
        # Draw and color the interior triangle mauve
        fill(a, b, "m")
        # Apply the process to the three "corner" triangles...
        newx = [x a a[[3,1,2]]]
        newy = [y b b[[3,1,2]]]
        for i = 1:3
            drawTriangle(newx[i,:], newy[i,:], level - 1)
        end
    end
end
```

    drawTriangle (generic function with 1 method)

```julia
# Equilateral triangle
x = [0, 1, 0.5]
y = [0, 0, 1/sqrt(2)]

drawTriangle(x, y, 5)
```

## Merge sort

Merge sort is a so-called Divide and Conquer algorithm. It divides an input array into two halves, calls itself for these two halves and then merges the two sorted halves. The method is illustrated below, for a simple test case with 7 elements:

(from <https://en.wikipedia.org/wiki/Merge_sort>)

In our implementation, the `mergeLR!(L, R, x)` function is used for merging two halves. It assumes that the arrays `L` and `R` are sorted, and merges them into one:

```julia
function mergeLR!(L, R, x)
    # Merge the *already sorted arrays* L and R into a sorted array x
    i = j = k = 1

    # Merge L and R into x
    while i <= length(L) && j <= length(R)
        if L[i] < R[j]
            x[k] = L[i]
            i += 1
        else
            x[k] = R[j]
            j += 1
        end
        k += 1
    end

    # Copy remaining elements
    while i <= length(L)
        x[k] = L[i]
        i += 1
        k += 1
    end
    while j <= length(R)
        x[k] = R[j]
        j += 1
        k += 1
    end
end
```

    mergeLR! (generic function with 1 method)

This can now be used to recursively split the array into two approximately equal-sized arrays until they have length 1, and then apply the `mergeLR!` function at each level:

```julia
function mergesort!(x)
    # Sort the elements of the array x using the Mergesort algorithm
    if length(x) <= 1
        return x
    else
        mid = length(x) ÷ 2   # Find the midpoint of the array
        L = x[1:mid]          # Divide array into 2 halves
        R = x[mid+1:end]

        mergesort!(L)         # Sort first half
        mergesort!(R)         # Sort second half

        mergeLR!(L, R, x)
    end
end
```

    mergesort! (generic function with 1 method)

```julia
# Example: Sort random integers
x = rand(1:1000, 10)
println(x)
mergesort!(x)
println(x)
```

    [336, 619, 143, 67, 680, 349, 849, 247, 738, 87]
    [67, 87, 143, 247, 336, 349, 619, 680, 738, 849]

It can be shown that the number of operations needed for this algorithm to sort an array of length $n$ is about a constant times $n \log_2 n$ (which is optimal for methods based on general comparisons). We can roughly verify this using the `@time` macro which measures execution time:

```julia
for array_size = Int64[1e3, 1e4, 1e5, 1e6, 1e7]
    x = rand(array_size)   # Random floating point numbers
    println("n = ", array_size)
    @time mergesort!(x)
end
```

    n = 1000
      0.080596 seconds (52.68 k allocations: 2.782 MiB, 99.71% compilation time)
    n = 10000
      0.002353 seconds (20.00 k allocations: 2.145 MiB)
    n = 100000
      0.038876 seconds (200.06 k allocations: 23.976 MiB, 28.18% gc time)
    n = 1000000
      0.593117 seconds (2.00 M allocations: 266.090 MiB, 47.88% gc time, 0.92% compilation time)
    n = 10000000
      3.937098 seconds (20.01 M allocations: 2.834 GiB, 11.18% gc time)

We can see that at least for the larger values, the execution time is a little more than 10 times larger when $n$ gets 10 times larger, showing slightly more than a linear dependency on $n$.

## Saving intermediate values in a recursion

The function below implements the so-called *McCarthy 91 function*:

$$
M(n) =
\begin{cases}
n-10, & \text{if }n > 100 \\
M(M(n+11)), & \text{if } n \le 100
\end{cases}
$$

While trivial to implement using recursion, it is not that easy to trace the recursive calls to the function. Therefore, we define a function `Mvalues(n)` which creates an empty array `returned_values`. Inside this function we define the actual recursive function `M(n)`, and each time it is called we push the value that it returns to the `returned_values` array. Note that the order of these numbers in the array will not be the same as the order in which `M(n)` is called, since the values are pushed to the array at the end of the function.

```julia
function Mvalues(n)
    returned_values = Int64[]

    function M(n)
        if n > 100
            newval = n - 10
        else
            newval = M(M(n + 11))
        end
        push!(returned_values, newval)
        return newval
    end

    M(n)
    return returned_values
end
```

    Mvalues (generic function with 1 method)

```julia
Mvalues(105)     # Easy - terminates immediately, M(105) = 95
```

    1-element Vector{Int64}:
     95

```julia
Mvalues(97)      # More complex - finally returns M(97) = 91
```

    9-element Vector{Int64}:
      98
      99
     100
     101
      91
      91
      91
      91
      91


```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
```

# Class 7: Deterministic Time Series Models I

Time series models are at the foundatation of dynamic macroeconomic theory. A time series model is an equation or system of equations that describes how the variables in the model change with time. Here, we examine some theory about **deterministic**, i.e., non-random, time series models and we explore methods for simulating them. Later, we'll examine the properties of **stochastic** time series models by introducing random variables to the discrete time models covered below.

## Discrete Versus Continuous Time

To begin, suppose that we are interested in a variable $y$ that takes on the value $y_t$ at date $t$. The date index $t$ is a real number. We'll say that $y_t$ is a **discrete time** variable if $t$ takes on values from a countable sequence; e.g. $t = 1, 2, 3 \ldots$ and so on. Otherwise, if $t$ takes on values from an uncountable sequence; e.g. $t\in \mathbb{R}^+$, then we'll say that $y_t$ is a **continuous time** variable. Discrete and continuous time models both have important places in macroeconomic theory, but we're going to focus on understanding discrete time models.

## First-Order Difference Equations

Now, suppose that the variable $y_t$ is determined by a linear function of $y_{t-1}$ and some other exogenously given variable $w_t$

\begin{align}
    y_{t} & =  \rho y_{t-1} + w_t,  \tag{1}\\
\end{align}

where $\rho$ is a constant. Equation (1) is an example of a **linear first-order difference equation**. As a *difference equation*, it specifies how $y_t$ is related to past values of $y$. The equation is a *first-order* difference equation because it specifies $y_t$ as a function only of $y_{t-1}$ and not $y_{t-2}$ or $y_{t-3}$ and so on. As simple as the equation looks, it's used in a wide range of modeling applications.

### Example: Compounding Interest

Suppose that you have an initial balance of $b_0$ dollars in a savings account that pays an interest rate $i$ per compounding period. Then, after the first compounding, your account will have $b_1 = (1+i)b_0$ dollars in it. Assuming that you never withdraw funds from the account, then your account balance in any subsequent period $t$ is given by the following difference equation:

\begin{align}
b_{t} & =  \left(1+i\right) b_{t-1}. \tag{2}
\end{align}

Equation (2) is linear first-order difference equation in the same form as Equation (1). You can see this by setting $y_t = b_t$, $\rho=1+i$, and $w_t=0$ in Equation (1).

### Example: Capital Accumulation

Let $K_t$ denote the amount of physical capital in a country at date $t$, let $\delta$ denote the rate at which the capital stock depreciates each period, and let $I_t$ denote the country's investment in new capital in date $t$. Then the law of motion for the stock of physical capital is:

\begin{align}
K_{t+1} & =  I_t + (1-\delta)K_t. \tag{3}
\end{align}

This standard expression for the law of motion for the capital stock is a linear first-order difference equation. To reconcile Equation (3) with Equation (1), set $y_t = K_{t+1}$, $\rho=1-\delta$, and $w_t=I_t$.

*Note*: There is a potentially confusing way in which we identified the $t+1$-dated variable $K_{t+1}$ with the $t$-dated variable $y_t$ in this example. We can do this because the value of $K_{t+1}$ truly is determined at date $t$ even though the capital isn't used for production until the next period.

## Computation

From Equation (1), it's easy to compute the value of $y_t$ as long as you know the value of the constant $\rho$ and the variables $y_{t-1}$ and $w_t$. To begin, let's suppose that $\rho=0.5$. Then Equation (1) in our example looks like this:

\begin{align}
y_{t} & =  0.5 y_{t-1} + w_t. \tag{4}\\
\end{align}

Now, suppose that the initial value of $y$ is $y_0=0$ and that $w$ is equal to 1 in the first period and equal to zero in subsequent periods. That is: $w_1=1$ and $w_2=w_3=\cdots =0$. Now, with what we have, we can compute $y_1$. Here's how:

```python
# Initialize values for rho y0, and w1
y0 = 0
rho = 0.5
w1 = 1

# Compute the value of y1
y1 = rho*y0 + w1

# Print the value of y1
print('y1 =',y1)
```

    y1 = 1.0

The variable `y1` in the preceding example stores the computed value for $y_1$. We can continue to *iterate* on Equation (4) to compute $y_2$, $y_3$, and so on. For example:

```python
# Initialize value for rw2
w2=0

# Compute the value of y2
y2 = rho*y1 + w2

# Compute the value of y2
# Print the result
print('y2 =',y2)
```

    y2 = 0.5

We can do this as many times as necessary to reach the desired value of $t$. Note that iteration is necesary. There is no *vectorized* way to compute an array of values $\left[y_1, y_2, y_3, \ldots, y_{T}\right]$  simultaneously. The linear first-order difference equation is an example of a *recursive* model and in general iteration is necessary for computing recursive models.

Of course, there are smart ways to write code to simulate models. Rather than creating a bunch of variables `y1`, `y2`, etc., let's define a function called `diff1_example()`that takes as arguments $\rho$, $y_0$m and an array of values for $w$, and returns an array of simulated $y$ values . The function should:

1. Definie a variable named `T` equal to the length of  `w`
2. Define an array of of length called `y` that is filled with zeros
3. Set the first vale of `y` to $y_0$
4. Fill-in the remaining values of `y` by iterating over:
        y[t+1]=rho*y[t]+w[t+1]
   for `t` in $[0,1,\ldots, T-1]$
5. Return `y`

```python
# Define a function that returns an array of y-values given rho, y0, and an array of w values.
def diff1_example(rho,w,y0):
    '''Function for computing the first-order difference equation y[t] = rho*y[t-1] + w[t]

    Args:
        rho (float):                Autoregressive coefficient on y[t-1]
        w (NumPy ndarray or list):  Array of exogenous values
        y0 (float):                 Initial value of the process

     Returns:
         NumPy ndarray
     '''

    T = len(w)
    y = np.zeros(T)
    y[0] = y0

    for t in range(T-1):
        y[t+1]=rho*y[t]+w[t+1]

    return y
```

### Example: Temporary Shock to $w_t$ with $\rho=0.5$

Now, use the function `diff1_example()` to simulate $y$ for 11 periods (i.e., for $t = 0, 1, \ldots 10$ with
1. $\rho=0.5$
2. $w_1 = 1$ and $w_0 = w_2 = \ldots = w_{10} = 0$
3. $y_0 = 0$

```python
# Initialize the variable 'T' to be equal to the number of periods to simulate
T = 11

# Create new variable 'rho' equal to the value for rho
rho = 0.5

# Create new variable 'y' equal to the initial value for y
y0 = 0

# Initialize variable 'w' and set initial value
w = np.zeros(T)
w[1]=1

# Simulate y
y = diff1_example(rho,w,y0)

# Construct figure with two line graphs plotting the trajectories of y and w with:
#    1. Plot for w in left panel and plot for y in right panel
#    2. Clear title, x-, and y-axis labels
#    3. y-axis limits [0,1.2]

# Create figure
fig = plt.figure(figsize=(12,4))

# w trajectory
ax = fig.add_subplot(1,2,1)
ax.plot(w,'-',lw=5,alpha = 0.75)
ax.set_title('$w_t$')
ax.set_ylabel('w')
ax.set_xlabel('t')
ax.set_ylim([0,1.2])
ax.grid()

# y trajectory
ax = fig.add_subplot(1,2,2)
ax.plot(y,'-',lw=5,alpha = 0.75)
ax.set_title('$y_t$ with $\\rho=0.5$')
ax.set_ylabel('y')
ax.set_xlabel('t')
ax.set_ylim([0,1.2])
ax.grid()

# Use fig.tight_layout() to manage whitespace in the figure
fig.tight_layout()
```

In period 1, the *shock* to $w_t$ creates an exogenous increase in the process $y_t$. Since $y_{t+1}$ depends on the value of $y_{t}$, the transient or temporary shock to $w_t$ has a persistent effect on the series $y_t$. We call plots like those in the previous figure **impulse responses** because they depict the *responses* of variables to a one-time shock or *impulse* in the exogenous variable.

### Example: Temporary Shock to $w_t$ for Different Values of $\rho$

Next, use the function `diff1_example()` to simulate $y$ for 11 periods with:
1. $\rho=0.2, 0.4, 0.8, 0.1$
2. $w_1 = 1$ and $w_0 = w_2 = \ldots = w_{10} = 0$
3. $y_0 = 0$

```python
# Note: Since T, w, and y0 are already defined, there's no need to re-define them

# Construct a line graph plotting the trajectories of y with:
#    1. Clear title, x-, and y-axis labels
#    2. Y-axis limits [0,1.2]
#    3. Add a legend outside of the plot with arguments: loc='center left', bbox_to_anchor=(1, 0.5)

fig = plt.figure()
ax = fig.add_subplot(1,1,1)

# Iterate over rho in [0.1, 0.2, 0.3, ... , 0.9]. Simulate y for each rho and plot. Set plot label equal to rho value
for rho in np.arange(0.1,1,0.1):
    y = diff1_example(rho,w,y0)
    plt.plot(y,'-b',lw=5,alpha = rho,label=round(rho,1))

ax.set_title('$y_t$')
plt.ylabel('y')
plt.xlabel('t')
# ax.set_ylim([0,1.2])
ax.legend(loc='center left', bbox_to_anchor=(1, 0.5))
ax.grid()
```

Apparently increasing $\rho$ also increases the degree of persistence in the process $y$ following the shock to $w$. By $t=10$, the effects of the one-time increase in $w$ on $y$ have essentially dissipated for the simulations with $\rho= 0.1, 0.2, \ldots, 0.7$. But for $\rho=0.8$, the value of the $y$ is still above $0.15$ in period $10$ and for $\rho=0.9$, the process $y$ is nearly at $0.4$; i.e., nearly $40$ percent of the shock still remains. But regardless, in all cases it appears that $y$ approaches zero in the limit.

### Example: Temporary Shock to $w_t$ for $\rho=-0.5$

Use the function `diff1_example()` to simulate $y$ for 11 periods with:
1. $\rho=-0.5$
2. $w_1 = 1$ and $w_0 = w_2 = \ldots = w_{10} = 0$
3. $y_0 = 0$

```python
# Note: Since T, w, and y0 are already defined, there's no need to re-define them

# Create new variable 'rho' equal to the value for rho
rho = -0.5

# Simulate y
y = diff1_example(rho,w,y0)

# Construct a line graph plotting the trajectory of y with:
#    1. Plot for y stacked on top of plot for w
#    2. Clear title, x-, and y-axis labels
#    3. Y-axis limits [-0.8,1.2]

# Create figure
fig = plt.figure()

# y trajectory
ax = fig.add_subplot(1,1,1)
ax.plot(y,'-',lw=5,alpha = 0.75)
ax.set_title('$y_t$ with $\\rho=-0.5$')
ax.set_ylabel('y')
ax.set_xlabel('t')
ax.set_ylim([-0.8,1.2])
ax.grid()
```

With $\rho=-0.5$, the sign of the pricess $y$ oscilates between positive and negative values but the absolute value of $y$ smoothly converges to zero in the limit.

### Example: Temporary Shock to $w_t$ for $\rho=1.5$ and $\rho=-1.5$

So far all of the values that we hav used for $\rho$ have been less than 1 in absolute value and in all cases, the process $y$ has returned to zero in the limit. But this may not always be the case. Let's consider the cases where $\rho=1.5$ and $\rho=-1.5$.

Use the function `diff1_example()` to simulate $y$ for 11 periods with:
1. $\rho=1.5$ (left panel) and $\rho=-1.5$ (right panel)
2. $w_1 = 1$ and $w_0 = w_2 = \ldots = w_{10} = 0$
3. $y_0 = 0$

```python
# Note: Since T, w, and y0 are already defined, there's no need to re-define them

# Create figure
fig = plt.figure(figsize=(12,4))

# Set rho=1.5
rho = 1.5

# Simulate y with rho = 1.5
y = diff1_example(rho,w,0)

# plot trajectory of y with rho = 1.5
ax = fig.add_subplot(1,2,1)

ax.plot(y,'-',lw=5,alpha = 0.75)
ax.set_title('$\\rho=1.5$')
ax.set_ylabel('y')
ax.set_xlabel('t')
ax.grid()

# Set rho=-1.5
rho = -1.5

# Simulate y with rho = -1.5
y = diff1_example(rho,w,0)

# plot trajectory of y with rho = -1.5
ax = fig.add_subplot(1,2,2)

ax.plot(y,'-',lw=5,alpha = 0.75)
ax.set_title('$\\rho=1.5$')
ax.set_ylabel('y')
ax.set_xlabel('t')
ax.grid()
```

Unlike the earlier simulations, the ones in the previous example do not return to zero and instead either explode toward infinity ($\rho = 1.5$) or oscillate towards positive and negative infinity. These two simulations are examples of *explosive processes* while the ones prior were examples of *stable processes*. In the next section, I elaborate on the notion of stability.

### Example: Permanent Shock to $w_t$ with $\rho=0.5$

So far we hav eonly considered temporary shocks to to $wt$. But sometimes we want to know what effect does a permanent change in $w_t$ have on $y_t$. To answer this, we set $w_0=0$ and $w_1 = w_2 = \cdots = 1$.

Now, use the function `diff1_example()` to simulate $y$ for 11 periods (i.e., for $t = 0, 1, \ldots 10$ with
1. $\rho=0.5$
2. $w_1 = 0$ and $w_1 = w_2 = \ldots = w_{10} = 1$
3. $y_0 = 0$

```python
# Initialize the variable 'T' to be equal to the number of periods to simulate
T = 11

# Create new variable 'rho' equal to the value for rho
rho = 0.5

# Create new variable 'y' equal to the initial value for y
y0 = 0

# Initialize variable 'w' as arrays of ones
w = np.ones(T-1)
w[0] = 0

# Simulate y
y = diff1_example(rho,w,y0)

# Construct figure with two line graphs plotting the trajectories of y and w with:
#    1. Plot for w in left panel and plot for y in right panel
#    2. Clear title, x-, and y-axis labels
#    3. y-axis limits [0,2.5]

# Create figure
fig = plt.figure(figsize=(12,4))

# w trajectory
ax = fig.add_subplot(1,2,1)
ax.plot(w,'-',lw=5,alpha = 0.75)
ax.set_title('$w_t$')
ax.set_ylabel('w')
ax.set_xlabel('t')
ax.set_ylim([0,2.5])
ax.grid()

# y trajectory
ax = fig.add_subplot(1,2,2)
ax.plot(y,'-',lw=5,alpha = 0.75)
ax.set_title('$y_t$ with $\\rho=0.5$')
ax.set_ylabel('y')
ax.set_xlabel('t')
ax.set_ylim([0,2.5])
ax.grid()

# Use fig.tight_layout() to manage whitespace in the figure
fig.tight_layout()
```

The cumulative effect on $y$ in period $T$ of the permanent change in $w_t$ is:

\begin{align}
y_T & = \rho^{T-1} + \rho^{T-2} + \rho^{T-3} + \cdots + \rho + 1\\
& = \frac{1}{1-\rho},
\end{align}

which, for the example above, is $2$ and that is confirmed by the simulation.

## Stability of First-Order Difference Equations

You might already see that the value of $\rho$ is important in determining the stability of the process $y$, but we can see clearly why. Suppose that $\mu=0$ (this just makes the argument more clear) and start with the equation for determining $y_1$:

\begin{align}
y_1 & = \rho y_0 + w_1
\end{align}

The value for $y_2$ is determined by an analogous equation:

\begin{align}
y_2 & = \rho y_1 + w_2
\end{align}

Then use the equation for $y_1$ to eliminate $y_1$ from the equation for $y_2$:

\begin{align}
y_2 & = \rho^2 y_0 + \rho w_1 + w_2
\end{align}

Continue this process to arrive at a formula that specifies the value of $y$ in period $T$ as a function of just $\rho$, $y_0$, and the values of $w$: $w_1, w_2, \ldots, w_T$:

\begin{align}
y_T & = \rho^T y_0 + \rho^{T-1} w_1 + \rho^{T-2} w_2 + \cdots + \rho w_{T-1} + w_T\\
\end{align}

Then, you can see that the marginal effect of $w_1$ on $y_T$ is:

\begin{align}
\frac{\partial y_T}{\partial w_1} & = \rho^{T-1}
\end{align}

From this, there are three facts:

1. If $|\rho|<1$, then the process $y$ is **stable** because $|\rho^{T-1}|\rightarrow 0$ as $T\rightarrow \infty$.
2. If $|\rho|>1$, then the process $y$ is **explosive** because $|\rho^{T-1}|\rightarrow \infty$ as $T\rightarrow \infty$.
2. If $|\rho|=1$, then the process $y$ is neither stable nor explosive because $|\rho^{T-1}|\rightarrow 1$ as $T\rightarrow \infty$.

The final case, with $|\rho^T|=1$, is closely related to the *random walk* stochastic process that is often used to describe finacial asset returns over short time horizons. We will encounter this process later.

Stability is a fundamental concept in economics and there are important roles for models with stable and explosive properties.

## Higher-order Difference Equations (Optional)

The $p$th-order linear difference equation is given by the following expression:

\begin{align}
y_t & = \rho_1 y_{t-1} + \rho_2 y_{t-2} + \cdots + \rho_p y_{t-p} + w_t,
\end{align}

where $\rho_1, \rho_2, \ldots, \rho_p$ are constants. In principle, computing the $p$th-order difference equation is the same as computing the first-order difference equation. The only difference real difference is that you need $p$ initial values of $y$ and not just 1. That is, in order to compute the period 1 value $y_1$, you need values for $y_0, y_{-1}, \ldots, y_{-(p-1)}$.

The function in the following cell simulates $p$-order difference equations so we can use it for the examples below.

```python
# Define a function that returns a DataFrame of y and w values given coefficients, y0, and an array of w values. CELL PROVIDED
def pth_order_diff(rho,y0,w):
    '''Function for computing the first-order difference equation:

            y[t] = C + rho[0]*y[t-1] + rho[1]*y[t-2] + ... + rho[p]*y[t-p] + w[t]

    Args:
        rho (NumPy ndarray or list):   Autoregressive coefficients on y[t-1], y[t-2], ... y[t-1]
        y0 (NumPy ndarray or list):    Initial value of the process (must be same length as rho)
        w (NumPy ndarray or list):     Array of exogenous values for t=0,..,T+1 where T is the end date

     Returns:
         NumPy ndarray with columns 'w' and 'y' and index: -(p-1), -(p-2), ..., T
     '''

    # Set order of the process. If rho is a list or array, set p to the length. Otherwise, p=1.
    try:
        p = len(rho)
    except:
        p = 1

    # Length of simulation
    try:
        T = len(w)
    except:
        T = 1

    # Construct series for y
    y = np.zeros(p+T-1)
    y[0:p] = y0

    # Construct series for w
    empty = np.empty(p-1)
    empty[:] = np.nan
    w = np.concatenate([empty,w[:T+1]])

    # Date index
    index = np.arange(-(p-1),T)

    # Initialize DataFrame
    df = pd.DataFrame({'w':w,'y':y},index=index)

    # Compute y
    for i in df.index[p:]:
        df.loc[i,'y'] = np.dot(rho,np.flip(df.loc[i-p:i-1,'y'])) + df.loc[i,'w']

    return df
```

### Example: A Second-Order Difference Equation

Consider the second order difference equation:

\begin{align}
y_t & = \rho_1 y_{t-1} + \rho_2 y_{t-2} + w_t,
\end{align}

As usual, take $w_1=1$ and $w_0=w_2=w_3 = \cdots$ and $y_0 = y_{-1} = 0$. Let's simulate the model with two different combinations of $\rho_1$ and $\rho_2$:

\begin{align}
y_t & = 0.5 y_{t-1} + 0.2 y_{t-2} + w_t,
\end{align}

and:

\begin{align}
y_t & = 0.6 y_{t-1} + -0.9 y_{t-2} + w_t,
\end{align}

Simulate the processes for 24 periods and, instead of using a line plot this time, use barcharts. Stack the axes for each vertically in a single figure.

```python
# Create figure that has dimensions 12x8
fig = plt.figure(figsize=(12,8))

# Initial y values
y0 = [0,0]

# End period of simulation
T = 24

# Create w array and set period 1 value
w = np.zeros(T+1)
w[1] = 1

# Specify coefficients
rho = [0.5,0.2]

# Simulate y and remove zeros from the simulated data
df = pth_order_diff(rho,y0,w)

# Remove zeros from the simulated data
df = df.replace(0,np.nan)

# Construct barchart of y values
ax = fig.add_subplot(2,1,1)
ax.bar(df.index,df['y'],alpha = 0.65)
ax.set_title('$\\rho_1 ='+str(rho[0])+'$ and $\\rho_2 = '+str(rho[1])+'$',fontsize=15)
ax.grid()

# Specify coefficients
rho = [0.6,-.9]

# Simulate y
df = pth_order_diff(rho,y0,w)

# Remove zeros from the simulated data
df = df.replace(0,np.nan)

# Construct barchart of y values
ax = fig.add_subplot(2,1,2)
ax.bar(df.index,df['y'],alpha = 0.65)
ax.set_title('$\\rho_1 ='+str(rho[0])+'$ and $\\rho_2 = '+str(rho[1])+'$',fontsize=15)
ax.grid()
```

Apparently under both parameterizations the second-order difference equation is stable. But change the sign of $\rho_2$ in the last simulation so that the model becomes:

\begin{align}
y_t & = 0.6 y_{t-1} + 0.9 y_{t-2} + w_t,
\end{align}

```python
# Create figure that has dimensions 12x4
fig = plt.figure(figsize=(12,4))

# Initial y values
y0 = [0,0]

# End period of simulation
T = 24

# Create w array and set period 1 value
w = np.zeros(T+1)
w[1] = 1

# Specify coefficients
rho = [0.6,0.9]

# Simulate y and remove zeros from the simulated data
df = pth_order_diff(rho,y0,w)

# Remove zeros from the simulated data
df = df.replace(0,np.nan)

# Construct barchart of y values
ax = fig.add_subplot(1,1,1)
ax.bar(df.index,df['y'],alpha = 0.65)
title = ''
for n,r in enumerate(rho):
    title+='$\\rho_'+str(n+1)+' = '+str(round(r,4))+'$, '
ax.set_title(title,fontsize=15)
ax.grid()
```

Evidently reversing the sign on the second coefficient has caused the model to become explosive. Determining the stability of second-order and higher-order difference equations is not trivial and requires math that is beyond what most students in the course will have seen and I won't go into the topic here. See Chapter 1 of James Hamilton's book [*Time Series Analysis*](https://press.princeton.edu/titles/5386.html]).

However, as the examples suggest, simulating a process can give strong clues about whether a process is stable.

### A Fifth-Order Difference Equation

Now consider the following fifth-order difference equation difference equation:

\begin{align}
y_t & =  -0.3948 y_{t-1} - 0.3719 y_{t-2} - 0.4126 y_{t-3} - 0.3845y_{t-4} +  0.2477 y_{t-5} + w_t
\end{align}

The coefficients were actually chosen randomly using NumPy as you can see in the code below.

```python
# Create figure that has dimensions 12x4
fig = plt.figure(figsize=(12,4))

# Initial y values
y0 = [0,0,0,0,0]

# End period of simulation
T = 28

# Create w array and set period 1 value
w = np.zeros(T+1)
w[1] = 1

# Specify coefficients
np.random.seed(126)
rho = np.random.uniform(size=5)-0.5
# rho = [0.5,0.2]

# Simulate y and remove zeros from the simulated data
df = pth_order_diff(rho,y0,w)

# Remove zeros from the simulated data
df = df.replace(0,np.nan)

# Construct barchart of y values
ax = fig.add_subplot(1,1,1)
ax.bar(df.index,df['y'],alpha = 0.65)
title = ''
for n,r in enumerate(rho):
    title+='$\\rho_'+str(n+1)+' = '+str(round(r,4))+'$, '
ax.set_title(title,fontsize=15)
ax.grid()
```