(c) October 2016 - This notebook was created by [Oriol Pujol Vila](http://www.maia.ub.es/~oriol).

# pill 8 - Stochastic Subgradient Methods

Outline:

+ A quick introduction to Stochastic subgradient methods
+ Mini batch approximation
+ Speeding up subgradient methods

# Stochastic subgradient methods

A vector $g \in {\bf R}^n$ is a *subgradient* of $f:{\bf R}^n\rightarrow{\bf R}$ at $x\in {\bf \text{dom}}\; f$ if for all $z\in {\bf \text{dom}}\; f$,

$$
f(z)\geq f(x) + g^T (z-x)
$$

The hyperplane with normal $(g,-1)$ supports ${\bf \text{epi}}\; f$ at $(x,f(x))$. Thus if $f$ is convex and differentiable, then its gradient at $x$ is a subgradient. Note that the subgradient can exist even when $f$ is not differentiable at $x$, and more of one subgradient may exist. A function $f$ is called *subdifferentiable* at $x$ if there exists at least one subgradient at $x$. The set of subgradietns of $f$ at the point $x$ is called the *subdifferential* of $f$ at $x$, and is denoted as $\partial f(x)$. A function $f$ is called subdifferentiable if it is subdifferentiable at all $x\in {\bf \text{dom}}\; f$. If $f$ is convex and differentiable at $x$, then $\partial f(x) = \{\nabla f(x)\}$, i.e. the gradient is the only subgradient.

## Minimum of a non-differentiable function

A point $x^*$ is a minimizer of a convex function $f$ if and only if $f$ is subdifferentiable at $x^*$ and
$$0 \in \partial f(x^*)$$
i.e. $0$ is a subgradient of $f$ at $x^*$. This follows from the definition of subgradient and the fact that $f(x)\geq f(x^*)$. If $f$ is differentiable at $x^*$, the equation is reduced to $\nabla f(x^*) = 0$.

## Using subgradients for solving convex optimization problems

In this section, the extension of the subgradient method is explained to solve an inequality constrained problem

$$
\begin{align}
\text{minimize } & \quad f_0(x)\\
\text{subject to } & \quad f_i(x)\leq 0, \quad  \; i = 1,\dots, m,
\end{align}
$$

where $f_i$ are convex. The algorithm takes the same form:

$$x^{(k+1)} = x^{(k)} - \alpha_k g^{(k)},$$

where $\alpha_k>0$ is a step size, and $g^{(k)}$ is a subgradient of the objective or one of the constraint functions at $x^{(k)}$. More specifically, we take

$$
g^{(k)}\in \left \{ \begin{align}
\partial f_0(x^{(k)}) & \quad f_i(x^{(k)})\leq 0, \; i= 1,\dots, m,\\
\partial f_j(x^{(k)}) & \quad f_j(x^{(k)}) > 0.\\
\end{align}   \right.
$$

i.e. if  $x^{(k)}$ is feasible ($f_i(x^{(k)})\leq 0, \; i= 1,\dots, m$), we use an objective subgradient. Otherwise, if the current point is infeasible, we choose any violated constraint, and use a subgradient of the associated constraint function. Note that we may choose any of the violated constraints. Observe also that the iterates are often infeasible (this does not happen in the projected subgradient method). We can alternatively use a projection of the solution into the feasible set, i.e. we may project such that the violated constrained is no longer violated. As in the unconstrained case, we may keep track of the best feasible point found so far,
$$f_{\mbox{best}}^{(k)} = \min \{f_0(x^{(i)})|\; x^{(i)} \mbox{feasible}, i = 1,\dots,k\}.$$
If there are no feasible points $f_{\mbox{best}}^{(k)} = \infty$.

## Stochastic subgradient methods

Supose we have a noisy problem and we can not evaluate exactly the value of the subgradient. For example, if we are dealing with a random variable the instantaneous gradient is not reliable. There may be different sources of noise, e.g. noise due to a finite sampling, measurement noise, uncertainty in data collection, etc.

Let $f: {\bf R}^n \rightarrow {\bf R}$ be a convex function. We say that $\tilde{g} \in {\bf R}^n$ is a *noisy (unbiased) subgradient* of $f$ at $x \in {\bf dom}\; f$ if $g = {\bf E} \,\tilde{g} \in \partial f(x)$, i.e. we have

$$f(z) \geq f(x) + ({\bf E}\, \tilde{g})^T(z-x).$$

This is, the expected value is a subgradient. Thus we can consider that $\tilde{g} = g + v$, where $g\in \partial f(x)$ is a subgradient and $v$ is a zero mean random "noise". If $x$ is a random variable, then we say that $\tilde{g}$ is a noisy subgradient if

$$\forall z \quad f(z)\geq f(x) + {\bf E}\, (\tilde{g}|x)^T (z-x),$$

holds almost surely. In other words, ${\bf E}\, (\tilde{g}|x) \in \partial f(x)$.

## The stochastic gradient method

Stochastic gradient methods are essentially the subgradient method using noisy subgradients. Consider the minimization of a convex function $f$,  the stochastic subgradient method uses the standard update rule,

$$x^{(k+1)} = x^{(k)} -\alpha_k \tilde{g}^{(k)},$$

where $x^{(k)}$ is the $k$th iterate, $\alpha_k > 0$ is the $k$th step size, and $\tilde{g}^{(k)}$ is a noisy subgradient of $f$ at $x^{(k)}$,

$${\bf E}\,(\tilde{g}^{(k)}| x^{(k)}) = g^{(k)} \in \partial f (x^{(k)})$$

We omit the convergence proof, but with square summable but not summable step size, e.g. $\alpha_k = a / (b + k)$, it can be proved convergence in expectation, i.e. ${\bf E}\, f_{best}^{(k)}\underset{k\rightarrow \infty}{\rightarrow} f^*$, and convergence in probability, i.e. $\underset{k\rightarrow \infty}{\lim} {\bf Prob} (f_{best}^{(k)}\geq f^* + \epsilon) = 0.$

## The algorithm

We can use two possible algorithms, either a full stochastic subgradient iteration or a combination of subgradient and projection methods. Let us consider the hybrid version for the sake of simplicity:

```
while no_ending (iterations k):
  1. Compute the subgradient of the loss function, g_tilde.
  2. Take the step
      x = x - 1/k * g_tilde
  3. Project the into the feasible set
      x = P(x)
```

## Solving problems using Stochastic Subgradient Methods (SSM)

Let us solve our well known least squares problem

```python
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt

def polyval(coefs, x):
    res = coefs[0]*np.ones((1,x.shape[0]))
    for i in range(1,len(coefs)):
        res = res + coefs[i]*np.power(x,i)
    return res

def chebys (coefs, x):
    pol = {}
    pol[0] = [1][::-1]
    pol[1] = [1, 0][::-1]
    pol[2] = [2, 0, -1][::-1]
    pol[3] = [4, 0, -3, 0][::-1]
    pol[4] = [8, 0, -8, 0, 1][::-1]
    pol[5] = [16, 0, -20, 0, 5, 0][::-1]
    pol[6] = [32, 0, -48, 0, 18, 0, -1][::-1]
    pol[7] = [64, 0, -112, 0, 56, 0, -7, 0][::-1]
    pol[8] = [128, 0, -256, 0, 160, 0, -32, 0, 1][::-1]
    pol[9] = [256, 0, -576, 0, 432, 0, -120, 0, 9, 0][::-1]
    pol[10] = [512, 0, -1280, 0, 1120, 0, -400, 0, 50, 0, -1][::-1]
    pol[11] = [1024, 0, -2816, 0, 2816, 0, -1232, 0, 220, 0, -11, 0][::-1]

    res = np.zeros((1,x.shape[0]))

    for i in range(len(coefs)):
        res= res + coefs[i]*polyval(pol[i],x)
    return res

x = np.linspace(0,1,100)

%matplotlib inline
import matplotlib.pyplot as plt

for i in range(11):
    co = np.zeros((11,))
    co[i]=1.
    plt.plot(x,chebys(co,x).ravel())

#Create a polynomial with noise
np.random.seed(42)
coefs = np.random.random(11)

x = np.linspace(0,1,100)

vals=chebys(coefs,x).ravel()
#plt.plot(x,vals)

N=15
idx = np.random.randint(0,100,N)
data = vals[idx]+0.5*np.random.normal(size=len(idx))
plt.figure()
plt.plot(x[idx],data,'ro')

x_train = x[idx][:,np.newaxis]
y_train = data[:,np.newaxis]
```

```python
X = np.c_[np.ones((15,1)),x_train,x_train**2,x_train**3,x_train**4,x_train**5,x_train**6,x_train**7,x_train**8,x_train**9,x_train**10].T

import numpy as np

#initialize weights
w = np.zeros((11,1))
conv = []
n_iters = 10000
step = 0.1

for k in range(1,n_iters):
    #get a noisy gradient by just taking one data point at random and get the instantaneous gradient
    i = np.random.randint(0,X.shape[1])
    x_sample = X[:,i][:,np.newaxis]
    y_sample = y_train[i,:]
    g_tilde = -2*x_sample*(y_sample-np.dot(x_sample.T,w))

    w = w - step * g_tilde

    J=np.sum((np.dot(X.T,w)-y_train)*(np.dot(X.T,w)-y_train))

    conv.append(J)

plt.plot(conv)
```

```python
x_draw = np.linspace(0,1,100)

x_extended = np.c_[np.ones((100,1)),x_draw,x_draw**2,x_draw**3,x_draw**4,x_draw**5,x_draw**6,x_draw**7,x_draw**8,x_draw**9,x_draw**10].T

y_hat = np.dot(x_extended.T,w)

plt.plot(x_train,y_train,'ro')
plt.plot(x_draw,y_hat)
plt.plot(x,vals)
```

**EXERCISE** Use a **pocket** strategy and keep the best weight so far. Plot the new convergence plot.

```python
#your code here
```

```python
x_draw = np.linspace(0,1,100)

x_extended = np.c_[np.ones((100,1)),x_draw,x_draw**2,x_draw**3,x_draw**4,x_draw**5,x_draw**6,x_draw**7,x_draw**8,x_draw**9,x_draw**10].T

y_hat = np.dot(x_extended.T,wopt)

plt.plot(x_train,y_train,'ro')
plt.plot(x_draw,y_hat)
plt.plot(x,vals)
```

# Checking the path

For the sake of simplicity we consider a 2D subproblem of the former problem and visualize the weight pass.

```python
import numpy as np

X = np.c_[np.ones((15,1)),x_train].T

#initialize weights
w = -np.ones((2,1))
conv = []
wpath = []
wpath.append(w)
n_iters = 100000
Jopt = 9e16
wopt = w

step = 0.001
for k in range(1,n_iters):
    #get a noisy gradient by just taking one data point at random and get the instantaneous gradient
    i = np.random.randint(0,X.shape[1])
    x_sample = X[:,i][:,np.newaxis]
    y_sample = y_train[i,:]
    g_tilde = -2*x_sample*(y_sample-np.dot(x_sample.T,w))

    w = w - step * g_tilde

    #Keep the best weight up to that point
    J=np.sum((np.dot(X.T,w)-y_train)*(np.dot(X.T,w)-y_train))
    wpath.append(w)
    conv.append(J)
    #Project if needed
    ## Add your projection code here

plt.plot(conv)

```

```python
#order = np.Inf
order = 10
delta = 0.5
xx = np.arange(-2.0, 2.0, delta)
yy = np.arange(-2.0, 2.0, delta)
XX, YY = np.meshgrid(xx, yy)
Xf = XX.flatten()
Yf = YY.flatten()
sz=XX.shape
data = np.concatenate([Xf[:,np.newaxis],Yf[:,np.newaxis]],axis=1);
Z = np.sum((np.dot(X.T,data.T)-y_train)*(np.dot(X.T,data.T)-y_train),axis = 0)/15.
print (np.min(Z),np.max(Z))
#print data.shape
#Z = np.linalg.norm(data,ord = order,axis = 1)
print (Z.shape)
Z.shape=sz
plt.imshow(Z, interpolation='bilinear', origin='lower', extent=(-2,2,-2,2),alpha=0.3, vmin=-10, vmax=10)
plt.contour(XX,YY,Z,[0,1,2,3,4,5])
fig = plt.gcf()
fig.set_size_inches(9,9)
plt.plot(-1,-1,'ro')
plt.plot(np.array(wpath)[:,0],np.array(wpath)[:,1])
plt.plot(np.array(wpath)[-1,0],np.array(wpath)[-1,1],'co')

```

```python
#order = np.Inf
order = 2
delta = 0.05
xx = np.arange(-4.0, 4.0, delta)
yy = np.arange(-4.0, 4.0, delta)
XX, YY = np.meshgrid(xx, yy)
Xf = XX.flatten()
Yf = YY.flatten()
sz=XX.shape
data = np.concatenate([Xf[:,np.newaxis],Yf[:,np.newaxis]],axis=1);
Z = np.sum((np.dot(X.T,data.T)-y_train)*(np.dot(X.T,data.T)-y_train),axis = 0)/15.
print (np.min(Z),np.max(Z))
#print data.shape
#Z = np.linalg.norm(data,ord = order,axis = 1)
print (Z.shape)
Z.shape=sz
plt.imshow(Z, interpolation='bilinear', origin='lower', extent=(-4,4,-4,4),alpha=0.3, vmin=-20, vmax=20, cmap = "PRGn")
CS = plt.contour(XX,YY,Z,[1.3,1.5,2,3,4,5,6,7,8], colors = 'k')
plt.clabel(CS, inline=1, fontsize=10)

fig = plt.gcf()
fig.set_size_inches(9,9)

fig.savefig('contours.jpg',dpi=300,bbox_inches='tight')
```

```python
%matplotlib inline

x_draw = np.linspace(0,1,100)

x_extended = np.c_[np.ones((100,1)),x_draw].T

y_hat = np.dot(x_extended.T,wopt)

plt.plot(x_train,y_train,'ro')
plt.plot(x_draw,y_hat)
plt.plot(x,vals)
```

**EXERCISE** Plot the path when you use the **pocket** strategy.

```python
#your code
```

```python
#my code

X = np.c_[np.ones((15,1)),x_train].T

import numpy as np

#initialize weights
w = np.zeros((2,1))
conv = []
wpath = []
wpath.append(w)
n_iters = 100000
Jopt = 9e16
wopt = w

step = 0.01
for k in range(1,n_iters):
    #get a noisy gradient by just taking one data point at random and get the instantaneous gradient
    i = np.random.randint(0,X.shape[1])
    x_sample = X[:,i][:,np.newaxis]
    y_sample = y_train[i,:]
    g_tilde = -2*x_sample*(y_sample-np.dot(x_sample.T,w))

    w = w - step * g_tilde

    #Keep the best weight up to that point
    J=np.sum((np.dot(X.T,w)-y_train)*(np.dot(X.T,w)-y_train))
    if J<Jopt:
        Jopt = J
        wopt = w

    wpath.append(wopt)
    #Project if needed
    ## Add your projection code here

plt.plot(conv)

#order = np.Inf
order = 2
delta = 0.5
xx = np.arange(-2.0, 2.0, delta)
yy = np.arange(-2.0, 2.0, delta)
XX, YY = np.meshgrid(xx, yy)
Xf = XX.flatten()
Yf = YY.flatten()
sz=XX.shape
data = np.concatenate([Xf[:,np.newaxis],Yf[:,np.newaxis]],axis=1);
Z = np.sum((np.dot(X.T,data.T)-y_train)*(np.dot(X.T,data.T)-y_train),axis = 0)/15.
print (np.min(Z),np.max(Z))
#print data.shape
#Z = np.linalg.norm(data,ord = order,axis = 1)
print (Z.shape)
Z.shape=sz
plt.imshow(Z, interpolation='bilinear', origin='lower', extent=(-2,2,-2,2),alpha=0.3, vmin=-10, vmax=10)
plt.contour(XX,YY,Z,[0,1,2,3,4,5])
fig = plt.gcf()
fig.set_size_inches(9,9)
plt.plot(0,0,'ro')
plt.plot(np.array(wpath)[:,0],np.array(wpath)[:,1])
plt.plot(np.array(wpath)[-1,0],np.array(wpath)[-1,1],'co')
```

# Mini batch

The noisy gradient can be as noisy as a single point but we can approximate the true function using more data. This is the mini-batch method. In this case we use the average gradient of the subset of the data. Observe that this is simply a sub sampling of the objective function.

```python
X = np.c_[np.ones((15,1)),x_train,x_train**2,x_train**3,x_train**4,x_train**5,x_train**6,x_train**7,x_train**8,x_train**9,x_train**10].T

import numpy as np

#initialize weights
w = np.zeros((11,1))
conv = []
wpath = []
wpath.append(w)
n_iters = 10000
Jopt = 9e16
wopt = w
mbatch_size=12
step = 0.005
for k in range(1,n_iters):
    #get a noisy gradient by just taking one data point at random and get the instantaneous gradient
    i = np.random.randint(0,X.shape[1],size=mbatch_size)
    x_sample = X[:,i]
    y_sample = y_train[i,:]
    g_tilde = -2*np.dot(x_sample,(y_sample-np.dot(x_sample.T,w)))

    w = w - step * g_tilde

    #Keep the best weight up to that point
    J=np.sum((np.dot(X.T,w)-y_train)*(np.dot(X.T,w)-y_train))
    if J<Jopt:
        wopt = w
        Jopt = J
    conv.append(J)
    wpath.append(w)
    #Project if needed
    ## Add your projection code here

plt.plot(conv)
```

```python
x_draw = np.linspace(0,1,100)

x_extended = np.c_[np.ones((100,1)),x_draw,x_draw**2,x_draw**3,x_draw**4,x_draw**5,x_draw**6,x_draw**7,x_draw**8,x_draw**9,x_draw**10].T

y_hat = np.dot(x_extended.T,wopt)

plt.plot(x_train,y_train,'ro')
plt.plot(x_draw,y_hat)
plt.plot(x,vals)
```

Observe now the path.

```python
X = np.c_[np.ones((15,1)),x_train].T

import numpy as np

#initialize weights
w = np.zeros((2,1))
conv = []
wpath = []
wpath.append(w)
n_iters = 100000
Jopt = 9e16
wopt = w
mbatch_size=30
step = 0.001
for k in range(1,n_iters):
    #get a noisy gradient by just taking one data point at random and get the instantaneous gradient

    i = np.random.randint(0,X.shape[1],size=mbatch_size)
    x_sample = X[:,i]
    y_sample = y_train[i,:]
    g_tilde = -2*np.dot(x_sample,(y_sample-np.dot(x_sample.T,w)))

    w = w - step * g_tilde

    #Keep the best weight up to that point
    J=np.sum((np.dot(X.T,w)-y_train)*(np.dot(X.T,w)-y_train))
    if J<Jopt:
        wopt = w
        Jopt = J
    conv.append(Jopt)
    wpath.append(wopt)
    #Project if needed
    ## Add your projection code here

plt.plot(conv)

```

```python

#order = np.Inf
order = 2
delta = 0.5
xx = np.arange(-2.0, 2.0, delta)
yy = np.arange(-2.0, 2.0, delta)
XX, YY = np.meshgrid(xx, yy)
Xf = XX.flatten()
Yf = YY.flatten()
sz=XX.shape
data = np.concatenate([Xf[:,np.newaxis],Yf[:,np.newaxis]],axis=1);
Z = np.sum((np.dot(X.T,data.T)-y_train)*(np.dot(X.T,data.T)-y_train),axis = 0)/15.
print (np.min(Z),np.max(Z))
#print data.shape
#Z = np.linalg.norm(data,ord = order,axis = 1)
print (Z.shape)
Z.shape=sz
plt.imshow(Z, interpolation='bilinear', origin='lower', extent=(-2,2,-2,2),alpha=0.3, vmin=-10, vmax=10)
plt.contour(XX,YY,Z,[0,1,2,3,4,5])
fig = plt.gcf()
fig.set_size_inches(9,9)
plt.plot(0,0,'ro')
plt.plot(np.array(wpath)[:,0],np.array(wpath)[:,1])
plt.plot(np.array(wpath)[-1,0],np.array(wpath)[-1,1],'co')
```

**EXERCISE** Change the `append` and use the optimal weights. Show the path of the optimal weights. How does it compare with the non-minibatch technique?

# Speeding up methods

These methods may suffer from a lack of speed convergence. There are several methods that allow to speed up the process.

## Momentum

In momentum we are trying to capture the notion of ‘inertia’ or that of ‘momentum’ (note that both are different concepts). The idea is that instead of moving in the direction of minus the gradient we will move in the direction of a combination of the previously accumulated directions combined with the new direction. This technique has a parameter $\gamma$ that is related to the decay or memory of the previous directions.

$$m^{(k+1)} = \gamma m^{(k)} - \alpha_k g^{(k)}$$

$$x^{(k+1)} = x^{(k)} + m^{(k+1)}$$

where $\gamma$ usually takes values around $0.9$.

**EXERCISE** Fill the code gap with the momentum.

```python
#your code
X = np.c_[np.ones((15,1)),x_train].T

import numpy as np

#initialize weights
w = np.zeros((2,1))
conv = []
wpath = []
wpath.append(w)
n_iters = 10000
Jopt = 9e16
wopt = w
mbatch_size=10
step = 0.0001
momentum = 0.95
m = 0.
for k in range(1,n_iters):
    #get a noisy gradient by just taking one data point at random and get the instantaneous gradient

    i = np.random.randint(0,X.shape[1],size=mbatch_size)
    x_sample = X[:,i]
    y_sample = y_train[i,:]
    g_tilde = -2*np.dot(x_sample,(y_sample-np.dot(x_sample.T,w)))

    ### PUT YOUR CODE FOR w UPDATE HERE

    #Keep the best weight up to that point
    J=np.sum((np.dot(X.T,w)-y_train)*(np.dot(X.T,w)-y_train))
    if J<Jopt:
        wopt = w
        Jopt = J
    conv.append(Jopt)
    wpath.append(w)
    #Project if needed
    ## Add your projection code here

plt.plot(conv)
```

```python

#order = np.Inf
order = 2
delta = 0.5
xx = np.arange(-2.0, 2.0, delta)
yy = np.arange(-2.0, 2.0, delta)
XX, YY = np.meshgrid(xx, yy)
Xf = XX.flatten()
Yf = YY.flatten()
sz=XX.shape
data = np.concatenate([Xf[:,np.newaxis],Yf[:,np.newaxis]],axis=1);
Z = np.sum((np.dot(X.T,data.T)-y_train)*(np.dot(X.T,data.T)-y_train),axis = 0)/15.
print (np.min(Z),np.max(Z))
#print data.shape
#Z = np.linalg.norm(data,ord = order,axis = 1)
print (Z.shape)
Z.shape=sz
plt.imshow(Z, interpolation='bilinear', origin='lower', extent=(-2,2,-2,2),alpha=0.3, vmin=-10, vmax=10)
plt.contour(XX,YY,Z,[0,1,2,3,4,5])
fig = plt.gcf()
fig.set_size_inches(9,9)
plt.plot(np.array(wpath)[0,0],np.array(wpath)[0,1],'ro')
plt.plot(np.array(wpath)[:,0],np.array(wpath)[:,1])
plt.plot(np.array(wpath)[-1,0],np.array(wpath)[-1,1],'co')
```

## Adagrad

A second family of methods tries to adapt the learning rate for each coordinate. This family of methods are of the form

$$x^{(k+1)} = x^{(k)} - \alpha_k D^{-1} g^{(k)}$$

where $D$ is a diagonal matrix. Observe that this independently adapts each coordinate. In particular in Adagrad, $D$ is defined as

$$D = \sqrt{\sum_{i=1}^k \text{diag}(g^{(i)})}$$

The algorithm can be written as follows,

$$D^{(k+1)} = D^{(k)} + \text{diag}(g^{(k)})^2$$

$$x^{(k+1)} = x^{(k)} - \alpha_k (D+\epsilon I)^{-1/2}g^{(k)}$$

where $\epsilon I$ is added for stability purposes. Note that if we implement this we do not build the diagonal matrix but operate element to element with the gradients.

```python
X = np.c_[np.ones((15,1)),x_train].T

import numpy as np

#initialize weights
#w = np.zeros((2,1))
w = -1.*np.ones((2,1))
conv = []
wpath = []
wpath.append(w)
n_iters = 100000
Jopt = 9e16
wopt = w
mbatch_size=10
step = 0.01
epsilon = 1e-16
D = np.zeros((2,1))
for k in range(1,n_iters):
    #get a noisy gradient by just taking one data point at random and get the instantaneous gradient

    i = np.random.randint(0,X.shape[1],size=mbatch_size)
    x_sample = X[:,i]
    y_sample = y_train[i,:]
    g_tilde = -2*np.dot(x_sample,(y_sample-np.dot(x_sample.T,w)))

    D = D + g_tilde*g_tilde

    w = w  - step * g_tilde / np.sqrt(D + epsilon)

    #Keep the best weight up to that point
    J=np.sum((np.dot(X.T,w)-y_train)*(np.dot(X.T,w)-y_train))
    if J<Jopt:
        wopt = w
        Jopt = J
    conv.append(Jopt)
    wpath.append(w)
    #Project if needed
    ## Add your projection code here

plt.plot(conv)
```

```python

#order = np.Inf
order = 2
delta = 0.5
xx = np.arange(-2.0, 2.0, delta)
yy = np.arange(-2.0, 2.0, delta)
XX, YY = np.meshgrid(xx, yy)
Xf = XX.flatten()
Yf = YY.flatten()
sz=XX.shape
data = np.concatenate([Xf[:,np.newaxis],Yf[:,np.newaxis]],axis=1);
Z = np.sum((np.dot(X.T,data.T)-y_train)*(np.dot(X.T,data.T)-y_train),axis = 0)/15.
print (np.min(Z),np.max(Z))
#print data.shape
#Z = np.linalg.norm(data,ord = order,axis = 1)
print (Z.shape)
Z.shape=sz
plt.imshow(Z, interpolation='bilinear', origin='lower', extent=(-2,2,-2,2),alpha=0.3, vmin=-10, vmax=10)
plt.contour(XX,YY,Z,[0,1,2,3,4,5])
fig = plt.gcf()
fig.set_size_inches(9,9)
plt.plot(np.array(wpath)[0,0],np.array(wpath)[0,1],'ro')
plt.plot(np.array(wpath)[:,0],np.array(wpath)[:,1])
plt.plot(np.array(wpath)[-1,0],np.array(wpath)[-1,1],'co')
```

## Adam

Adam combines both momentum and adagrad.

$$m^{(k+1)} = \gamma_1 m^{(k)} - (1-\gamma_1) g^{(k)}$$

$$D^{(k+1)} = \gamma_2 D^{(k)} + (1-\gamma_2)\text{diag}(g^{(k)})^2$$

$$x^{(k+1)} = x^{(k)} + \alpha_k (D^{(k+1)}+\epsilon I)^{-1/2}m^{(k+1)}$$

Usual values of $\gamma_1$ are $0.9$ and $\gamma_2$ are $0.999$, $\epsilon$ can take very small values.

**EXERCISE** Code the Adam according to the former equations.

```python
#your code
X = np.c_[np.ones((15,1)),x_train].T

import numpy as np

#initialize weights
w = np.zeros((2,1))
conv = []
wpath = []
wpath.append(w)
n_iters = 100000
Jopt = 9e16
wopt = w
mbatch_size=10
step = 0.0001
epsilon = 1e-16
D = w
m = w
g1 = 0.9
g2 = 0.999
for k in range(1,n_iters):
    #get a noisy gradient by just taking one data point at random and get the instantaneous gradient

    i = np.random.randint(0,X.shape[1],size=mbatch_size)
    x_sample = X[:,i]
    y_sample = y_train[i,:]
    g_tilde = -2*np.dot(x_sample,(y_sample-np.dot(x_sample.T,w)))

    #### YOUR ADAM CODE HERE

    #Keep the best weight up to that point
    J=np.sum((np.dot(X.T,w)-y_train)*(np.dot(X.T,w)-y_train))
    if J<Jopt:
        wopt = w
        Jopt = J
    conv.append(Jopt)
    wpath.append(w)
    #Project if needed
    ## Add your projection code here

plt.plot(conv)
```

```python

#order = np.Inf
order = 2
delta = 0.5
xx = np.arange(-2.0, 2.0, delta)
yy = np.arange(-2.0, 2.0, delta)
XX, YY = np.meshgrid(xx, yy)
Xf = XX.flatten()
Yf = YY.flatten()
sz=XX.shape
data = np.concatenate([Xf[:,np.newaxis],Yf[:,np.newaxis]],axis=1);
Z = np.sum((np.dot(X.T,data.T)-y_train)*(np.dot(X.T,data.T)-y_train),axis = 0)/15.
print (np.min(Z),np.max(Z))
#print data.shape
#Z = np.linalg.norm(data,ord = order,axis = 1)
print (Z.shape)
Z.shape=sz
plt.imshow(Z, interpolation='bilinear', origin='lower', extent=(-2,2,-2,2),alpha=0.3, vmin=-10, vmax=10)
plt.contour(XX,YY,Z,[0,1,2,3,4,5])
fig = plt.gcf()
fig.set_size_inches(9,9)
plt.plot(np.array(wpath)[0,0],np.array(wpath)[0,1],'ro')
plt.plot(np.array(wpath)[:,0],np.array(wpath)[:,1])
plt.plot(np.array(wpath)[-1,0],np.array(wpath)[-1,1],'co')
```

```python
X = np.c_[np.ones((15,1)),x_train,x_train**2,x_train**3,x_train**4,x_train**5,x_train**6,x_train**7,x_train**8,x_train**9,x_train**10].T
import numpy as np

#initialize weights
w = np.ones((11,1))
conv = []
wpath = []
wpath.append(w)
n_iters = 100000
Jopt = 9e16
wopt = w
mbatch_size=10
step = 0.001
epsilon = 1e-16
D = w
m = w
g1 = 0.9
g2 = 0.999
for k in range(1,n_iters):
    #get a noisy gradient by just taking one data point at random and get the instantaneous gradient

    i = np.random.randint(0,X.shape[1],size=mbatch_size)
    x_sample = X[:,i]
    y_sample = y_train[i,:]
    g_tilde = -2*np.dot(x_sample,(y_sample-np.dot(x_sample.T,w)))

    # UPDATE w

    #Keep the best weight up to that point
    J=np.sum((np.dot(X.T,w)-y_train)*(np.dot(X.T,w)-y_train))
    if J<Jopt:
        wopt = w
        Jopt = J
    conv.append(Jopt)
    wpath.append(w)
    #Project if needed
    ## Add your projection code here

plt.plot(conv)
```

```python
x_draw = np.linspace(0,1,100)

x_extended = np.c_[np.ones((100,1)),x_draw,x_draw**2,x_draw**3,x_draw**4,x_draw**5,x_draw**6,x_draw**7,x_draw**8,x_draw**9,x_draw**10].T

y_hat = np.dot(x_extended.T,wopt)

plt.plot(x_train,y_train,'ro')
plt.plot(x_draw,y_hat)
plt.plot(x,vals)
```

```python
w
```

    array([[1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.],
           [1.]])

**EXERCISE** Plot the convergence curves for all methods at the same time.

```python

```


## Why Metaprogramming?

- Metaprogramming is used to write code at parsing time in order to help the compiler achieve good performance
- Metaprogramming is used to write Domain-Specific Languages (DSLs) to make easy syntax for complicated functions

## Examples of Improving Performance

- `@fastmath 3*5`
- ParallelAccelerator's `@acc`
- `@simd`
- `@parallel`
- `Threads.@threads`
- Even DSLs will use symbolic mathematics to automatically simplify a problem for you!

## Examples of Domain-Specific Languages:

### JuliaStats

`~` for Formulas was actually in in-place macro!

### DifferentialEquations.jl

For differential equations packages, the standard way to define a differential equation like the Lorenz equation:

$$
\begin{align}
x\prime &= \sigma (y-x)\\
y\prime &= x (\rho - z) - y\\
z\prime &= xy - \beta z\\
\end{align}
$$

is the following:

```julia
f = (t,u,du) -> begin
 du[1] = 10.0(u[2]-u[1])
 du[2] = u[1]*(28.0-u[3]) - u[2]
 du[3] = u[1]*u[2] - (8/3)*u[3]
end
```

However, we can define a DSL to make this easier to do:

```julia
g = @ode_def begin
  dx = σ*(y-x)
  dy = x*(ρ-z) - y
  dz = x*y - β*z
end σ=>10. ρ=>28. β=>2.66
```

### JuMP.jl

JuMP has a DSL which allows you to define constrainted optimization problems without reference to the package or type of equation which is being solved. You can then swap out different optimization packages to use as the solver without care.

```julia
using JuMP

m = Model()
@variable(m, 0 <= x <= 2 )
@variable(m, 0 <= y <= 30 )

@objective(m, Max, 5x + 3*y )
@constraint(m, 1x + 5y <= 3.0 )

print(m)

status = solve(m)

println("Objective value: ", getobjectivevalue(m))
println("x = ", getvalue(x))
println("y = ", getvalue(y))
```

## Writing Your Own Macros

An easy way to write macros is the "QuoteLater" method:

- Write a macro, make it work (in the REPL)
- Later, quote it and interpolate in the value

The Julia manual is very complete on Metaprogramming: http://docs.julialang.org/en/release-0.5/manual/metaprogramming/

One fact that it misses: How to interpolate an expression as an expression (not as a value) into another expression.

```julia
ex_new = Meta.quot(ex)
quote
    still_expression = $(esc(ex_new))
end
```

### Metaprogramming Project

You have a way to compute least squares, how about letting it work like an `R` function? Define your own version `solve_least_squares(Y,X)` using your previous result, and define the `@~` macro so that way `solve_least_squares(Y~1+X1+X2+X4)` solves the least square problem on the data matrix `X` on its columns 1, 2, and 4.

Metaprogramming is hard the first few times you do it. You might want to work in pairs!

Hints:

- You cannot "get a character" from a symbol
- You can use `string` to change a symbol to a string
- You can `parse` a string to a number
- Check out the `args` field of an expression: it's an array that holds a bunch of goodies!

```julia
y = rand(10)
X = rand(10,4)
y~1+X1+X2+X4 # Make a tuple
solve_least_squares(y~1+X1+X2+X4) # Solve the least square problem on a tuple
```


>>> Work in Progress (Following are the lecture notes of Prof Andrew Ng - CS229 - Stanford. This is my interpretation of his excellent teaching and I take full responsibility of any misinterpretation/misinformation provided herein.)

## Lecture 17

#### Outline
- Reinforcement Learning
  - MDP (recap)
  - Value function
  - Value iteration/Policy iteration
  - Learning state transition probability/putting it all together

### Value function for policy $\pi$
- One of the challenges of finding optimal policy is if we have 11 states and 4 actions, there is a exponentially large number of possible policies $4^{11}$
- how to find the best policy
- we need to define 3 things
  - $V^{\pi}$ - For policy $\pi$, $V^{\pi}: S \mapsto R $ is s.t. $V^{\pi}(s)$ is the expected total payoff for starting in state s and executing $\pi$ (take actions according to policy $\pi$) is
    > $V^{\pi}(s) = E[R(s_{0}, a_{0}) + \gamma R(s_{1}, a_{1}) + \gamma^{2} R(s_{2}, a_{2}) + ... | \pi, s_{0}=s]$
    - $V^{\pi}$ is called "Value function for policy $\pi$"
  - $V^{*}$ - is the optimal value function
  - $\pi^{*}$ - is the optimal policy
  - +1/-1 - is called absorbing state. there is no more policy once it reaches to this state

$\tiny{\text{YouTube-Stanford-CS229-Andrew Ng}}$

#### Bellman's equation
- governs the value function

- The intution is - robot woke up at $s_{0}$ state, the reward you get is $R(s_{0})$, which is the _immediate reward_. Then it takes some action and ends up at state $s_{1}$, the _future reward_ you get is $\gamma R(s_{1})$. And then $\gamma R(s_{2})$ and so on. given that we execute the policy $\pi$ and we start off in this state $s_{0}$. This can be written as:

> $\begin{equation}\\
\begin{aligned}\\
V^{\pi}(s) &= E[R(s_{0}) + \gamma R(s_{1})+ \gamma^{2} R(s_{2}) + .. |\pi, s_{0}]\\
&= E[R(s_{0}) + \gamma \{ R(s_{1})+ \gamma^{1} R(s_{2}) + ..\} |\pi, s_{0}]\\
&= E[R(s_{0}) + \gamma V^{\pi}(s_{1}) |\pi, s_{0}]\\
\end{aligned}\\
\end{equation}\\
$

- the bellman's equation says that the expected total payoff you get if the robot wakes up at state s is the immediate reward plus gamma times the expected future rewards

- the mapping between these two equation will be:
> $s \rightarrow s_{0}$(current state), $s' \rightarrow s_{1}$(future state).
- So the equation with relevant mapping will be:
> $V^{\pi}(s) = E[R(s) + \gamma V^{\pi}(s')]$

- What is s' drawn from which distribution?
> $s' \sim P_{s \pi (s)}$
- in state s, take action $a=\pi(s)$. so the distribution from which s' is drawn from
> $s' \sim P_{s a}$

- so the bellman equation becomes:
> $V^{\pi}(s) = R(s) + \gamma\sum\limits_{s'\in S}P_{s\pi(s)}(s')V^{\pi}(s')$

- given $\pi$, this gives us a linear system of equation in terms of $V^{\pi}(s)$
- the unknown variables gives us a system of 11 linear equations with 11 unknowns
- can use linear algebra solver to solve this

$\tiny{\text{YouTube-Stanford-CS229-Andrew Ng}}$

### Optimal Value Function V*
- Look at all the possible combinatorial exponential policies for this MDP, and the max of all policies is the optimal value function
> $V^{*}(s) = \max\limits_{\pi}V^{\pi}(s)$

#### Bellman's equation
- there is a different version of Bellman's equation
> $V^{\pi}(s) = R(s) + \max\limits_{a}\gamma\sum\limits_{s'\in S}P_{sa}(s')V^{*}(s')$

### Optimal policy
- Lets say we have a way of how to calculate $V^{*}$

- $\pi^{*}$ is the optimal policy
> $\pi^{*} = \text{arg}\max\limits_{a}\sum\limits_{s'\in S}P_{sa}(s')V^{*}(s')$

$\tiny{\text{YouTube-Stanford-CS229-Andrew Ng}}$

- For every $\pi$, s; the optimal value for state S is equal to
> $V^{*}(s) = V^{\pi *}(s) \ge V^{\pi}(s)$

### Strategy for finding optimal policy
  - a) find $V^{*}$
  - b) use arg max equation to find $\pi^{*}$

#### Value Iteration
- Value iteration gives us the $V^{*}$. The algorithm is as follows. It uses synchronous or asynchronous update. synchronous update is the more common one, where you update all 11 values at the same time.

$\tiny{\text{YouTube-Stanford-CS229-Andrew Ng}}$

#### Policy Iteration

- the focus is policy rather than value
- solve for the value function for the policy $\pi$, which was linear system of equations $V^{\pi}$
- assume V is the optimal value function and update $\pi(s)$, using Bellman's equation
- iterate until it converges

$\tiny{\text{YouTube-Stanford-CS229-Andrew Ng}}$

#### Pros and cons
- policy iteration
  - is relatively easy for small state space
- value iteration
  - is relatively better for large state space

### State transition probability
- is not known in advance
- estimate $P_{sa}(s')$ from data
> $P_{sa}(s') = \frac{\text{# times took action a in state s and got to state s'}}{\text{# times took action a in state s}}$

$\tiny{\text{YouTube-Stanford-CS229-Andrew Ng}}$

#### Exploration vs Exploitation
- may converge to a local optima
- exploitation
  - how aggressive or how greedy you should be at taking actions to maximize your rewards
- exploration
  - the process of taking actions that may appear less optimal at the outset, maybe if it tries some new things it's never tried before, maybe it will find a new thing
- this is the exploitation/exploration trade-off

##### epsilon-greeedy
- 0.9 chance wrt $\pi$
- 0.1 chance act randomly

- Search for "intrinsic motivation"


```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
```

# Class 12: Introduction to the `linearsolve` Module

In general, dynamic stochastic general equilibrium (DSGE) models do not admit analytic (i.e., pencil-and-paper) solutions and they are time-consuming to work with. The `linearsolve` module approximates, solves, and simulates DSGE models and therefore makes DSGE models easier to use.

## Installing `linearsolve`

`linearsolve` is not included in the Anaconda Python installation and so before you can import it, you need to download and install the `linearsolve` package from PyPI, the Python Package Index. In Windows, open the Anaconda Prompt and in Mac, open the Terminal and run the following commmand:

    pip install linearsolve

You only have to install the package once.

```python
# Import the linearsolve under the 'ls' namespace
import linearsolve as ls
```

## Example: A One-Equation Model of TFP

Consider the following AR(1) specification for $\log$ TFP:

\begin{align}
\log A_{t+1} & = \rho \log A_t + \epsilon_{t+1},
\end{align}

where $\epsilon_{t+1} \sim \mathcal{N}(0,\sigma^2)$. The goal is to simulate

\begin{align}
\log (A_{t}/\bar{A}) & \approx \frac{A_t - \bar{A}}{\bar{A}},
\end{align}

where $\bar{A}$ is the nonstochastic (i.e., $\epsilon_t = 0$) steady state value of $A_t$. Let's compute the model simulation with `linearsolve`. To do this we need to do several things:

1. Create a Pandas series that stores the names of the parameters of the model.
2. Define a function that returns the equilibrium conditions of the model solved for zero.
3. Initialize an instance of the `linearsolve.model` class
4. Compute and input the steady state of the model.
5. Approximate and solve the model.
6. Compute simulations of the model.

Use the following values for the simulation:

| $$\rho$$ | $$\sigma$$ |
|----------|------------|
| 0.95     | 0.01       |

**Step 1:** Create a variable called `parameters` that stores parameter values as a Pandas Series.

```python
# Create a variable called 'parameters' that stores the model parameter values in a Pandas Series
parameters = pd.Series(dtype=float)
parameters['rho'] = .95

# Print stored parameter values
print(parameters)
```

    rho    0.95
    dtype: float64

```python
# Create a variable called 'sigma' that stores the value of sigma
sigma = 0.01
```

```python
# Create variable called 'var_names' that stores the variable names in a list with state variables ordered first
var_names = ['a']

# Create variable called 'shock_names' that stores an exogenous shock name for each state variable.
shock_names = ['e_a']
```

**Step 2:** Define a function called `equilibrium_equations` that evaluates the equilibrium
equations of model solved for zero. The function should accept three arguments:

1. `variables_forward`: Values of $t+1$-dated variables
2. `variables_current`: Values of $t$-dated variables
3. `parameters`: Pandas Series with the parameters for the model

The function should return a NumPy array of the model's equilibrium conditions solved for zero.

```python
# Define a function that evaluates the equilibrium conditions of the model solved for zero. PROVIDED
def equilibrium_equations(variables_forward,variables_current,parameters):

    # Create a variable called 'p' that storres the model's parameters. PROVIDED.
    p = parameters

    # Create variable called 'cur' that stores the values of current (date t) variables. PROVIDED.
    cur = variables_current

    # Create variable called 'fwd' that stores the values of one-period-ahead (date t+1) variables. PROVIDED.
    fwd = variables_forward

    # Create variable called 'tfp_proc' that returns the law of motion for TFP solved for zero (exogenous shock excluded)
    tfp_proc = p.rho*np.log(cur.a) - np.log(fwd.a)

    # Return equilibrium conditions stacked in a numpy array. PROVIDED.
    return np.array([
        tfp_proc
        ])
```

**Step 3:** Initialize a model instance using the `ls.model()` function. The

1. `equations`: Name of function that stores the equilibrium conditions
2. `n_states`: Number of *state* variables (i.e., variables that are *predetermined*)
3. `var_names`: List of the names of the endogenous variables
4. `shock_names`: List of the names of the exogenous shocks
5. `parameters`: Pandas Series of parameter values

```python
# Initialize the model into a variable named 'ar1_model' using the ls.model() function
ar1_model = ls.model(equations = equilibrium_equations,
                     n_states=1,
                     var_names=var_names,
                     shock_names=shock_names,
                     parameters = parameters)
```

**Step 4:** Set the steady state of the model. Either use the `.compute_ss()` method which requires an initial guess of what the steady state is. Or set the steady state `.ss` attribute directly.

```python
# Compute the steady state numerically using .compute_ss() method of ar1_model
ar1_model.compute_ss([1])
print(ar1_model.ss)
```

    a    1.0
    dtype: float64

```python
# Set the .ss attribute of ar1_model directly
ar1_model.ss = pd.Series({'a':1})
```

**Step 5:** Compute a transform the model into a log-linear approximation around the nonstochastic steady state. Then rewrite the equilibrium conditions so that all endogenous variables are expressed as linear functions of state variables and exogenous shocks.

```python
# Appproximate and solve using the .approximate_and_solve() method of ar1_model
ar1_model.approximate_and_solve()
```

**Step 6:** Simulate the model using one of the following methods:

1. `.impulse()`: Compute impulse responses to a one-time shock to each exogenous variables. Results stored in `.irs` attribute.
2. `.stoch_sim()`: Compute stochastic simulation. Results stored in `.simulated` attribute.

First, we'll compute an impulse response simulation. Let's consider the effect of a one time shock to $\epsilon$ of 0.01 in period 5. Simulate 41 periods.

```python
# Compute impulse response of a to a one-time shock to e_a
ar1_model.impulse(T=41,t0=5,shocks=[0.01])
```

The impulse response simulations are stored in the `.irs` attribute as a dictionary with keys equal to the names of the exogenous shocks.

```python
# Print ar1_model.irs
print(ar1_model.irs)
```

    {'e_a':      e_a         a
    0   0.00  0.000000
    1   0.00  0.000000
    2   0.00  0.000000
    3   0.00  0.000000
    4   0.00  0.000000
    5   0.01  0.010000
    6   0.00  0.009500
    7   0.00  0.009025
    8   0.00  0.008574
    9   0.00  0.008145
    10  0.00  0.007738
    11  0.00  0.007351
    12  0.00  0.006983
    13  0.00  0.006634
    14  0.00  0.006302
    15  0.00  0.005987
    16  0.00  0.005688
    17  0.00  0.005404
    18  0.00  0.005133
    19  0.00  0.004877
    20  0.00  0.004633
    21  0.00  0.004401
    22  0.00  0.004181
    23  0.00  0.003972
    24  0.00  0.003774
    25  0.00  0.003585
    26  0.00  0.003406
    27  0.00  0.003235
    28  0.00  0.003074
    29  0.00  0.002920
    30  0.00  0.002774
    31  0.00  0.002635
    32  0.00  0.002503
    33  0.00  0.002378
    34  0.00  0.002259
    35  0.00  0.002146
    36  0.00  0.002039
    37  0.00  0.001937
    38  0.00  0.001840
    39  0.00  0.001748
    40  0.00  0.001661}

Let's look at the first 10 rows of the `'e_a'` element of `ar1_model.irs`.

```python
# Print first 10 rows of the element in ar1_model.irs that corresponds to the shock to TFP
print(ar1_model.irs['e_a'].head())
```

       e_a    a
    0  0.0  0.0
    1  0.0  0.0
    2  0.0  0.0
    3  0.0  0.0
    4  0.0  0.0

```python
# Plot simulated impulse response to e_a
ar1_model.irs['e_a'][['e_a','a']].plot(lw=3,alpha=0.5,grid=True).legend(loc='upper right',ncol=2)
```

Next, we'll use the `.stoch_sim()` method to compute a stochastic simulation. The method takes arguments:

1. `seed`: Seed of NumPy RNG. (Optional)
2. `T`: Number of periods to simulate
3. `covMat`: Covariance matrix for exogenous shock process

The simulation should be for 201 periods.

```python
# Compute stochastic simulation
ar1_model.stoch_sim(seed=126,T=201,cov_mat= [[sigma**2]])

# Print the first 10 rows of `model.simulated`
print(ar1_model.simulated.head(10))
```

            e_a         a
    0 -0.006006  0.003344
    1  0.007683  0.010860
    2 -0.006501  0.003816
    3 -0.000440  0.003185
    4 -0.000819  0.002207
    5  0.003070  0.005167
    6  0.006283  0.011191
    7  0.001451  0.012083
    8  0.002981  0.014459
    9 -0.008517  0.005219

The stochastic simulations are stored in the `.stoch_sim` attribute as a Pandas `DataFrame`.

```python
# Plot the stochastic simulation
ar1_model.simulated[['e_a','a']].plot(lw=3,alpha=0.5,grid=True).legend(loc='upper right',ncol=2)
```

## Example 2: The Stochastic Solow Growth Model Revisited

Now consider the following system of equations:

\begin{align}
Y_t & = A_t K_t^{\alpha} \\
K_{t+1} & = sY_t + (1-\delta) K_t\\
\log A_{t+1} & = \rho \log A_t + \epsilon_{t+1}
\end{align}

where $\epsilon_{t+1} \sim \mathcal{N}(0,\sigma^2)$. Let's simulate the model with `linearsolve`. Before proceding, let's also go ahead and rewrite the model with all variables moved to the lefthand side of the equations:

\begin{align}
0 & = A_t K_t^{\alpha} - Y_t \\
0 & = sY_t + (1-\delta) K_t - K_{t+1}\\
0 & = \rho \log A_t + \epsilon_{t+1} - \log A_{t+1}
\end{align}

Capital and TFP are called *state variables* because they're $t+1$ values are predetermined. Output is called a *costate* or *control* variable. Note that the model as 3 endogenous variables with 2 state variables.

Use the following values for the simulation:

| $$\rho$$ | $$\sigma$$ | $$s$$ | $$\alpha$$ | $$\delta $$ | $$T$$  |
|----------|------------|-------|------------|-------------|--------|
| 0.75     | 0.006      | 0.1   | 0.35       |  0.025      | 201    |

### Initialization, Approximation, and Solution

The next several cells initialize the model in `linearsolve` and then approximate and solve it.

```python
# Create a variable called 'parameters' that stores the model parameter values in a Pandas Series
parameters = pd.Series(dtype=float)
parameters['rho'] = .75
parameters['alpha'] = 0.35
parameters['delta'] = 0.025
parameters['s'] = 0.1

# Print the model's parameters
print(parameters)
```

    rho      0.750
    alpha    0.350
    delta    0.025
    s        0.100
    dtype: float64

```python
# Create a variable called 'sigma' that stores the value of sigma
sigma = 0.006
```

```python
# Create variable called 'var_names' that stores the variable names in a list with state variables ordered first
var_names = ['a','k','y']

# Create variable called 'shock_names' that stores an exogenous shock name for each state variable.
shock_names = ['e_a','e_k']
```

```python
# Define a function that evaluates the equilibrium conditions of the model solved for zero. PROVIDED
def equilibrium_equations(variables_forward,variables_current,parameters):

    # Parameters. PROVIDED
    p = parameters

    # Current variables. PROVIDED
    cur = variables_current

    # Forward variables. PROVIDED
    fwd = variables_forward

    # Production function
    production_function = cur.a*cur.k**p.alpha - cur.y

    # Capital evolution
    capital_evolution = p.s*cur.a*cur.k**p.alpha + (1 - p.delta)*cur.k - fwd.k

    # Exogenous tfp
    tfp_process = p.rho*np.log(cur.a) - np.log(fwd.a)

    # Stack equilibrium conditions into a numpy array
    return np.array([
        production_function,
        capital_evolution,
        tfp_process
        ])
```

Next, initialize the model.

```python
# Initialize the model into a variable named 'solow_model'
solow_model = ls.model(equations = equilibrium_equations,
                 n_states=2,
                 var_names=var_names,
                 shock_names=shock_names,
                 parameters = parameters)
```

```python
# Compute the steady state numerically using .compute_ss() method of solow_model
guess = [1,4,1]
solow_model.compute_ss(guess)

# Print the computed steady state
print(solow_model.ss)
```

    a    1.000000
    k    8.438129
    y    2.109532
    dtype: float64

```python
# Find the log-linear approximation around the non-stochastic steady state and solve using .approximate_and_solve() method of solow_model
solow_model.approximate_and_solve()
```

### A Few Details About the Approximation (Optional)

The previous step constructs a log-linear approximation of the model and then solves for the endogenous variables as functions of the state variables and exogenous shocks only.

View the approximated model by calling the `.approximated()` method.

```python
# Print the log-linear approximation to the models's equilibrium conditions
print(solow_model.approximated())
```

    Log-linear equilibrium conditions:

                 0 = -2.1095·a[t]-0.7383·k[t]+2.1095·y[t]

    -8.4381·k[t+1] = -0.211·a[t]-8.301·k[t]

           -a[t+1] = -0.75·a[t]

Each variable represents the log-deviation from the steady state of the respective variable in our model. For example, the variable `y[t]` means $\log(Y_t) - \log(Y)$ in terms of the stochastic Solow model. But how do these equations relate to the original model?

The first equation appears to be:

\begin{align}
0 &= -2.1095\cdot a_t-0.7383\cdot k_t + 2.1095\cdot y_t
\end{align}

Note that dividing by 2.1095 and solving for $y_t$ yields:

\begin{align}
y_t &= a_t +0.3499\cdot k_t,
\end{align}

so the coefficient on $k_t$ appears to be close to $\alpha=0.35$. We can derive this linear equation directly. First, start with the production function:

\begin{align}
Y_t &= A_t K_t^{\alpha}.
\end{align}

Then divide both sides by steady state output:

\begin{align}
\frac{Y_t}{Y} &= \frac{A_t K_t^{\alpha}}{AK^{\alpha}} \, = \, \frac{A_t}{A}\frac{K_t^{\alpha}}{K^{\alpha}}.
\end{align}

Then, take the log of both sides:
\begin{align}
\log\left(\frac{Y_t}{Y} \right)&= \log\left(\frac{A_t}{A}\right) + \alpha\log\left(\frac{K_t}{K}\right).
\end{align}

finally, letting $y_t = \log(Y_t/Y)$, k_t = $\log(K_t/K)$, and $a_t = \log(A_t/A)$, we have:

\begin{align}
y_t &= a_t + \alpha k_t.
\end{align}

However, understanding this process isn't as important as being able to interpret the graphs and statistics that compute using the output of `linearsolve`.

### A Few Details About the Solution (Optional)

It's also worth seeing what it means for a model to be *solved*. After `linearsolve` computes the log-linear approximation to the model, it solves for each endogenous variable as a function of state variables only. View the solved model by calling the `.solved()` method.

```python
# Print the solved model
print(solow_model.solved())
```

    Solution to the log-linear system:

    a[t+1] = 0.75·a[t]+e_a[t+1]

    k[t+1] = 0.025·a[t]+0.9837·k[t]+e_k[t+1]

      y[t] = a[t]+0.35·k[t]

### Impulse Responses

Compute a 41 period impulse responses of the model's variables to a 0.01 unit shock to TFP in period 5.

```python
# Compute impulse responses
solow_model.impulse(T=41,t0=5,shocks=[0.01,0])

# Print the first 10 rows of the computed impulse responses.
print(solow_model.irs['e_a'].head(10))
```

        e_a         a         k         y
    0  0.00  0.000000  0.000000  0.000000
    1  0.00  0.000000  0.000000  0.000000
    2  0.00  0.000000  0.000000  0.000000
    3  0.00  0.000000  0.000000  0.000000
    4  0.00  0.000000  0.000000  0.000000
    5  0.01  0.010000  0.000000  0.010000
    6  0.00  0.007500  0.000250  0.007588
    7  0.00  0.005625  0.000433  0.005777
    8  0.00  0.004219  0.000567  0.004417
    9  0.00  0.003164  0.000663  0.003396

```python
# Plot the computed impulse responses to a TFP shock
fig = plt.figure(figsize=(12,4))

ax1 = fig.add_subplot(1,2,1)
solow_model.irs['e_a'][['a','y','k']].plot(lw=3,alpha=0.5,grid=True,ax = ax1).legend(loc='upper right',ncol=2)
ax1.set_title('Endogenous Variables')

ax2 = fig.add_subplot(1,2,2)
solow_model.irs['e_a'][['e_a']].plot(lw=3,alpha=0.5,grid=True,ax = ax2).legend(loc='upper right',ncol=2)
ax2.set_title('TFP Shock Process')

fig.tight_layout()
```

### Stochastic Simulation

Compute a 201 period stochastic simulation of the model's variables. Set the variance of $\epsilon_t$ to $\sigma^2$ and the variance of the shock to capital to 0 so that the covariance matrix for the shock process is:

\begin{align}
\text{Covariance matrix} & = \left[\begin{array}{cc} \sigma^2 & 0\\ 0 & 0\end{array} \right]
\end{align}

```python
# Compute stochastic simulation and print the simulated values.
solow_model.stoch_sim(seed=126,T=201,cov_mat= [[sigma**2,0],[0,0]])

# Print first 10 rows of model.simulated
print(solow_model.simulated.head(10))
```

            e_a  e_k         a         k         y
    0  0.001484  0.0  0.003862  0.000649  0.004089
    1  0.001427  0.0  0.004323  0.000735  0.004580
    2  0.003155  0.0  0.006397  0.000831  0.006688
    3  0.011305  0.0  0.016103  0.000978  0.016445
    4 -0.001778  0.0  0.010299  0.001364  0.010777
    5 -0.013211  0.0 -0.005486  0.001600 -0.004927
    6  0.004054  0.0 -0.000060  0.001437  0.000442
    7  0.006153  0.0  0.006108  0.001412  0.006602
    8 -0.001336  0.0  0.003245  0.001541  0.003785
    9 -0.006801  0.0 -0.004367  0.001598 -0.003808

```python
# Plot the computed stochastic simulation
fig = plt.figure(figsize=(12,4))

ax1 = fig.add_subplot(1,2,1)
solow_model.simulated[['a','y','k']].plot(lw=3,alpha=0.5,grid=True,ax = ax1).legend(loc='upper right',ncol=3)
ax1.set_title('Endogenous Variables')

ax2 = fig.add_subplot(1,2,2)
solow_model.simulated[['e_a']].plot(lw=3,alpha=0.5,grid=True,ax = ax2).legend(loc='upper right',ncol=2)
ax2.set_title('TFP Shock Process')

fig.tight_layout()
```

```python
# Compute standard deviations of simulated TFP, output, and capital
solow_model.simulated[['a','y','k']].std()
```

    a    0.009478
    y    0.009484
    k    0.001385
    dtype: float64

```python
# Compute correlation coefficients of simulated TFP, output, and capital
solow_model.simulated[['a','y','k']].corr()
```

      a
      y
      k

      a
      1.000000
      0.998694
      -0.012365

      y
      0.998694
      1.000000
      0.038745

      k
      -0.012365
      0.038745
      1.000000


# Proyecto 2

### En la primera parte, vamos a exportar la paquetería que se necesitá para la resolución del problema, para esto, primero se va a revisar que los datos en el archivo estén todos correctos. Vamos a leerlos y, posteriormente, vamos a identificar posibles problemas en estos

```python
# Used to read the data file
import pandas as pd

# Used in the definition of the model and its initial condition
import numpy as np

# Used for plots other than those generated by the parameter estimation module
import matplotlib.pyplot as plt
%matplotlib inline
import PDEparams as pde
```

```python
df = pd.read_csv('CoV2019.csv')
```

```python
df.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 41 entries, 0 to 40
    Data columns (total 8 columns):
    Date of report    41 non-null object
    Days              41 non-null int64
    Total             41 non-null int64
    China             41 non-null int64
    Death China       41 non-null int64
    Other             41 non-null int64
    Death Outside     41 non-null int64
    Death Globally    41 non-null int64
    dtypes: int64(7), object(1)
    memory usage: 2.7+ KB

```python
df.head()
```

      Date of report
      Days
      Total
      China
      Death China
      Other
      Death Outside
      Death Globally

      0
      01/20/20
      21
      282
      278
      6
      4
      0
      6

      1
      01/21/20
      22
      346
      341
      6
      4
      0
      6

      2
      01/23/20
      24
      581
      571
      17
      7
      0
      17

      3
      01/24/20
      25
      846
      830
      25
      11
      0
      25

      4
      01/25/20
      26
      1320
      1297
      41
      23
      0
      41

Los archivos en primera instancia se ven bastante bien, por lo cual, ahora se va a utilizar el módelo ISR para este sistema, el cual viene dada por:

$$\begin{align}
\frac{\mathrm{d} S}{\mathrm{d} t} &= -\beta\, S\, I\,\\
\frac{\mathrm{d} I}{\mathrm{d} t} &= +\beta\,S\,I-\gamma\,I\\
\frac{\mathrm{d} R}{\mathrm{d} t} &= \gamma I
\end{align}$$

donde $S$ es la población susceptible, $I$ es la población infectada y $R$ es la población inmune o fallecida.

En este caso, dado que solo conocemos la población fallecida y no inmune, entonces, solo se cuentan con los datos de la población infectada por día, la cual, viene dada por el feature de "China" en nuestros datos, es decir, "China" corresponde a $I$ en nuestro modelo.

```python
df['I']=''
df['I'].iloc[0]=df['China'].iloc[0]
for i in range(1,41):
    df['I'].iloc[i]=df['China'].iloc[i]-df['China'].iloc[i-1]
df['I']=df['I']/79579
df
```

      Date of report
      Days
      Total
      China
      Death China
      Other
      Death Outside
      Death Globally
      I

      0
      01/20/20
      21
      282
      278
      6
      4
      0
      6
      0.003493

      1
      01/21/20
      22
      346
      341
      6
      4
      0
      6
      0.000792

      2
      01/23/20
      24
      581
      571
      17
      7
      0
      17
      0.002890

      3
      01/24/20
      25
      846
      830
      25
      11
      0
      25
      0.003255

      4
      01/25/20
      26
      1320
      1297
      41
      23
      0
      41
      0.005868

      5
      01/26/20
      27
      2014
      1985
      56
      29
      0
      56
      0.008645

      6
      01/27/20
      28
      2798
      2761
      80
      37
      0
      80
      0.009751

      7
      01/28/20
      29
      4593
      4537
      106
      56
      0
      106
      0.022317

      8
      01/29/20
      30
      6065
      5997
      132
      68
      0
      132
      0.018347

      9
      01/30/20
      31
      7818
      7736
      170
      82
      0
      170
      0.021852

      10
      01/31/20
      32
      9826
      9720
      213
      106
      0
      213
      0.024931

      11
      02/01/2020
      33
      11953
      11821
      259
      132
      0
      259
      0.026401

      12
      02/02/2020
      34
      14557
      14411
      304
      146
      1
      305
      0.032546

      13
      02/03/2020
      35
      17391
      17238
      361
      153
      1
      362
      0.035524

      14
      02/04/2020
      36
      20630
      20471
      425
      159
      1
      426
      0.040626

      15
      02/05/2020
      37
      24554
      24363
      491
      191
      1
      492
      0.048907

      16
      02/06/2020
      38
      28276
      28060
      564
      216
      1
      565
      0.046457

      17
      02/07/2020
      39
      31481
      31211
      637
      270
      1
      638
      0.039596

      18
      02/08/2020
      40
      34886
      34598
      723
      288
      1
      724
      0.042561

      19
      02/09/2020
      41
      37558
      37251
      812
      307
      1
      813
      0.033338

      20
      02/10/2020
      42
      40554
      40235
      909
      319
      1
      910
      0.037497

      21
      02/11/2020
      43
      43103
      42708
      1017
      395
      1
      1018
      0.031076

      22
      02/12/2020
      44
      45171
      44730
      1114
      441
      1
      1115
      0.025409

      23
      02/13/20
      45
      46997
      46550
      1368
      447
      1
      1369
      0.022870

      24
      02/14/20
      46
      49053
      48548
      1381
      505
      2
      1383
      0.025107

      25
      02/15/20
      47
      50580
      50054
      1524
      526
      2
      1526
      0.018925

      26
      02/16/20
      48
      51857
      51174
      1666
      683
      3
      1669
      0.014074

      27
      02/17/20
      49
      71429
      70635
      1772
      794
      3
      1775
      0.244549

      28
      02/18/20
      50
      73332
      72528
      1870
      804
      3
      1873
      0.023788

      29
      02/19/20
      51
      75204
      74280
      2006
      924
      3
      2009
      0.022016

      30
      02/20/20
      52
      75748
      74675
      2121
      1073
      8
      2129
      0.004964

      31
      02/21/20
      53
      76769
      75569
      2239
      1200
      8
      2247
      0.011234

      32
      02/22/20
      54
      77794
      76392
      2348
      1402
      11
      2359
      0.010342

      33
      02/23/20
      55
      78811
      77042
      2445
      1769
      17
      2462
      0.008168

      34
      02/24/20
      56
      79331
      77262
      2595
      2069
      23
      2618
      0.002765

      35
      02/25/20
      57
      80239
      77780
      2666
      2459
      34
      2700
      0.006509

      36
      02/26/20
      58
      81109
      78191
      2718
      2918
      44
      2762
      0.005165

      37
      02/27/20
      59
      82294
      78630
      2747
      3664
      57
      2804
      0.005517

      38
      02/28/20
      60
      83652
      78961
      2791
      4691
      67
      2858
      0.004159

      39
      02/29/20
      61
      85403
      79394
      2838
      6009
      86
      2924
      0.005441

      40
      03/01/2020
      62
      87137
      79579
      2873
      7169
      104
      2977
      0.002325

Ahora, se va a definir la función del modelo ISR y las condiciones iniciales para $I$, $S$ y $R$.

```python
def ISR(z,t,b,g):
    S , I , R = z
    return [-b*S*I, (b*S*I)-g*I, g*I]
```

```python
def S_initial():
    return 1
def R_initial():
    return 0
def I_initial():
    return 0.003493
```

```python
df1=pd.DataFrame(data={'0': df['Days'],'2':df['I']})
df1
```

      0
      2

      0
      21
      0.003493

      1
      22
      0.000792

      2
      24
      0.002890

      3
      25
      0.003255

      4
      26
      0.005868

      5
      27
      0.008645

      6
      28
      0.009751

      7
      29
      0.022317

      8
      30
      0.018347

      9
      31
      0.021852

      10
      32
      0.024931

      11
      33
      0.026401

      12
      34
      0.032546

      13
      35
      0.035524

      14
      36
      0.040626

      15
      37
      0.048907

      16
      38
      0.046457

      17
      39
      0.039596

      18
      40
      0.042561

      19
      41
      0.033338

      20
      42
      0.037497

      21
      43
      0.031076

      22
      44
      0.025409

      23
      45
      0.022870

      24
      46
      0.025107

      25
      47
      0.018925

      26
      48
      0.014074

      27
      49
      0.244549

      28
      50
      0.023788

      29
      51
      0.022016

      30
      52
      0.004964

      31
      53
      0.011234

      32
      54
      0.010342

      33
      55
      0.008168

      34
      56
      0.002765

      35
      57
      0.006509

      36
      58
      0.005165

      37
      59
      0.005517

      38
      60
      0.004159

      39
      61
      0.005441

      40
      62
      0.002325

Ahora, se va a ajustar el sistema de ecuaciones diferenciales a los datos

```python
my_model = pde.PDEmodel(df1, ISR, [S_initial,I_initial,R_initial], bounds=[(0.01,6),(0.01,6)],
                        param_names=[r'$b$', r'$g$'], nvars=3, ndims=0, nreplicates=1, obsidx=1, outfunc=None)
```

```python
my_model.initial_condition
```

    array([1.      , 0.003493, 0.      ])

Ahora, obtenemos $\beta$ y $\gamma$.

```python
my_model.fit()
```

            $b$       $g$
    0  0.368779  0.282126

```python
my_model.best_params
```

      $b$
      $g$

      0
      0.368779
      0.282126

```python
my_model.best_error
```

    0.0012824810484898903

Veamos ahora la identificabilidad de los parametros, para esto, calculamos el likelihood de los parámetros

```python
my_model.likelihood_profiles()
```

    HBox(children=(HTML(value='parameters'), FloatProgress(value=0.0, max=2.0), HTML(value='')))

    HBox(children=(HTML(value='values within parameters'), FloatProgress(value=0.0), HTML(value='')))

    HBox(children=(HTML(value='values within parameters'), FloatProgress(value=0.0), HTML(value='')))

```python
my_model.result_profiles
```

      parameter
      value
      error

      0
      $b$
      0.010000
      0.001998

      1
      $b$
      0.070505
      0.001532

      2
      $b$
      0.131010
      0.001510

      3
      $b$
      0.191515
      0.001472

      4
      $b$
      0.252020
      0.001409

      ...
      ...
      ...
      ...

      195
      $g$
      5.757980
      0.001999

      196
      $g$
      5.818485
      0.001999

      197
      $g$
      5.878990
      0.001999

      198
      $g$
      5.939495
      0.001999

      199
      $g$
      6.000000
      0.001999

200 rows × 3 columns

```python
my_model.plot_profiles()
```

Como observamos, hay identificabilidad de los parámetros, ya que, existe un mínimo para $\beta$ y $\gamma$.

```python

```