# Implementing Walsh and Haar Transforms Using Python

## Table of Contents
* [Walsh Transform](#Walsh)
    * [Introduction](#WalshIntroduction)
    * [Python Implementation](#WalshImplementation)
    * [Testing](#WalshTesting)
* [Haar Transform](#Haar)
    * [Introduction](#HaarIntroduction)
    * [Python Implementation](#HaarImplementation)
    * [Testing](#HaarTesting)

## The Walsh Transform

### Introduction

The Walsh Transform is one way of transforming a signal/image from a space-domain to its corresponding frequency-domain. The DFT is a complex, sinusoidal transform while DCT is a transform where the imaginary values are eliminated and the real ones are retained. In the case of Walsh, it is a square waveform transform. In addition, the base/kernel functions in the Walsh tranformation are orthogonal, orthonormal and symmetric. Those in DFT are also symmetric. The forward walsh transform is as follows (Assume that N = $2^n$):

$
\begin{align}
H(u,v) = \frac{1}{N}\sum_{x=0}^{N-1}\sum_{y=0}^{N-1}f(x,y) (-1)^{ \beta } \;\\ where \; u,v=0,1,2,...N-1 \\
\beta = \sum_{i=0}^{n-1}b_i(x)b_i(u) + \sum_{i=0}^{n-1}b_i(y)b_i(v)
\end{align}
$

Since the kernels of the walsh transform are real and orthogonal, they are separable. The forward transform can be rewritten as:

$H(u,v) = k.f(x,y).k $

$ Where \; u,v=0,1,2,...N-1 \; and \; k = \sum_{i=0}^{n-1}b_i(x)b_i(u) $

Similarly, the inverse walsh transform will look as follows:

$f(x,y) = k.H(u,v).k $

This is exactly the same as that of the forward transformation as the kernels are real and symmetric.

### Python Implementation

First of all, let's import the necessary python libraries

```python
import numpy as np

import matplotlib.pyplot as plt
%matplotlib inline

#import matplotlib.image as img
import PIL.Image as Image

import math
import cmath

import time

import csv

from numpy import binary_repr

```

Now let's start with creating common image functions.

```python
def generateImagesWithResizedWhite(imge):
    """
    Generates images with the same size as the original but with a resized white part of them.
    """

    N = imge.shape[0]

    imges = []
    i = N/2
    while i >= 4:
        j = (N - i)/2

        #Starting and ending indices for the white part.
        indx1 = j
        indx2 = j+i

        #Draw the image.
        imgeNew = np.zeros([N, N],dtype=int)
        imgeNew[indx1:indx2, indx1:indx2] = np.ones([i, i], dtype=int)*255

        #Add the image to the list.
        imges.append(imgeNew)

        i = i/2

    return imges

def generateBlackAndWhiteSquareImage(imgSize):
    """
    Generates a square-sized black and white image with a given input size.

    Parameters
    ----------
    imgSize : int
        Input number that stores the dimension of the square image to be generated.

    Returns
    -------
    imge : ndarray
        The generated black and white square image.
    """

    #Creating a matrix with a given size where all the stored values are only zeros (for initialization)
    imge = np.zeros([imgSize, imgSize], dtype=int)

    #Starting and ending indices of the white part of the image.
    ind1 = imgSize/4
    ind2 = ind1 + (imgSize/2)

    #Make a part of the image as white (255)
    imge[ind1:ind2, ind1:ind2] = np.ones([imgSize/2, imgSize/2], dtype=int)*255

    #return the resulting image
    return imge

def generateImages(imgSizes=[128, 64, 32, 16, 8]):
    """
    Generates images of different sizes."""
    #Create an empty list of images to save the generated images with different sizes.
    images = []

    #Generate the first and biggest image
    imge = generateBlackAndWhiteSquareImage(imgSizes[0])

    #Add to the images list
    images.append(imge)

    #Generate the resized and smaller images with different sizes.
    for i in range(1, len(imgSizes)):
        size = imgSizes[i]
        images.append(resizeImage(imge, size))

    return images

def resizeImage(imge, newSize):
    """
    Reduces the size of the given image.

    Parameters
    ----------
    imge : ndarray
        Input array that stores the image to be resized.

    Returns
    -------
    newSize : int
        The size of the newly generated image.
    """

    #Compute the size of the original image (in this case, only # of rows as it is square)
    N = imge.shape[0]

    #The ratio of the original image as compared to the new one.
    stepSize = N/newSize

    #Creating a new matrix (image) with a black color (values of zero)
    newImge = np.zeros([N/stepSize, N/stepSize])

    #Average the adjacent four pixel values to compute the new intensity value for the new image.
    for i in xrange(0, N, stepSize):
        for j in xrange(0, N, stepSize):
            newImge[i/stepSize, j/stepSize] = np.mean(imge[i:i+stepSize, j:j+stepSize])

    #Return the new image
    return newImge
```

Next, we are going to implement the walsh transform:

```python
class Walsh(object):
    """
    This class Walsh implements all the procedures for transforming a given 2D digital image
    into its corresponding frequency-domain image (Walsh Transform)
    """

    @classmethod
    def __computeBeta(self, u, x, n):
        uBin = binary_repr(u, width=n)
        xBin = binary_repr(x, width=n)
        beta = 0
        for i in xrange(n):
            beta += (int(xBin[i])*int(uBin[i]))

        return beta

    #Compute walsh kernel (there is only a single kernel for forward and inverse transform
    #as it is both orthogonal and symmetric).
    @classmethod
    def computeKernel(self, N):
        """
        Computes/generates the walsh kernel function.

        Parameters
        ----------
        N : int
            Size of the kernel to be generated.

        Returns
        -------
        kernel : ndarray
            The generated kernel as a matrix.
        """

        #Initialize the kernel
        kernel = np.zeros([N, N])
        #Compute each value of the kernel...
        n = int(math.log(N, 2))
        for u in xrange(N):
            for x in xrange(N):
                beta = Walsh.__computeBeta(u, x, n)
                kernel[u, x] = (-1)**beta

        #To make the kernel orthonormal, we can divide it by sqrt(N)
        #kernel /= math.sqrt(N)

        #Return the resulting kernel
        return kernel

    @classmethod
    def computeForward2DWalshTransform(self, imge):
        """
        Computes/generates the 2D Walsh transform.

        Parameters
        ----------
        imge : ndarray
            The input image to be transformed.

        Returns
        -------
        final2DWalsh : ndarray
            The transformed image.
        """

        N = imge.shape[0]
        kernel = Walsh.computeKernel(N)

        imge1DWalsh = np.dot(kernel, imge)
        final2DWalsh = np.dot(imge1DWalsh, kernel)

        return final2DWalsh/N

    @classmethod
    def computeInverse2DWalshTransform(self, imgeWalsh):
        """
        Computes/generates the inverse of 2D Walsh transform.

        Parameters
        ----------
        imgeWalsh : ndarray
            The Walsh transformed image.

        Returns
        -------
        imgeInverse : ndarray
            The inverse of the transformed image.
        """

        N = imgeWalsh.shape[0]
        kernel = Walsh.computeKernel(N)

        imge1DInverse = np.dot(kernel, imgeWalsh)
        imgeInverse = np.dot(imge1DInverse, kernel)

        return imgeInverse/N

```

### Testing the Code

Let's try to compute the 4x4 walsh kernel.

```python
Walsh.computeKernel(4)
```

    array([[ 1.,  1.,  1.,  1.],
           [ 1., -1.,  1., -1.],
           [ 1.,  1., -1., -1.],
           [ 1., -1., -1.,  1.]])

Note: if we want to make the kernel orthonormal, we divide it by $\sqrt N$. Now, we will compute the walsh transform of a given image.  First we will read an image from a file.

```python
#Read an image file
imgeCameraman = Image.open("Images/cameraman.tif") # open an image

#Convert the image file to a matrix
imgeCameraman = np.array(imgeCameraman)
```

```python
imgeWalsh = Walsh.computeForward2DWalshTransform(imgeCameraman)
```

The inverse of the transformed image can also be computed as follows:

```python
imgeInverse = Walsh.computeInverse2DWalshTransform(imgeWalsh)
```

The minimum and maximum values of the transformation are:

```python
np.min(np.absolute(imgeWalsh)), np.max(np.absolute(imgeWalsh))
```

    (0.0, 60576.76953125)

We can visualize the original, the transformed and inverse transformed images as follows:

```python
fig, axarr = plt.subplots(1, 3, figsize=(10,7))

axarr[0].imshow(imgeCameraman, cmap=plt.get_cmap('gray'))
axarr[0].set_title('Original Image')

axarr[1].imshow(np.absolute(imgeWalsh), cmap=plt.get_cmap('gray'))
axarr[1].set_title('Forward Transformed Image')

axarr[2].imshow(imgeInverse, cmap=plt.get_cmap('gray'))
axarr[2].set_title("Inverse Transformed Image")
plt.show()
```

Now, let's compute the running time of Walsh tranform using images with different size

```python
#Generate images
imgSizes = [128, 64, 32, 16, 8]
images = generateImages(imgSizes)
```

```python
# A list that stores the running time of the DCT algorithm for images with different size.
runningTimeWalsh = []

#For each image...
for i, imge in enumerate(images):

    #Compute the image size
    N = imge.shape[0]

    print "Computing for ", N, "x", N, "image..."

    #Save the starting time.
    startTime = time.time()

    #Compute the DCT of the image.
    walshImge = Walsh.computeForward2DWalshTransform(imge)

    #Save the running time
    runningTimeWalsh.append((time.time() - startTime)/60.0)
```

    Computing for  128 x 128 image...
    Computing for  64 x 64 image...
    Computing for  32 x 32 image...
    Computing for  16 x 16 image...
    Computing for  8 x 8 image...

```python
result = zip(imgSizes, runningTimeWalsh)
np.savetxt("RunningTimes/runningTimeWalsh.csv", np.array(result), delimiter=',')
```

The running time will be visualized and compared with the other transformation methods in the next section. As it will be shown, Walsh is faster than DFT as it is only a square waveform where DFT is a complex, sinusoidal transformation.

## The Haar Transform

### Introduction

The Haar transform is also one type of the transformation methods. In contrast to Walsh, the kernel functions in Haar are not symmetric. So we need to transpose the first kernel when finding the second one.

The forward transform is computed as follows:

$ H(0, 0, x) = \frac{1}{N} $

$
H(r, m, x) = \left\{
  \begin{array}{l l}
   \frac{2^{r/2}}{\sqrt N} & \quad \text{$ \frac{m-1}{2^r} \leq x \leq \frac{m-1/2}{2^r}$ } \\
   -\frac{2^{r/2}}{\sqrt N} & \quad \text{$ \frac{m-1/2}{2^r} \leq x \leq \frac{m}{2^r}$ }\\
   0 & \quad \text{otherwise}
  \end{array} \right.
$

Where $0 \leq x < 1$, $0 \leq r < log_2 N $ and $1 \leq m \leq 2^r$

### Python Implementation

It's implementation is as follows:

```python
class Haar(object):
    """
    This class Haar implements all the procedures for transforming a given 2D digital image
    into its corresponding frequency-domain image (Haar Transform)
    """

    #Compute the Haar kernel.
    @classmethod
    def computeKernel(self, N):
        """
        Computes/generates the haar kernel function.

        Parameters
        ----------
        N : int
            Size of the kernel to be generated.

        Returns
        -------
        kernel : ndarray
            The generated kernel as a matrix.
        """

        i = 0
        kernel = np.zeros([N, N])
        n = int(math.log(N, 2))

        #Fill for the first row of the kernel
        for j in xrange(N):
            kernel[i, j] = 1.0/math.sqrt(N)

        # For the other rows of the kernel....
        i += 1
        for r in xrange(n):
             for m in xrange(1, (2**r)+1):
                j=0
                for x in np.arange(0, 1, 1.0/N):
                    if (x >= (m-1.0)/(2**r)) and (x < (m-0.5)/(2**r)):
                        kernel[i, j] = (2.0**(r/2.0))/math.sqrt(N)
                    elif (x >= (m-0.5)/(2**r)) and (x < m/(2.0**r)):
                        kernel[i, j] = -(2.0**(r/2.0))/math.sqrt(N)
                    else:
                        kernel[i, j] = 0
                    j += 1
                i += 1
        return kernel

    @classmethod
    def computeForward2DHaarTransform(self, imge):
        """
        Computes/generates the 2D Haar transform.

        Parameters
        ----------
        imge : ndarray
            The input image to be transformed.

        Returns
        -------
        final2DHaar : ndarray
            The transformed image.
        """

        N = imge.shape[0]
        kernel = Haar.computeKernel(N)

        imge1DHaar = np.dot(kernel, imge)

        #Transpose the kernel as it is not symmetric
        final2DHaar = np.dot(imge1DHaar, kernel.T)

        return final2DHaar/N

    @classmethod
    def computeInverse2DHaarTransform(self, imgeHaar):
        """
        Computes/generates the inverse of 2D Haar transform.

        Parameters
        ----------
        imgeHaar : ndarray
            The Haar transformed image.

        Returns
        -------
        imgeInverse : ndarray
            The inverse of the transformed image.
        """

        N = imgeHaar.shape[0]
        kernel = Haar.computeKernel(N)

        imge1DInverse = np.dot(kernel.T, imgeHaar)
        imgeInverse = np.dot(imge1DInverse, kernel)

        return imgeInverse/N

```

### Testing the Code

Now let's try the 4x4 haar kernel:

```python
xKernel = Haar.computeKernel(4)
xKernel
```

    array([[ 0.5       ,  0.5       ,  0.5       ,  0.5       ],
           [ 0.5       ,  0.5       , -0.5       , -0.5       ],
           [ 0.70710678, -0.70710678,  0.        ,  0.        ],
           [ 0.        ,  0.        ,  0.70710678, -0.70710678]])

Now, we will compute the haar transform of a given image.

```python
imgeHaar = Haar.computeForward2DHaarTransform(imgeCameraman)
```

Now, let's return back to its original form by using the inverse Haar transform

```python
imgeHaarInverse = Haar.computeInverse2DHaarTransform(imgeHaar)
```

The range of the results of the Haar transform is:

```python
np.min(np.absolute(imgeHaar)), np.max(np.absolute(imgeHaar))
```

    (0.0, 118.31400299072266)

We can now visualize the original, forward and inverse transformed images as follows:

```python
fig, axarr = plt.subplots(1, 3, figsize=(10,7))

axarr[0].imshow(imgeCameraman, cmap=plt.get_cmap('gray'))
axarr[0].set_title('Original Image')

axarr[1].imshow(np.absolute(imgeHaar), cmap=plt.get_cmap('gray'))
axarr[1].set_title('Forward Transformed Image')

axarr[2].imshow(imgeHaarInverse, cmap=plt.get_cmap('gray'))
axarr[2].set_title("Inverse Transformed Image")
plt.show()
```

Now, we will compute the running time of Haar Transform using images with different sizes:

```python
#Generate images
imgSizes = [128, 64, 32, 16, 8]
images = generateImages(imgSizes)

# A list that stores the running time of the DCT algorithm for images with different size.
runningTimeHaar = []

#For each image...
for i, imge in enumerate(images):

    #Compute the image size
    N = imge.shape[0]

    print "Computing for ", N, "x", N, "image..."

    #Save the starting time.
    startTime = time.time()

    #Compute the Haar of the image.
    haarImge = Haar.computeForward2DHaarTransform(imge)

    #Save the running time
    runningTimeHaar.append((time.time() - startTime)/60.0)

result = zip(imgSizes, runningTimeHaar)
np.savetxt("RunningTimes/runningTimeHaar.csv", np.array(result), delimiter=',')
```

    Computing for  128 x 128 image...
    Computing for  64 x 64 image...
    Computing for  32 x 32 image...
    Computing for  16 x 16 image...
    Computing for  8 x 8 image...

```python
#Load the running time for DFT, Walsh, Haar Transforms
runningTimeDFT = np.loadtxt("RunningTimes/runningTimeDFT.csv", delimiter =',')
runningTimeWalsh = np.loadtxt("RunningTimes/runningTimeWalsh.csv", delimiter =',')
runningTimeHaar = np.loadtxt("RunningTimes/runningTimeHaar.csv", delimiter =',')
```

```python
#Plot the running times
plt.plot(xrange(runningTimeDFT.shape[0]), runningTimeDFT[:,1], '-d')
plt.hold
plt.plot(xrange(runningTimeWalsh.shape[0]), runningTimeWalsh[:,1], '-d')
plt.plot(xrange(runningTimeHaar.shape[0]), runningTimeHaar[:,1], '-d')

xlabels = [str(int(imge)) + 'x' + str(int(imge)) for imge in runningTimeDFT[:, 0]]
print xlabels
plt.xticks(xrange(len(runningTimeDFT[:, 0])), xlabels)
plt.xlabel("Image Size(Pixels)")
plt.ylabel("Time(Sec)")
plt.legend(['DFT', 'Walsh', 'Haar'])
plt.show()
```

As we can see from the graph above, both Walsh and Haar are faster than DFT but are not as precise as the DFT. Because DFT is a complex sinusoidal wave.

```python

```


# Visualizing the effect of $L_1/L_2$ regularization

We use a toy example with two weights $(w_0, w_1)$ to illustrate the effect $L_1$ and $L_2$ regularization has on the solution to a loss minimization problem.

## Table of Contents

1. [Draw the data loss and the L1/L2L1/L2 regularization curves](#Draw-the-data-loss-and-the-%24L_1%2FL_2%24-regularization-curves)
2. [Plot the training progress](#Plot-the-training-progress)
3. [L1L1 -norm regularization leads to "near-sparsity"](#%24L_1%24-norm-regularization-leads-to-%22near-sparsity%22)
4. [References](#References)

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import axes3d
import matplotlib.animation as animation
import matplotlib.patches as mpatches

from torch.autograd import Variable
import torch
import math
```

```python
from IPython.display import set_matplotlib_formats
set_matplotlib_formats('pdf', 'png')
plt.rcParams['savefig.dpi'] = 75

plt.rcParams['figure.autolayout'] = False
plt.rcParams['figure.figsize'] = 10, 6
plt.rcParams['axes.labelsize'] = 18
plt.rcParams['axes.titlesize'] = 20
plt.rcParams['font.size'] = 16
plt.rcParams['lines.linewidth'] = 2.0
plt.rcParams['lines.markersize'] = 8
plt.rcParams['legend.fontsize'] = 14

# plt.rcParams['text.usetex'] = True
plt.rcParams['font.family'] = "Sans Serif"
plt.rcParams['font.serif'] = "cm"
```

## Draw the data loss and the $L_1/L_2$ regularization curves

We choose just a very simple convex loss function for illustration (in blue), which has its minimum at W=(3,2):

$$
\begin{equation}
loss(W) = 0.5(w_0-3)^2 + 2.5(w_1-2)^2
\label{eq:loss} \tag{1}
\end{equation}
$$

The L1-norm regularizer (aka lasso regression; lasso: least absolute shrinkage and selection operator):

$$
\begin{equation}
L_1(W) = \sum_{i=1}^{|W|} |w_i|
\label{eq:eq1} \tag{2}
\end{equation}
$$

The L2 regularizer (aka Ridge Regression and Tikhonov regularization), is not is the square of the L2-norm, and this little nuance is sometimes overlooked.

$$
\begin{equation}
L_2(W) = ||W||_2^2= \sum_{i=1}^{|W|} w_i^2
\label{eq:eq3} \tag{3}
\end{equation}
$$

```python
def loss_fn(W):
    return 0.5*(W[0]-3)**2 + 2.5*(W[1]-2)**2

# L1 regularization
def L1_regularization(W):
    return abs(W[0]) + abs(W[1])

def L2_regularization(W):
    return np.sqrt(W[0]**2 + W[1]**2)

fig = plt.figure(figsize=(10,10))
ax = fig.gca(projection="3d")

xmesh, ymesh = np.mgrid[-3:9:50j,-2:6:50j]
loss_mesh = loss_fn(np.array([xmesh, ymesh]))
ax.plot_surface(xmesh, ymesh, loss_mesh);

l1_mesh = L1_regularization(np.array([xmesh, ymesh]))
ax.plot_surface(xmesh, ymesh, l1_mesh);

l2_mesh = L2_regularization(np.array([xmesh, ymesh]))
ax.plot_surface(xmesh, ymesh, l2_mesh);
```

## Plot the training progress

The diamond contour lines are the values of the L1 regularization.  Since this is a contour diagram, all the points on a contour line have the same L1 value.

In otherwords, for all points on a contour line:
$$L_1(W) = \left|w_0\right| + \left|w_1\right| == constant$$
This is called the L1-ball.

L2-balls maintain the equation: $$L_2(W) = w_0^2 + w_1^2 == constant$$

The oval contour line are the values of the data loss function.  The regularized solution tries to find weights that satisfy both the data loss and the regularization loss.

```alpha``` and ```beta``` control the strengh of the regularlization loss versus the data loss.
To see how the regularizers act "in the wild", set ```alpha``` and ```beta``` to a high value like 10.  The regularizers will then dominate the loss, and you will see how each of the regularizers acts.

Experiment with the value of alpha to see how it works.

```python
initial_guess = torch.Tensor([8,5])
W = Variable(initial_guess, requires_grad=True)
W_l1_reg = Variable(initial_guess.clone(), requires_grad=True)
W_l2_reg = Variable(initial_guess.clone(), requires_grad=True)

def L1_regularization(W):
    return W.norm(1)

def L2_regularization(W):
    return W.pow(2).sum()

lr = 0.04
alpha = 0.75 # 1.5 # 4 # 0.4
beta = 0.75
num_steps = 1000

def train(W, lr, alpha, beta, num_steps):
    guesses = []
    for i in range(num_steps):
        # Zero the gradients of the weights
        if W.grad is not None:
            W.grad.data.zero_()

        # Compute the loss and the gradients of W
        loss = loss_fn(W) + alpha * L1_regularization(W) + beta * L2_regularization(W)
        loss.backward()

        # Update W
        W.data = W.data - lr * W.grad.data
        guesses.append(W.data.numpy())
    return guesses

# Train the weights without regularization
guesses = train(W, lr, alpha=0, beta=0, num_steps=num_steps)
# ...and with L1 regularization
guesses_l1_reg = train(W_l1_reg, lr, alpha=alpha, beta=0, num_steps=num_steps)
guesses_l2_reg = train(W_l2_reg, lr, alpha=0,     beta=beta, num_steps=num_steps)

fig = plt.figure(figsize=(15,10))
plt.axis("equal")

# Draw the contour maps of the data-loss and regularization loss
CS = plt.contour(xmesh, ymesh, loss_mesh, 10, cmap=plt.cm.bone)
# Draw the L1-balls
CS2 = plt.contour(xmesh, ymesh, l1_mesh, 10, linestyles='dashed', levels=list(range(5)));
# Draw the L2-balls
CS3 = plt.contour(xmesh, ymesh, l2_mesh, 10, linestyles='dashed', levels=list(range(5)));

# Add green contour lines near the loss minimum
CS4 = plt.contour(CS, levels=[0.25, 0.5], colors='g')

# Place a green dot at the data loss minimum, and an orange dot at the origin
plt.scatter(3,2, color='g')
plt.scatter(0,0, color='r')

# Color bars and labels
plt.xlabel("W[0]")
plt.ylabel("W[1]")
CB  = plt.colorbar(CS, label="data loss", shrink=0.8, extend='both')
CB2 = plt.colorbar(CS2, label="reg loss", shrink=0.8, extend='both')
# Label the contour lines
plt.clabel(CS, fmt = '%2d', colors = 'k', fontsize=14) #contour line labels
plt.clabel(CS2, fmt = '%2d', colors = 'red', fontsize=14) #contour line labels

# Plot the two sets of weights (green are weights w/o regularization; red are L1; blue are L2)
it_array = np.array(guesses)
unregularized = plt.plot(it_array.T[0], it_array.T[1], "o", color='g')
it_array = np.array(guesses_l1_reg)
l1 = plt.plot(it_array.T[0], it_array.T[1], "+", color='r')
it_array = np.array(guesses_l2_reg)
l2 = plt.plot(it_array.T[0], it_array.T[1], "+", color='b')

# Legends require a proxy artists in this case
unregularized = mpatches.Patch(color='g', label='unregularized')
l1 = mpatches.Patch(color='r', label='L1')
l2 = mpatches.Patch(color='b', label='L2')
plt.legend(handles=[unregularized, l1, l2])

# Finally add the axes, so we can see how far we are from the sparse solution.
plt.axhline(0, color='orange')
plt.axvline(0, color='orange')

print("solution: loss(%.3f, %.3f)=%.3f" % (W.data[0], W.data[1], loss_fn(W)))
print("solution: l1_loss(%.3f, %.3f)=%.3f" % (W.data[0], W.data[1], L1_regularization(W)))
print("regularized solution: loss(%.3f, %.3f)=%.3f" % (W_l1_reg.data[0], W_l1_reg.data[1], loss_fn(W_l1_reg)))
print("regularized solution: l1_loss(%.3f, %.3f)=%.3f" % (W_l1_reg.data[0], W_l1_reg.data[1], L1_regularization(W_l1_reg)))
print("regularized solution: l2_loss(%.3f, %.3f)=%.3f" % (W_l2_reg.data[0], W_l2_reg.data[1], L2_regularization(W_l2_reg)))
```

## $L_1$-norm regularization leads to "near-sparsity"

$L_1$-norm regularization is often touted as sparsity inducing, but it actually creates solutions that oscillate around 0, not exactly 0 as we'd like.

To demonstrate this, we redefine our toy loss function so that the optimal solution for $w_0$ is close to 0 (0.3).

```python
def loss_fn(W):
    return 0.5*(W[0]-0.3)**2 + 2.5*(W[1]-2)**2

# Train again
W = Variable(initial_guess, requires_grad=True)
guesses_l1_reg = train(W, lr, alpha=alpha, beta=0, num_steps=num_steps)
```

When we draw the progress of the weight training, we see that $W_0$ is gravitating towards zero.

```python
# Draw the contour maps of the data-loss and regularization loss
CS = plt.contour(xmesh, ymesh, loss_mesh, 10, cmap=plt.cm.bone)

# Plot the progress of the training process
it_array = np.array(guesses_l1_reg)
l1 = plt.plot(it_array.T[0], it_array.T[1], "+", color='r')

# Finally add the axes, so we can see how far we are from the sparse solution.
plt.axhline(0, color='orange')
plt.axvline(0, color='orange');
plt.xlabel("W[0]")
plt.ylabel("W[1]")
```

But if we look closer at what happens to $w_0$ in the last 100 steps of the training, we see that is oscillates around 0, but never quite lands there.  Why?

Well, $dL1/dw_0$ is a constant (```lr * alpha``` in our case), so the weight update step:

```W.data = W.data - lr * W.grad.data```

can be expanded to

```W.data = W.data - lr * (alpha + dloss_fn(W)/dW0)``` where ```dloss_fn(W)/dW0)```
is the gradient of loss_fn(W) with respect to $w_0$.

The oscillations are not constant (although they do have a rythm) because they are influenced by this latter loss.

```python
it_array = np.array(guesses_l1_reg[int(0.9*num_steps):])
for i in range(len(it_array)):
    print("%.4f\t(diff=%.4f)" % (it_array.T[0][i], abs(it_array.T[0][i]-it_array.T[0][i-1])))

```

## References

 **Ian Goodfellow and Yoshua Bengio and Aaron Courville**.
    [*Deep Learning*](http://www.deeplearningbook.org),
    MIT Press,
    2016.


```python
import sympy as sp
```

```python
x = sp.symbols("x")
sp.integrate(sp.exp(-x**2), (x, -sp.oo, sp.oo))
```

$\displaystyle \sqrt{\pi}$

```python
x, L, n = sp.symbols("x, L, n")
g = sp.Function("g")

sp.integrate(sp.Sum(sp.exp(-sp.I*sp.pi/L*n*x), (n, -sp.oo, sp.oo)).doit(), (x, 0, L))
```

$\displaystyle \int\limits_{0}^{L} \sum_{n=-\infty}^{\infty} e^{- \frac{i \pi n x}{L}}\, dx$

```python
x, k, kappa, L = sp.symbols(r"x, k, \kappa, L", real=True)

sp.integrate(sp.cosh(kappa*x)*sp.sin(k*x)*x, (x, -L/2, L/2))
```

$\displaystyle \begin{cases} 0 & \text{for}\: \left(\kappa = 0 \wedge k = 0\right) \vee \left(\kappa = 0 \wedge \kappa = - i k \wedge k = 0\right) \vee \left(\kappa = 0 \wedge \kappa = i k \wedge k = 0\right) \vee \left(\kappa = 0 \wedge \kappa = - i k \wedge \kappa = i k \wedge k = 0\right) \\\frac{L \sin^{2}{\left(\frac{L k}{2} \right)}}{4 k} - \frac{L \cos^{2}{\left(\frac{L k}{2} \right)}}{4 k} + \frac{\sin{\left(\frac{L k}{2} \right)} \cos{\left(\frac{L k}{2} \right)}}{2 k^{2}} & \text{for}\: \left(\kappa = 0 \wedge \kappa = - i k\right) \vee \left(\kappa = 0 \wedge \kappa = i k\right) \vee \left(\kappa = - i k \wedge \kappa = i k\right) \vee \left(\kappa = - i k \wedge k = 0\right) \vee \left(\kappa = i k \wedge k = 0\right) \vee \left(\kappa = 0 \wedge \kappa = - i k \wedge \kappa = i k\right) \vee \left(\kappa = - i k \wedge \kappa = i k \wedge k = 0\right) \vee \kappa = - i k \vee \kappa = i k \\\frac{L \kappa^{3} \sin{\left(\frac{L k}{2} \right)} \sinh{\left(\frac{L \kappa}{2} \right)}}{\kappa^{4} + 2 \kappa^{2} k^{2} + k^{4}} - \frac{L \kappa^{2} k \cos{\left(\frac{L k}{2} \right)} \cosh{\left(\frac{L \kappa}{2} \right)}}{\kappa^{4} + 2 \kappa^{2} k^{2} + k^{4}} + \frac{L \kappa k^{2} \sin{\left(\frac{L k}{2} \right)} \sinh{\left(\frac{L \kappa}{2} \right)}}{\kappa^{4} + 2 \kappa^{2} k^{2} + k^{4}} - \frac{L k^{3} \cos{\left(\frac{L k}{2} \right)} \cosh{\left(\frac{L \kappa}{2} \right)}}{\kappa^{4} + 2 \kappa^{2} k^{2} + k^{4}} - \frac{2 \kappa^{2} \sin{\left(\frac{L k}{2} \right)} \cosh{\left(\frac{L \kappa}{2} \right)}}{\kappa^{4} + 2 \kappa^{2} k^{2} + k^{4}} + \frac{4 \kappa k \cos{\left(\frac{L k}{2} \right)} \sinh{\left(\frac{L \kappa}{2} \right)}}{\kappa^{4} + 2 \kappa^{2} k^{2} + k^{4}} + \frac{2 k^{2} \sin{\left(\frac{L k}{2} \right)} \cosh{\left(\frac{L \kappa}{2} \right)}}{\kappa^{4} + 2 \kappa^{2} k^{2} + k^{4}} & \text{otherwise} \end{cases}$

```python
sp.integrate(sp.sinh(kappa*x)*sp.cos(k*x)*x, (x, -L/2, L/2))
```

$\displaystyle \begin{cases} 0 & \text{for}\: \left(\kappa = 0 \wedge k = 0\right) \vee \left(\kappa = 0 \wedge \kappa = - i k \wedge k = 0\right) \vee \left(\kappa = 0 \wedge \kappa = i k \wedge k = 0\right) \vee \left(\kappa = 0 \wedge \kappa = - i k \wedge \kappa = i k \wedge k = 0\right) \\i \left(- \frac{L \sin^{2}{\left(\frac{L k}{2} \right)}}{8 k} + \frac{L \cos^{2}{\left(\frac{L k}{2} \right)}}{8 k} - \frac{\sin{\left(\frac{L k}{2} \right)} \cos{\left(\frac{L k}{2} \right)}}{4 k^{2}}\right) - i \left(\frac{L \sin^{2}{\left(\frac{L k}{2} \right)}}{8 k} - \frac{L \cos^{2}{\left(\frac{L k}{2} \right)}}{8 k} + \frac{\sin{\left(\frac{L k}{2} \right)} \cos{\left(\frac{L k}{2} \right)}}{4 k^{2}}\right) & \text{for}\: \left(\kappa = 0 \wedge \kappa = - i k\right) \vee \left(\kappa = - i k \wedge \kappa = i k\right) \vee \left(\kappa = - i k \wedge k = 0\right) \vee \left(\kappa = 0 \wedge \kappa = - i k \wedge \kappa = i k\right) \vee \left(\kappa = - i k \wedge \kappa = i k \wedge k = 0\right) \vee \kappa = - i k \\- i \left(- \frac{L \sin^{2}{\left(\frac{L k}{2} \right)}}{8 k} + \frac{L \cos^{2}{\left(\frac{L k}{2} \right)}}{8 k} - \frac{\sin{\left(\frac{L k}{2} \right)} \cos{\left(\frac{L k}{2} \right)}}{4 k^{2}}\right) + i \left(\frac{L \sin^{2}{\left(\frac{L k}{2} \right)}}{8 k} - \frac{L \cos^{2}{\left(\frac{L k}{2} \right)}}{8 k} + \frac{\sin{\left(\frac{L k}{2} \right)} \cos{\left(\frac{L k}{2} \right)}}{4 k^{2}}\right) & \text{for}\: \left(\kappa = 0 \wedge \kappa = i k\right) \vee \left(\kappa = i k \wedge k = 0\right) \vee \kappa = i k \\\frac{L \kappa^{3} \cos{\left(\frac{L k}{2} \right)} \cosh{\left(\frac{L \kappa}{2} \right)}}{\kappa^{4} + 2 \kappa^{2} k^{2} + k^{4}} + \frac{L \kappa^{2} k \sin{\left(\frac{L k}{2} \right)} \sinh{\left(\frac{L \kappa}{2} \right)}}{\kappa^{4} + 2 \kappa^{2} k^{2} + k^{4}} + \frac{L \kappa k^{2} \cos{\left(\frac{L k}{2} \right)} \cosh{\left(\frac{L \kappa}{2} \right)}}{\kappa^{4} + 2 \kappa^{2} k^{2} + k^{4}} + \frac{L k^{3} \sin{\left(\frac{L k}{2} \right)} \sinh{\left(\frac{L \kappa}{2} \right)}}{\kappa^{4} + 2 \kappa^{2} k^{2} + k^{4}} - \frac{2 \kappa^{2} \cos{\left(\frac{L k}{2} \right)} \sinh{\left(\frac{L \kappa}{2} \right)}}{\kappa^{4} + 2 \kappa^{2} k^{2} + k^{4}} - \frac{4 \kappa k \sin{\left(\frac{L k}{2} \right)} \cosh{\left(\frac{L \kappa}{2} \right)}}{\kappa^{4} + 2 \kappa^{2} k^{2} + k^{4}} + \frac{2 k^{2} \cos{\left(\frac{L k}{2} \right)} \sinh{\left(\frac{L \kappa}{2} \right)}}{\kappa^{4} + 2 \kappa^{2} k^{2} + k^{4}} & \text{otherwise} \end{cases}$

```python
sp.integrate(sp.cosh(kappa*x)*sp.cos(k*x), (x, -L/2, L/2))
```

$\displaystyle \begin{cases} L & \text{for}\: \left(\kappa = 0 \wedge k = 0\right) \vee \left(\kappa = 0 \wedge \kappa = - i k \wedge k = 0\right) \vee \left(\kappa = 0 \wedge \kappa = i k \wedge k = 0\right) \vee \left(\kappa = 0 \wedge \kappa = - i k \wedge \kappa = i k \wedge k = 0\right) \\\frac{L \sin^{2}{\left(\frac{L k}{2} \right)}}{2} + \frac{L \cos^{2}{\left(\frac{L k}{2} \right)}}{2} + \frac{\sin{\left(\frac{L k}{2} \right)} \cos{\left(\frac{L k}{2} \right)}}{k} & \text{for}\: \left(\kappa = 0 \wedge \kappa = - i k\right) \vee \left(\kappa = 0 \wedge \kappa = i k\right) \vee \left(\kappa = - i k \wedge \kappa = i k\right) \vee \left(\kappa = - i k \wedge k = 0\right) \vee \left(\kappa = i k \wedge k = 0\right) \vee \left(\kappa = 0 \wedge \kappa = - i k \wedge \kappa = i k\right) \vee \left(\kappa = - i k \wedge \kappa = i k \wedge k = 0\right) \vee \kappa = - i k \vee \kappa = i k \\\frac{2 \kappa \cos{\left(\frac{L k}{2} \right)} \sinh{\left(\frac{L \kappa}{2} \right)}}{\kappa^{2} + k^{2}} + \frac{2 k \sin{\left(\frac{L k}{2} \right)} \cosh{\left(\frac{L \kappa}{2} \right)}}{\kappa^{2} + k^{2}} & \text{otherwise} \end{cases}$

```python
sp.integrate(sp.sinh(kappa*x)*sp.sin(k*x), (x, -L/2, L/2))
```

$\displaystyle \begin{cases} 0 & \text{for}\: \left(\kappa = 0 \wedge k = 0\right) \vee \left(\kappa = 0 \wedge \kappa = - i k \wedge k = 0\right) \vee \left(\kappa = 0 \wedge \kappa = i k \wedge k = 0\right) \vee \left(\kappa = 0 \wedge \kappa = - i k \wedge \kappa = i k \wedge k = 0\right) \\i \left(- \frac{L \sin^{2}{\left(\frac{L k}{2} \right)}}{4} - \frac{L \cos^{2}{\left(\frac{L k}{2} \right)}}{4} + \frac{\sin{\left(\frac{L k}{2} \right)} \cos{\left(\frac{L k}{2} \right)}}{2 k}\right) - i \left(\frac{L \sin^{2}{\left(\frac{L k}{2} \right)}}{4} + \frac{L \cos^{2}{\left(\frac{L k}{2} \right)}}{4} - \frac{\sin{\left(\frac{L k}{2} \right)} \cos{\left(\frac{L k}{2} \right)}}{2 k}\right) & \text{for}\: \left(\kappa = 0 \wedge \kappa = - i k\right) \vee \left(\kappa = - i k \wedge \kappa = i k\right) \vee \left(\kappa = - i k \wedge k = 0\right) \vee \left(\kappa = 0 \wedge \kappa = - i k \wedge \kappa = i k\right) \vee \left(\kappa = - i k \wedge \kappa = i k \wedge k = 0\right) \vee \kappa = - i k \\- i \left(- \frac{L \sin^{2}{\left(\frac{L k}{2} \right)}}{4} - \frac{L \cos^{2}{\left(\frac{L k}{2} \right)}}{4} + \frac{\sin{\left(\frac{L k}{2} \right)} \cos{\left(\frac{L k}{2} \right)}}{2 k}\right) + i \left(\frac{L \sin^{2}{\left(\frac{L k}{2} \right)}}{4} + \frac{L \cos^{2}{\left(\frac{L k}{2} \right)}}{4} - \frac{\sin{\left(\frac{L k}{2} \right)} \cos{\left(\frac{L k}{2} \right)}}{2 k}\right) & \text{for}\: \left(\kappa = 0 \wedge \kappa = i k\right) \vee \left(\kappa = i k \wedge k = 0\right) \vee \kappa = i k \\\frac{2 \kappa \sin{\left(\frac{L k}{2} \right)} \cosh{\left(\frac{L \kappa}{2} \right)}}{\kappa^{2} + k^{2}} - \frac{2 k \cos{\left(\frac{L k}{2} \right)} \sinh{\left(\frac{L \kappa}{2} \right)}}{\kappa^{2} + k^{2}} & \text{otherwise} \end{cases}$

```python
a, b = sp.symbols("a, b")
sp.simplify(a*sp.sin(a)*sp.sinh(b) + b*sp.cos(a)*sp.cosh(b))
```

$\displaystyle a \sin{\left(a \right)} \sinh{\left(b \right)} + b \cos{\left(a \right)} \cosh{\left(b \right)}$


# Basic Symbolic Quantum Mechanics with [SymPy](http://sympy.org)

We first load the IPython extensions that enable LaTeX-based mathematical printing
of SymPy objects, and then import the quantum computing libraries from SymPy.

```
%load_ext sympyprinting
```

```
from sympy import sqrt, symbols, Rational
from sympy import expand, Eq, Symbol, simplify, exp, sin
from sympy.physics.quantum import *
from sympy.physics.quantum.qubit import *
from sympy.physics.quantum.gate import *
from sympy.physics.quantum.grover import *
from sympy.physics.quantum.qft import QFT, IQFT, Fourier
from sympy.physics.quantum.circuitplot import circuit_plot
```

Bras and Kets

Create symbolic states

```
phi, psi = Ket('phi'), Ket('psi')
alpha, beta = symbols('alpha beta', complex=True)
```

Create a superposition

```
state = alpha*psi + beta*phi; state
```

Dagger the superposition and multiply the original

```
ip = Dagger(state)*state; ip
```

Distribute

```
qapply(expand(ip))
```

Operators

Create symbolic operators

```
A = Operator('A')
B = Operator('B')
C = Operator('C')
```

Test commutativity

```
A*B == B*A
```

    False

Distribute A+B squared

```
expand((A+B)**2)
```

Create a commutator

```
comm = Commutator(A,B); comm
```

Carry out the commutator

```
comm.doit()
```

Create a more fancy commutator

```
comm = Commutator(A*B,B+C); comm
```

Expand the commutator

```
comm.expand(commutator=True)
```

Carry out and expand the commutators

```
_.doit().expand()
```

Take the dagger

```
Dagger(_)
```


# Matrices 3D

### Definición:

La multiplicación se define como:

\begin{equation}
(AB)C
\end{equation}

**Ejemplo 1**:Multiplicar:

\begin{equation}
\left [
\begin{matrix}
1 \\\
2 \\\
3
\end{matrix}
\right ]
\left [
\begin{matrix}
4 & 5 & 6
\end{matrix}
\right ]
\left [
\begin{matrix}
7 & 8 & 9
\end{matrix}
\right ]
\end{equation}

Primero se multiplican $A$ y $B$:

\begin{equation}
\left [
\begin{matrix}
1 \\\
2 \\\
3
\end{matrix}
\right ]
\left [
\begin{matrix}
4 & 5 & 6
\end{matrix}
\right ] = \left [
\begin{matrix}
4 & 5 & 6 \\\
8 & 10 & 12 \\\
12 & 15 & 18
\end{matrix}
\right ]
\end{equation}

Luego se multiplican todas las columnas de $AB$ con $C$.

\begin{equation}
\left [
\begin{matrix}
4 \\\
8 \\\
12
\end{matrix}
\right ]
\left [
\begin{matrix}
7 & 8 & 9
\end{matrix}
\right ] = \left [
\begin{matrix}
28 & 32 & 36 \\\
56 & 64 & 72 \\\
84 & 96 & 108
\end{matrix}
\right ]
\end{equation}

\begin{equation}
\left [
\begin{matrix}
5 \\\
10 \\\
15
\end{matrix}
\right ]
\left [
\begin{matrix}
7 & 8 & 9
\end{matrix}
\right ] = \left [
\begin{matrix}
35 & 40 & 45 \\\
70 & 80 & 90 \\\
105 & 120 & 135
\end{matrix}
\right ]
\end{equation}

\begin{equation}
\left [
\begin{matrix}
6 \\\
12 \\\
18
\end{matrix}
\right ]
\left [
\begin{matrix}
7 & 8 & 9
\end{matrix}
\right ] = \left [
\begin{matrix}
42 & 48 & 54 \\\
84 & 96 & 108 \\\
126 & 144 & 162
\end{matrix}
\right ]
\end{equation}

El resultado es:

\begin{equation}
\left [
\begin{matrix}
1 \\\
2 \\\
3\
\end{matrix}
\right ]
\left [
\begin{matrix}
4 & 5 & 6
\end{matrix}
\right ]
\left [
\begin{matrix}
7 & 8 & 9
\end{matrix}
\right ] = \left [
\left [
\begin{matrix}
28 & 32 & 36 \\\
56 & 64 & 72 \\\
84 & 96 & 108
\end{matrix}
\right ]
\left [
\begin{matrix}
35 & 40 & 45 \\\
70 & 80 & 90 \\\
105 & 120 & 135
\end{matrix}
\right ]
\left [
\begin{matrix}
42 & 48 & 54 \\\
84 & 96 & 108 \\\
126 & 144 & 162
\end{matrix}
\right ]
\right ]
\end{equation}

### Ejemplo gráfico:

```
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt

fig = plt.figure()
ax = fig.gca(projection='3d')

ax.text(1, 0, 0, "1", color='red')
ax.text(2, 0, 0, "2", color='red')
ax.text(3, 0, 0, "3", color='red')

ax.text(0, 1, 0, "4", color='blue')
ax.text(0, 2, 0, "5", color='blue')
ax.text(0, 3, 0, "6", color='blue')

ax.text(0, 0, 1, "7", color='green')
ax.text(0, 0, 2, "8", color='green')
ax.text(0, 0, 3, "9", color='green')

ax.set_xlim3d(0, 5)
ax.set_ylim3d(0, 5)
ax.set_zlim3d(0, 5)

ax.set_xlabel('Eje x')
ax.set_ylabel('Eje y')
ax.set_zlabel('Eje z')

ax.view_init(17, 24)

plt.show()
```

```
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt

fig = plt.figure()
ax = fig.gca(projection='3d')

ax.text(1, 0, 0, "1", color='red')
ax.text(2, 0, 0, "2", color='red')
ax.text(3, 0, 0, "3", color='red')

ax.text(0, 1, 0, "4", color='blue')
ax.text(0, 2, 0, "5", color='blue')
ax.text(0, 3, 0, "6", color='blue')

ax.text(0, 0, 1, "7", color='green')
ax.text(0, 0, 2, "8", color='green')
ax.text(0, 0, 3, "9", color='green')

## 4  5  6
## 8 10 12
##12 15 18

ax.text(1, 1, 0, "4", color='magenta')
ax.text(2, 1, 0, "8", color='magenta')
ax.text(3, 1, 0, "12", color='magenta')

ax.text(1, 2, 0, "5", color='orange')
ax.text(2, 2, 0, "10", color='orange')
ax.text(3, 2, 0, "15", color='orange')

ax.text(1, 3, 0, "6", color='purple')
ax.text(2, 3, 0, "12", color='purple')
ax.text(3, 3, 0, "18", color='purple')

ax.set_xlim3d(0, 5)
ax.set_ylim3d(0, 5)
ax.set_zlim3d(0, 5)

ax.set_xlabel('Eje x')
ax.set_ylabel('Eje y')
ax.set_zlabel('Eje z')

ax.view_init(17, 24)

plt.show()
```

```
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt

fig = plt.figure()
ax = fig.gca(projection='3d')

ax.text(1, 0, 0, "1", color='red')
ax.text(2, 0, 0, "2", color='red')
ax.text(3, 0, 0, "3", color='red')

ax.text(0, 1, 0, "4", color='blue')
ax.text(0, 2, 0, "5", color='blue')
ax.text(0, 3, 0, "6", color='blue')

ax.text(0, 0, 1, "7", color='green')
ax.text(0, 0, 2, "8", color='green')
ax.text(0, 0, 3, "9", color='green')

##28 32  36
##56 64  72
##84 96 108
##
## 35  40  45
## 70  80  90
##105 120 135
##
## 42  48  54
## 84  96 108
##126 144 162

ax.text(1, 1, 0, "28", color='magenta')
ax.text(2, 1, 0, "56", color='magenta')
ax.text(3, 1, 0, "84", color='magenta')
ax.text(1, 1, 1, "32", color='magenta')
ax.text(2, 1, 1, "64", color='magenta')
ax.text(3, 1, 1, "96", color='magenta')
ax.text(1, 1, 2, "36", color='magenta')
ax.text(2, 1, 2, "72", color='magenta')
ax.text(3, 1, 2, "108", color='magenta')

ax.text(1, 2, 0, "35", color='orange')
ax.text(2, 2, 0, "70", color='orange')
ax.text(3, 2, 0, "105", color='orange')
ax.text(1, 2, 1, "40", color='orange')
ax.text(2, 2, 1, "80", color='orange')
ax.text(3, 2, 1, "120", color='orange')
ax.text(1, 2, 2, "45", color='orange')
ax.text(2, 2, 2, "90", color='orange')
ax.text(3, 2, 2, "135", color='orange')

ax.text(1, 3, 0, "42", color='purple')
ax.text(2, 3, 0, "84", color='purple')
ax.text(3, 3, 0, "126", color='purple')
ax.text(1, 3, 1, "48", color='purple')
ax.text(2, 3, 1, "96", color='purple')
ax.text(3, 3, 1, "144", color='purple')
ax.text(1, 3, 2, "54", color='purple')
ax.text(2, 3, 2, "108", color='purple')
ax.text(3, 3, 2, "162", color='purple')

ax.set_xlim3d(0, 5)
ax.set_ylim3d(0, 5)
ax.set_zlim3d(0, 5)

ax.set_xlabel('Eje x')
ax.set_ylabel('Eje y')
ax.set_zlabel('Eje z')

ax.view_init(17, 24)

plt.show()
```


# Dimensionality reduction example

The goal of this script is to illustrate dimensionality reduction of timeseries
data using the singular value decomposition to derive a maximum variance
representation.

For this, do the following:

    1. Definitions and imports
    2. Create covariance function
    3. Simulate the data
    4. Dimensionality reduction
    5. Plots and illustrations

The following entries may be changed to explore the script:

    * sigma_noise , sigma_signal  : Both together define the signal-to-noise ration in the data
    * cov_type                    : The form of the correlation function. Only the strings 'sq_exp',
                                    'exp', and 'white' are allowed.
    * corr_length_signal          : A quantity describing how slow the correlation between values drops
                                    depending on temporal separation. Increase to generate a smoother signal.
    * cov_type_guess              : The form of the guessed correlation function. Only the strings 'sq_exp',
                                    'exp', and 'white' are allowed.
    * corr_length_signal_guess    : A quantity describing how slow assumedly the correlation between values drops
                                    depending on temporal separation. Increase to generate a smoother signal.
    * n_expansion                 : The maximum amount of expansion coefficients for the dimensionality reduction.
                                    The bigger this quantity, the more terms are used to represent the signal.
    * n_show                      : The amount of reconstructions / elements shown in figures

This script is given out as part of the Machine Learning Tutorial during IV2020, Munich. Please consult the slides for some background information regarding motivation and a possible physical setup for this toy example.

Written by Jemil Butt, Zan Gojcic, ETH Zurich.

# 1. Definitions and imports ------------------------------------------------

Define the length $n_{datapoints}$ of the timeseries $x$, the standarddeviation of noise $\sigma_{noise}$ and the correlation length $\rho_{signal}$ of the simulated signal. Different correlation lengths $\rho_{signal}$ lead to different regularity properties of the data that is to be reconstructed in later steps. Different types of correlation structures can be chosen.
 $$\begin{align}\text{'sq_exp'}: k_{signal}(s,t)&=\exp\left[-\left(\frac{s-t}{\rho_{signal}}\right)^2\right] &&\text{leads to smooth signals} \\
 \text{'exp'}: k_{signal}(s,t)&=\exp\left[-\frac{|s-t|}{\rho_{signal}}\right] &&\text{leads to continuous, nondifferentiable signals}\\
   \text{'white'}: k_{signal}(s,t)&=\delta_{st} =\mathbb{1}_{s=t}(s,t) &&\text{leads to uncorrelated signals} \end{align}$$

Example signal 'sq_exp'       $\hspace{5cm}$     Example signal 'exp' $\hspace{5cm}$ Example signal 'white'

Adjusting $\sigma_{noise}$ leads to different signal to noise ratios. The bigger $\sigma_{noise}$ is the more noisy the data looks like and the harder it is for the algorithm to reconstruct the data due to the unpredictability of the white noise. For reconstructing the data, $n_{expansion}$ basis functions $u_k(\cdot)$ are employed. They are superimposed in the following way.
$$ x_{data}\approx \sum_{k=1}^{n_{expansion}}\alpha_k u_{k}(\cdot)$$ The number n_{expansion} of basis functions to be used is adjustable and the quality of the reconstruction improves with increasing $n_{expansion}$. The basis functions $u_k(\cdot)$ are not known a priori and finding them so as to reconstruct $x_{data}$ with the minimum number of coefficients is part of the problem.

```

# 1.1 Import numerical and plotting libraries
import numpy as np
import matplotlib.pyplot as plt

# 1.2 Define the number of observations and the time at which they take place
#       (Time interval is arbitrarily set to [0,1] to avoid scaling issues)
n_datapoints=100
time=np.linspace(0,1,n_datapoints)

#1.3 Parameters for simulationa and estimation

# Parameters for simulation             # The following parameters may be changed to explore the method
sigma_noise=0.0                         # Quantifies the average magnitude of the noise
cov_type='sq_exp'                       # Quantifies the correlation structure of the data
corr_length_signal=0.3                  # A quantity describing how slow the correlation between values drops
                                        # depending on temporal separation.

# Parameters for dimensionality reduction
cov_type_guess='sq_exp'                 # A guess for the correlation structure of the data
corr_length_signal_guess=0.3            # A guess for the smoothness of the signal. Is used to construct a
                                        # covariance matrix whose eigenfunctions reconstruct the signal
n_expansion=10                          # The amount of eigenfunctions used to reconstruct the measured signal

# Parameters for visualization
n_show=3

print('Packages imported. The parameters for simulation and representation have been defined and are now \
ready for later use.')

```

    Packages imported. The parameters for simulation and representation have been defined and are now ready for later use.

# 2. Create covariance function  ------------------------------------------------

Covariance functions $k(\cdot,\cdot)$ are a compact way of encoding the spatial dependence of the correlation structure. They are used to construct the covariance matrices necessary for inference and prediction by evaluating $k$ via
$$k(\cdot,\cdot):T\times T\ni (s,t)\mapsto k(s,t)\in \mathbb{R}$$
to generate the covariance matrix $K$ with elements $(K)_{ij}=k(t_i,t_j)$. For the simple case of an 1-D interval $T$ in time, $k(\cdot,\cdot)$ is a two-dimensional function specifying the correlation between $t$ and $s$ for all $(s,t)\in T\times T$. Some typical structures are plotted below.

Covariance matrix 'sq_exp'       $\hspace{3cm}$     Covariance matrix 'exp' $\hspace{3cm}$ Covariance matrix 'white'

```

def Create_covariance_matrix(cov_type,input_vector,correlation_length,dimensions):

    # 2.1 Rename for comfort
    n=dimensions
    t=input_vector

    # 2.2 Case based evaluation

    if cov_type == 'white':
        cov_fun=lambda t_1,t_2: np.multiply(1,t_1==t_2)
    elif cov_type == 'sq_exp':
        cov_fun=lambda t_1,t_2: np.exp(-np.power(((t_1-t_2)/correlation_length),2))
    elif cov_type == 'exp':
        cov_fun=lambda t_1,t_2: np.exp(-np.abs(((t_1-t_2)/correlation_length)))
    else:
        print('Wrong type of covariance function specified.')

    # 2.3 Assemble matrix and return

    Cov_mat=np.zeros((n,n))
    for i in range(n):
        for j in range(n):
            Cov_mat[i,j]=cov_fun(t[i],t[j])

    return Cov_mat

print('Defined a function for assembling covariance matrices from correlation functions.')

```

    Defined a function for assembling covariance matrices from correlation functions.

# 3. Simulate the data ------------------------------------------------------

Simulate randomly chosen smooth signals by considering them to be drawn from a multivariate Gaussian distribution with dimension $n_{timesteps}$. With e.g. $n_{timesteps}=100$, each throw of the dice produces a vector of length 100 that we interpret as a function mapping dimension indices to function values. We then draw $x_{signal}$ according to
$$ x_{signal}\sim\mathcal{N}(0,K)$$
where $0\in \mathbb{R}^{100}$ and $K\in \mathbb{R}^{100 \times 100}$. $\mathcal{N}$ symbolizes the multivariate Gaussian distribution. Even though each draw is random and the different realizations are independent of each other, there exists significant spatial correlation between neighboring values. This can be exploited not only for interpolation and estimation (as done in the regression example). It also to means that of the $100$ elements few are truly uncorrelated and many redundant - therefore the $100$ elements in $x_{signal}$ can be represented by significantly less numbers. More explanations of the intuition behind the simulation process may be found in the notes associated to the regression example.

```

# 3.1 Define the covariance matrices of signal an noise for simulation

Cov_mat_signal=Create_covariance_matrix(cov_type,time,corr_length_signal,n_datapoints)
Cov_mat_noise=pow(sigma_noise,2)*Create_covariance_matrix('white',time,0,n_datapoints)

# 3.2 Define a guess for the covariance matrix of the signal
Cov_mat_signal_guess=Create_covariance_matrix(cov_type_guess,time,corr_length_signal_guess,n_datapoints)

# 3.3 Do the simulation - first simulate random signal ...
x_true=np.random.multivariate_normal(np.zeros([n_datapoints]),Cov_mat_signal)

# ... then add noise to create the synthetic measurements
x_measured=x_true+np.random.multivariate_normal(np.zeros([n_datapoints]),Cov_mat_noise)

print('Covariance matrices have been constructed. Simulations were carried out by sampling from \
multivariate normal distributions.')

```

    Covariance matrices have been constructed. Simulations were carried out by sampling from multivariate normal distributions.

# 4. Dimensionality reduction -----------------------------------------------

The goal is to find an orthonormal sequence of basis vectors $\{u_j(\cdot)\}_{j=1}^k$ such that reconstruction of the signal $x_{signal}$ via $x_{reconstruct}=\sum_{j=1}^k\langle x_{signal},u_j\rangle u_j$ is possible up to a small residual RMSE with the smallest number $k$ of basis vectors. Since our data is stochastic and the exact form of $x_{signal}$ is unknown beforehand, we need to pose the optimization problem in terms of expected values. The task is to solve
$$ u_1, ... , u_k =\underset{v_1, ... , v_k \text{ orthonormal}}{\operatorname{argmin}} E\left[ \|\sum_{j=1}^k \langle x_{signal},v_j\rangle v_j - x_{signal}\|\right]. $$

Knowledge of the covariance matrix $K$ is sufficient to reconstruct any randomly chosen signal $x_{signal}$ with known correlation structure and in this case a closed form solution can be derived. It is based on the spectral decomposition of $K$.

$$\text{Spectral theorem } \hspace{2cm} K= U \Lambda U^T \hspace{2cm}\text{where } \Lambda \text{ diagonal and } U \text{ unitary}$$

Denoting the elements of $\Lambda$ by $\lambda_i$ one finds the energy of the expected reconstruction error to be
$$ \begin{align}
E[\|x_{signal}-x_{reconstruct}\|^2_2]=E\left[\|\sum_{j=k+1}^{n_{timesteps}}\langle x_{signal},u_j\rangle u_j\|^2_2\right]&=\sum_{j=k+1}^{n_{timesteps}}u_jE\left[x_{signal}x_{signal}^T\right]u_j \\
&=\sum_{j=k+1}^{n_{timesteps}} \lambda_j
\end{align}$$

The reconstruction error decreases therefore monotonically with $k$ and the first eigenvector decreases the energy of the residuals $x_{signal} - x_{reconstruct}$ by $\lambda_1$ - the largest amount possible. This means that - on average - no other single function $u(\cdot)$ can be used better to approximate $x_{signal}$ than $u_1(\cdot)$.
When trying to find the best $k$ vectors $v_1 , ... ,v_k$ to reconstruct $x_{signal}$ via $x_{reconstruct}=\sum_{j=1}^k \alpha_j v_j$, the eigenvectors $u_1, ... ,u_k$ minimize $E\left[\|x_{signal}-x_{reconstruct}\|^2_2\right]$ among all possible vectors and are therefore the best choice.

```

# 4.1 Extract basis elements
u,s,vh=np.linalg.svd(Cov_mat_signal_guess)
basis_elements=u[:,0:n_expansion]

# 4.2 Represent signal in new basis
coeffs=basis_elements.T@x_measured
signal_reconstruct=np.zeros([n_datapoints,n_expansion])
reconstruction_rmse=np.zeros(n_expansion)

for k in range(n_expansion):
    signal_reconstruct[:,k]=basis_elements[:,0:k+1]@coeffs[0:k+1]
    reconstruction_rmse[k]=np.linalg.norm(signal_reconstruct[:,k]-x_measured)/np.sqrt(n_datapoints);

print('Singular value decomposition performed and basis functions extracted. Reconstruction using the \
most efficient n_expansion basis functions.')

```

    Singular value decomposition performed and basis functions extracted. Reconstruction using the most efficient n_expansion basis functions.

# 5. Plots and illustrations ------------------------------------------------

Illustrate the decomposition and reconstruction procedure by first plotting the randomly chosen signal to be processed. The eigenvectors corresponding to the largest eigenvalues of the covariance matrices are plotted in the second figure. They correspond to the basis vectors with the most explanatory power.

Figure 3 shows the increasingly more faithful reconstructions whose RMSE monotonically decreases with increasing number of basis functions used for reconstruction. For smooth functions with much correlations structure to exploit, $10$ coefficients may be sufficient for an almost perfect reconstruction amounting to a compression of factor $10$. The actual decay of the RMSE with increasing amount of basis elements used for reconstruction is shown in figure 4.

```

# 5.1 Figure showing the measured data and the true underlying signal
plt.figure(1)
plt.scatter(time,x_measured,s=60, facecolors='none', edgecolors='k')
plt.plot(time,x_true,color='xkcd:cerulean',linewidth=2)
plt.title('True signal and measured data')
plt.legend(['x_true','x_measured'])
plt.xlabel('Time')
plt.ylabel('Function value')

# 5.2 Figure showing the basis elements used to reconstruct the measured signal
plt.figure(3)
index_show=np.round(np.linspace(0,n_expansion-1,n_show)).astype(int)
index_basis=index_show+np.ones(3)
plt.plot(time,basis_elements[:,index_show])
plt.title('Basis elements used for approximation')
plt.legend(['Basis element %d'% index_basis[0],'Basis element %d'% index_basis[1], 'Basis element %d' % index_basis[2]])
plt.xlabel('Time')
plt.ylabel('Function value')

# 5.3 Figure showing the reconstructions of the measured data in terms of an
# increasing sequence of basis elements
plt.figure(2)
plt.plot(time,signal_reconstruct[:,index_show])
plt.title('Approximations to data using basis elements')
plt.legend(['Approximation using %d elements' % index_basis[0],'Approximation using %d elements' % index_basis[1],\
            'Approximation using %d elements' % index_basis[2]])
plt.xlabel('Time')
plt.ylabel('Function value')

# 5.4 Figure showing the decay of the rmse with increasing amount of basis elements
# used for reconstruction
plt.figure(4)
plt.plot(np.insert(reconstruction_rmse,0,np.linalg.norm(x_measured)/np.sqrt(n_datapoints)))
plt.title('Root mean square error')
plt.legend(['RMSE'])
plt.xlabel('# Expansion coefficients')
plt.ylabel('RMSE')
```


# Classifieur logistique

## Concepts

Le classifieur logistique est une classifieur linéaire, $X$ est l'entrée (par exemple les pixels d'une image) et il applique un transformation linéaire pour effectuer ses prédictions $y$.

\begin{align}
WX+b = y
\end{align}

- $W \in \mathbb{R}^{n \times m}$ est une matrice contenant la matrice des poids,
- $X \in \mathbb{R}^{m}$ est un vecteur d'entrée,
- $b \in \mathbb{R}^{n}$ est le biais,
- $y \in \mathbb{R}^{n}$ est la sortie prédite.

L'objectif est de trouver les matrices $W$ et $b$ permettant de réaliser les meilleurs prédictions.

Le problème est que $y$ n'est évidemment pas une probabilité d'appartenance à une classe. $\sum_{i=1}^{n} y_i \ne 1$. Nous allons utiliser la fonction $softmax$ sur les éléments de sortie, pour d'une part normaliser les valeurs de sortie, mais aussi "exagérer" les écarts de probabiblité:

\begin{align}
S(y_i) =  \frac{e^{y_i}}{\sum_j e^{y_j}}
\end{align}

Les valeurs obtenues sont appelées **logits**.

Nous avons donc maintenant des probabilités d'appartenance à une classe (ou plutôt une estimation, d'où le ^ sur le vecteur $p$:

\begin{align}
S(WX+b) = \hat{p}
\end{align}

Lors de l'apprentissage, la probabilité de la sortie désirée est donc encodée de la manière triviale suivante: nous utilisons un vecteur ayant une probabilité de $1$ pour la classe désirée, et $0$ pour les autres classes. Cet encodage est appelé **one hot encoding**. Notons $p$ ce vecteur de probabilité désiré.

L'objectif est maintenant de mesurer une "distance" entre le vecteur de probabilité de la sortie désirée, et celle de la sortie effective. La mesure couremment utilisée à ces fins est la **cross entropie**.

\begin{align}
D(\hat{p},p) = - \sum_i p_i log (\hat{p}_i)
\end{align}

**Important** Notez que $\hat{p}$ est placé dans le $log$, ce point est inportant car $p$ contient $n-1$ valeurs $0$... et $log(0)= -\infty$.

## Apprentissage

La première méthode est de minimiser l'erreur moyenne sur tous les exemples:

\begin{align}
\mathcal{L}(X,b) = \frac{1}{n} \sum_i D(S(WX_i+b),P_i)
\end{align}

Ce que nous recherchons, c'est: $argmin_{W,b} \mathcal{L}(W,b)$.

Cf: [Exercice n°1: 1_notmnist.ipynb](1_notmnist.ipynb)

# Mesure de la performance

**Ensemble d'apprentissage**: un ensemble d'exemples utilisés pour l'apprentissage: pour s'adapter aux paramètres du classificateur. Dans le cas des NN, nous utiliserions l'ensemble d'apprentissage pour trouver les poids synaptiques «optimaux».

**Ensemble de validation**: un ensemble d'exemples utilisés pour ajuster les hyper-paramètres paramètres d'un classificateur. Dans le cas des NN, nous utiliserions l'ensemble de validation pour trouver le nombre «optimal» de couches cachées ou déterminer un point d'arrêt pour l'algorithme de rétropropagation.

**Ensemble de tests**: un ensemble d'exemples utilisés uniquement pour évaluer la performance d'un classificateur entièrement formé. Dans le cas des NN, nous utiliserions le test pour estimer le taux d'erreur après avoir choisi le modèle final (taille du NN et poids synaptiques).

Pourquoi séparer les ensembles de test et de validation? Pour éviter un sur-apprentissage par over-fitting des hyper-paramètres du modèle.

En vrac:
- Pour annoncer une amélioration, il doit y avoir 30 données correctement classées dans l'ensemble de validation pour conclure que le modèle s'est amélioré. Soit 0.1% d'amélioration sur 30000 données dans l'ensemble de validation.

# Descente de gradient vs Descente de gradient stochastique

## Description

\begin{align}
\mathcal{L}(X,b) = \frac{1}{n} \sum_i D(S(WX_i+b),P_i)
\end{align}

Il faut calculer $\Delta \mathcal{L}(W,b)$ pour tous les exemples... et de nombreuses fois. Ce processus est top couteux, au lieu de calculer $\mathcal{L}$, nous allons calculer à chaque itération une estimation des cette valeur : $\hat{\mathcal{L}}$ sur un sous ensemble des données d'apprentissage (entre 1 et 1000 données choisies aléatoirement à chaque itération). Cette méthode est appelée descente de gradient stochastique ou **SGD**.

## Contraintes pour la SGD

Pour que la **SGD** fonctionne, il faut:

- Des entrées:
    - de moyenne nulle
    - de variance faible
- Des poids initialisés:
    - de manière aléatoire
    - de moyenne nulle
    - de variance faible

## Astuce pour la SGD

### Momentum

Utiliser l'inertie (mumentum) plutôt que de mettre à jour les poids simplement avec $\delta_t = - \alpha \Delta \mathcal{L}(W)$, de la manière suivante : $\delta_t = (1-\beta) \Delta \mathcal{L}(W) + \beta \delta_{t-1}$ avec $\beta \in [0,1]$. Une valeur de $\beta = 0.9$ est généralement un bon choix.

### Learning rate decay

L'objectif est de diminuer la taille des pas au fil des itérations. De nombreuses méthodes existent:

- Exponential decay : Il a la forme mathématique $lr = lr_0  e^{- kt}$, où lr, k sont des hyperparamètres et t est le nombre d'itération.
- Step Decay : consiste par exemple à réduire le taux d'apprentissage de moitié toutes les 10 epoch.
- ...

Dans tous les cas si l'apprentissage se passe mal... commencez par diminuer la vitesse d'apprentissage.

**ADAGRAD** est un algorithme d'optimisation des descente de gradient, cet algorithme évite d'avoir à gérer un certain nombre d'hyper paramètres tels que *la vitesse d'apprentissage initiale*, le *momentum*, le *decay*.
Il existe beaucoup [d'autres algorithmes](http://ruder.io/optimizing-gradient-descent/index.html#adagrad) de ce type.

## Eviter le sur-apprentissage

L'objectif des techniques suivantes est de minimiser le sur-apprentissage.

### Early termination

Cette technique consiste à stopper l'apprentissage dès que la précision sur l'ensemble d'apprentissage diminue.

### Techniques de regularisation

Techniques ajoutant explicitement des contraintes sur les poids, par exemple :

- **L2 régularisation**: On ajoute un nouveau terme à la fonction de cout pénalisant des poids trop élevés:

\begin{align}
\mathcal{L}_{reg} = \mathcal{L} + \beta \frac{1}{2} \| W \|_2^2
\end{align}

$\beta$ est donc un autre hyper paramètre.

La dérivée de l'expression de régularisation ajouté est simple à calculer et vaut simplement: $\beta W$.

- **Dropout**: Le réseau neuronal se voit aléatoirement amputé d'une partie de ses neurones pendant la phase d'entrainement (leur valeur de sortie estimée à 0). Ce processus est répété à chaque itération. Une probabilité de $0.5$ est couramment utilisée pour le dropout. Si la technique de dropout ne fonctionne pas sur un modèle, c'est qu'il ne comporte pas suffisamment de poids.

```python

```


```python
from __future__ import print_function
from sympy import *
from eqn_manip import move_terms, move_terms_left, divide_terms, mult_eqn, get_coeff_for
from IPython.display import display
init_printing()
```

### Cubic spline equations
First step is make the a straightforward derivation of some cubic spline equations.  These could be further simplified to a tridiagonal matrix and efficiently solved.  For small numbers of knots and for testing purposes, it is sufficient to use a larger matrix and a general solver.

The first derivation will involve knots with constant separation of 1.  Later the derivation will be repeated with arbitrary knot positions.

```python
# Distance from the previous knot, for the case of uniform knot spacing
t = Symbol('t')

# Number of knots
n = Symbol('n', integer=True)
i = Symbol('i', integer=True)
# Function values to intepolated at the knots
y = IndexedBase('y')

# Coefficients of the spline function
a,b,c,d = [IndexedBase(s) for s in 'a b c d'.split()]

# Cubic spline function
s = a + b*t + c*t*t + d*t**3
display(Eq(y,s))

# With indexed variables
si = a[i] + b[i]*t + c[i]*t*t + d[i]*t**3
display(Eq(y[i],si))
```

$$y = t^{3} d + t^{2} c + t b + a$$

$${y}_{i} = t^{3} {d}_{i} + t^{2} {c}_{i} + t {b}_{i} + {a}_{i}$$

### Set up equations

```python
# Value at knots (t=0)
eq1 = Eq(si.subs(t,0), y[i])
eq1
```

$${a}_{i} = {y}_{i}$$

```python
# Value at knots (t=1)
eq2 = Eq(si.subs(t,1), y[i+1])
eq2
```

$${a}_{i} + {b}_{i} + {c}_{i} + {d}_{i} = {y}_{i + 1}$$

```python
# Continuity of first derivatives at knots
dsi = diff(si, t)
eq3 = Eq(dsi.subs(t,1), dsi.subs(i, i+1).subs(t,0))
eq3
```

$${b}_{i} + 2 {c}_{i} + 3 {d}_{i} = {b}_{i + 1}$$

```python
# Continuity of second derivatives at knots
d2si = diff(si, t, 2)
eq4 = Eq(d2si.subs(t,1), d2si.subs(i, i+1).subs(t,0))
eq4
```

$$2 {c}_{i} + 6 {d}_{i} = 2 {c}_{i + 1}$$

### Boundary conditions

```python
# Natural BC (second derivative is zero at both ends)
eq5 = Eq(d2si.subs(i,0).subs(t,0), 0)
display(eq5)
eq6 = Eq(d2si.subs(i,n-1).subs(t,1), 0)
eq6
```

$$2 {c}_{0} = 0$$

$$2 {c}_{n - 1} + 6 {d}_{n - 1} = 0$$

```python
# Or
# Specified first derivatives at boundaries
eq5b = Eq(dsi.subs(i,0).subs(t,0),0)
display(eq5b)
eq6b = Eq(dsi.subs(i,n-1).subs(t,1),0)
eq6b
```

$${b}_{0} = 0$$

$${b}_{n - 1} + 2 {c}_{n - 1} + 3 {d}_{n - 1} = 0$$

### Set up equations

There are 4$n$ unknowns.  To get the right number of equations, we need
 - one of eq5 and eq6 (boundary conditions) (2 equations)
 - eq1 and eq2 (values at beginning and end of each interval) for every knot ($2n$ equations)
 - eq3 and eq4 (continuity of 1st and 2nd derivatives) for every interior knot ($2(n-1)$ equations)

This gives the necessary $4n$ equations

```python
display(eq1)
display(eq2)
display(eq3)
display(eq4)
```

$${a}_{i} = {y}_{i}$$

$${a}_{i} + {b}_{i} + {c}_{i} + {d}_{i} = {y}_{i + 1}$$

$${b}_{i} + 2 {c}_{i} + 3 {d}_{i} = {b}_{i + 1}$$

$$2 {c}_{i} + 6 {d}_{i} = 2 {c}_{i + 1}$$

```python
# Equations 3 and 4 need to be rearranged so all the unknowns are the left-hand side
sym_lhs = [a[i],b[i],c[i],d[i],a[i+1],b[i+1],c[i+1],d[i+1]]
sym_rhs = [y[i]]
eq3b = divide_terms(eq3, sym_lhs, sym_rhs)
display(eq3b)
eq4b = divide_terms(eq4, sym_lhs, sym_rhs)
display(eq4b)
```

$$- {b}_{i + 1} + {b}_{i} + 2 {c}_{i} + 3 {d}_{i} = 0$$

$$- 2 {c}_{i + 1} + 2 {c}_{i} + 6 {d}_{i} = 0$$

```python
# Set up matrix with concrete number of knots
nknots = 3
nval = nknots -1
neqn = 4*nval
m = Matrix.eye(neqn, neqn)
rhs = Matrix.eye(neqn,1)
unknowns = Matrix.eye(neqn,1)
```

```python
use_natural_bc = True
symlist = list()
for idx in range(nval):
    symlist.append(a[idx])
    symlist.append(b[idx])
    symlist.append(c[idx])
    symlist.append(d[idx])

for idx,sym in enumerate(symlist):
    unknowns[idx] = sym

# First rows
for idx,sym in enumerate(symlist):
    if use_natural_bc:
        m[0,idx] = get_coeff_for(eq5.lhs.subs(i,0), sym , symlist)
    else:
        m[0,idx] = get_coeff_for(eq5b.lhs.subs(i,0), sym, symlist)
    m[1,idx] = get_coeff_for(eq1.lhs.subs(i,0), sym,symlist)
    m[2,idx] = get_coeff_for(eq2.lhs.subs(i,0), sym,symlist)

rhs[0] = eq5.rhs.subs(i,0)
rhs[1] = eq1.rhs.subs(i,0)
rhs[2] = eq2.rhs.subs(i,0)

jdx = 3
for nv in range(1,nval):
    for idx,sym in enumerate(symlist):
        m[jdx+0,idx] = get_coeff_for(eq1.lhs.subs(i,nv), sym,symlist)
        m[jdx+1,idx] = get_coeff_for(eq2.lhs.subs(i,nv), sym,symlist)
        m[jdx+2,idx] = get_coeff_for(eq3b.lhs.subs(i,nv-1), sym,symlist)
        m[jdx+3,idx] = get_coeff_for(eq4b.lhs.subs(i,nv-1), sym,symlist)
    rhs[jdx+0] = eq1.rhs.subs(i,nv)
    rhs[jdx+1] = eq2.rhs.subs(i,nv)
    rhs[jdx+2] = eq3b.rhs.subs(i,nv)
    rhs[jdx+3] = eq4b.rhs.subs(i,nv)
    jdx += 4

# Last rows
for idx,sym in enumerate(symlist):
    if use_natural_bc:
        m[jdx,idx] = get_coeff_for(eq6.lhs.subs(n,nval), sym , symlist)
    else:
        m[jdx,idx] = get_coeff_for(eq6b.lhs.subs(n,nval), sym , symlist)
rhs[jdx] = eq6.rhs.subs(n,nval)

# Display of matrix-vector multiplication is not working - pretend the first equals is a multiplication
with evaluate(False):
    display(Eq(Eq(m,unknowns),rhs))
```

$$\left[\begin{matrix}0 & 0 & 2 & 0 & 0 & 0 & 0 & 0\\1 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\1 & 1 & 1 & 1 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 1 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 1 & 1 & 1 & 1\\0 & 1 & 2 & 3 & 0 & -1 & 0 & 0\\0 & 0 & 2 & 6 & 0 & 0 & -2 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 2 & 6\end{matrix}\right] = \left[\begin{matrix}{a}_{0}\\{b}_{0}\\{c}_{0}\\{d}_{0}\\{a}_{1}\\{b}_{1}\\{c}_{1}\\{d}_{1}\end{matrix}\right] = \left[\begin{matrix}0\\{y}_{0}\\{y}_{1}\\{y}_{1}\\{y}_{2}\\0\\0\\0\end{matrix}\right]$$

```python
# Solve
sln = m.LUsolve(rhs)
sln
```

$$\left[\begin{matrix}{y}_{0}\\- \frac{5 {y}_{0}}{4} + \frac{3 {y}_{1}}{2} - \frac{{y}_{2}}{4}\\0\\\frac{{y}_{0}}{4} - \frac{{y}_{1}}{2} + \frac{{y}_{2}}{4}\\{y}_{1}\\- \frac{{y}_{0}}{2} + \frac{{y}_{2}}{2}\\\frac{3 {y}_{0}}{4} - \frac{3 {y}_{1}}{2} + \frac{3 {y}_{2}}{4}\\- \frac{{y}_{0}}{4} + \frac{{y}_{1}}{2} - \frac{{y}_{2}}{4}\end{matrix}\right]$$

```python
# Can substitute concrete knot values
ysubs = {y[0]:0.5, y[1]:1.1, y[2]:1.5}
ysubs
```

$$\left \{ {y}_{0} : 0.5, \quad {y}_{1} : 1.1, \quad {y}_{2} : 1.5\right \}$$

```python
# Create substitutions for spline coefficients
subslist = dict(zip(symlist,sln))
subslist
```

$$\left \{ {a}_{0} : {y}_{0}, \quad {a}_{1} : {y}_{1}, \quad {b}_{0} : - \frac{5 {y}_{0}}{4} + \frac{3 {y}_{1}}{2} - \frac{{y}_{2}}{4}, \quad {b}_{1} : - \frac{{y}_{0}}{2} + \frac{{y}_{2}}{2}, \quad {c}_{0} : 0, \quad {c}_{1} : \frac{3 {y}_{0}}{4} - \frac{3 {y}_{1}}{2} + \frac{3 {y}_{2}}{4}, \quad {d}_{0} : \frac{{y}_{0}}{4} - \frac{{y}_{1}}{2} + \frac{{y}_{2}}{4}, \quad {d}_{1} : - \frac{{y}_{0}}{4} + \frac{{y}_{1}}{2} - \frac{{y}_{2}}{4}\right \}$$

```python
# For a particular interval
si0 = si.subs(i,0).subs(subslist)
display(si0)
si0b = si0.subs(ysubs)
display(si0b)
```

$$t^{3} \left(\frac{{y}_{0}}{4} - \frac{{y}_{1}}{2} + \frac{{y}_{2}}{4}\right) + t \left(- \frac{5 {y}_{0}}{4} + \frac{3 {y}_{1}}{2} - \frac{{y}_{2}}{4}\right) + {y}_{0}$$

$$- 0.05 t^{3} + 0.65 t + 0.5$$

```python
# Can take the seconnd derivative, to compare with the output values from some spline solvers
# At t=0, values is zero, as expected from natural boundary conditions
display(diff(si0b, t, 2).subs(t,0))
# Compute at t=1
display(diff(si0b, t, 2).subs(t,1))
```

$$0$$

$$-0.3$$

# Derive for general knot spacing

```python
# Distance from the previous knot
# t[i] ranges from 0 to x[i+1]-x[i]
t = IndexedBase('t')

# Number of knots
n = Symbol('n', integer=True)
i = Symbol('i', integer=True)
# Location of knots
x = IndexedBase('x')
# Function values to intepolated at the knots
y = IndexedBase('y')

# Coefficients of the spline function
a,b,c,d = [IndexedBase(s) for s in 'a b c d'.split()]

# With indexed variables
si = a[i] + b[i]*t[i] + c[i]*t[i]*t[i] + d[i]*t[i]**3
display(Eq(y[i],si))
```

$${y}_{i} = {a}_{i} + {b}_{i} {t}_{i} + {c}_{i} {t}_{i}^{2} + {d}_{i} {t}_{i}^{3}$$

```python
# Value at knots (t[i]=0)
eq1 = Eq(si.subs(t[i],0), y[i])
eq1
```

$${a}_{i} = {y}_{i}$$

```python
# Value at knots (t[i]=x[i+1])
eq2 = Eq(si.subs(t[i],x[i+1]-x[i]), y[i+1])
eq2
```

$$\left({x}_{i + 1} - {x}_{i}\right)^{3} {d}_{i} + \left({x}_{i + 1} - {x}_{i}\right)^{2} {c}_{i} + \left({x}_{i + 1} - {x}_{i}\right) {b}_{i} + {a}_{i} = {y}_{i + 1}$$

```python
# Continuity of first derivatives at knots
dsi = diff(si, t[i])
eq3 = Eq(dsi.subs(t[i],x[i+1]-x[i]), dsi.subs(i, i+1).subs(t[i+1],0))
eq3
```

$$3 \left({x}_{i + 1} - {x}_{i}\right)^{2} {d}_{i} + 2 \left({x}_{i + 1} - {x}_{i}\right) {c}_{i} + {b}_{i} = {b}_{i + 1}$$

```python
# Continuity of second derivatives at knots
d2si = diff(si, t[i], 2)
eq4 = Eq(d2si.subs(t[i],x[i+1]-x[i]), d2si.subs(i, i+1).subs(t[i+1],0))
eq4
```

$$6 \left({x}_{i + 1} - {x}_{i}\right) {d}_{i} + 2 {c}_{i} = 2 {c}_{i + 1}$$

```python
# Natural BC (second derivative is zero at both ends)
eq5 = Eq(d2si.subs(i,0).subs(t[0],0), 0)
display(eq5)
eq6 = Eq(d2si.subs(t[i],x[i+1]-x[i]).subs(i,n-1), 0)
eq6
```

$$2 {c}_{0} = 0$$

$$6 \left(- {x}_{n - 1} + {x}_{n}\right) {d}_{n - 1} + 2 {c}_{n - 1} = 0$$

```python
# Or
# Specified first derivatives at boundaries
eq5b = Eq(dsi.subs(i,0).subs(t[0],0),0)
display(eq5b)
eq6b = Eq(dsi.subs(t[i],x[i+1]-x[i]).subs(i,n-1),0)
eq6b
```

$${b}_{0} = 0$$

$$3 \left(- {x}_{n - 1} + {x}_{n}\right)^{2} {d}_{n - 1} + 2 \left(- {x}_{n - 1} + {x}_{n}\right) {c}_{n - 1} + {b}_{n - 1} = 0$$

```python
# Equations 3 and 4 need to be rearranged so all the unknowns are the left-hand side
sym_lhs = [a[i],b[i],c[i],d[i],a[i+1],b[i+1],c[i+1],d[i+1]]
sym_rhs = [y[i]]
eq3b = divide_terms(eq3, sym_lhs, sym_rhs)
display(eq3b)
eq4b = divide_terms(eq4, sym_lhs, sym_rhs)
display(eq4b)
```

$$3 \left({x}_{i + 1} - {x}_{i}\right)^{2} {d}_{i} + 2 \left({x}_{i + 1} - {x}_{i}\right) {c}_{i} - {b}_{i + 1} + {b}_{i} = 0$$

$$6 \left({x}_{i + 1} - {x}_{i}\right) {d}_{i} - 2 {c}_{i + 1} + 2 {c}_{i} = 0$$

```python
display(eq1)
display(eq2)
display(eq3b)
display(eq4b)
```

$${a}_{i} = {y}_{i}$$

$$\left({x}_{i + 1} - {x}_{i}\right)^{3} {d}_{i} + \left({x}_{i + 1} - {x}_{i}\right)^{2} {c}_{i} + \left({x}_{i + 1} - {x}_{i}\right) {b}_{i} + {a}_{i} = {y}_{i + 1}$$

$$3 \left({x}_{i + 1} - {x}_{i}\right)^{2} {d}_{i} + 2 \left({x}_{i + 1} - {x}_{i}\right) {c}_{i} - {b}_{i + 1} + {b}_{i} = 0$$

$$6 \left({x}_{i + 1} - {x}_{i}\right) {d}_{i} - 2 {c}_{i + 1} + 2 {c}_{i} = 0$$

```python
# Set up matrix with concrete number of knots
nval = 2
nknots = nval + 1
neqn = 4*nval
m = Matrix.eye(neqn, neqn)
rhs = Matrix.eye(neqn,1)
unknowns = Matrix.eye(neqn,1)
```

```python
use_natural_bc = True
symlist = list()
for idx in range(nval):
    symlist.append(a[idx])
    symlist.append(b[idx])
    symlist.append(c[idx])
    symlist.append(d[idx])

for idx,sym in enumerate(symlist):
    unknowns[idx] = sym

# First rows
for idx,sym in enumerate(symlist):
    if use_natural_bc:
        m[0,idx] = get_coeff_for(eq5.lhs.subs(i,0), sym , symlist)
    else:
        m[0,idx] = get_coeff_for(eq5b.lhs.subs(i,0), sym, symlist)
    m[1,idx] = get_coeff_for(eq1.lhs.subs(i,0), sym,symlist)
    m[2,idx] = get_coeff_for(eq2.lhs.subs(i,0), sym,symlist)

rhs[0] = eq5.rhs.subs(i,0)
rhs[1] = eq1.rhs.subs(i,0)
rhs[2] = eq2.rhs.subs(i,0)

jdx = 3
for nv in range(1,nval):
    for idx,sym in enumerate(symlist):
        m[jdx+0,idx] = get_coeff_for(eq1.lhs.subs(i,nv), sym,symlist)
        m[jdx+1,idx] = get_coeff_for(eq2.lhs.subs(i,nv), sym,symlist)
        m[jdx+2,idx] = get_coeff_for(eq3b.lhs.subs(i,nv-1), sym,symlist)
        m[jdx+3,idx] = get_coeff_for(eq4b.lhs.subs(i,nv-1), sym,symlist)
    rhs[jdx+0] = eq1.rhs.subs(i,nv)
    rhs[jdx+1] = eq2.rhs.subs(i,nv)
    rhs[jdx+2] = eq3b.rhs.subs(i,nv)
    rhs[jdx+3] = eq4b.rhs.subs(i,nv)
    jdx += 4

# Last rows
for idx,sym in enumerate(symlist):
    if use_natural_bc:
        m[jdx,idx] = get_coeff_for(eq6.lhs.subs(n,nval), sym , symlist)
    else:
        m[jdx,idx] = get_coeff_for(eq6b.lhs.subs(n,nval), sym , symlist)
rhs[jdx] = eq6.rhs.subs(n,nval)

# Display just the matrix, displaying other pieces doesn't work, and they are the same as previously
m
```

$$\left[\begin{matrix}0 & 0 & 2 & 0 & 0 & 0 & 0 & 0\\1 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\1 & - {x}_{0} + {x}_{1} & \left(- {x}_{0} + {x}_{1}\right)^{2} & \left(- {x}_{0} + {x}_{1}\right)^{3} & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 1 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 1 & - {x}_{1} + {x}_{2} & \left(- {x}_{1} + {x}_{2}\right)^{2} & \left(- {x}_{1} + {x}_{2}\right)^{3}\\0 & 1 & - 2 {x}_{0} + 2 {x}_{1} & 3 \left(- {x}_{0} + {x}_{1}\right)^{2} & 0 & -1 & 0 & 0\\0 & 0 & 2 & - 6 {x}_{0} + 6 {x}_{1} & 0 & 0 & -2 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 2 & - 6 {x}_{1} + 6 {x}_{2}\end{matrix}\right]$$

```python
# Solve
sln = m.LUsolve(rhs)
# Display is large - skip for now, uncomment if you want to see it
#sln
```

```python
# Create substitutions for spline coefficients
subslist = dict(zip(symlist,sln))
#subslist
```

```python
# Can substitute concrete knot values
ysubs = {y[0]:0.5, y[1]:1.1, y[2]:1.5}
display(ysubs)
# Use uniform knots to verify answer is the same as before
xsubs = {x[0]:0.0, x[1]:1.0, x[2]:2.0}
display(xsubs)
```

$$\left \{ {y}_{0} : 0.5, \quad {y}_{1} : 1.1, \quad {y}_{2} : 1.5\right \}$$

$$\left \{ {x}_{0} : 0.0, \quad {x}_{1} : 1.0, \quad {x}_{2} : 2.0\right \}$$

```python
# For a particular interval
si0 = si.subs(i,0).subs(subslist)
#display(si0)
si0b = si0.subs(ysubs).subs(xsubs).subs(t[0],t)
display(si0b)
```

$$- 0.05 t^{3} + 0.65 t + 0.5$$

```python

```

```python

```


### DEMDP06
# Deterministic Optimal Economic Growth Model

Welfare maximizing social planner must decide how much society should consume and invest.  Model is of special interest because it has a known closed-form solution.

- States
    -     s       stock of wealth
- Actions
    -     k       capital investment
- Parameters
    -     beta    capital production elasticity
    -     delta   discount factor

```python
import numpy as np
import matplotlib.pyplot as plt
from compecon import BasisChebyshev, DPmodel, DPoptions, qnwnorm, demo
```

### Model parameters

Assuming that the marginal productivity of capital is $\beta=0.5$ and the discount factor is $\delta=0.9$

```python
β, δ = 0.5, 0.9
```

## Analytic results

The steady-state values for this model are

```python
sstar = (β * δ) ** (β / (1 - β))   # steady-state wealth
kstar = β * δ * sstar                    # steady-state capital investment
vstar = np.log(sstar - kstar) / (1 - δ)     # steady-state value
pstar = 1 / (sstar * (1 - β * δ))        # steady-state shadow price
b = 1 / (1 - δ * β)

print('\n\nSteady-State')
for var, value in zip(['Wealth','Investment','Value','Shadow price'], [sstar,kstar,vstar,pstar]):
    print(f'\t{var:12s} = {value:8.4f}')
```

    Steady-State
    	Wealth       =   0.4500
    	Investment   =   0.2025
    	Value        = -13.9634
    	Shadow price =   4.0404

The true value function is
\begin{equation}
V(s) = v^* + \frac{1}{1-\delta\beta}\left(\log(s) -\log(s^*)\right)
\end{equation}

```python
def vtrue(wealth): # analytic value function
    return vstar + b * (np.log(wealth) - np.log(sstar))
```

The true policy function is
\begin{equation}
k(s) = \delta\beta s
\end{equation}

```python
def ktrue(wealth): #analytic policy function
    return δ*β*wealth
```

## Numeric results

### State space
The state variable is s="Wealth", which we restrict to $0\in[0.2, 1.0]$.

Here, we represent it with a Chebyshev basis, with $n=15$ nodes.

```python
n, smin, smax = 15, 0.2, 1.0
basis = BasisChebyshev(n, smin, smax, labels=['Wealth'])
```

### Action space
The choice variable k="Investment" must be nonnegative.

```python
def bounds(s, i=None, j=None):
    return np.zeros_like(s), s[:]
```

### Reward function
The reward function is the utility of consumption=$s-k$.

```python
def reward(s, k, i=None, j=None):
    sk = s - k
    u = np.log(sk)
    ux= - sk ** -1
    uxx = - sk ** -2
    return u, ux, uxx
```

### State transition function
Next period, wealth will be equal to production from available initial capital $k$, that is $s' = k^\beta$

```python
def transition(s, k, i=None, j=None, in_=None, e=None):
    g = k ** β
    gx = β * k **(β - 1)
    gxx = (β - 1) * β * k ** (β - 2)
    return g, gx, gxx
```

### Model structure
The value of wealth $s$ satisfies the Bellman equation
\begin{equation*}
V(s) = \max_k\left\{\log(s-k) + \delta V(k^\beta)  \right\}
\end{equation*}

To solve and simulate this model,use the CompEcon class `DPmodel`

```python
growth_model = DPmodel(basis, reward, transition, bounds,
                       x=['Investment'],
                       discount=δ)
```

### Solving the model

Solving the growth model by collocation, using *Newton* algorithm and a maximum of 20 iterations

```python
options = dict(print=True,
               algorithm='newton',
               maxit=20)

snodes = growth_model.Value.nodes
S = growth_model.solve(vtrue(snodes), ktrue(snodes), **options)
```

    Solving infinite-horizon model collocation equation by Newton's method
    iter change       time
    ------------------------------
       0       4.7e-07    0.0313
       1       1.6e-12    0.0313
    Elapsed Time =    0.03 Seconds

`DPmodel.solve` returns a pandas `DataFrame` with the following data:

```python
S.head()
```

      Wealth
      value
      resid
      Investment

      Wealth

      0.200000
      0.200000
      -15.437865
      2.766030e-07
      0.090000

      0.205369
      0.205369
      -15.389699
      -2.099164e-07
      0.092416

      0.210738
      0.210738
      -15.342776
      -2.488242e-07
      0.094832

      0.216107
      0.216107
      -15.297033
      -1.102768e-07
      0.097248

      0.221477
      0.221477
      -15.252412
      5.644805e-08
      0.099664

We are also interested in the shadow price of wealth (the first derivative of the value function) and the approximation error.

To analyze the dynamics of the model, it also helps to compute the optimal change of wealth.

```python
S['shadow price'] = growth_model.Value(S['Wealth'],1)
S['error'] = S['value'] - vtrue(S['Wealth'])
S['D.Wealth'] = transition(S['Wealth'], S['Investment'])[0] - S['Wealth']
S.head()
```

      Wealth
      value
      resid
      Investment
      shadow price
      error
      D.Wealth

      Wealth

      0.200000
      0.200000
      -15.437865
      2.766030e-07
      0.090000
      9.090758
      4.241867e-07
      0.100000

      0.205369
      0.205369
      -15.389699
      -2.099164e-07
      0.092416
      8.853206
      -3.099040e-08
      0.098631

      0.210738
      0.210738
      -15.342776
      -2.488242e-07
      0.094832
      8.627699
      -4.010327e-08
      0.097210

      0.216107
      0.216107
      -15.297033
      -1.102768e-07
      0.097248
      8.413362
      1.253504e-07
      0.095739

      0.221477
      0.221477
      -15.252412
      5.644805e-08
      0.099664
      8.209399
      3.151234e-07
      0.094220

### Solving the model by Linear-Quadratic Approximation

The `DPmodel.lqapprox` solves the linear-quadratic approximation, in this case arround the steady-state. It returns a LQmodel which works similar to the DPmodel object.

We also compute the shadow price and the approximation error to compare these results to the collocation results.

```python
growth_lq = growth_model.lqapprox(sstar, kstar)
L = growth_lq.solution(basis.nodes)
L['shadow price'] = L['value_Wealth']
L['error'] = L['value'] - vtrue(L['Wealth'])
L['D.Wealth'] = L['Wealth_next']- L['Wealth']
L.head()
```

      Wealth
      Investment
      value
      value_Wealth
      Wealth_next
      shadow price
      error
      D.Wealth

      Wealth

      0.202191
      0.202191
      -0.020528
      -15.014820
      4.444953
      0.202192
      4.444953
      0.403234
      2.977733e-07

      0.219577
      0.219577
      -0.004880
      -14.937786
      4.416570
      0.219578
      4.416570
      0.330284
      2.769166e-07

      0.253590
      0.253590
      0.025731
      -14.788512
      4.361045
      0.253590
      4.361045
      0.217716
      2.361148e-07

      0.302742
      0.302742
      0.069968
      -14.576129
      4.280804
      0.302742
      4.280804
      0.107984
      1.771510e-07

      0.364886
      0.364886
      0.125897
      -14.313256
      4.179353
      0.364886
      4.179353
      0.031397
      1.026024e-07

```python
growth_lq.G
```

    array([[1.]])

## Plotting the results

### Optimal Policy

```python
fig1 = demo.figure('Optimal Investment Policy', 'Wealth', 'Investment')
plt.plot(S.Investment, label='Chebychev Collocation')
plt.plot(L.Investment, label='L-Q Approximation')
demo.annotate(sstar, kstar,'$s^*$ = %.2f\n$V^*$ = %.2f' % (sstar, kstar),'bo', (10, -17),ms=10)
plt.legend(loc= 'upper left')
```

### Value Function

```python
fig2 = demo.figure('Value Function', 'Wealth', 'Value')
plt.plot(S.value, label='Chebychev Collocation')
plt.plot(L.value, label='L-Q Approximation')

demo.annotate(sstar, vstar, f'$s^* = {sstar:.2f}$\n$V^* = {vstar:.2f}$', 'bo', (10, -17),ms=10)
plt.legend(loc= 'upper left')
```

### Shadow Price Function

```python
fig3 = demo.figure('Shadow Price Function', 'Wealth', 'Shadow Price')
plt.plot(S['shadow price'], label='Chebychev Collocation')
plt.plot(L['shadow price'], label='L-Q Approximation')
demo.annotate(sstar, pstar,f'$s^* = {sstar:.2f}$\n$\lambda^* = {pstar:.2f}$', 'bo', (10, 17),ms=10)
plt.legend(loc= 'upper right')
```

### Chebychev Collocation Residual and Approximation Error vs. Linear-Quadratic Approximation Error

```python
fig4 = plt.figure(figsize=[12, 6])
demo.subplot(1, 2, 1, 'Chebychev Collocation Residual\nand Approximation Error', 'Wealth', 'Residual/Error')
plt.hlines(0,smin,smax,'k',linestyles='--')
plt.plot(S[['resid', 'error']])
plt.legend(['Residual','Error'], loc='lower right')
plt.ticklabel_format(style='sci', axis='y', scilimits=(-1,1))

demo.subplot(1, 2, 2, 'Linear-Quadratic Approximation Error', 'Wealth', 'Error')
plt.hlines(0,smin,smax,'k',linestyles='--')
plt.plot(L['error'], label='Error')
plt.legend(loc='upper left')
plt.ticklabel_format(style='sci', axis='y', scilimits=(-1,1))
```

### Wealth dynamics

Notice how the steady-state is stable in the Chebyshev collocation solution, but unstable in the linear-quadratic approximation. In particular, simulated paths of wealth in the L-Q approximation will converge to zero, unless the initial states is within a small neighborhood of the steady-state.

```python
fig5 = demo.figure('Wealth dynamics', 'Wealth', 'Wealth change', figsize=[8,5])
plt.plot(S['D.Wealth'], label='Chebychev Collocation')
plt.plot(L['D.Wealth'], label='L-Q Approximation')
plt.hlines(0,smin,smax,linestyles=':')

demo.annotate(sstar, 0, f'$s^* = {sstar:.2f}$\n$\Delta s^* = {0:.2f}$', 'bo', (10, 10),ms=10,fs=11)
plt.legend(loc= 'lower left')
```

## Simulating the model

We simulate 20 periods of the model starting from $s=s_{\min}$

```python
T = 20
data = growth_model.simulate(T, smin)
```

### Simulated State and Policy Paths

```python
opts = dict(spec='r*', offset=(-5, -5), fs=11, ha='right')

fig6 = demo.figure('State and Policy Paths','Period', 'Wealth/Investment',[0, T + 0.5])
plt.plot(data[['Wealth', 'Investment']])
demo.annotate(T, sstar, 'steady-state wealth\n = %.2f' % sstar, **opts)
demo.annotate(T, kstar, 'steady-state investment\n = %.2f' % kstar, **opts)
```

```python
#demo.savefig([fig1,fig2,fig3,fig4,fig5,fig6])
```

notebooks/dp/06 Deterministic Optimal Economic Growth Model.ipynb