# Finding the Optimal Learning Rate on K-MNIST using Baysian Optimization

Within this blog, I am giving a short introduction into Bayesian optimization to find a near optimal learning rate. There exists a lot of great tutorials regarding the theory of Bayesian optimization. The main objective of this blog is to give a hands-on tutorial for hyperparameter optimization. As I will cover the theory only very briefly, it is recommend to read about the latter first before going through this tutorial. I am training a small ResNet implemented in PyTorch on the Kuzushiji-MNIST (or K-MNIST) dataset. This tutorial covers the following steps:

1. **Download and import the K-MNIST dataset into our project**
2. **Define a small ResNet in Pytorch**
3. **Define everything needed for Bayesian Optimization**
4. **Using Bayesian Optimization to find the optimal learning rate**
4. **Some practical approaches for learning rate optimization (logarithmic feature transformation)**

What this tutorial will **not** cover:
* Introduction to PyTorch
* Gaussian Processes

A basic understanding of Python and PyTorch are required.
***
But first things first, let's make some general imports and set the seed so we do not need to worry about them later:

# Finding the Optimal Learning Rate on K-MNIST using Baysian Optimization

Within this blog, I am giving a short introduction into Bayesian
optimization to find a near optimal learning rate. There exists a
lot of great tutorials regarding the theory of Bayesian optimization.
The main objective of this blog is to give a hands-on tutorial for
hyperparameter optimization. As I will cover the theory only very briefly,
it is recommend the read about the latter first before going through this
tutorial. I am training a small ResNet implemented in PyTorch on the
Kuzushiji-MNIST (or K-MNIST) dataset. This tutorial covers the following
steps:

1. **Download and import the K-MNIST dataset into our project**
2. **Define a small ResNet in Pytorch**
3. **Define everything needed for Bayesian Optimization**
4. **Using Bayesian Optimization to find optimal learning rate**
4. **Some practical approaches for learning rate optimization**

What this tutorial will **not** cover:
* Introduction to PyTorch
* Gaussian Processes

***
A basic understanding of Python and PyTorch are required. But first things first, let's make some general imports and set some seeds so we do not need to worry about them later:

```python
import os
import torch
import torchvision
import numpy as np
from tqdm.notebook import tqdm
import matplotlib.pyplot as plt

# set seeds
seed=94
np.random.seed(seed)
torch.manual_seed(seed)
```

    <torch._C.Generator at 0x7f870008ad70>

Done! So let's dive into the first step...
***
## Download and import the K-MNIST dataset

As mentioned before, we are training on the K-MNIST dataset. Luckily, this dataset is part of the `torchvision` package, which we have just imported! This makes it very straightforward to create a training and validation dataset as the dataset is downloaded autonomously (if it is not already downloaded) and imported in the desired format. We can pass each dataset to PyTorch's `DataLoader`, which represents an iterable over the dataset.

```python
# define a batch size
batch_size = 32

# define transformations we want to apply to our images
transform = torchvision.transforms.Compose(
    [torchvision.transforms.ToTensor()])

# create a dataset and dataloader for training
train_ds = torchvision.datasets.KMNIST(root='./data', train=True,
                                       download=True, transform=transform)
train_loader = torch.utils.data.DataLoader(train_ds, batch_size=batch_size,
                                           shuffle=True, num_workers=2)

# create a dataset and dataloader for validation
val_ds = torchvision.datasets.KMNIST(root='./data', train=False,
                                     download=True, transform=transform)
val_loader = torch.utils.data.DataLoader(val_ds, batch_size=batch_size,
                                         shuffle=True, num_workers=2)
```

That's it! We have just prepared our data. Let's see how our images look like.

```python
def show_batch(images):
    """
    Prints one batch of images in a single image.

    Parameters
    ----------
    images:
        Batch of images in training data from the DataLoader.
    """
    images = torchvision.utils.make_grid(images).numpy()
    plt.imshow(np.transpose(images, (1, 2, 0)))
    plt.show()

# get one batch
images, labels = next(iter(train_loader))

# plot
show_batch(images)
```

Looks like some japanese characters! **Make sure to run the DataLoader cell again, as we took one batch already.**

It's time to create the model.
___
## Defining a small ResNet

We use a small ResNet9 (8 convolutional layers and 1 fully-connected layer) as it is small yet provides reasonable performance. This is the structure of the network:

#### Convolutional Blocks
In my eyes, it looks less cluttered if we aggregate multiple layers to blocks. So let's start by defining the convolutional blocks shown above. These consist of a convolutional layer, batch normalization, ReLU activation and (eventually) MaxPooling:

```python
import torch.nn as nn
import torch.nn.functional as F

class ConvBlock(nn.Module):
    """
    Block consisting of a convolutional layer, batch-norm, relu activation and max-pooling (if needed).
    """
    def __init__(self, in_channels, out_channels, kernel_size=3, padding=1, pool=False, pool_kernel_size=2):
        super(ConvBlock, self).__init__()

        self.conv = nn.Conv2d(in_channels, out_channels, kernel_size=kernel_size, padding=padding)
        self.conv_bn = nn.BatchNorm2d(out_channels)

        if pool:
            self.pooling = nn.MaxPool2d(pool_kernel_size)
        else:
            self.pooling = None

    def forward(self, x):
        out = F.relu(self.conv_bn(self.conv(x)))
        if self.pooling is not None:
            out = self.pooling(out)
        return out
```

#### Residual Blocks

Next, let's define the residual blocks shown above. These blocks consists of two convolutional blocks without MaxPooling:

```python
class ResidualBlock(nn.Module):
    """
    Residual block consisting of 2 convolutional blocks.
    """
    def __init__(self, in_channels, out_channels, kernel_size=3, padding=1):
        super(ResidualBlock, self).__init__()

        self.conv_block1 = ConvBlock(in_channels, out_channels, kernel_size, padding)
        self.conv_block2 = ConvBlock(in_channels, out_channels, kernel_size, padding)

    def forward(self, x):
        residual = x
        out = self.conv_block1(x)
        out = self.conv_block2(out)
        out += residual
        return out
```

### ResNet9

Now it is straightforward to define our ResNet. Simply concatenate the blocks as shown above and add an additional MaxPolling and a fully-connected layer at the end. Note that we do not add a Softmax layer at the end, as the Cross-Entropy loss, which we are going to use later, includes this already:

```python
class ResNet9(nn.Module):
    """
    ResNet consisting of 8 convolutional layers, 1 fully-connected layer and some forward paths for residuals.
    """
    def __init__(self, in_channels, num_classes):
        super(ResNet9, self).__init__()

        # 1st and 2nd convolutional layer
        self.conv_block1 = ConvBlock(in_channels, 64)
        self.conv_block2 = ConvBlock(64, 128, pool=True)

        # residual block consisting of the 3rd and 4th convolutional layer
        self.res_block1 = ResidualBlock(128, 128)

        # 5th and 6th convolutional layers
        self.conv_block3 = ConvBlock(128, 256, pool=True)
        self.conv_block4 = ConvBlock(256, 512, pool=True)

        # residual block consisting of the 7th and 8th convolutional layer
        self.res_block2 = ResidualBlock(512, 512)

        # final fully-connected layer
        self.classifier = nn.Sequential(nn.MaxPool2d(3),
                                        nn.Flatten(),
                                        nn.Linear(512, num_classes))

    def forward(self, x):
        out = self.conv_block1(x)
        out = self.conv_block2(out)
        out = self.res_block1(out)
        out = self.conv_block3(out)
        out = self.conv_block4(out)
        out = self.res_block2(out)
        out = self.classifier(out)
        return out
```

That's it! We have our model now. So let's dive into the next step.

___

## Bayesian Optimization

A naive solution to find promising learning rates is to sample learning rates equidistantly or randomly in the search space. This is the concept behind grid and random search. While this is easy to use when function evaluation is cheap, it becomes infeasible when the function evaluation is costly. The latter is typically the case in deep learning. However, we can do better using Bayesian optimization. Bayesian optimization uses probability theory to predict the *most promising* learning rate candidate based on previously evaluated learning rates.

### Objective Function and Surrogate Model

But how do we predict the most promising next learning rate? Well, if you think about it, what we actually would like to know is some kind of function, which maps a learning rate to a performance metric, for instance the loss. If we would have such a function, we can simply take the minimum of it to find the best learning rate possible. Let's call the latter the *objective function*. Obviously, we don't have access to the objective function (otherwise, you wouldn't be here). And evaluating the objective function for a huge number of learning rates is also infeasible, as we already said. However, what we can do is to evaluate a few learning rates and try to fit a *model* to the objective function. The key idea is to fit the model until it decently represents the objective function. Then, we can instead search for the model's minimum to find a surrogate optimal learning rate. This is why we call such a model a *surrogate model* in Bayesian optimization. We are going to use a *Gaussian Process (GP)* as our surrogate model. Without observation noise, a GP can be interpreted as an interpolator, which - in contrast to other interpolators - additionally gives us information about its uncertainty between two data samples. The uncertainty measure between two data samples is crucial and one of the most distinct features of Bayesian optimization. The latter is used to exert exploration in the search space(or rather the learning rate's space). The higher the uncertainty within a certain search space region the more exploration we need to do. Note that we are not going to implement GPs ourselves here. There are tons of libraries out there. Instead, we are using `sklearn`'s implementation, as shown later.

### Acquisition Function

Okay, assume for now that we have a surrogate model, which does not yet fits the objective function very well. How do we actually choose the next most promising learning rate to evaluate? This is where the *aquisition funtion* comes into play. We are using it to determine what learning rate is the most promising for the *current* GP fitting. Hence, the acquisition function can be interpreted as a one-step utility measure. A popular choice for the acquisition function is *Expected Improvement* (EI). For our task, the improvement is defined as the improvement over the current best learning rate. Hence, the improvement $I$ at time step $t$ is defined as

\begin{equation}
I^{(t)}(\lambda) = \max (0, L_{inc}^{(t)} - L(\lambda)) \,, \label{eq:improvement}
\end{equation}

where $\lambda$ is the learning rate and $L_{inc}^{(t)}$ is the best loss experienced so far, which we call current incumbent. The corresponding learning rate is $\lambda_{inc}^{(t)} = \mathrm{argmin}_{\lambda' \in \mathcal{D}^{(t)}} L(\lambda')$, where $\mathcal{D}^{(t)}$ is the dataset containing all learning rates $\lambda'$ evaluated until time step $t$. This equation has an intuitive appeal; an improvement is achieved if our model predicts a loss smaller than the loss of the current incumbent. The best improvement possible can be achieved at the smallest loss, $\min\, L(\lambda)$.

The Expected Improvement additionally considers uncertainty and is defined - as the name suggests - as the expectation over the improvement $I^{(t)}$

\begin{equation}
u_{EI}^{(t)}(\lambda) = \mathop{\mathbb{E}}[I^{(t)}(\lambda)] = \int_{-\infty}^{\infty} p^{(t)}(L|\lambda) \times I^{(t)}(\lambda) \, dL \,.
\end{equation}

The latter can be computed in the closed form yielding:

\begin{equation}
u_{EI}^{(t)}(\lambda)=
\begin{cases}
\sigma^{(t)}(\lambda) [ Z \Phi(Z) + \phi(Z) ],& \text{if } \sigma^{(t)}(\lambda) > 0\\
0, & \text{if } \sigma^{(t)}(\lambda) = 0
\end{cases} \,,
\end{equation}

where $Z = \frac{L_{inc}^{(t)} - \mu^{(t)} (\lambda) - \xi }{\sigma^{(t)}(\lambda)}$ and $\xi$ is an optional exploration parameter. Note that $\phi$ is the PDF and $\Phi$ is the CDF of the standard normal distribution.

Now we can predict the next promising learning rate using our utility function

\begin{equation}
\lambda^{(t+1)} = \mathrm{argmax}_{\lambda \in \Lambda} u_{EI}^{(t)}(\lambda) \,,
\end{equation}

where $\Lambda$ is the search space.

That's it! We now know everything to write our own Bayesian optimizer. Let's start coding! We are going to define a class, which contains everything needed for Bayesian Optimization. Below, you can see the respective class. Let me first show you the code before explaining.

```python
from scipy.stats import norm
from matplotlib import gridspec

class BayesianOptimizer:
    """
    This is a Bayesian Optimizer, which takes in a function to optimize, and finds the
    maximum value of a parameter within a bounded search space. It uses Expected Improvement as the
    acquisition function.

    Attributes
    ----------
    f: function
        Function to optimize.

    gp: GaussianProcessRegressor
        Gaussian Process used for regression.

    mode: str
        Either "linear" or "logarithmic".

    bound: list
        List containing the lower and upper bound of the search space. IMPORTANT: If mode is "logarithmic",
        the bound specifies the minimum and maximum exponents!

    size_search_space: int
        Number of evaluation points used for finding the maximum of the acquisition function. Can be interpreted
        as the size of our discrete search space.

    search_space: ndarray
        Vector covering the search space.

    gp_search_space: ndarray
        The search space of GP might be transformed logarithmically depending on the mode, which is why it
        might differ from our defined search space.

    dataset: list
        List containing all data samples used for fitting (empty at the beginning).

    states: list
        List containing the state of each iteration in the optimization process (used for later plotting).
    """

    def __init__(self, f, gp, mode, bound, size_search_space=250):
        if mode not in ["linear", "logarithmic"]:
            raise ValueError("%s mode not supported! Chose either linear or logarithmic." % mode)
        else:
            self.mode = mode
        self.f = f
        self.gp = gp
        self.min = bound[0]
        self.max = bound[1]
        self.size_search_space = size_search_space
        if mode == "linear":
            self.search_space = np.linspace(self.min, self.max, num=size_search_space).reshape(-1, 1)
            self.gp_search_space = self.search_space
        else:
            self.search_space = np.logspace(self.min, self.max, num=size_search_space).reshape(-1, 1)
            self.gp_search_space = np.log10(self.search_space)
        self.dataset = []
        self.states = []

    def _ei(self, c_inc, xi=0.05):
        """
        Expected Improvement (EI) acquisition function used for maximization.

        Parameters
        ----------
        c_inc: float
            Utility of current incumbent.

        xi: float
            Optional exploration parameter.

        Returns
        -------
        util: ndarray
            Utilization given the current Gaussian Process and incumbent
        """
        # calculate the current mean and std for the search space
        mean, std = self.gp.predict(self.gp_search_space, return_std=True)
        std = np.array(std).reshape(-1, 1)

        # calculate the utilization
        a = (mean - c_inc - xi)
        z = a / std
        util = a * norm.cdf(z) + std * norm.pdf(z)
        return util

    def _max_acq(self):
        """
        Calculates the next best incumbent for the current dataset D.

        Returns
        -------
        x_max: float
            Location (x-coordinate) of the next best incumbent

        util_max: float
            Utility of the next best incumbent.

        util: ndarray
            Utility function for the search space.
        """
        # get the value of the current best incumbent
        c_inc = np.max(np.array(self.dataset)[:, 1])

        # calculate the utility function
        util = self._ei(c_inc)

        # check if the utilization is all zero
        if np.all((util == 0.)):
            print("Warning! Utilization function is all zero. Returning a random point for evaluation.")
            x_max = self.search_space.reshape(-1)[np.random.randint(len(self.search_space))]
            util_max = 0.0
        else:
            # get the maximum's location and utility
            x_max = self.search_space.reshape(-1)[util.argmax()]
            util_max = util.max()

        return x_max, util_max, util

    def eval(self, n_iter=10, init_x_max=None):
        """
        Runs n_iter evaluations of function f and optimizes its parameter using Bayesian Optimization.

        Parameters
        ----------
        n_iter: int
            Number of iterations used for optimization

        init_x_max: float
            Initial guess of the parameter. If none, a random initial guess is sampled in the search space.

        Returns
        -------
        best_return_x: float
            Best sample found during optimization

        best_return_param:
            Parameters defining the best function (e.g., torch model).
        """

        # get a random initial value for the incumbent from our search space if not specified
        if not init_x_max:
            x_max = self.search_space[np.random.randint(len(self.search_space))]
            x_max = x_max.item()
        else:
            x_max = init_x_max

        # for storing the best return and some parameters specifying it
        best_return = None
        best_return_x = None
        best_return_param = None

        for i in range(n_iter):

            # print some information
            print("\nBO Iteration %d --> Chosen parameter: %f %s" % (i, x_max,
                                                                     "" if (init_x_max or i != 0) else "(randomly)"))
            # evaluate the function
            y, param = self.f(x_max)

            # store if it is the best
            if not best_return or y > best_return:
                best_return = y
                best_return_x = x_max
                best_return_param = param

            # add the new sample to the dataset
            self.dataset.append([x_max, y])

            # get all the data samples in the dataset
            xs = np.array(self.dataset)[:, 0].reshape(-1, 1)
            ys = np.array(self.dataset)[:, 1].reshape(-1, 1)

            # fit the GP with the updated dataset
            if self.mode == "linear":
                self.gp.fit(xs, ys)
            else:
                self.gp.fit(np.log10(xs), ys)

            # calculate the maximum utilization and its position
            x_max, util_max, util = self._max_acq()

            # save the state for later plotting
            self.states.append({"dataset": self.dataset.copy(),
                                "util": util,
                                "GP": self.gp.predict(self.gp_search_space, return_std=True)})

        return best_return_x, best_return_param

    def save_all_plots(self):
        """
        Saves all plots.
        """
        self.plot_all(show=False, save=True)

    def plot_all(self, show=True, save=True):
        """
        Plots all states/iterations made during optimization until now.

        Parameters
        ----------
        show: bool
            If true, plot is shown directly.

        save: bool
            If true, plot is saved.
        """
        for id, state in enumerate(self.states):
            self.plot_state(state, id, show=False, save=save)
        if show:
            plt.show()

    def plot_iteration(self, it, show=True, save=True):
        """
        Plots a certain iteration of the optimization process.

        Parameters
        ----------
        it: int
            Iteration of the optimization process

        show: bool
            If true, plot is shown directly.

        save: bool
            If true, plot is saved.
        """
        # get the corresponding state
        state = self.states[it]
        self.plot_state(state, it, show=show, save=save)

    def plot_state(self, state, fig_number, show=True, save=True, additional_func=None):
        """
        Plots a state of the optimization process.

        Parameters
        ----------
        state: dict
            Dictionary storing the dataset, utilization and GP describing one state during optimization.

        fig_number: int
            Id of the figure to plot.

        show: bool
            If true, plot is shown directly.

        save: bool
            If true, plot is saved.

        additional_func: (function, name)
            Additional function to plot.
        """

        # reshape search space as this is more convenient for plotting
        search_space = self.search_space.reshape(-1)

        # get all information of the corresponding state
        dataset = state["dataset"]
        util = state["util"].reshape(-1)
        gp = state["GP"]

        # create figure with two plots (ax1: GP fitting, ax2: utility function)
        figure = plt.figure(fig_number)
        gs = gridspec.GridSpec(nrows=2, ncols=1, height_ratios=[3, 1], figure=figure)
        ax1 = figure.add_subplot(gs[0])
        ax1.set_xticklabels([])  # turn off x labeling of upper plot
        ax1.set_title("Iteration %d" % fig_number)
        ax2 = figure.add_subplot(gs[1])

        # check if we need to set a logarithmic scale
        if self.mode == "logarithmic":
            ax1.set_xscale("log")
            ax2.set_xscale("log")

        # adjust borders to make it look better
        figure.subplots_adjust(bottom=.14, top=.95)

        # plot an additional function if given
        if additional_func:
            func, name = additional_func
            add_ys, _ = func(search_space)
            ax1.plot(search_space, add_ys, color="red", label=name)

        # plot the GP mean and std
        mu, std = gp
        mu = mu.reshape(-1)
        ax1.plot(search_space, mu,
                 color="blue", label="GP mean")
        ax1.fill_between(search_space,
                         mu - (std * 1), mu + (std * 1),
                         color="blue", alpha=0.3, label="GP std")

        # plot the dataset
        xs = np.array(dataset)[:, 0]
        ys = np.array(dataset)[:, 1]
        ax1.scatter(xs, ys, color="blue", label="Dataset")

        # plot the utility function
        ax2.plot(search_space, util, color="green", label="Utility function")
        ax2.fill_between(search_space,
                         np.zeros_like(util),
                         util.reshape(-1), alpha=0.3, color="green")

        figure.legend(loc="lower center", ncol=5 if additional_func else 4)

        if save:
            if not os.path.exists('./plots'):
                os.makedirs('./plots')
            fig_name = "./plots/BO_iteration_%d" % fig_number
            plt.savefig(fig_name)
        if show:
            plt.show()

```

The latter might look overwhelming at first, but it's actually straightforward.  Let's go through it function by function:

- `__init__()`: Here we initialize everything needed for our optimizer. The most important parts are the objective function (`self.f`), the Gaussian proccess (`self.gp`, defined later), the search space (`self.search_space`) and the search space for the Gaussian process (`self.gp_search_space`). But why do we have two search spaces? Well, you'll see later that it might be very beneficial to transform the GP's search space to a logarithmic space. More on that later!

- `_ei()`: This function defines the Expected Improvement (EI) acquisition function as described above.

- `_max_acq()`: This function calculates the best next incumbent based on the acquisition function. It simply calculates the utility function for our bounded and discrete search space $\Lambda$ (`self.search_space`) and determines where the maximum is.

- `eval()`:  This function evaluates the given function (`self.f`), fits the GP and determines the next incumbent using `_max_acq`. This is done for `n_iter` iterations.

Note that we have defined the Bayesian optimizer in a way that it is *maximizing* the objective function. That is, we need to take the negative of the objective function in case of a minimization problem (as it is the case for the loss).

***
### Testing our Bayesian Optimizer

Okay, now we have our Bayesian optimizer. Let's try it on a simple example. Therefore, we need to define an objective function first:

```python
def objective(x):
    return x**2 * np.sin(5 * np.pi * x)**6.0, None
```

I have taken this objective function from another [blog](https://machinelearningmastery.com/what-is-bayesian-optimization/), which provides a great tutorial for basic Bayesian optimization. I can recommend to check it out as well!

Note that the objective function returns a tuple consisting of the actual return of the function and an additional parameter, which is `None` in this case. The latter is used later when we want to know, which ResNet model yielded what loss in order to save its parameters and probably continue training from there on. Let's take a quick look at our objective function:

```python
def plot_function(func, xs, fig_id=0):
    fig = plt.figure(0)
    ax = fig.add_subplot()
    ys, _ = objective(xs)
    ax.plot(xs, ys, color="red")
    ax.set_title("Objective Function")

xs = np.linspace(0,1, 250)
plot_function(objective, xs)
plt.show()
```

Okay, seems like our objective has several maxima from which the one at $x=0.9$ is the best in our interval. Let's see if our optimizer can find it.

However, we need to define the kernel, the Gaussian Process and some bounds first. We use a product kernel here consisting of a constant kernel and a Radial Basis Function (RBF) kernel. This is the default setting for sklearn as well.

```python
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import RBF, ConstantKernel as C

kernel = C(1.0, (1e-5, 1e5)) * RBF(10, (1e-2, 1e2))
gp = GaussianProcessRegressor(kernel, n_restarts_optimizer=10)
bo = BayesianOptimizer(objective, gp, mode="linear", bound=[0, 1])
```

As mentioned above, we are not going to talk much about the theory of GPs. If you want to learn more, you might want to checkout this [blog](https://peterroelants.github.io/posts/gaussian-process-tutorial/). However, we are going to talk a bit about the importance of kernels later.

For now, let's run 10 iterations of Bayesian optimization using our class and plot the results. Note that our optimizer stores everything we need to know about an iteration during the optimization process in `self.states`, so that plotting is easier.

```python
# sklearn might throw some annoying warnings, let's surpress them
import warnings
warnings.filterwarnings("ignore")

# give an initial guess of 0.5 and optimize
maximum, _ = bo.eval(10, 0.5)
print("\n--> Best Value found: ", maximum)

# plot the results
for i, state in enumerate(bo.states, start=1):
    bo.plot_state(state, i, show=False, additional_func=(objective, "Objective"))
plt.show()
```

The red line shows the objective function (as shown above), the blue line shows the mean of the GP, the light blue area shows the GP's standard deviation and the green line shows the EI utility function. Go through the images and track the way our optimizer works. On the last image you can see that our optimizer found $0.892$ as the best result after 10 iterations, which is quite near the global maximum at $0.9$. However, it is not guaranteed that the optimizer finds the global maximum on a small number of iterations. It might only find a local maximum, as the one at $0.5$. Okay, now we can go on to the main part of this blog.

___
# Using Bayesian Optimization to find the Optimal Learning Rate

We want to find an optimal (or near optimal) learning rate for our classification task on K-MNIST. Therefore, we need to think more thoroughly about what our objective functions is. As mentioned before, we are using the loss $L$. But which loss exactly? The one calculated on a batch? Or the one after ten batches? Are we using the training or the validation loss?

Well, our main goal in a classification task is to decrease the loss on validation data. And even though, function evaluation is expensive, K-MNIST is a rather small dataset. This is why we are going to evaluate on the **validation loss after training one epoch**. In doing so, we are optimizing the learning rate with respect to the loss we care most about and on all data provided.

That is, our function to evaluate, which is given to the Bayesian optimizer, takes the learning rate and the dataset (training + validation) as the input and returns the average validation loss (as well as the torch model). As we are evaluating on one epoch, the function is called `run_one_epoch()`, as shown below.

Note that, our function is returning the negative loss since our optimizer tries to maximize the objective function (and we are interested in a small loss). Moreover, we are also calculating the accuracy, as it is more human-readable.

```python
def accuracy(pred, true):
    class_index_pred = pred.detach().numpy().argmax(axis=1)
    return np.sum(true.detach().numpy() == class_index_pred) / len(pred)

def run_one_epoch(lr, train_l, val_l, seed):
    """
    Runs one epoch of training using the specified learning rate lr and returns the negative average
    validation loss.

    Parameters
    ----------
    lr: float
        Learning rate of the model.

    train_l: DataLoader
        Torch's DataLoaders constituting an iterator over the training dataset.

    val_l: DataLoader
        Torch's DataLoaders constituting an iterator over the validation dataset.

    seed: int
        Seed for Numpy and Torch.

    Returns
    -------
    Tuple containing the negative validation loss and the model trained on the specified learning rate.
    """
    # set the seed to initialize same model and randomness on all epochs to allow fair comparison
    np.random.seed(seed)
    torch.manual_seed(seed)

    # get our model and define the optimizer as well as the loss criterion
    model = ResNet9(in_channels=1, num_classes=10)
    optimizer = torch.optim.SGD(model.parameters(), lr=lr)
    criterion = torch.nn.CrossEntropyLoss()

    train_loop = tqdm(train_l)  # tqdm wrapper used to print progress bar
    for data in train_loop:
        # unpack images and labels
        images, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # calculate loss
        outputs = model(images)
        loss = criterion(outputs, labels)

        # calculate and apply the gradients
        loss.backward()
        optimizer.step()

        # print some training information
        train_loop.set_postfix({"Loss": loss.item(), "Accuracy": accuracy(outputs, labels)})

    # let's validate our model
    print("Validating ...")
    with torch.no_grad():
        cum_val_loss = 0.0
        cum_acc = 0.0
        for data in val_l:
            # unpack images and labels
            images, labels = data

            # calculate loss
            outputs = model(images)
            cum_val_loss += criterion(outputs, labels)
            cum_acc += accuracy(outputs, labels)

        # print some validation information
        avg_val_loss = cum_val_loss / len(val_loader)
        avg_val_acc = cum_acc / len(val_loader)
        print("---> Validation-Loss: %.4f & Validation-Accuracy: %.4f" % (avg_val_loss, avg_val_acc))
        print("\n", "-"*60,"\n")

    return -avg_val_loss, model
```

Basically, the `run_one_epoch()` method consists of two loops; the training and the validation loop. While the model is optimized during the training loop, it is kept fixed during validation.  We have everything needed now to find the optimal learning rate. However, as for our example, we need to define a kernel, a GP and some bounds. As can be seen from our bounds defined below, our search space covers learning rates from $0.00001$ to $1.0$ since learning rates smaller than that are very uncommon on the first epoch. Let's run it for 10 iterations and see what happens!

```python
n_iter = 10  # number of iterations
np.random.seed(seed) # set seed to allow fair comparison to logarithmic

# define the GP
kernel = C(1.0, (1e-5, 1e5)) * RBF(10, (1e-2, 1e2))
gp = GaussianProcessRegressor(kernel, n_restarts_optimizer=10)

# optmize the learning rate
bo = BayesianOptimizer(lambda x: run_one_epoch(x, train_loader, val_loader, seed),
                       gp,
                       mode="linear",
                       bound=[1e-5, 1.0])

found_lr, best_model = bo.eval(n_iter=10)
print("\n--> Found learning-rate after %d iterations: %f" % (n_iter, found_lr))

# plot all iterations
bo.plot_all()

# save the best model (this is the one returned from BO)
torch.save(best_model.state_dict(), "./ResNet9_linear")
```

Okay, let's take a look at the results. The optimizer is giving us a learning rate of $0.205$ that was achieved in the 8th iteration with a validation loss of $0.2946$ and an accuracy of $91.47\%$, which is quite good for the first epoch. However, I don't know about you, but I was wondering a bit about the way the optimizer was searching for the learning rate. If you take a closer look at the graph, you'll see that nearly all evaluations were done between $0.1$ and $1.0$. If you have ever tried to manually tune a learning rate on a classification task or at least read about commonly used learning rates, you'll find these learning rates quite huge. Moreover, you'll find the way the optimizer is searching for the optimal learning rate quite counterintuitive. Usually, we tune learning rates with an exponential decay, e.g., starting from $1\times 10^{-1}$ and going smaller to  $1\times 10^{-2}$,  $1\times 10^{-3}$, and so on.

Let's take a moment and think about why the optimizer is evaluating learning rates in a way, which might look counterintuitive to us. At first, let's take a closer look at the search space defined in the `BayesianOptimizer` class (for the linear case, ignore the logarithmic case for the moment). It uses **linear spacing**. Let's take a look at such a search space with the same bounds yet a smaller number of samples:

```python
np.linspace(1e-5, 1.0, 100)
```

    array([1.00000000e-05, 1.01109091e-02, 2.02118182e-02, 3.03127273e-02,
           4.04136364e-02, 5.05145455e-02, 6.06154545e-02, 7.07163636e-02,
           8.08172727e-02, 9.09181818e-02, 1.01019091e-01, 1.11120000e-01,
           1.21220909e-01, 1.31321818e-01, 1.41422727e-01, 1.51523636e-01,
           1.61624545e-01, 1.71725455e-01, 1.81826364e-01, 1.91927273e-01,
           2.02028182e-01, 2.12129091e-01, 2.22230000e-01, 2.32330909e-01,
           2.42431818e-01, 2.52532727e-01, 2.62633636e-01, 2.72734545e-01,
           2.82835455e-01, 2.92936364e-01, 3.03037273e-01, 3.13138182e-01,
           3.23239091e-01, 3.33340000e-01, 3.43440909e-01, 3.53541818e-01,
           3.63642727e-01, 3.73743636e-01, 3.83844545e-01, 3.93945455e-01,
           4.04046364e-01, 4.14147273e-01, 4.24248182e-01, 4.34349091e-01,
           4.44450000e-01, 4.54550909e-01, 4.64651818e-01, 4.74752727e-01,
           4.84853636e-01, 4.94954545e-01, 5.05055455e-01, 5.15156364e-01,
           5.25257273e-01, 5.35358182e-01, 5.45459091e-01, 5.55560000e-01,
           5.65660909e-01, 5.75761818e-01, 5.85862727e-01, 5.95963636e-01,
           6.06064545e-01, 6.16165455e-01, 6.26266364e-01, 6.36367273e-01,
           6.46468182e-01, 6.56569091e-01, 6.66670000e-01, 6.76770909e-01,
           6.86871818e-01, 6.96972727e-01, 7.07073636e-01, 7.17174545e-01,
           7.27275455e-01, 7.37376364e-01, 7.47477273e-01, 7.57578182e-01,
           7.67679091e-01, 7.77780000e-01, 7.87880909e-01, 7.97981818e-01,
           8.08082727e-01, 8.18183636e-01, 8.28284545e-01, 8.38385455e-01,
           8.48486364e-01, 8.58587273e-01, 8.68688182e-01, 8.78789091e-01,
           8.88890000e-01, 8.98990909e-01, 9.09091818e-01, 9.19192727e-01,
           9.29293636e-01, 9.39394545e-01, 9.49495455e-01, 9.59596364e-01,
           9.69697273e-01, 9.79798182e-01, 9.89899091e-01, 1.00000000e+00])

You will quickly realize that most samples lie between $1\times 10^{-1}$ and $1\times 10^{0}$. This is due to the fact that linear spacing causes equidistant spreading, which is different to our logarithmic way of tuning. As you might know, this is an easy fix. We can just use logarithmic spacing (note that `np.logspace` is expecting you to give *exponents* as the bound):

```python
# uses base 10 by default
np.logspace(-5,0, 100)
```

    array([1.00000000e-05, 1.12332403e-05, 1.26185688e-05, 1.41747416e-05,
           1.59228279e-05, 1.78864953e-05, 2.00923300e-05, 2.25701972e-05,
           2.53536449e-05, 2.84803587e-05, 3.19926714e-05, 3.59381366e-05,
           4.03701726e-05, 4.53487851e-05, 5.09413801e-05, 5.72236766e-05,
           6.42807312e-05, 7.22080902e-05, 8.11130831e-05, 9.11162756e-05,
           1.02353102e-04, 1.14975700e-04, 1.29154967e-04, 1.45082878e-04,
           1.62975083e-04, 1.83073828e-04, 2.05651231e-04, 2.31012970e-04,
           2.59502421e-04, 2.91505306e-04, 3.27454916e-04, 3.67837977e-04,
           4.13201240e-04, 4.64158883e-04, 5.21400829e-04, 5.85702082e-04,
           6.57933225e-04, 7.39072203e-04, 8.30217568e-04, 9.32603347e-04,
           1.04761575e-03, 1.17681195e-03, 1.32194115e-03, 1.48496826e-03,
           1.66810054e-03, 1.87381742e-03, 2.10490414e-03, 2.36448941e-03,
           2.65608778e-03, 2.98364724e-03, 3.35160265e-03, 3.76493581e-03,
           4.22924287e-03, 4.75081016e-03, 5.33669923e-03, 5.99484250e-03,
           6.73415066e-03, 7.56463328e-03, 8.49753436e-03, 9.54548457e-03,
           1.07226722e-02, 1.20450354e-02, 1.35304777e-02, 1.51991108e-02,
           1.70735265e-02, 1.91791026e-02, 2.15443469e-02, 2.42012826e-02,
           2.71858824e-02, 3.05385551e-02, 3.43046929e-02, 3.85352859e-02,
           4.32876128e-02, 4.86260158e-02, 5.46227722e-02, 6.13590727e-02,
           6.89261210e-02, 7.74263683e-02, 8.69749003e-02, 9.77009957e-02,
           1.09749877e-01, 1.23284674e-01, 1.38488637e-01, 1.55567614e-01,
           1.74752840e-01, 1.96304065e-01, 2.20513074e-01, 2.47707636e-01,
           2.78255940e-01, 3.12571585e-01, 3.51119173e-01, 3.94420606e-01,
           4.43062146e-01, 4.97702356e-01, 5.59081018e-01, 6.28029144e-01,
           7.05480231e-01, 7.92482898e-01, 8.90215085e-01, 1.00000000e+00])

Great! We can now evaluate each exponent equally in our search space, similar to what we would do when manually tuning the learning rate.

However, we can't just use logarithmic spacing. Something you might not know (because we skipped it here) is that the main component for calculating the covariance in GPs is the *kernel*. As mentioned before, we took a RBF kernel (Constant kernel is not important here). Let's take a look at the kernel function for our case

\begin{equation}
k(\lambda_i, \lambda_j) = \exp\left( - \frac{d(\lambda_i, \lambda_j)^2}{2l^2} \right)\, .
\end{equation}

$\lambda_i$ and $\lambda_j$ are two learning rates, $d(.,.)$ is the Euclidean distance between those points and $l$ is the length parameter for scaling the covariance.

The part I want to point your attention to is the distance $d(.,.)$. As for a lot of kernels, this is the main metric for calculating the covariance between two points. Our intention when using logarithmic spacing is that we would like to explore each exponent equally. However, because our kernel is using the distance for calculating the covariance, it yields higher covariance for greater distance and vice versa. And since Expected Improvement yields higher utility with higher variance (c.f., equations above), our optimizer would still favour greater exponents.

However, the fix is easy here as well. We can simply **transform the search space logarithmically** when working with GPs. That is, the kernel is computed on a transformed learning rate $\psi(\lambda) = (\log_{10}(\lambda))$. The kernel is then

\begin{equation}
k(\psi(\lambda_i), \psi(\lambda_j)) = \exp\left( - \frac{d(\psi(\lambda_i), \psi(\lambda_j))^2}{2l^2} \right)\, .
\end{equation}

Note that we do not transform the targets during GP fitting. This means that the loss is equal for both cases $L(\lambda) = L(\psi(\lambda))$.

Let's recap briefly before we try that.

1. We are using logarithmic *spacing* to have a search space, which contains the same amount of data points for each exponent. In doing so, we pay equal attention to each exponent.

2. We are using logarithmic *feature transformation* when working with GPs. This turns our non-linearly (here logarithmically) spaced search space into a linearly spaced search space. In doing so, we are encouraging the optimizer to search on small exponents as well.

Okay, enough theory! Let's see the results. Everything needed is already implemented in the `BayesianOptimizer` class. We only need to switch the mode from linear to logarithmic and change the bounds to exponents.

```python
n_iter = 10  # number of iterations
np.random.seed(seed) # set seed to allow fair comparison to linear

# define the GP
kernel = C(1.0, (1e-5, 1e5)) * RBF(10, (1e-2, 1e2))
gp = GaussianProcessRegressor(kernel, n_restarts_optimizer=10)

# optmize the learning rate
bo = BayesianOptimizer(lambda x: run_one_epoch(x, train_loader, val_loader, seed),
                       gp,
                       mode="logarithmic",
                       bound=[-5, 0])

found_lr, best_model = bo.eval(n_iter=10)
print("\n--> Found learning-rate after %d iterations: %f" % (n_iter, found_lr))

# plot all iterations
bo.plot_all()

# save the best model (this is the one returned from BO)
torch.save(best_model.state_dict(), "./ResNet9_log")
```

Yeay! We were able to decrease the loss from $0.2946$ to $0.1843$ and increase our accuracy from $91.47\%$ to $94.45\%$. The learning rate found is significantly smaller then the one found before: linear-mode $\rightarrow 0.204$ vs. logarithmic-mode $\rightarrow 0.0037$! Especially in the first iterations, you can see the that the variance is high in *both* directions, which is exactly what we wanted. In the last iteration, you can see that our optmizer paid equal attention to all exponents in our search space.

## Conclusion

One might argument that the difference between linear and logarithmic mode in performance isn't that high. Actually, that's true. However, paying more attention to smaller learning rates becomes more important in later epochs, where greater learning rates often cause stagnation in learning.

Finally, I want to point out that, even though it might be useful to exploit our domain knowledge to transform the learning rate's search space logarithmically, it might be different for other hyperparameters, which we want to optimize as well. So, be careful with that!


```python
%load_ext autoreload
```

```python
autoreload 2
```

```python
%matplotlib inline
```

```python
import matplotlib.pyplot as plt
import numpy as np
import sympy as sym
import seaborn as sn

import pycollocation
```

## The basic model: amplification and persistance

```python
# need to specify some production function for gatherers
def output(k, alpha):
    return k**alpha

def marginal_product_capital(k, alpha, **params):
    return alpha * k**(alpha - 1)

def K_dot(t, K, B, q, u, a, R):
    return (1 / u) * ((a + q) * K - R * B) - K

def B_dot(t, K, B, q, u, R):
    return (1 / R) * q * K - B

def q_dot(t, K, B, q, u, R):
          return (R - 1) * q - R * u

def u_dot(t, K, B, q, u, m, K_bar, R, **params):
    return (1 / R) * marginal_product_capital((1 / m) * (K_bar - K), **params) - u

def rhs(t, K, B, q, u, a, m, K_bar, R, **params):
    out = [K_dot(t, K, B, q, u, a, R),
           B_dot(t, K, B, q, u, R),
           q_dot(t, K, B, q, u, R),
           u_dot(t, K, B, q, u, m, K_bar, R, **params)]
    return out
```

```python
def steady_state_capital(a, m, K_bar, R, alpha, **params):
    return K_bar - m * (alpha / (a * R))**(1 / (1 - alpha))

def steady_state_debt(a, m, K_bar, R, **params):
    Kstar = steady_state_capital(a, m, K_bar, R, **params)
    return (a / (R - 1)) * Kstar

def steady_state_land_price(a, R, **params):
    return (R / (R - 1)) * a

def steady_state_user_cost(a, **params):
    return a

def bcs_lower(t, K, B, q, u, K0, **params):
    return [K - K0]

def bcs_upper(t, K, B, q, u, a, m, K_bar, R, **params):
    Bstar = steady_state_debt(a, m, K_bar, R, **params)
    qstar = steady_state_land_price(a, R)
    ustar = steady_state_user_cost(a)
    return [B - Bstar, q - qstar, u - ustar]
```

```python
params = {'a': 1.01, 'm': 10.0, 'alpha': 0.33, 'R': 1.5, 'K_bar': 100, 'K0': 95}
```

```python
Kstar
```

    98.97176708265565

```python
B0
```

    142.5

```python
Bstar
```

    199.9229695069644

```python
# specify an initial guess
domain = [0, 10]
ts = np.linspace(domain[0], domain[1], 1000)
Kstar = steady_state_capital(**params)
Ks = Kstar - (Kstar - params['K0']) * np.exp(-ts)
initial_capital_poly = np.polynomial.Chebyshev.fit(ts, Ks, 5, domain)

# initial value of debt is some multiple of capital stock
B0 = 1.5 * params['K0']
Bstar = steady_state_debt(**params)
Bs = Bstar - (Bstar - B0) * np.exp(-ts)
initial_debt_poly = np.polynomial.Chebyshev.fit(ts, Bs, 5, domain)

# starting with K0 < Kstar, must be that u0 > ustar
ustar = steady_state_user_cost(**params)
u0 = 1.5 * ustar
us = ustar - (ustar - u0) * np.exp(-ts)
initial_user_cost_poly = np.polynomial.Chebyshev.fit(ts, us, 5, domain)

# starting with K0 < Kstar, must be that q0 > qstar
qstar = steady_state_land_price(**params)
q0 = 1.05 * qstar
qs = qstar + (qstar - q0) * np.exp(-ts)
initial_land_price_poly = np.polynomial.Chebyshev.fit(ts, qs, 5, domain)

initial_coefs = np.hstack([initial_capital_poly.coef, initial_debt_poly.coef,
                           initial_user_cost_poly.coef, initial_land_price_poly.coef])
```

```python
nodes = pycollocation.PolynomialSolver.collocation_nodes(5, domain, "Chebyshev")
```

```python
problem = pycollocation.TwoPointBVP(bcs_lower, bcs_upper, 1, 4, rhs, params)
```

```python
solution = pycollocation.PolynomialSolver.solve({'kind': "Chebyshev"},
                                                initial_coefs,
                                                domain,
                                                nodes,
                                                problem)
```

```python
pycollocation.PolynomialSolver._array_to_list(initial_coefs, 4)
```

    [array([ 98.25192341,   1.29704214,  -0.91771876,   0.5470026 ,
             -0.25019508,   0.10814735]),
     array([ 189.51562177,   18.75236135,  -13.26818405,    7.90844809,
              -3.61726759,    1.56357157]),
     array([ 1.10152628, -0.16491558,  0.11668559, -0.06954998,  0.03181166,
            -0.01375066]),
     array([ 3.00254212,  0.04947467, -0.03500568,  0.02086499, -0.0095435 ,
             0.0041252 ])]

```python
initial_capital_poly.coef
```

    array([ 98.25192341,   1.29704214,  -0.91771876,   0.5470026 ,
            -0.25019508,   0.10814735])

```python
solution.result
```

      status: 5
     success: False
         qtf: array([ 118.11507587,  -73.62805396,   34.42542314,   91.23327592,
            -21.58362966,  -46.31673836, -312.73178766,   49.73195376,
            -63.00744997,   17.69122132,  -29.50229648,    6.74219956,
           -130.3121271 ,   76.17524295,   -0.89808815,    1.0618052 ,
              4.798498  ,    8.7009872 ,    8.70516571,   13.22207324,
              3.44698481,   -3.86957298,   10.27331385,    4.00065662])
        nfev: 59
           r: array([ nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
            nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
            nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
            nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
            nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
            nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
            nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
            nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
            nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
            nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
            nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
            nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
            nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
            nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
            nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
            nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
            nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
            nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
            nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
            nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
            nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
            nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
            nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
            nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
            nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
            nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
            nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
            nan,  nan,  nan])
         fun: array([  9.91315830e+01,   1.27135105e+02,   1.31413999e+02,
             1.32020647e+02,   1.32290429e+02,   1.05707121e+02,
             1.30513462e+02,   1.32691054e+02,   1.32959922e+02,
             1.35856846e+02,   3.31235599e+00,   3.91420017e+00,
             4.02867648e+00,   4.04545691e+00,   4.02224183e+00,
             2.61978349e+00,   2.24958520e+00,   2.05253856e+00,
             2.01347433e+00,   2.01616608e+00,   1.31817469e-01,
             9.31581637e-01,  -2.02819269e+00,   2.02245781e+00])
           x: array([  9.82519234e+01,   1.29704214e+00,  -9.17718765e-01,
             5.47002603e-01,  -2.50195078e-01,   1.08147352e-01,
             1.89515622e+02,   1.87523613e+01,  -1.32681840e+01,
             7.90844809e+00,  -3.61726759e+00,   1.56357157e+00,
             1.10152628e+00,  -1.64915583e-01,   1.16685588e-01,
            -6.95499784e-02,   3.18116626e-02,  -1.37506585e-02,
             3.00254212e+00,   4.94746748e-02,  -3.50056763e-02,
             2.08649935e-02,  -9.54349878e-03,   4.12519754e-03])
     message: 'The iteration is not making good progress, as measured by the \n  improvement from the last ten iterations.'
        fjac: array([[ -7.23449284e-02,  -1.30682715e-01,  -1.39759317e-01,
             -1.41050339e-01,  -1.41202692e-01,   3.95426739e-01,
              3.01616125e-01,   2.86438732e-01,   2.84266727e-01,
              2.84010180e-01,   0.00000000e+00,   0.00000000e+00,
              0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
              2.73561927e-02,   1.47345086e-01,   2.56011589e-01,
              2.81930285e-01,   2.85267356e-01,  -4.22620469e-01,
              0.00000000e+00,   0.00000000e+00,   0.00000000e+00],
           [ -2.66774214e-02,  -2.23107591e-02,  -1.19285758e-01,
             -2.23129477e-01,  -2.87427339e-01,  -4.32154370e-01,
             -1.92558236e-01,   2.77589130e-02,   2.36578697e-01,
              3.65436465e-01,   0.00000000e+00,   0.00000000e+00,
              0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
             -2.98970708e-02,  -9.40681674e-02,   2.48102042e-02,
              2.34633815e-01,   3.67053739e-01,   4.87750569e-01,
              0.00000000e+00,   0.00000000e+00,   0.00000000e+00],
           [ -2.97178802e-01,  -2.85558434e-01,  -1.72785009e-01,
              1.52402057e-01,   4.59196777e-01,  -2.14202659e-01,
              1.87158167e-01,   3.72466946e-01,   1.41007900e-01,
             -2.11049032e-01,   0.00000000e+00,   0.00000000e+00,
              0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
             -1.48188023e-02,   9.14298210e-02,   3.32901526e-01,
              1.39848677e-01,  -2.11983431e-01,   3.13422790e-01,
              0.00000000e+00,   0.00000000e+00,   0.00000000e+00],
           [  5.28146766e-01,   1.95267774e-01,  -2.43643372e-01,
             -1.07104666e-01,   5.24213406e-01,   1.67066461e-02,
             -3.38591405e-01,  -1.94745874e-02,   2.91408159e-01,
             -2.19860019e-02,   0.00000000e+00,   0.00000000e+00,
              0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
              1.15586606e-03,  -1.65408163e-01,  -1.74058898e-02,
              2.89012865e-01,  -2.20825123e-02,  -1.57839943e-01,
              0.00000000e+00,   0.00000000e+00,   0.00000000e+00],
           [  5.52677579e-01,  -2.21286529e-01,  -1.69796027e-01,
              3.48154980e-01,  -4.44166173e-01,  -1.07787892e-01,
             -1.27528422e-01,   3.43715089e-01,  -1.21986510e-01,
             -1.06626072e-01,   0.00000000e+00,   0.00000000e+00,
              0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
             -7.45696437e-03,  -6.22993180e-02,   3.07202752e-01,
             -1.20984683e-01,  -1.07098758e-01,  -3.95499428e-02,
              0.00000000e+00,   0.00000000e+00,   0.00000000e+00],
           [ -4.08984368e-01,   4.90842459e-01,  -4.45438456e-01,
              3.35336614e-01,  -2.94714927e-01,   1.46249558e-01,
             -2.21291912e-01,  -5.99810589e-03,   1.67251343e-01,
             -1.67153214e-01,   0.00000000e+00,   0.00000000e+00,
              0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
              1.01176778e-02,  -1.08105235e-01,  -5.36096591e-03,
              1.65879718e-01,  -1.67895451e-01,  -1.58579220e-02,
              0.00000000e+00,   0.00000000e+00,   0.00000000e+00],
           [ -1.70265717e-01,  -3.19460977e-01,  -2.73805037e-01,
             -3.10048709e-01,  -7.61346773e-02,  -1.85920395e-01,
             -2.55209155e-01,  -2.65093370e-01,  -2.67416326e-01,
             -3.32673475e-01,   0.00000000e+00,   0.00000000e+00,
              0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
              1.70557503e-02,   8.65885515e-02,   1.49584906e-01,
              1.63683377e-01,   1.00225145e-01,  -2.93057307e-01,
             -4.32456489e-01,   0.00000000e+00,   0.00000000e+00],
           [  2.47590809e-01,   2.74408382e-01,   2.45881859e-02,
             -2.11383274e-01,  -9.35247906e-02,   3.25113573e-01,
              3.49629266e-01,   9.61799178e-02,  -1.05265609e-01,
             -3.79273507e-01,   0.00000000e+00,   0.00000000e+00,
              0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
             -2.47760040e-02,  -5.19934032e-02,   6.89406236e-03,
              1.72431489e-01,   1.27599927e-01,   4.45254886e-01,
             -4.11842488e-01,   0.00000000e+00,   0.00000000e+00],
           [  9.37327963e-02,  -1.46606110e-01,  -4.01193779e-01,
             -1.65580856e-02,   7.06490470e-02,   3.80745437e-01,
              1.58479795e-02,  -4.44632021e-01,  -2.83474846e-01,
              3.35637525e-01,   0.00000000e+00,   0.00000000e+00,
              0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
             -1.18334213e-02,   1.28269849e-01,   2.71991467e-01,
             -8.46985520e-03,  -1.71263185e-01,   3.32432157e-01,
              1.92577229e-01,   0.00000000e+00,   0.00000000e+00],
           [ -2.83414228e-02,   3.26163402e-01,  -1.46957875e-01,
             -1.18446516e-01,   9.00695487e-02,  -3.16900138e-01,
              3.03232421e-01,   2.06809388e-01,  -5.56230375e-01,
              3.17559118e-01,   0.00000000e+00,   0.00000000e+00,
              0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
             -7.13296098e-03,  -3.32661064e-01,   6.27590573e-02,
              1.63904858e-01,  -1.42600460e-01,  -2.03965721e-01,
             -1.54617258e-02,   0.00000000e+00,   0.00000000e+00],
           [  2.34564474e-02,  -1.38943727e-01,  -1.27915550e-01,
              4.62546429e-02,  -4.78450016e-02,  -1.27137189e-01,
              4.33101023e-01,  -4.14795567e-01,   3.12863768e-01,
             -1.88015531e-01,   0.00000000e+00,   0.00000000e+00,
              0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
             -5.11635946e-02,  -6.02898633e-01,   1.55122842e-01,
             -1.74006048e-01,   1.01433347e-01,  -9.22605186e-02,
              1.05250693e-01,   0.00000000e+00,   0.00000000e+00],
           [  2.19519761e-01,   2.61633451e-01,  -1.05877212e-01,
              2.89874769e-02,  -1.07016927e-01,  -4.13756530e-01,
              3.74783977e-01,  -2.76762319e-01,   1.49644156e-01,
             -1.03647216e-01,   1.11022302e-16,   0.00000000e+00,
              0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
              6.04121077e-02,   6.20930332e-01,   1.31972814e-02,
              1.15730685e-01,  -1.08153678e-01,  -1.01030969e-01,
              1.17425727e-01,   0.00000000e+00,   0.00000000e+00],
           [ -2.39397034e-02,  -1.16204377e-02,  -1.01631154e-02,
             -2.98988479e-01,  -8.70629369e-02,   4.89410703e-02,
             -7.07384754e-02,   1.18028512e-01,  -2.14251036e-01,
             -4.29743435e-01,  -9.92360879e-03,  -9.92360879e-03,
             -9.92360879e-03,  -9.92360879e-03,  -9.92360879e-03,
             -2.96430220e-03,  -3.07783560e-02,  -4.18985610e-03,
              1.98638147e-01,   1.67680810e-01,   1.45585961e-02,
              7.62007751e-01,   1.98472181e-02,   0.00000000e+00],
           [ -2.58930933e-02,   1.68292599e-01,   3.62741618e-01,
              4.68711891e-01,   1.51572398e-01,  -2.12050715e-02,
             -9.69750249e-02,  -2.20265277e-01,  -1.95251997e-01,
             -2.18331324e-02,  -1.45873117e-01,  -1.21836806e-01,
             -8.29452362e-02,  -4.40536423e-02,  -2.00173794e-02,
             -9.23296414e-03,   1.19499959e-02,   4.29331555e-01,
              2.63268276e-01,   4.47081542e-01,  -1.79328112e-02,
             -1.67287295e-03,  -1.93751070e-02,   0.00000000e+00],
           [  5.37137391e-02,  -2.59418901e-01,  -3.35850517e-01,
              4.20931617e-01,   1.17590691e-01,   1.65422264e-03,
              1.76535094e-01,   2.33379245e-02,  -1.87082809e-01,
             -1.28016433e-02,   1.14854101e-01,   3.92577084e-02,
             -3.73712344e-02,  -5.75259730e-02,  -4.17452344e-02,
              3.35038875e-02,   3.20095051e-02,  -6.03835028e-01,
              1.53909149e-01,   3.81368362e-01,   1.88648133e-02,
             -3.02161020e-04,  -4.12720037e-02,   0.00000000e+00],
           [ -3.29272081e-02,   2.73455770e-01,  -3.31833205e-01,
             -1.00278309e-01,   1.98147384e-01,  -4.18105071e-02,
             -4.13318757e-02,   1.62985929e-01,  -4.29569771e-02,
             -4.10103400e-02,  -1.24235383e-01,   1.23016014e-02,
              4.25628092e-02,  -3.46191422e-02,  -7.19288349e-02,
             -1.79453150e-02,   1.52906886e-01,   1.51852840e-01,
             -6.70229735e-01,   4.56695234e-01,  -1.12316617e-02,
              8.40942075e-04,  -5.21690839e-02,   0.00000000e+00],
           [ -9.52388969e-03,   5.06361338e-02,   5.63077523e-02,
             -2.75038299e-02,   2.29377868e-03,  -2.22927176e-03,
             -3.01627462e-02,  -1.16180410e-02,   7.26877680e-03,
             -4.68706154e-03,   7.00835097e-01,  -1.55779542e-01,
             -5.39777423e-04,  -8.47090663e-02,  -5.26455924e-01,
              1.54634859e-01,  -1.07415713e-02,   1.12691533e-01,
             -2.49437772e-02,   6.06917592e-04,   4.31703604e-03,
              5.70128458e-03,  -3.93851734e-01,   0.00000000e+00],
           [ -7.47531264e-03,   5.27229490e-02,  -2.62943060e-03,
              1.38653385e-02,   3.25091872e-02,  -6.53558011e-03,
             -1.74325321e-02,   1.92192393e-03,  -1.73892260e-02,
             -8.58267023e-03,   5.47574061e-01,  -2.42327311e-01,
              1.98678694e-01,  -2.44500792e-01,   6.84787257e-01,
             -1.86242104e-02,   1.94956118e-02,   6.70149459e-02,
             -6.16480684e-02,   8.20344211e-02,  -3.92623880e-03,
              3.95966079e-03,   2.29467993e-01,   0.00000000e+00],
           [ -3.08190762e-04,  -1.09237189e-04,   7.63461611e-03,
              2.47482619e-02,   1.50596252e-02,   4.48223885e-04,
             -1.44205517e-03,  -6.43194087e-03,  -1.88340432e-02,
             -1.03163355e-02,  -1.70167697e-02,   6.77440774e-02,
              2.29976100e-01,   5.23745904e-01,   2.76184012e-01,
              4.97008734e-01,  -3.50913420e-02,   1.95960332e-02,
              1.37974090e-03,   4.81302440e-02,   2.49159052e-02,
              1.33031637e-02,  -3.37537680e-01,   4.73879042e-01],
           [ -5.29923115e-03,   2.98045194e-02,   5.75729863e-02,
              4.42660428e-02,   2.17533492e-02,  -2.26555002e-03,
             -1.88820351e-02,  -2.80204253e-02,  -2.11249678e-02,
             -4.95504063e-03,   1.37781563e-01,   6.37843429e-01,
              3.02493165e-01,  -3.06232622e-02,  -1.08532422e-01,
              4.38415361e-01,  -3.12736361e-02,   8.94905097e-02,
              1.29148511e-02,   5.99677300e-02,   1.92633017e-02,
              1.93534689e-03,   3.53419003e-01,  -3.62538419e-01],
           [ -6.69678975e-03,   3.99755074e-02,   2.03416154e-02,
             -2.02766580e-02,   6.74238335e-03,  -3.10430317e-03,
             -1.88668987e-02,  -8.73358081e-04,   4.79013105e-03,
             -2.10129790e-03,   1.92690456e-01,   5.30658249e-01,
             -6.73296231e-01,  -1.85540440e-01,   1.47823187e-01,
             -2.80536678e-02,   7.90341297e-03,   6.77377449e-02,
             -3.68989997e-02,   1.09044061e-02,  -3.86038025e-03,
              3.19199653e-04,  -2.04000084e-03,   4.02272262e-01],
           [  3.44016404e-03,  -2.97622175e-02,   2.24635192e-02,
             -1.65146416e-02,  -2.54129774e-02,   5.00644272e-03,
              5.27701594e-03,  -6.95841347e-03,   1.50585787e-02,
              5.18835064e-03,  -2.66725599e-01,   7.85506046e-02,
              3.74645672e-01,  -7.59281144e-01,  -3.06562708e-02,
              1.41407356e-01,  -2.59402851e-02,  -1.87365649e-02,
              4.65463165e-02,  -6.48722269e-02,   8.55963111e-03,
             -8.96414257e-04,  -2.53726266e-01,   3.32202295e-01],
           [ -3.06493631e-03,   2.23544608e-02,   3.40913110e-02,
              5.61908371e-02,   1.22625693e-02,  -3.38185295e-03,
             -1.03684830e-02,  -2.30733532e-02,  -2.38365678e-02,
             -3.84450350e-03,   1.76846786e-01,   2.73456889e-01,
              4.50031196e-01,   1.97469281e-01,  -1.93699069e-01,
             -6.21928043e-01,   4.85708635e-02,   2.71809688e-02,
              4.50957143e-02,   4.27767672e-02,  -3.32320185e-02,
              4.17854289e-03,   1.86599830e-01,   4.23077296e-01],
           [ -2.44277376e-03,   1.09185342e-02,   2.91809620e-02,
              1.32301721e-02,  -2.94210441e-03,   3.94976745e-05,
             -9.24834627e-03,  -1.21019043e-02,  -1.00694510e-02,
             -8.74592901e-03,   1.30078168e-02,   3.55520016e-01,
              9.86585043e-02,   3.63597046e-02,   3.13252697e-01,
             -3.43533510e-01,   2.23627076e-02,   2.13305680e-02,
              3.56848323e-02,   4.55686761e-03,  -1.80944132e-02,
              1.63984065e-02,  -6.70069847e-01,  -4.39122776e-01]])

```python
K_hat, B_hat, q_hat, u_hat = solution.functions
```

```python
pts = np.linspace(domain[0], domain[1], 1000)
fig, axes = plt.subplots(4, 1)
axes[0].plot(pts, K_hat(pts))
axes[1].plot(pts, B_hat(pts))
axes[2].plot(pts, q_hat(pts))
axes[3].plot(pts, q_hat(pts))

fig.tight_layout()
plt.show()
```

```python
K_resids, B_resids, q_resids, u_resids = solution.residuals(pts)
```

```python
pts = np.linspace(domain[0], domain[1], 1000)
fig, axes = plt.subplots(4, 1)
axes[0].plot(pts, K_resids)
axes[1].plot(pts, B_resids)
axes[2].plot(pts, q_resids)
axes[3].plot(pts, u_resids)

fig.tight_layout()
plt.show()
```

```python
basic_model_solver.solve(kind="Chebyshev",
                         coefs_dict=initial_coefs,
                         domain=domain)

```

```python
basic_model_solver.result["success"]
```

    True

```python
basic_model_viz = pycollocation.Visualizer(basic_model_solver)
```

```python
basic_model_viz.interpolation_knots = np.linspace(domain[0], domain[1], 1000)
basic_model_viz.solution.plot(subplots=True, style=['r', 'b'])
plt.show()
```

Solution is not as accurate as I would like...

```python
basic_model_viz.residuals.plot(subplots=True, style=['r', 'b'])
plt.show()
```

...actually, when using noramlized residuals eveything looks great!

```python
basic_model_viz.normalized_residuals.plot(logy=True, sharey=True)
plt.show()
```

```python
assets = basic_model_viz.solution[['q', 'K']].prod(axis=1)
liabilities = basic_model_viz.solution.B
equity = assets - liabilities
leverage = assets / equity
```

```python
leverage.plot()
```

```python
def credit_cycles(t, X, a, m, alpha, R, K_bar):
    out = np.array([(1 / X[3]) * ((a + X[2]) * X[0] - R * X[1]) - X[0],
                    (1 / R) * X[2] * X[0] - X[1],
                    (R - 1) * X[2] - R * X[3],
                    (alpha / R) * ((1 / m) * (K_bar - X[0]))**(alpha - 1) - X[3]])
    return out

def jacobian(t, X, a, m, alpha, R, K_bar):
    out = np.array([[((a + X[2]) / X[3]) - 1.0, -R / X[3], X[0] / X[3], -X[3]**(-2)],
                    [(1 / R) * X[2], -1.0, (1 / R) * X[0], 0.0],
                    [0.0, 0.0, R - 1, -R],
                    [-(1 / m) * (alpha - 1) * (alpha / R) * ((1 / m) * (K_bar - X[0]))**(alpha - 2), 0.0, 0.0, -1.0]])
    return out

def Kstar(a, m, alpha, R, K_bar):
    return K_bar - m * (alpha / (a * R))**(1 / (1 - alpha))

def Bstar(a, m, alpha, R, K_bar):
    return (a / (R - 1)) * Kstar(a, m, alpha, R, K_bar)

```

```python
initial_condition = np.array([Kstar(a, m, alpha, R, K_bar), Bstar(a, m, alpha, R, K_bar), (R / (R - 1)) * a, a])
```

```python
initial_condition
```

    array([K_bar - m*(alpha/(R*a))**(1/(-alpha + 1)),
           a*(K_bar - m*(alpha/(R*a))**(1/(-alpha + 1)))/(R - 1), R*a/(R - 1),
           a], dtype=object)

```python
credit_cycles(0, initial_condition)
```

    array([ -8.64019967e-12,   0.00000000e+00,   0.00000000e+00,
            -9.88098492e-14])

```python
jacobian(0, initial_condition)
```

    array([[  1.01000000e+02,  -9.61904762e-01,   9.52163239e+02,
             -9.07029478e-01],
           [  1.05000000e+02,  -1.00000000e+00,   9.89872674e+02,
              0.00000000e+00],
           [  0.00000000e+00,   0.00000000e+00,   1.00000000e-02,
             -1.01000000e+00],
           [  2.15879931e+00,   0.00000000e+00,   0.00000000e+00,
             -1.00000000e+00]])

```python
from scipy import linalg
```

```python
from IPython.html.widgets import fixed, interact, FloatSliderWidget
```

    :0: FutureWarning: IPython widgets are experimental and may change in the future.

```python
def eigenvalues(a=1.0, m=1.0, alpha=0.33, R=1.05, K_bar=10.0):
    steady_state = np.array([Kstar(a, m, alpha, R, K_bar),
                             Bstar(a, m, alpha, R, K_bar),
                             (R / (R - 1)) * a,
                             a])
    vals, vecs = linalg.eig(jacobian(0, steady_state, a, m, alpha, R, K_bar))
    print vals
```

```python
interact(eigenvalues, a=(0.0, 1e3, 1e0), m=(0.0, 1e2, 1e-1), R=(0.0, 1e2, 1e-2), K_bar=(0.0, 1e4, 1e1))
```

    [  4.96886453e+04+86405.36943471j   4.96886453e+04-86405.36943471j
      -9.93761281e+04    +0.j           4.01197590e-08    +0.j        ]

```python
params = 2.0, 0.5, 0.33, 1.01, 500.0
problem = ivp.IVP(credit_cycles, jacobian)
problem.f_params = params
problem.jac_params = params
```

<h2> Full model </h2>

```python
lamda, pi, phi, = sym.symbols('lamda, pi, phi')

# full model from Kiyotaki and Moore "credit-cycles" paper
K_dot = (pi / (phi + u)) * ((a + q + lamda * phi) * K - R * B) - pi * lamda * K
B_dot = (R - 1) * B + (phi * (1 - lamda) - a) * K
q_dot = (R - 1) * q - R * u
u_dot = (1 / R) * mpk.subs({k: (1 / m) * (K_bar - K)}) - u

rhs = {'K': K_dot, 'B': B_dot, 'q': q_dot, 'u': u_dot}
```

```python
bcs = {}

ustar = ((pi * a - (1 - lamda) * (1 - R + pi * R) * phi) /
         (lamda * pi + (1 - lamda) * (1 - R + pi * R)))
qstar = (R / (R - 1)) * ustar
Kstar = K_bar - m * (alpha / (ustar * R))**(1 / (1 - alpha))
Bstar = ((a - (1 - lamda) * phi) / (R - 1)) * Kstar

# initial conditions for K and B are given
K0 = 75
bcs['lower'] = [K - K0]

# boundary conditions on B, q and u can be written in terms of steady state values
bcs['upper'] = [B - Bstar, q - qstar, u - ustar]
```

```python
params = {'a': 1.05, 'pi': 0.05, 'phi': 20.0, 'lamda': 0.975,'m': 1.0, 'alpha': 0.16,
          'R': 1.01, 'K_bar': 100}
```

```python
# set up the model and solver
full_model = pycollocation.SymbolicBoundaryValueProblem(dependent_vars=['K', 'B', 'q', 'u'],
                                                        independent_var='t',
                                                        rhs=rhs,
                                                        boundary_conditions=bcs,
                                                        params=params)

full_model_solver = pycollocation.OrthogonalPolynomialSolver(full_model)
```

```python
def Kstar(a, phi, R, alpha, pi, m, lamda, K_bar):
    return K_bar - m * (alpha / (ustar(a, phi, R, alpha, pi, m, lamda, K_bar) * R))**(1 / (1 - alpha))

def Bstar(a, phi, R, alpha, pi, m, lamda, K_bar):
    return ((a - (1 - lamda) * phi) / (R - 1)) * Kstar(a, phi, R, alpha, pi, m, lamda, K_bar)

def qstar(a, phi, R, alpha, pi, m, lamda, K_bar):
    return (R / (R - 1)) * ustar(a, phi, R, alpha, pi, m, lamda, K_bar)

def ustar(a, phi, R, alpha, pi, m, lamda, K_bar):
    u = ((pi * a - (1 - lamda) * (1 - R + pi * R) * phi) /
         (lamda * pi + (1 - lamda) * (1 - R + pi * R)))
    return u

# specify an initial guess
domain = [0, 25]
ts = np.linspace(domain[0], domain[1], 1000)
Ks = Kstar(**params) - (Kstar(**params) - K0) * np.exp(-ts) * np.cos(2.0 * np.pi * ts)
initial_capital_poly = np.polynomial.Chebyshev.fit(ts, Ks, 25, domain)

# initial value of debt is some multiple of capital stock
B0 = 1.5 * K0
Bs = Bstar(**params) - (Bstar(**params) - B0) * np.exp(-ts) #* np.cos(2.0 * np.pi * ts)
initial_debt_poly = np.polynomial.Chebyshev.fit(ts, Bs, 25, domain)

# starting with K0 > Kstar, must be that u0 > ustar
us = ustar(**params) - (ustar(**params) - 1.5 * ustar(**params)) * np.exp(-ts) #* np.cos(2.0 * np.pi * ts)
initial_user_cost_poly = np.polynomial.Chebyshev.fit(ts, us, 25, domain)

# starting with K0 > Kstar, must be that q0 > qstar
qs = qstar(**params) - (qstar(**params) - 1.5 * qstar(**params)) * np.exp(-ts) #* np.cos(2.0 * np.pi * ts)
initial_land_price_poly = np.polynomial.Chebyshev.fit(ts, qs, 25, domain)

initial_coefs = {'K': initial_capital_poly.coef, 'B': initial_debt_poly.coef,
                 'u': initial_user_cost_poly.coef, 'q': initial_land_price_poly.coef}
```

```python
def jacobian(t, X, a, phi, R, alpha, pi, m, lamda, K_bar):
    out = np.array([[(pi / (phi + X[3])) * (a + X[2] + lamda * phi) - pi * lamda, -(pi / (phi + X[3])) * R, (pi / (phi + X[3])) * X[0], -(pi / (phi + X[3])**2) * ((a + X[2] + lamda * phi) * X[0] - R * X[1])],
                    [(R - 1) * X[1] + (phi * (1 - lamda) - a), (R - 1), 0.0, 0.0],
                    [0.0, 0.0, R - 1, -R],
                    [-(1 / m) * (alpha - 1) * (alpha / R) * ((1 / m) * (K_bar - X[0]))**(alpha - 2), 0.0, 0.0, -1.0]])
    return out

def eigenvalues(a=1.0, phi=20.0, pi=0.05, lamda=0.975, m=1.0, alpha=0.33, R=1.05, K_bar=10.0):
    steady_state = np.array([Kstar(a, phi, R, alpha, pi, m, lamda, K_bar),
                             Bstar(a, phi, R, alpha, pi, m, lamda, K_bar),
                             qstar(a, phi, R, alpha, pi, m, lamda, K_bar),
                             ustar(a, phi, R, alpha, pi, m, lamda, K_bar)])
    vals, vecs = linalg.eig(jacobian(0, steady_state, a, phi, R, alpha, pi, m, lamda, K_bar))
    print vals
    print np.absolute(vals)
```

```python
interact(eigenvalues, a=(1.0, 2.0, 1e-2), alpha=(1e-2, 1-1e-2, 1e-2), m=(0.0, 1e2, 1e-1), R=(0.0, 1e2, 1e-2), K_bar=(0.0, 1e4, 1e1))
```

    [-0.96239413+0.j          0.03119783+0.31974317j  0.03119783-0.31974317j
      0.05000000+0.j        ]
    [ 0.96239413  0.32126158  0.32126158  0.05      ]

    <function __main__.eigenvalues>

```python

```

```python
full_model_solver.solve(kind="Chebyshev",
                        coefs_dict=initial_coefs,
                        domain=domain)

```

```python
full_model_solver.result["success"]
```

    False

```python
full_model_viz = pycollocation.Visualizer(full_model_solver)
```

```python
full_model_viz.interpolation_knots = np.linspace(domain[0], domain[1], 1000)
full_model_viz.solution.plot(subplots=True)
```

```python
full_model_viz.normalized_residuals.plot(subplots=True)
plt.show()
```

```python

```